Specify the start of the A/B Test
Impression criteria is an optional step when setting up an A/B test. It essentially allows you to add a pre-filter on who is tracked in your A/B test analytics.
So, if you want to limit your A/B test to just users that have completed a certain Event (made a purchase, or viewed the signup page), or reached a certain State in your app (a level in a game), then impression criteria can be very powerful, and yield very focused results.
Note: If you set impression criteria, you will only be able to see metrics for users that meet that criteria.
It is, however, completely acceptable not to specify an impression criteria for your experiment.
How Leanplum evaluates qualified users
To be clear, users are still evaluated against the targets and assigned to any relevant A/B tests upon app start. They just won’t be shown in the analytics until triggering the impression criteria, if one is specified.
When to use impression criteria
We urge caution, and generally suggest not using it just to be on the safe side. It cannot be removed (for that specific A/B test) without skewing the results.
Generally, the choice boils down to two things:
- How certain are you with what Events/States should trigger the analytics tracking
- How deep within your app are you testing
Setting an impression criteria is considered a best practice if you are testing a feature which is unlikely to be encountered shortly after a user starts the app. And, if you're only concerned with users that have reached a certain State or completed a certain Event (e.g. Add to Cart).
For example, if a gaming company is testing a change in the difficulty of Level 10 in their game, they may want to set an impression criteria of 'User enters Level 10' since many users will not reach that point in the game, and the test should be focused on just tracking those users.
However, if you're testing something on the home page, or aren't quite sure what should trigger the tracking, then we recommend skipping impression criteria and using filters in Analytics to narrow the results down -- after the fact -- as you wish.
For example, if you are testing a new user onboarding flow, it may not be necessary to specify an impression criteria as all new users who open the app will almost immediately be exposed to the test.