Impression criteria is an optional step when setting up an A/B test. Impression criteria essentially allow you to add a pre-filter on who is tracked in your A/B test analytics.
So, if you want to limit your A/B test to just users that have completed a certain Event (made a purchase, or viewed the signup page), or reached a certain State in your app (a level in a game), then impression criteria can help yield more focused results.
Note: If you set impression criteria, you will only be able to see metrics for users who meet that criteria. Users in your target who do not reach the impression criteria will still be assigned to your test, but you won't see them in analytics. It is completely acceptable not to specify an impression criteria for your experiment.
How Leanplum evaluates qualified users
To be clear, users are still evaluated against your test targets and assigned to any relevant A/B tests upon app start. These users just won’t show up in analytics until they trigger the impression criteria (if one is specified).
When to use impression criteria
Use caution: an impression criteria cannot be removed from an A/B test without skewing your results. If you aren't sure if you need one, it's probably safer to create a test without an impression criteria.
Generally, the choice boils down to two things:
- How certain you are of the Events/States that should trigger analytics tracking
- How deep in your app are you testing
Setting an impression criteria may be a good idea if you are testing a feature that's unlikely to be encountered shortly after a user starts the app—or if you're only concerned with users that have reached a certain State or Event (e.g. Add to Cart).
For example, if a gaming company is testing a change in the difficulty of Level 10 in their game, they may want to set an impression criteria of "User enters Level 10" since many users will not reach that point in the game, and the test should be focused on just tracking those users.
However, if you're testing something on the home page, or aren't quite sure what should trigger the tracking, we recommend skipping impression criteria and using filters in Analytics to narrow your results down after the fact.
For example, if you are testing a new user onboarding flow, it may not be necessary to specify an impression criteria, as all new users who open the app will be exposed to the test almost immediately.