Impression criteria is an optional step when setting up an A/B test. Impression criteria allows you to define when a user is counted towards the analytics of an A/B test.
So, if you want to limit your A/B test to just users that have completed a certain Event (made a purchase, or viewed the signup page), or reached a certain State in your app (a level in a game), then impression criteria can help yield more focused results.
Example: Say you’re testing something on the settings section. When a user opens the app and enters the test, they may never see what’s being tested on the settings page, but would still count in the metrics/analytics. By setting an impression criteria of event “Click on Settings Page”, only users who fire that event will impact the test results/analytics, making it more accurate.
Note: If you set impression criteria, you will only be able to see metrics for users who meet that criteria. Users in your test's target who do not reach the impression criteria will still be assigned to your test, but you won't see them in analytics. It is completely acceptable not to specify an impression criteria for your experiment.
How Leanplum evaluates qualified users
Users are still evaluated against your test targets and assigned to any relevant A/B tests upon app start. Users who enter your test just won’t show up in analytics until they trigger the impression criteria (if one is specified).
How it works: User opens the app -> Leanplum is initialized -> test targets are evaluated -> If user meets the targeting criteria, they are randomly placed in the control or the variant -> user triggers the event that has been chosen as the impression criteria -> user counts in the analytics at that point.
When to use impression criteria
Use caution: an impression criteria cannot be removed from an A/B test without skewing your results. If you aren't sure if you need one, it's probably safer to create a test without an impression criteria.
Generally, the choice boils down to two things:
- How certain you are of the Events/States that should trigger analytics tracking
- How deep in your app are you testing
Setting an impression criteria may be a good idea if you are testing a feature that's unlikely to be encountered shortly after a user starts the app—or if you're only concerned with users that have reached a certain State or Event (e.g. Add to Cart).
For example, if a gaming company is testing a change in the difficulty of Level 10 in their game, they may want to set an impression criteria of "User enters Level 10" since many users will not reach that point in the game, and the test should be focused on just tracking those users.
However, if you're testing something on the home page, or aren't quite sure what should trigger the tracking, we recommend skipping impression criteria and using filters in Analytics to narrow your results down after the fact.
For example, if you are testing a new user onboarding flow, it may not be necessary to specify an impression criteria, as all new users who open the app will be exposed to the test almost immediately.