A/B testing troubleshooting and FAQ

What is the control variant? 

The control variant is what will be used to compare your result against in the A/B test. It is typically the standard experience served to the user, whether it is an experience in the app or a message.  

If testing a variable, the control will be the override value on the Variables Dashboard. If no override has been set, it defaults to the default in the code value.

 

What happens when a user logs in on two devices and is a part of an A/B test?

See the article here: https://support.leanplum.com/hc/en-us/articles/209895446-Handle-same-User-login-on-different-devices

 

Can a user enter into two tests at the same time?

If a user fits the targeting for more than one test, then yes, that user could be sorted into multiple tests at once. This can cause problems when testing variables, which is why we recommend that you only run one test per variable at any given time.


Why am I seeing an unknown segment in the variant of my A/B test?

Situation:

You have started an a/b test. You later notice that a segment is listed within the variant of the a/b test but you cannot remove it. Why is the segment there and how can I remove it? 

Example:
The unknown segment here is targeting version 2.0 and has a default value of "Open".

Reason: 
The segment was added directly to the variable on the variables page. When you add a segment to the variable on the variables page, that segment also gets ported over to the a/b test it's live in so you don't risk changing that specific group's experience. 

Result:
The "override" value applied on the variables page to the segment will be ported over the a/b test in the control variant. 

If you remove the segment from the variable on the variables page, you'll then be able to remove it from the test as well. 

 

Is it possible for a user to see multiple different variants in a single test?

If your test's target is sticky (magnet button is selected/blue), your users will remain in whatever variant (or control) group they are originally assigned. Even if the user no longer fits the target criteria for your test, they will remain in that variant, which means they will only see the variant experience until you finish the test.

If your test is not sticky, a user will remain in their randomly assigned test group until they no longer fit the target criteria for the test. They would no longer receive their assigned variant experience once they exited the target group. If that user's target status changed again, and they became eligible for the test again, they would be treated as a new user and randomly sorted into a variant. This means it is possible for them to see multiple variant experiences, especially if their target status changes often.

For example, if the target of a non-sticky test was "Los Angeles users," and one of the users started in LA then went to Chicago, they would no longer be in the test. If that user returned to Los Angeles, they would fit the target criteria once again, and they could be sorted into one of the variants at random.

 

If I delete an event, what happens to the A/B test results and analytics related to that event? Can I still see these results?

You may want to go through and delete the events that you no longer use in your app at some point. If you delete an event, note that you won't be able to see that event in analytics or in the results for an A/B test.

If you want to save the results related to these events, you can export statistics for A/B tests and other reports before you delete the event(s). See API export methods for more.


Was this article helpful?
Have more questions? Submit a request