There are a few different ways to manage your A/B tests. Read on to learn more about pausing, copying, or deleting a test.
Pausing an A/B test
You may want to pause a test to reconsider your strategy or to prevent users from seeing a variant.
What happens when you pause a test? When you pause a test, all of the users in the test will be served the control group's experience on their next app start. Once you restart your test, the users will return to their original variant groups. (Note: if the message you are testing is live, pausing your test may cause all Targeted users to start receiving the message. To prevent this, make sure your message is paused before you pause a test.)
Pausing a test can also affect the integrity of your results. For example, let's say you were testing a new messaging strategy to maximize the number of returning users. Then, you pause your test for one day and restart it. This seems like a minor change, but what if you paused on a Saturday, when more users tend to use your app? Or, what if you paused the test early in the morning, before some users go to work? Different types of users interact with your app at different times, and pausing can skew your test results more than anticipated.
Different types of users interact with your app at different times, and pausing can skew your test results more than anticipated.
Note that the status of messages in your test are independent of the test's status. So if you pause the A/B test but leave the messages active, messages will still go out to all users that match the message's targeting criteria (even those in a test group that had the message disabled). See this article for more on A/B testing messages.
How do I keep new users from entering a test without affecting the experience of my current users?
To keep new users from entering a test without affecting the experience of those already in the test, you can change the target of the test to exclude virtually all of your users (e.g. set the target to only one specific userID). If your test is sticky (magnet button activated), all the users who were previously in the test will remain in their same variant groups, but no new users will be able to join.
This is different from pausing because rather than serving everyone the control experience, all the users in your test will remain in their original variants.
Copying an A/B test
To make a new test based on one of your existing tests, you can make a copy. Click the ( ... ) button in the upper right corner of the screen, then select Copy from the dropdown that appears.
You can copy tests within your app, or you can copy a test from one app into another one of your organization's apps.
This will create a new version of the test with all of your old test's Targets, Goals, Distribution, and variants pre-loaded into the settings. You can alter these settings and the test's title before sending the new test out to your users.
Note: Always double check all settings and variants before starting any A/B test — especially when using copies. If an old test had a message with scheduled delivery, for example, the dates may no longer be correct.
Also, be aware of the end result of the original test you are copying. If you already rolled out a variant or message to your users, starting a new test based on the old copy may conflict with a change you already made.
Deleting an A/B test
Click the ( ... ) button in the upper right corner of the test, then click Delete. Note: Once you delete a test, there is no way to undo it.
If you delete a test that is currently live, all users will return to the default experience (control) on their next app start. To avoid this, it's better to finish your test and select a winning variant before deleting the test.
Sometimes mistakes happen, and you may want to stop users from receiving a message variant that already went live. There's no way to unsend a message that has already been delivered, but "Abort send" can stop some messages that haven't been delivered yet.
Click the ( ... ) button in the upper corner of a completed test to reveal Abort send.
If your aborted message is sent again, users who already received the message will not get another one.