A/B testing a message

When A/B testing a message, there are a few additional settings to consider when setting up a test. Check out the following how-to's for more on setting up A/B tests for messages.

Create a test for a message

There are two ways to add a message to an A/B test:

  1. From within the message itself, click on the A/B Test button in the top right-hand corner. This will create a new A/B test with a control group and a variant, with the message enabled for the variant and disabled for the control.
  2. From within a pre-existing A/B test, click the Add dropdown on the lower left-hand side, and select +Message. You can then select the message you would like to add from the Select Message drop-down menu.


Setting up an A/B test for a message

Because messages can operate separately from A/B tests, it's important to check the following settings in your message and your test. Doing so will help ensure that your content is going to the right users. 

To prevent the users outside the test from getting the message, make sure the targets for your test and your message are the same. You may also need to make sure the message you are testing is paused after publishing. See below for more detail.


If the target audience for your message is different from the target for your A/B test, the dashboard will display a warning to alert you that users outside of your test will likely receive your message. 


For example, if you target "All Users" on the message, but restrict the A/B test to US users, all users outside the US would receive the message, but only US users would be entered into the A/B test. 

Live VS. paused

To prevent users outside of your test from receiving your message, make sure the message is paused after you publish your test. This is especially important when using the audience slider. 
Live messages are sent to all targeted users who aren't in your A/B test. The audience slider keeps a portion of your audience out of your test, so these users will receive your message if it is live/published. Make sure your message is paused to prevent the users outside the test from getting the message.

Note on pausing an A/B test: If you pause an A/B test but leave the message active, your message will go out to all users that match the message's target (even those who were previously in a test group that had the message disabled).

Audience slider

The audience slider determines what percentage of your Target audience will be in your A/B test. It does not, however, control the audience for the message itself. This means that if your message is live, the portion of your target audience that is held out of the test will still receive your message. 


To prevent users outside of your test from receiving your message, you should pause the message you are testing.

Consider using the message Holdback beta feature to make it easier to test a new message on a portion of your audience. 

Disable the message for test groups

When setting up an A/B test for your message, you'll typically want to test the effects of not sending the message (the variant) against actually sending it (the control).

If you want to disable the message for the control group, uncheck the Enabled setting underneath the control group, and enable for in the variant. 

Next, learn how to analyze the results of your A/B test.

Was this article helpful?
Have more questions? Submit a request