A/B testing your message

When A/B testing a message, there are a few additional settings to consider when setting up your test. Check out the following how-to's for more on setting up an A/B tests for messages.

Adding a message to an A/B test

There are two ways to add a message to an A/B test:

  1. From within the message itself, click on the A/B Test button in the top right-hand corner. This will create a new A/B test with a control group and a variant, with the message enabled for the variant and disabled for the control.
  2. From within a pre-existing A/B test, click the Add button from the lower left-hand side and select +Message. You can then select the message you would like to add from the Select Message drop-down menu.

 

Setting up an A/B test with a message

Because messages can operate separately from your A/B test, it's important to check the following settings in your message and your test. Doing so will help ensure that your content is going to the right users. 

Targeting. When the target for your message is the same as your test, your message will only be delivered to users who are in your A/B test. If the two targets are different, the dashboard will display a warning to alert you that users outside of your test will likely receive your message. 

Screen_Shot_2017-08-09_at_1.08.49_PM.png

For example, if you target "All Users" on the message, but restrict the A/B test to US users, all users outside the US would receive the message, while only US users would be entered into the A/B test. 

Live VS. paused status. To prevent users outside of your test from receiving your message, you should pause your message right after you start it.
Screen_Shot_2017-09-22_at_1.50.51_PM.png
Live messages can be sent to any targeted users who aren't in your A/B test. This means that using the audience slider with a live message can cause users outside your test audience to receive the message. 

If you pause the A/B test, but leave the message active, messages will go out to all users that match the message's target (even those in a test group that had the message disabled).

Audience slider. The audience slider determines what percentage of your Target audience will be in your A/B test. It does not, however, control the audience for the message itself. This means that if your message is live, the portion of your target audience that is held out of the test will still receive your message. 

Screen_Shot_2017-09-22_at_1.57.31_PM.png

To prevent users outside of your test from receiving your message, you should pause your message before testing.

Consider using the message Holdback beta feature to make it easier to test a new message on a portion of your audience.

 

Disable the message for test groups

When setting up an A/B test for your message, you'll typically want to test the effects of not sending the message (the variant) against actually sending it (the control).

If you want to disable the message for the control group, uncheck the Enabled setting underneath the control group, and enable for in the variant. 

Next, learn how to analyze the results of your A/B test.


Was this article helpful?
Have more questions? Submit a request