When A/B testing a message, there are a few additional settings to consider when setting up a test. Check out the following how-to's for more on setting up A/B tests for messages.
Create a test for a message
There are two ways to add a message to an A/B test:
- From within the message itself, click the A/B Test button in the top right-hand corner. This will create a new A/B test with a control group and a variant, with the message enabled for the variant and disabled for the control. Using this method ensures that your test and message targets are the same.
- From within a pre-existing A/B test, click the Add dropdown on the lower left-hand side, and select +Message. You can then select the message you would like to add from the Select Message drop-down menu.
When testing a message, there are a few extra settings to consider in the A/B testing composer.
To prevent the users outside the test from getting the message, make sure the targets for your test and your message are the same. If you create the test directly from the message composer, your message targets are loaded automatically into the A/B test composer, so they will already match.
If the target audience for your message is larger than the target for your A/B test, the dashboard will display a warning to alert you that users outside of your test will likely receive your message.
For example, if you target "All Users" on the message, but restrict the A/B test to US users, all users outside the US would receive the message, but only US users would be entered into the A/B test.
You may also need to make sure the message you are testing is paused after publishing. This is especially important when using the audience slider. See below for more detail.
The audience slider determines what percentage of your Target audience will be in your A/B test. You can use it to send the test to a smaller portion of your audience, or to slowly roll out a test to your full audience.
When using the audience slider, Leanplum will prompt you to pause your message before publishing your test. This is because if your message is live, the portion of your message’s audience that is held out of the test will still receive your message. Just follow the prompt or return to your message to make sure it is paused.
Note: The audience slider does not work with immediate-send messages. To send an immediate message to a smaller portion of your users, consider using User Buckets in your target (and make sure your test and message targets match!).
The message Holdback beta feature can also make it easier to test out a new message on a portion of your audience.
Disable the message for test groups
When setting up an A/B test for your message, you may want to test the effects of not sending the message (the variant) against actually sending it (the control).
If you want to disable the message for the control group, uncheck the Enabled setting underneath the control group, and enable for in the variant.
Next, learn how to analyze the results of your A/B test.