A/B testing your message

When setting up an A/B test for your message, you'll typically want to test the effects of not sending the message (the variant) against actually sending it (the control). Leanplum makes this a snap. 

Adding a message to an A/B test

There are two ways to add a message to an A/B test:

  1. From within the message itself, click on the A/B Test button in the top right-hand corner. This will create a new A/B test with a control group and a variant, with the message enabled for the variant and disabled for the control.
  2. From within a pre-existing A/B test, click the Add button from the lower left-hand side and select +Message. You can then select the message you would like to add from the Select Message drop-down menu.

 

Status and Targets: Message vs. A/B Test 

When you add a message to an A/B test, the statuses of the message(s) and the A/B test are still independent from one another. If you pause the A/B test, but leave the messages active, messages will still go out to all users that match the message's targeting criteria (even those in a test group that had the message disabled).

Similarly, the targeting remains independent. Leanplum does default the A/B test targets to the message's targets if you create the Test from the Message Composer. Still, it’s important to keep in mind that if the targeting criteria differ, the test and message will act independently. For example, if you target "All Users" on the message, but restrict the A/B test to users in the US only, all users outside the US would receive the message, while only US users would be entered into the A/B test. 

We typically recommend that you keep your A/B test status and targets inline with messages included in the test.

Disable the message for test groups

If you want to disable the message in the control group, then you can simply uncheck the Enabled setting underneath the control group, and enable it in the variant.

Next, learn how to analyze the results of your A/B test.


Was this article helpful?
Have more questions? Submit a request