Note: Holdbacks require Leanplum SDK version iOS 1.4.1 / Android 1.3.1 or later.
At Leanplum, we believe that every piece of content should be A/B tested. That's why we wanted to make it easy to A/B test your messages and see how successful they are at driving specific user behavior and ROI.
By withholding your message from a group of users, you can measure how successful your message is at driving behavior compared to the holdback group. Using the holdback feature in conjunction with a Message goal can give an even clearer picture of the user behavior generated by your message.
Adding a Holdback
To add a holdback group to your message, click Add Holdback in the upper right corner of the Target. By default, 10% of targeted users will be held back. To change the percentage of users to be held back from the message, click the current percentage and type in the percent of users you want to be held back.
Users who are assigned to the holdback will not receive the message — no need to set up a separate A/B test. We handle all of that behind the scenes! Leanplum will create a control group that doesn't receive the message and compare results to the test group that does.
You can also change the holdback in the advanced holdback settings.
- Click Add Holdback, then click the gear icon. Select View Holdback Test to open your advanced settings.
- To change the holdback amount for your message, use the Distribution slider to adjust the percentage of users in the control group. The control users are the holdback group, and they will not receive your message. (The Enabled checkbox for the control group should stay unchecked.)
You can also use the holdback to send a message to a smaller portion of users. For example, to send your message to just 20% of your audience, move the control group's distribution (the holdback group) up to 80%.
Analyzing the results
Both the holdback group and the users who received your message will be included in your results. The Holdback test is configured to account for both of groups of users automatically.
Leanplum will track several Events for users targeted in a message with a Holdback. If a user receives the message, Leanplum will trigger a Holdback impression event and a message Sent event. The impression event simply notes that the user was included in the Holdback test.
For users that do not receive the message, Leanplum will trigger a Held back event at the time the message would have been sent. These events will be visible in their user profile in the Users tab, as well as in the Analytics for the message itself.
Note: For immediate and scheduled messages, we will automatically finish the Holdback test two days after the last message occurrence. For triggered messages, we will automatically finish the test 30 days after you finish the message.
You can also use Holdbacks for more advanced message use cases. Note: we recommend adding a Holdback group to all of your messages to best determine the impact of your messages.
Two different message copy versions
You can easily add additional variants to the Holdback if you wish to test multiple versions of copy, images or other message content.
- Click Add Holdback in your message.
- Click the gear icon next to Holdback to view the Holdback Test.
- Add another variant to the test and make any desired message copy changes.
- Start your message.
Note: Holdbacks can't be added to previously-active messages. To add a Holdback to an existing message, please create a new copy of your message.
We recommend using Holdback in conjunction with Message Goals to help you understand the full impact that your message campaigns are having on users.