Note: Holdbacks require Leanplum SDK version iOS 1.4.1 / Android 1.3.1 or later. Your app must be whitelisted to use this feature. Please reach out to firstname.lastname@example.org or directly to your customer success manager to to be whitelisted for Message Holdbacks.
At Leanplum, we believe that every piece of content that users see should be A/B tested. That's why we wanted to make it easy to A/B test your messages, allowing you to see how successful they are at driving specific user behavior and ROI. Not showing your message to a group of users will allow you to measure how successful your campaign was compared to the holdback group.
Adding a Holdback
To add a holdback, click Add Holdback in the upper right corner of the Target. By default, 10% of targeted users will be held back. To change the percentage of users to be held back from the message, click the current percentage and type in the percent of users you want to be held back.
Users who are assigned to the holdback will not receive the message. No need to set up a separate A/B test and add your messages manually. We handle all of that behind the scenes! Leanplum will create a control group that doesn't receive the message and compare results to a variant group that does.
You can also change the holdback in the advanced holdback settings.
- Click Add Holdback, then click the gear icon. Select View Holdback Test to open your advanced settings.
- To change the holdback amount for your message, use the Distribution slider to adjust the percentage of users in the control group. The control users are the holdback group, and they will not receive your message. (The Enabled checkbox for the control group should stay unchecked.)
Important: In your advanced holdback settings, the audience slider won't exclude users from a live message — the slider only works for A/B tests where the message being tested is paused. Since your holdback is in a normal message that won't be paused, the audience should be 100%.
For example, let's say you wanted to send your message to only 20% of your users (holdback 80%). If you set the Audience slider to "Include 20% of targeted users," that 20% of users will be entered into your Holdback test and split between the control and variant group, but the other 80% of users will still receive the original message.
To send your message to 20% of your audience, leave the audience slider at 100% and move the control group's distribution up to 80%.
Analyzing the Results
Users who were either sent a message or were in the Holdback and would have been sent the message will be included in results. We automatically configure the Holdback test to account for both of these groups of users.
Leanplum will track several Events for users targeted in a message with a Holdback. If a user receives the message, Leanplum will trigger a Holdback impression event and then a message Sent event. The impression event simply notes that the use is included in the Holdback test. For users that do not receive the message, Leanplum will trigger a Held Back event at the time the message would have been sent. These events will be visible in their user profile in the Users tab, as well as in Analytics for the message itself.
Note: For immediate and scheduled messages, we will automatically finish the Holdback test 2 days after the last message occurrence. For triggered messages, we will automatically finish the test 30 days after you finish the message.
You can use Holdbacks for more advanced message use cases. Note: we recommend adding a Holdback group to all of your messages to best determine the impact of your messages.
Two different message copy versions
You can easily add additional variants to the Holdback if you wish to test multiple versions of copy, images or other message content.
- Click to add a Holdback to your message.
- Click the gear icon next to Holdback > View Holdback Test.
- Add a variant to the test and make any desired message copy changes.
- Start your message.
Note: Holdbacks can't be added to previously-active messages. To add a Holdback, please create a copy of your message.
We recommend using Holdback in conjunction with Message Goals in order to best understand the impact that your message campaigns are having on users.