Note: Holdbacks require Leanplum SDK version iOS 1.4.1 / Android 1.3.1 or later.
Thanks for agreeing to be a beta tester for one of our newest features. Your feedback is essential to the continued improvement of the Leanplum Platform.
At Leanplum, we believe that every piece of content that users see should be A/B tested. Therefore, we wanted to make it ridiculously easy to A/B test your messages so you can see how successful they are at driving specific user behavior and ROI. Not showing your message to a group of users will allow you to measure how successful your campaign was compared to the holdback group.
Adding a Holdback
It only takes one click to add a holdback to your message. By default, users who are assigned to the holdback will not receive the message. No need to set up a separate A/B test and add your messages manually. We handle all of that behind the scenes!
You can also adjust the percentage of users to be held back from the message. By default, 10% of targeted users will be held back.
Analyzing the Results
Users who were either sent a message or were in the Holdback and would have been sent the message will be included in results. We automatically configure the Holdback test to account for both of these groups of users.
Note: For immediate and scheduled messages, we will automatically finish the Holdback test 2 days after the last message occurrence. For triggered messages, we will automatically finish the test 30 days after you finish the message.
You can use Holdbacks for more advanced message use cases. Note: we recommend adding a Holdback group to all of your messages to best determine the impact of your messages.
Two different message copy versions
You can easily add additional variants to the Holdback if you wish to test multiple versions of copy, images or other message content.
- Click to add a Holdback to your message.
- Click the gear icon next to Holdback > View Holdback Test.
- Add a variant to the test and make any desired message copy changes.
- Start your message.
Note: Holdbacks can not be added to previously-active messages. To add a Holdback, please create a copy of your message.
We recommend using Holdback in conjunction with Message Goals in order to best understand the impact that your message campaigns are having on users.
Please reach out to firstname.lastname@example.org or directly to your customer success manager to to be whitelisted for Message Holdbacks.