Gradually Rolling Out A/B Tests to Your Audience

The rate at which you rollout your A/B tests to new users should depend on the size of your total audience as well as how fast you want to get results. Most customers choose to implement a two to three stage rollout process, starting with a small portion of their total audience and slowly increasing the sample size to a higher percentage of the audience, until finally reaching 100%. This type of staged rollout plan attempts to minimize the risks associated with testing a new feature while still aiming to produce results as fast as possible.

Beginning the test with a smaller audience

The newly improved Audience slider is the best way to gradually rollout your A/B tests. Audience represents the percentage of targeted users that should be included in the experiment. Included users are randomly assigned to either the control or one of the variants.

By default, 100% of users who meet the targeting requirement that you've defined are included in the A/B test. If you would like to rollout starting with a smaller audience, you should reduce the percentage prior to starting the test. Notice that when we reduce the audience size to 1%, the Excluded percentage appears adjacent to Variant 1. This represents the percentage of targeted users that are excluded from the experiment.

Note that once you publish your test, you cannot retroactively reduce the audience size. This is due to the fact that targeting changes only effect new users, not users that have already been assigned to the test. So be sure to start with a smaller audience before publishing your test for the first time!

Increasing the size of the audience step-by-step

If you are ready to increase the sample size in your experiment, you may then choose to roll it out to a wider audience. In this example we increase the audience from 1% to 50%, and then again from 50% all the way up to 100%.

Notice that each time we increase the audience, we do not modify the distribution, or the rate at which users are assigned to each variant. It is best practice to NOT modify the distribution among variants mid-test, as it can bias your variant groups and make it difficult to interpret test results.

Choosing a winner and finishing the test

If you have rolled out to 100% of the audience and are ready to pick a winning variant, we recommend that you finish your test and roll out the winning variant to the desired group of users.

Please reach out to or directly to your customer success manager with any questions.

Was this article helpful?
Have more questions? Submit a request