Leanplum provides an estimated length of time that will be required for your experiment to reach statistically significant results.
Three inputs determine the length of time estimate:
- the experiment goals
- the experiment targets
- the number and size of variants
Modifying any of these three inputs could shorten or lengthen the test time estimate.
This will likely have the most substantial impact on reducing the estimated time until significant results. Click the details button next to Time in your A/B test to reveal the estimated time calculation specific to each selected goal.
These details give insight into which goals are affecting the time estimate most. You may choose to add or remove new goals to achieve an acceptable amount of time until completion.
Change the significance of the test:
Significance represents the likelihood that when Leanplum identifies a winning variant, it is in fact a winning variant. If you choose a higher significance, you reduce the risk of a false positive (i.e. concluding your goal was reached when in actually was not), but your experiment will need to run for more time.
Our defaults of 80 percent power and 95 percent significance work well for most applications, and we recommend you only change them if requested by your data science team. We calculate significance with the Welsh T-test.
Changing the targets:
Sometimes the target audience is too small to reach statistically significant results fast. If you can include a larger portion of your total audience in the test, you can modify the targets to speed up the time estimate.
Changing the number of variants:
The more variants you have, less users will fall into each group. A smaller sample size means it will take longer for your test to reach significant results. Click the X in the variant table to delete a variant, or click the Add ( + ) Variant button to add one.
Contact firstname.lastname@example.org with any other questions.