In the world of A/B testing, picking winners is about finding the right patterns in data. It’s about seeing the forest and the trees. It’s about having the right context.
Leanplum's signal strength indicator is part of our automation platform which bubbles up the most relevant insights and patterns to make the most effective data-driven decisions.
Some metrics have too few users when calculated, particularly when they are event-based and farther down a user flow. For example, optimizing your virtual economy might mean measuring specific purchase behaviors during discrete moments in the game. The possibility can always arise where there are too few users who trigger an event to make a meaningful impact to app-level metrics like retention.
With signal strength indicators, you can easily tell which statistically significant metrics have the biggest impact on your optimization goals and which ones you can safely ignore.
A brief overview of statistically significant metrics
Statistically significant metrics are already automatically highlighted and clustered together to enable end-users to not only see the optimization goals that are significant, but any other measured event or in-app behavior that is also materially affected.
Significance is based off the confidence intervals in Leanplum, which is set to 95% as a default.
The tiles are ordered by the magnitude of the significance for the given metric (highest to lowest magnitude).
The added value of signal strength indicators
The power of signal strength indicators lies in separating signal from noise. Signal strength indicators drive value for two related use cases:
- Detecting false signals. You can safely ignore any statistically significant metrics that have too few users to move the needle on your optimization goals.
- Prioritizing focus. The ordering of the statistically significant metric tiles at the bottom of the reports page now takes into account signal strength to ensure you are focusing on the highest impact metrics.
Keep in mind that running an experiment to optimize purchase rates will result in a lower signal indicator (1, 2 or 3) versus an experiment higher in the funnel like on-boarding optimization where there is a larger expected DAU sample size (4 or 5).