A/B testing analytics are very similar to general user activity Analytics so read up on the basics. To see analytics for your A/B test, go to the Analytics tab in the side navigation, or click View Results directly from your A/B test. This will bring you to the Analytics Dashboard, where you can view all sorts of metrics for your test.
The metric tiles in Analytics are split into three different sections: Goals, Favorites, and Significant Changes. If you are testing a message, you'll also see the message or campaign's default analytics in the A/B Test metrics.
The metric you select (either via the dropdown above the chart, or the metric tiles below) will be displayed on the graph. The graph defaults to displaying Daily Active Users. Above the graph, you'll find labels for the control and all of the variants, as well as the percent change each variant experienced compared to the control. Checking the circle next to each variant adds it to the chart comparison.
Below each graph, there are tabs representing each variant. All metric tiles represent the percent change compared to the control group. Navigate between these tabs to see the percent change for metrics across variants.
When looking through your metric tiles, color coding indicates a positive or negative change for each metric. Once a metric change has reached statistical significance, the color will be changed to Green or Red. Green indicates a positive change in the metric while Red indicates a negative change in the metric.
For any A/B Test or Holdback Message, you can select as many goals as you need. Since all data will be captured and analyzed for this test by default, goals can be added or removed from the test at any time, without any loss of historical data.
If you are running a test that includes a message, you will automatically get the default messaging metrics within the A/B Test. These metrics show statistical significance as well, making it very quick and easy to tell if one message variant is encouraging more engagement than another.
Just like every report in Leanplum, your favorite metrics will be shown for quick analysis. Favorites can be changed and updated at any time, from any report. Use this section to save your most important metrics which can be used to analyze the results of any test or campaign.
Significant Changes Metrics
Significant Changes is a section that Leanplum automatically populates with any metric that undergoes a statistically significant change as a result of this test. This helps you avoid unforeseen consequences. For example, a test may positively affect your goal metrics, but negatively affect other metrics that you didn't expect.
Signal strength indicators
The signal strength of a metric represents the portion of your audience that a metric reaches. Signal strength appears as a small grey bar in the upper right corner of a metric tile. (Signal strength only appears on tiles that relate to your custom events, not on standard out-of-the-box metrics.)
For example, metrics that reach a large portion of your audience, such as the percent of users per day that trigger "main menu," would have a strong signal strength. However, there might be a weak signal strength for a metric such as the percent of users per day that trigger Language Settings, if less users tend to access that portion of the app.
Signal strength is designed to put metrics into perspective, allowing you to consider the effect a change has on your audience as a whole. Even if a metric has a very drastic change in terms of statistical significance, you may decide that it's not as important as other metrics you are watching if it only affects a very small portion of your audience.