Statistical Significance

A brief overview of how we calculate statistical significance in AB Testing and MVT at Webtrends Optimize

AB Tests

In AB testing statistical significance represents the likelihood that an observed difference in conversion rates between a variation and the control is not purely due to chance. For AB tests (and ABn) Webtrends Optimize uses a students t-test to calculate statistical significance.

Specifically this is the more robust two tailed approach, meaning that both positive and negative directions are considered. Further calculations are made at a 95% level of confidence meaning that the risk of a false positive is limited.

In addition, we report on the the probability of an experiment beating control. Effectively this is the likelihood that the variation outperforms the live site.

Multivariate Tests

Webtrends Optimize offers both Full and Fractional MVT approaches. Our Fractional Factorial approach leverages a highly tuned Design of Experiments approach coupled with a standard predictive analytics engine to deliver actionable results in a fraction of the time required for Full Factorial tests.

This approach is intended to overcome the limitations of the Taguchi Fractional technique by making over 200 additional test designs available for use in campaigns. All of the test designs offered are constructed for balance and orthogonal testing with the goal of enabling us to calculate statistical significance within the results as accurately as possible.

Non-Binomial Calculations

To calculate statistical significance for non-binomial metrics (such as order value, time on page, units per transaction) requires specific methodologies. Comparing just the variation averages can be misleading as they do not consider the impact of outliers.

For this reason Webtrends Optimize will calculate statistical significance of non-binomial KPI using both the Wilcoxon and Kruskal-Wallis non-parametric methods. This again ensures that improvements in average order value are not down to chance.

Get in touch

If you have any more questions about how we calculate statistical significance for AB Tests or MVT feel free to send us a message, or call us on 0333 444 5502.

Send message