When any ecommerce site is looking to optimise digital experiences, AB testing and Multi-Armed-Bandits testing (MAB) are two of the most valuable experimentation techniques. But the key problem is here, when should you use each method? They each have pros and cons that make them better suited for certain use cases, which I will explain throughout this blog.
As a recap, AB testing a common experimentation method for comparing two or more variations to determine which one performs better. In this method we split visitors evenly between a control and version(s) of a page to see which performs better on key metrics like conversions. Multi-Armed Bandit is an algorithmic approach used in decision-making problems to balance exploration and exploitation. MAB dynamically shifts more traffic to better performing variants, so your visitors are always seeing the best performing variant.
Comparisons
AB Testing and MAB both have their places as testing tools for website optimisation, but their different strengths mean they are more suitable for certain scenarios than others.
AB Testing strengths
- It is a simple and easy method for comparing two different variations of a website
- AB Testing splits users evenly from the start into different groups and measures performance
- You can use AB testing for low traffic websites and still achieve reliable results
- AB Testing doesn’t rely on leveraging any sort of historical data, so has no dependencies
Multi-Armed Bandits strengths
- MAB can continually adapt, and update decisions as new data is collected, so the winning variation can be instantly implemented rather than waiting for the test to finish
- When using MAB, you can easily test multiple variations on a website
- MAB is an effective method for website where there is lots of dynamic contents on the page
- You will likely choose to use MAB if you have a limited time to find a winning variation or you want it implemented as soon as possible
AB Testing is typically the most common approach to compare variations as it is easy to implement and determine who the winner is. Multi-Armed Bandits is a specific algorithmic approach within the realm of AB testing that balances exploration and exploitation. So, whilst you might naturally lean towards AB testing as a straightforward and reliable testing method, MAB is still a good option to consider as a dynamic approach and when you want the winning variation to be implemented as soon as possible.
Practical examples of AB Testing
Pricing strategy
When you need to test something very critical like different pricing strategies, AB testing would be preferable. AB testing allows you to compare the performance of different price points on a subset of customers and measure their impact on sales, revenue, and customer behaviour accurately.
Website redesign or significant layout changes
If you are considering making significant changes to your website design or layout, AB testing is recommended. It enables you to test different variations of the design on separate user groups and gather quantitative data on metrics like click-through rates, conversion rates, and bounce rates to determine the optimal design.
Product descriptions or imagery
When optimising important pages like product descriptions, imagery, or product listing pages, AB testing provides a controlled environment to measure the impact of changes on customer engagement, conversion rates, and overall sales performance.
Checkout process optimisation
Improving the checkout process is crucial for reducing cart abandonment and improving customer satisfaction. AB testing allows you to test different variations of the checkout flow, such as different form layouts, payment options, or guest checkout options, to identify the most effective configuration that maximises conversions.
Practical examples of Multi-Armed Bandits
Personalisation
Instead of comparing fixed variants in AB testing, Multi-Armed-Bandit can dynamically personalise content based on user preferences. As a specific example, news websites often aim to provide customised content recommendations for each reader. However, AB testing a few headline or thumbnail variations does not capture individual interests during testing. In contrast, a multi-armed bandit algorithm can learn to promote personalised content by optimising for each user's clicks and engagement. As the user's interests change, the bandit adapts its article recommendations accordingly.
Marketing Campaigns
Short online marketing campaigns must capture user attention immediately to be effective. However, running a few AB test variations of an ad fails to adapt to real-time user responses. Multi-armed bandits better optimise ad variations by treating each variation like a slot machine "arm" that can be played to maximise reward. Online ads have short lifetimes, so Multi-Armed Bandits excel at rapidly finding the right match between ad creative and user. The limited scope of AB tests pales in comparison to the continuous optimisation of MAB. By constantly learning and adapting ad variants, bandits drive better advertising results.
Recommendation Systems
The multi-armed bandit (MAB) method is particularly well-suited for recommendation systems, where the goal is to suggest the most relevant items (e.g., products, movies, articles) to users based on their preferences and behaviours. In this context, each item or set of recommendations can be considered an "arm" of the bandit problem. By utilising a MAB approach, recommendation systems can efficiently explore different recommendation algorithms while simultaneously exploiting the best-performing ones. The system continuously updates the reward estimates (e.g., user engagement, click-through rates, or ratings) for each recommendation and dynamically adjusts the recommendation strategy to prioritise the options that have proven to be most effective for individual users or user segments.
Website Content Optimisation
Optimising the content layout, design elements, or call-to-action buttons on a website to maximise user engagement or conversion rates. MAB can adaptively select the most effective variations based on user interactions, continuously improving the website's performance.
What can you learn from this?
AB and Multi-Armed Bandit testing both have advantages in certain situations. For experimentation strategies to be successful, leveraging the right methodology based on the circumstances, what your goals are and the timeframe you need to achieve it in, is crucial.
With a strategic testing approach, companies can continuously learn about their user’s behaviour and provide them with the best experience. Knowing when to opt for AB Testing or MAB can move optimisation efforts forward whilst making the most impact.
