Multivariate testing is a great tool to use as part of your conversion rate optimisation strategy. But what are the scenarios when you should be considering using MVT?
Stakeholders can be understandably apprehensive about using MVT tests. MVT tests are sometimes perceived as more complex projects that need a lot of traffic, time and resources.
In this blog we are going to look at five multivariate testing examples when it is absolutely the best possible choice of experiment, and address the most common concerns to running MVTs that we have sometimes heard over the years.
First things first
Firstly, we need to remind ourselves that, as with each and every optimisation project, we should start an MVT by formulating our hypotheses. One of the advantages of a multivariate test is that several hypotheses can be tested at the same time.
This is important!
We are not testing because we have an array (aka experiment matrix) to fill in but because there is a genuine need to explore multiple hypotheses within the same project. Once you have formulated your goals, setting up your factors and levels becomes a much easier task.
Multivariate testing examples
So now let’s look at some key multivariate testing examples…
1. Testing individual elements on the page & understanding what matters
You may be very familiar with the first of these multivariate testing examples.
You have undertaken a radically different redesign of a whole new page and you tested it using an ABn test. As a result of the project, you identified your optimal. Now is time to fine tune the improvements. This is when you can use an MVT test to explore further options on how to design and optimise different elements within the page.
Or you would like to change the page but instead of changing the whole layout you would like to see how each element on the page impacts on its performance.
Factors for further testing can vary from hero images, frozen or moving carousels, sizes of the images, to colour schemes, fonts, the position of the call to action and many more.
This is one of the most common multivariate testing examples and it can run literally on almost any page of your website e.g. landing pages, SEO pages, home pages, product pages.
2. When you need to tidy up a page
The scenario describes a situation when there is a complex ‘busy’ page and you would like to understand what elements create friction and what elements are important to your customers. How do you decide what has the biggest impact on the performance of the page and what is redundant?
This time running MVT is the best option to understand the key drivers on the page.
Below is an example of a basket page on B&Q. In this example an MVT can help to answer questions such as:
- Is the search bar needed on the basket page?
- Should we keep the main nav?
- Do we need product recommendation examples here?
- Do payment card icons provide reassurance, or do they cause friction?
- Does the footer cause distraction?
Multivariate testing evaluates the performance of each factor and its influence on the conversions rate.
It also helps us to evaluate all possible scenarios of the page layout and the relation of each factor with each other.
Overall, MVT is a more objective way of doing such a test and it eliminates a degree of subjectivity when the designs are chosen based on personal preferences.
3.Testing elements with high information density
There are some areas on the page where the density of information is very high. These examples include price and discount presentation, or multiple CTAs in proximity.
This is one of the best multivariate testing examples as one element appears to be interlinked with another.
A good illustration of this scenario is the price and discount presentation on product listings pages on allsaints.com.
MVT would be the appropriate choice of test here and could help us to identify:
- Whether making the price after discount more prominent can help conversions
- Whether changing the place of the final price has an impact
- Is crossing out the original price more impactful than listing a % discount?
An important word of caution should be added here: Before designing a test, you should always check that all elements can be tested independently. If there is a case of interdependence and changing one element breaks another then you should maybe explore other testing options.
4. Testing Call to Actions
Button tests used to be popular. Nowadays, I often even hear that these tests are passé, and CTAs have no real impact on user behaviour… Yet, in my experience, even very recently we have had some astonishing results with button tests. A MVT is the perfect tool to allow you to test multiple button options at the same time:
- Copy variations – more direct, or more playful, changing depending on the weather or visitor segments
- Colour, shimmer or any other special effects
- Size
These multivariate testing examples can also cover progress bars, hello bars and other single elements. In short, a carefully designed MVT test allows us to test copy, colour, size, background, shape of the same element in one single test. When traffic allows, this is the most comprehensive approach to testing.
5. Testing timing and position in addition to design
The last of the multivariate testing examples we’re looking at today includes exit intent and promo overlays, email subscription pop-ups or chat boxes.
The advantage of an MVT test in this scenario is that it allows to test design options together with time and place where they appear.
Design aside we also know that the timing and position of a message can have a significant impact on performance. With MVT we can explore different multi-dimensional factors too.
Let us look at this example of a subscription pop up on www.made.com.
The email sign up pop-up can be tested to answer some of these questions:
- What is the best layout option?
- When is the good time to show it?
- Where is the appropriate place for it to appear?
So what are the objections to MVT?
Now that we have talked about the scenarios where running a multivariate test is a better choice of test, let’s address the objections that CRO specialists may encounter when presenting their MVT ideas to their stakeholders.
The most frequent objection to an MVT refers to the volume of traffic needed to reach significance. However, this does not have to be the case.
Firstly, at Webtrends Optimize we have the capability to run fractional factorial tests rather than just full factorial. With arrays based on a proprietary algorithm in which each level of each factor is tested at least in one experiment.
Using this approach an MVT test of 4 or 8 experiments may still be larger than a regular ABn test, but the benefits and the quality of the experiments easily outweigh the drawbacks.
Whilst MVT may not be the right fit for smaller sites. Larger e-commerce businesses usually have less traffic restrictions and therefore, don’t have to give up doing an MVT on these grounds.
The Second main objection evolves around MVT complexity as well as development and QA time.
Whilst an MVT test is a more complex job, it is still more likely to save you time and effort if you compare it with the number of AB tests you would need to run to investigate all the options you are able to test in a single MVT.
You will often find that you save yourself some time and effort at the end of your testing cycle.
Conclusion
With all the detailed insight you are going to get using the above multivariate testing examples as well many others, it is certainly very rewarding and will save you time and effort in the long run.
If you are interested to explore how Webtrends Optimize can help you to devise the right conversion rate optimisation strategy for your business and help you to choose the right project for you needs, please speak with us.
I and the team here are always happy to have a chat, even to just provide advice and guidance.
