Multivariate Testing

Test multiple variables on a page at the same time with multivariate testing

What is Multivariate testing (MVT)?

Whereas AB testing is when we test complete versions of a web page against alternatives, multivariate testing is a testing technique which helps you to optimise various components within the web page or experience itself. These web page components which are tested are referred to as 'factors'.

A simple MVT example

Say you have 3 factors on a simple web page. A header banner at the top, some text on the left and an image on the right. For each factor you could have 3 different variations (a 'control' and 2 alternatives). These tested variations are referred to as 'levels' in multivariate testing.

The total number of experiments (the array) in your multivariate test is then calculated by compounding the number of levels for each factor. Sticking with this same example, a web page with 3 factors, each with 3 levels would contain 27 different experiments (3x3x3) as illustrated in this graphic, with blue, orange and green tiles representing the different levels and factors.

The number of levels per factor isn’t always the same of course. A slightly different example, still with the 3 factors (header banner, image, text) could have 4 variations (levels) of header banner and 3 each of the image and text. In this case the total experiments would be 36 (4x3x3).

Full Factorial multivariate testing

The approach in the example above is usually referred to as being full factorial multivariate testing. This is where all combinations of factors/levels are tested providing a detailed understanding on the changes made in each factor.

However, as you can imagine, as the number of factors and levels increase so does the number of experiments, rapidly. For this reason, a full factorial MVT is usually only recommended for sites with a lot of traffic and conversion events - and even then, usually for relatively small testing arrays.

Where traffic is limited and a more efficient approach is required, a fractional factorial multivariate testing methodology is best employed.

Fractional factorial MVT

Fractional factorial multivariate testing reduces the number of rotating combinations by testing and accelerating test duration. By using a 'Design of Experiments' methodology the number of variations is reduced to just what is needed to prove the hypothesis.

The efficiency depends on the testing array. So, for example a test with 7 factors, each with 2 levels (including control) would generate 128 experiments at full factorial (2x2x2x2x2x2x2). A fractional factorial methodology could reduce this to 8, thus reducing test duration by 16 times.

Multivariate testing using fractional factorial is therefore often thought of as considerably more effective. You test each variation (level) a balanced/equal number of times and make some predictions about its overall impact in other combinations.

Multivariate testing with Webtrends Optimize

Webtrends Optimize supports both full and fractional factorial multivariate testing.

The platform will automatically generate the reduced fractional factorial array required and report comprehensively on each factor and level in order that the optimal combination can be efficiently identified.

Take a look at this short video to see how easy it is to set up a simple multivariate test in Webtrends Optimize.

Why and when to use an MVT & common misconceptions

Why choose a multivariate test?

When to consider multivariate testing

Dispelling some MVT myths

#1 - We don't have the traffic

So, a full factorial MVT that tests every combination will dilute traffic where sample is a consideration. However, as described above a fractional multivariate test uses a highly efficient experiment array. If you find your site can support an ABn test, then a fractional MVT should form part of your considerations too.

#2 - They are so complex to build

Multivariate tests support a bigger array of changes – so the effort does scale. However, if this is a big concern, concentrating on less dynamic pages and simpler elements such as copy and imagery will reduce effort and has proven to deliver rewarding results.

Get the most from your Multivariate Testing

Drive customer acquisition & increase conversion

Drive customer acquisition & increase conversion

Optimise your forms, landing pages, registration flow or other experiences on your site to acquire more customers. By making the experience more relevant, you'll increase conversions and gain valuable insight into your visitors' behaviour.

Build better experiences across channels & devices

Build better experiences across channels & devices

Create great mobile experiences. Using our support for Android and iOS app testing, you can run multivariate testing on your mobile experiences to understand which combination of elements performs the best and which experience will yield the greatest results.

An essential tool in the digital marketer's toolbox

Multivariate testing (and AB testing) provides reliable evidence to support your decisions based on real visitor interaction. By testing you can see exactly what worked best and which version of your content delivered the most conversions.

Webtrends Optimize allows you to test and personalise all aspects of your digital assets using these tools plus our powerful social proof and product recommendations engines, exit intent and welcome messaging and much more - and all are available as part of our standard offering. No sneaky 'upgrades' required, or features hidden behind paywalls.

View Pricing

Get in touch

Want to know more about Multivariate Testing, or how Webtrends Optimize can help your increase your conversion? Click below to send us a message or call us on 0333 444 5502.

Send message