A key part of any successful experimentation programme is the research.
Without research, you are experimenting based on hunches and opinions, and your success rate will significantly drop.
Paula Sappington spoke about Hilton's journey to customer-driven optimisation at Experimentation Elite this year, and how there is a stark difference for them between the win rates for experiments with and without user research (43% and 22% respectively). The opportunity to double your likelihood of success is just too good to miss.
But, with so many different types of research out there, which one is the best for you? The truth is that the best programmes adopt a wide spectrum of research types. I've always found it helpful to split them into two camps:
- Quantitative Research - In short, numbers. This is what you use to understand what is happening on your website, where and when. Looking at a quantitative analytics tool will help you to understand where there are leaks in key conversion funnels, where users are spending their time, what they are interacting with, where they are coming from, plus how this has changed over time. However, there is one key gap, which is why we turn to...
- Qualitative Research - This provides the why, which completes the picture. It's all very well knowing that 30% of people are dropping out at stage 2 of your checkout, but without the context of why, you are reduced to guesswork, and that much smaller chance of resolving it with an experiment.
In the remainder of this article, we will touch on different techniques in each of these categories. Remember, the more different types of research methods you adopt, the more complete picture you will build of user frictions on your website and opportunities, allowing for easier ideation and prioritisation of experiments, and a greater chance of success.
Web Analytics
This is one of your best sources of quantitative research. Two providers dominate the market, Google and Adobe (although providers like Matomo are becoming increasingly popular as concerns are raised about where user data is stored).
Your web analytics are only as good as your setup - and there are plenty of people who can provide more context here. Bhavik Patel has some particularly useful resources, and Luke Carthy recently presented at Experimentation Elite about how to get the best out of the much-maligned GA4.
The work that you did in understanding the metrics that matter to you will be helpful in getting your setup right. Once you are tracking the right metrics, you will want to understand how different segments are performing, including dimensions like device, browser, traffic source, and many more, as well as how metrics have performed over time (has there been a recent drop in a key metric?). Funnels, and their leaks, are also incredibly important. For a retailer, you will want to look at areas like your checkout funnel, but also a broader funnel from Landing Page > PDP > Add To Cart > Checkout > Thank You Page (as well as some stages in-between).
For a business who relies heavily on forms, you might also want to invest in form analytics as a better way to track and understand interaction here.
Session Replay
Session Replay tools are a great way to understand why things are happening on your website. Being able to watch replays of real customer sessions can be really illuminating and help to reveal why you saw a particular metric in your web analytics.
However, any website of a decent size produces far too many replays to be able to review them efficiently. You need to be strategic.
At a basic level, you need to filter your replays for particular events, for example, abandonment on the cart page for Safari users in the last week. Your web analytics will guide you here.
However, this can still generate a high amount of replays, which is why some providers have introduced struggle scoring into their platforms, detecting customer frictions, allowing you to prioritise which sessions you watch.
Heatmapping
One other common research method for websites is heatmapping. This allows you to understand customer behaviour on a page in a much more visual manner, in aggregate.
A scroll map can be particularly useful for understanding how far people are getting down a page, what they are seeing and what they miss. More advanced tools help you to understand click-through-rates, which reveal if some lesser seen items are disproportionally clicked, indicating that they may need to be moved up the page. Historically, I've found that scroll heatmaps have been particular revealing for mobiles, with scroll depth much lower than many people think.
Click maps are great for understanding what people have been clicking and tapping on, and sometimes even the order that items have been interacted with. These build an understanding of what customers are interested in, especially when combined with attention heatmaps, showing where they hover the mouse (often analogous to where they are reading). Some tools also provide revenue per click, which helps you to prioritise the order of items in menus for maximum benefit and assess the performance of particular banners and campaign content.
Voice of the Customer
So far, these research methods have looked at what customer behaviour can tell us, without actually interacting with the customer themselves.
One way to involve the customer voice is with Voice of the Customer (VoC) surveys, deployed on your website.
These can collect customer sentiment, like CSAT and NPS scores, but there is great value in asking your customers questions and getting their feedback.
There is an art to asking the right questions here, and when to deploy the popup, but many great insights have been obtained through exit intent pop-ups with questions around what prevented the visitor from purchasing today. Post-purchase surveys can also garner some great insight.
User Testing
Another great way of understanding your customer and why they do what they do is user testing, whether that is remote, or in person.
In person testing will, arguably, gather deeper insights as you can read more of their cues that you would be able to through a screen, but you can often achieve larger scale at a lower price with remote user testing.
One advantage of user testing is that you can gather feedback on things that you haven't even fully developed yet, understanding how users react to your new designs, even before you reach MVP stage.
User Interviews
If you want to get even more in-depth with your users than user testing can, there is an incredible amount that you can gain from interviewing your existing users and your target market.
Experts like Els Aerts and Abi Hough can give you a lot more here than I ever could, but understanding user motivations, what causes them to buy with you, buy with a competitor, or not buy at all, as well as much more, is incredibly valuable.
Select the people you interview with care, understand that what they say and what they do are two different things, and remember the dangers of making big decisions on a small sample size. However, combining user interviews with all of the other research methods above will give you unparalleled insights into your users and where friction points and opportunities in your website might be.
Some research is better than none...
I hope you found this dive into research methods and the different techniques you can incorporate when learning more about your users. I'm sure there are research methods/types I have missed, including journey mapping, and there will be more nuances than I've had time to go through here. I suggest checking out some of the experts I've referenced for more information and insight on their specialisms.
Remember, although this sounds like a lot, don't be intimidated. Some research is better than none, and there are plenty of individuals, agencies and tools who can help you with the above tasks to make your experimentation programme a success. Get in touch to learn more, or feel free to reach out with any questions.
