AI is a hot topic at the moment, but it can be difficult to determine how best it can be used. And whilst there are lots of good uses for it, it is also important to manage your expectations of what it can and can’t do.
A recent webinar with our Head of Product, Sandeep Shah, Conversion Founder, Stephen Pavlovich, plus CRO experts Craig Sullivan and Iqbal Ali , explored the ways that AI can be used to add more to experimentation programmes.
In this blog, we will be exploring (and summarising from the webinar) how AI can be used in the real world to upscale, be used for research and prioritisation, augment your workload and more.
Copilot and ChatGPT
The first topic that was explored was about developer research tools, such Copilot and Github, and also ChatGPT, which has quickly become one of the most popular language understanding and generation tools. As Sandeep led the discussion on this section, he started off by demonstrating the power of these tools.
He explained how these tools can be embedded directly into your code editor and with a bit of instruction, they can be used to make changes to your website by actually giving you CSS you would need to make the changes. On the slightly more advanced side, it can create entire functions for you with very large chunks of code.
The power of this becomes evident when you look at the scale of work involved in having to code quite large applications. This can make life a lot easier and really speed up the time it takes to code. And the reality of this is that it could save businesses up to 30/40% in developer costs.
You can also get even more out it by asking ChatGPT to explain the code that has been given. You can then validate the code, which will mean you have a full iteration of the code cycle, ensuring you are learning development very quickly. So essentially, AI has taken out the friction points, for a quicker and more seamless process.
So not only can it act as another resource, but it is also an effective way to add power to your workforce, scale the amount of work you are doing and speed up the time it takes. So, whilst it is not going to teach you to write code from scratch, it will help to direct you to learning better coding and augment your ability already.
Enthusiastic Intern
Next, Craig Sullivan explored the idea of viewing AI as an ‘enthusiastic intern’ and how the perspective ensures you get the most out of it whilst having realistic standards.
He goes on to explain, that whilst AI is an incredibly intelligent bit of technology, it is not as experienced as we are as humans. It’s important to remember that AI is not doing things for you, but rather is helping you and providing assistance where you need it. For example, it can help you with market research, competitor analysis, copywriting, SEO, and more.
Craig goes on to say that it’s not going to tell you how to think, but it will help you to figure out what words and phrases are being used around a certain topic and how it is being spoken about. Moreover, you can utilise a variety of tools that can take on elements of a task that is time consuming, non-value adding, a bit boring, and generally not the bit you want to be doing. This could be audio or video transcripts, note taking, doing recruitment automation, summarising content, and more. So again, it’s not doing the work for you or even thinking for you, it’s helping to minimise the boring and monotonous workload.
Good research and asking the right questions can ensure you find out all the information you need, which can then help you with whatever you want to do next. But it is important to remember that it is not perfect and there are still things which you need to check to be sure on the accuracy of the data. You can access the full slides from the webinar, where Craig goes into more detail and outlines a ‘Checklist for AI Survival’ to help avoid some of the common mistakes and pitfalls of AI.
Confidence AI
Stephen Pavlovich, CEO and Founder of Conversion, then explored the idea of Confidence AI. He starts by explaining that when it comes to CRO and experimentation, the easy part is coming up with ideas. The challenge comes from knowing which ideas, concepts and experimentations to run first. Often, we will have a lot more idea’s than we do capacity, so prioritisation is one of the most important things we can do when running a CRO programme. Which then begs the question, what is the best way to prioritise?
A lot of people will be familiar with the idea of prioritising based on impact and ease. Impact is where you try to estimate how impactful the experiment will be, and ease says how easy it is to build and run. However, as Stephen explains, this is quite a subjective task and it’s hard to estimate how impactful an experiment is going to be. Hence, why we are running the experiment in the first place.
Something to bear in mind with this is that people can be quite bias towards their own ideas, and often think they are going to be more impactful than they are.
At Conversion.com, they wanted to find a way to take data from all of the 1000+ experiments they run and then apply prioritisation in the future, using AI. They started to categorise experiments using a huge number of variables. From this, they start to identify the core principle of the experiment that they are running. The Lever Framework they have developed means they can categorise experiments based on the core principle they think affects user behaviour.
Then, they take the data which they have categorised across thousands of experiments, and they are training a machine learning model to be able to predict the confidence in experiments. This means they can harness data from all the experiments they have ever run, from all the clients they have worked with, to then determine how they prioritise experiments based on how successful they think they will be. So, the more experiments they run, the better abled they are to predict which experiments are going to be successful.
Summarisation
Next, Iqbal explores how AI can help to understand complicated text and ideas. He says that AI is clever when it comes to rewriting text in a clean way and making it much clearer. The key is also giving it good quality context, for example, information about the company, to provide as much detail as possible.
Iqbal then introduces the idea of Chained Processing. He explains this as initially taking some complicated and messy text and giving it some good quality context behind the text. Then we take some badly formatted text (essentially raw input) and split it into multiple chunks. This is done because people quite often mention multiple themes in the same text. So, we would use AI to split things into specific points, and then you can do some pretty aggressive summarisation on that split without removing the context. Once you have done this, you can then also group all of the text into clusters, and then you can do progressive grouping of the broken text. By having the process laid out in a very specific way, it allows you to validate the output at each step and see if any data set is not working correctly.
Iqbal suggests it could be used as a way to sort through and understand big volumes of data. So, for example, he has used it previously to sort through a whole bunch of reviews for a company that has hundreds of products. This means it is hard to decipher what product the review is referring to, so you can’t see the positive or negative things being said about specific things. AI can be used to sort through this and extract the useful text from it. For example, you could ask AI ‘What is the delivery company that most customers complain about?’ and then tailor your visitors overall experience to ensure they don’t encounter any issues. ChatGPT can also produce graphs and detailed analysis, so you can get the most out of it.
All of this is done through the extremely clever AI app that Iqbal is building. He currently uses this app when working with clients to help summarise their large corpus of text.
Sentiment Analysis
Iqbal also explained the idea of Sentiment Analysis. This is essentially the process of analysing digital text to determine if the emotional tone of the message is positive, negative, or neutral.
In terms of writing stories or something similar, you can put a description of the story or a description of some of the text that you want to put on a website. Then, you can use an AI tool like ChatGPT to ask how likely it is to be interpreted and what themes are suggested by the text. A lot of the time, it’s very accurate, but it’s important to remember that it might not always be. And it usually gives you good things to think about and unpick.
The idea is that there is lots of stuff that’s locked away in the text, like sentiment that you can then get access to.
In-Context Semantic Search
Finally, the idea of In-Context Semantic Search was explored. Stephen explained how at Conversion, they have noticed a challenge in getting insights from the many documents, emails, experimentations, etc. You can only get value from it when it is retrieved, which can be a difficult task. However, AI now gives us the opportunity to have this kind of contextual semantic search which increases the efficiency and effectivesness of that insight and data retrieval. It can use natural language processing to really understand the intent behind the question.
Stephen also shared something they have been working on at Conversion. The tool works by providing a way to search back through other experiments for insights and ideas. For example, if you’ve observed that there are concerns around pricing for example, then you can type in a term like ‘high pricing’ and access every experiment and every bit of insight around that specific topic. It can bring up different solutions to similar insight and provide inspiration and ideas around the topic. A big advantage of this is that it eradicates the learning curve and get up to speed a lot quicker by having access to years of data and insight.
The Power of AI
AI is an incredibly powerful tool and there are lots of different ways you can use it to enhance your experimentation program. It can help you to gain insights from your experiments, as well speed up the time to takes to do menial tasks, as well as cutting developer costs and acting as another resource. Essentially, the possibilities are endless.
If you want to speak in more detail about any of the topics covered, get in touch and we are happy to help.
You can also watch the full recording of the webinar on our YouTube channel.