Over the past 5 years A/B testing has become standard industry practice. According to a study recently conducted by Econsultancy, 67% of businesses use A/B testing to optimise their conversion rates. However, despite this, 90% of A/B testers say that they’re unsatisfied with the results! With this alarming statistic in mind, we decided to ask the question: does A/B testing work?

There is another way for us to think about these figures. If 90% of testers were unsure about the results, that means 10% of people saw an improvement in their website’s performance! If we allow for unrealistic expectations and imperfect test design, there is a strong chance that A/B testing can help to optimise a website. Even so, the statistics are a reminder that website optimisation involves more than strolling from A to B.

Does A/B Testing Work?

Of course, A/B testing is not a ‘magic bullet’. There is no wand-waving or secret formula involved when you subscribe to AB Tasty or Optimizely.

Many people are surprised at the work that can be involved. For example, take a look at this account written by Laurence, a marketing professional who works for an online shoe shop. He discovered A/B testing in the early days.

A/B Testing Case Study: Laurence, Her 1st AB Test

Laurence Chooses AB Tasty

After some Google research on A/B testing services available, I made up a shortlist comprised of: AB Tasty, Kameleoon, Optimizely and VWO.

In the end I went for AB Tasty because our office is based in France… 

The starter pack wasn’t expensive: only €29 for testing 5,000 visitors. Seemed pretty reasonable for a first test…

Laurence Designs a Test

Most of the online articles I’ve read talk about how simply changing the colour of a Call to Action button can increase conversions by 300% so I thought I’d start with something simple like this. After just 3 hours, my credit of 5,000 visitors had run out and the AB Tasty interface was showing a very low rate of reliability, which meant I would need to do further testing. But how many more times would be sufficient? … I changed to a €99 pack to try and get some satisfying results from this first test.

The next day, the reliability level had gone up to ‘Green’ with an improvement rate of 27% showing! I was elated at this result and I went straight to my Manager to give them the good news: a 27% increase in revenue! Except, the thing that I hadn’t realised is that this 27%  increase wasn’t particularly visible in the end of month revenue report. In fact, the increase barely showed in the figures and my Manager wasn’t convinced that any increase was down to my A/B testing…

Laurence’s Conclusion

In the end, my first experience with A/B testing wasn’t particularly conclusive, which made me realise that signing up to an A/B testing solution probably isn’t the way to go unless you really know what it is you want to test and know how to manage the test correctly.

Laurence’s experience is a classic one. A/B testing solutions do not offer a magic solution for improving your conversion rates. So, does A/B testing work? Well, if you follow these 5 simple pointers, they can offer a valuable tool.

5 things to remember about A/B testing:

1.  Unless you have a coherent method for optimising conversion rates, using an A/B testing solution service will only be a waste of time and money. At Convertize we provide a library of neuromarketing principles with which to structure your tests.
2. Signing up to a pack that only tests 5,000 visitors is unlikely to be of any use. Such a small sample size would only provide you with statistically significant results if you achieved an enormous increase in conversions. Achieving an increase of 10% would be a remarkable feat for a first-time tester. You can read about sample sizes and statistical significance in our blog. 
   
3. Bloggers often claim that simply changing the colour of a button will increase conversions. Whilst this is a surprisingly powerful tactic, such a small change on its own is unlikely to sustain an improved rate.
4. You should always check the significance of your tests. Otherwise, you risk waking up each morning thinking you’ve doubled your sales!
5. A/B testing isn’t always applicable. If your site does not generate enough conversions, then starting A/B tests won’t be of much help as you will never get a fully representative statistic. You may be much better off applying proven solutions from Neuromarketing and Nudge marketing experts.

Does A/B Testing Work?

The short answer is: ‘yes…in the right circumstances.’ There is a reason why companies with big marketing budgets use A/B testing software; it provides insights that no other method can. For most marketing websites, there is something to be learned through A/B testing. Whether A/B testing can work for smaller businesses with lower-traffic websites is more open to question. 

by Philippe Aimé

Philippe is the CEO of Convertize. He created his first website in 1998 and has spent the past 20 years finding ways to make digital marketing more persuasive. He now heads a team of CRO consultants.