Time to Reconsider A/B Testing

A/B testing isn’t making marketers very happy. Once a promising solution to website optimization, A/B testing is now seen as faulty, time-consuming and difficult. While setting up A/B testing is easy with the right tools, getting the answers you need takes time, practice and expertise. In fact, most A/B tests are never published because they don’t increase conversions! Marketers are seeing much more benefit from focusing optimization efforts on more modern, user-centric testing methods. Here’s why you should reconsider A/B testing.


the latest on A/B testing


It takes time to get right

Each A/B test requires a hypothesis around how your variations will measure up to the control – and repeated rounds of testing to gain usable insights. It’s a time-consuming process best suited to helping you learn about your target audience with incremental steps. But remember, true lifts in conversions come from understanding the user and serving relevant and valuable offers (something that may be more easily done with data and personas).


It’s error-prone

Did you know that only 1 out of 8 A/B tests have driven significant change? A/B testing is so wonky because you’d need to run tests until you reach statistical confidence – the probability that a test result is accurate. (Most researchers look for a 95% confidence level in order to reach a conclusion.) Otherwise, results may be influenced by random factors and produce false data. Calculating statistical confidence can be too complex for those of us who are not statisticians.


It’s not for every business

Another reason A/B testing fails may be sample size. There are tools available for determining the correct number, but some estimates suggest that 25,000 visitors may be needed to reach statistical confidence, and smaller businesses are unlikely to have that kind of traffic. This is one of the reasons that Optimizely, one of the most popular A/B testing tools, is discontinuing its free platform in favor of one more focused on large enterprises.


Contact our team today to learn more about understanding your key audiences – and making sure your websites are living up to their promise.  

Meghann Porter - Digital Marketing Director

By: Meghann Porter

Digital Marketing Director

Meghann has over 10 years of experience developing digital marketing programs. At Signal, she manages a wide range of digital initiatives – including SEM, social, display, retargeting, SEO, mobile, user testing and conversion optimization. She’s an integral part of our team, working across industries and clients to contribute to the design and build of all web and mobile projects. Meghann is known for her masterful understanding of online analytics.

919-348-2131 LinkedIn

Like this article?

Get Signal’s monthly insights directly in your inbox!

More from the Signal Blog