What is an A/B test?

What is an A/B test?

One of the most powerful tools there is to optimize your web page or web site is to use one or more A/B tests (also called split test). Basically, it’s very easy.

Instead of threshing and arguing about what a page should look like, you choose at least two of your best suggestions and test them against each other. The page that gives the best results, based on data from real visits, wins. Then you refine, make new sub-variants out of the winning page and test again. Then you have created a loop for conversion optimization, based on data.

A/B test on your real visitors

What makes A/B testing so amazing compared to other test methods is mainly two things:

  1. You test on your real visitors and no artificial ”panel” or something like that.
  2. The visitors do not notice the test, so you avoid the uneven distribution that often is the result when the participants of the test are är aware that they are being observed.

A/B tests – This is how it works

You add a simple test script (JavaScript) to the page or pages you want to test. You also add a goal script on the page the visitor arrives to when the conversion goal is reached, for example a receipt page or a Thank you page.

Here are some examples on tools you can use for A/B testing: Visual Website Optimizer, Optimizely, Convert and Google Content Experiments.

When the test has started, the visitors are randomly distributed to the different variants of the test. Then you follow the statistics in the testing tool to see which variant has the best conversion.

This is how an ongoing test could look like, where Control (original page) is measured against Variation (test page):


The test tool keeps track of the visitors with cookies, so that a visitor always gets to see the same variant even when the same visitor returns to the site multiple times. You do not want the same visitor to see different variants, because then you won’t know which variant led to the conversion in the end.

Multivariate testing – for the advanced

In an A/B test you test different pages against each other. In a multivariate test you test different components on a page against each other. For example, you can have three different variants of an image, two different headlines and two different colors for a button. This will lead up to 4 x 2 x 2 = 16 different test combinations.

It is a bit trickier to implement the test scripts for a multivariate test, and it quickly turns into many different combinations, which makes it more difficult to get statistic significance in the results.

Common errors with A/B testing

1. The number of variants is too big for the amount of traffic you have

First of all, you must consider how much traffic and how many conversions you have on your site. For example, if you only have a couple of conversions per day, it will take many weeks (or even months?) before the test is finished. You need to balance how many things you want to test against the amount of traffic you have. A lot of traffic/conversions makes it possible to test many variants and vice versa.

2. You test too many things at the same time

Let’s say you want to test a new image, a new headline and a new button. You don’t have enough traffic for a multivariate test so you run a simple A/B test with the old page against the new page, with all the new elements.

The test results shows no difference between the original and the variant. What conclusion should you make from this? Perhaps the headline was good and the button was bad? Or the opposite? You have no idea.

3. You test the wrong things and make changes that are too small

The hardest part of making tests that are good, is not setting up the test technique – it is to come up with what is worth testing – To create good test hypotheses.

Changes that are less important for conversion, or not “radical” enough, leads you to a test result that will not be valuable for you besides learning to dare more. To be able to get amazing test results, you often need to ”rewind the tape” even more and dare to make even bigger changes.

If you don’t see improvement right away – remember that the most important thing about A/B testing is that you learn. Even from your mistakes.
We have experience from more than 400 conversion projects and are the only “Two star solution partner” in the Nordics for Optimizely, and also certified partner of VWO. Read more on how we succeed with A/B testing!

Any questions about A/B testing? Ask our CRO-experts: