Find out where it’s possible to A/B test your site with the Experiment Feasibility Calculator

Simon Dahla

Numerous sites in Sweden don’t have enough traffic to A/B test, but some can test certain pages on their site. If you can A/B test or not depends primarily on what goals you want to measure, for example the traffic volume or the expected increase in conversion. Below is a step by step guide on how to use the calculator to see where on your site it is possible to A/B test.

We A/B test a lot and are also teaching others how to work with A/B testing correctly. The problem today is that a lot of sites run experiments, draw conclusions and get results that not are reflecting reality at all. Or perhaps even more tragic: starts an experiment that never reaches statistical significance*. Ouch!

*Statistical significance = Shows whether a result is representative given your predefined values (frequentist statistics). Given the expected result, you can tell with some certainty if the results are reliable or not (always with a certain standard deviation with high or low probability).

Find meaningful experiments

There are many tools today that help you determine how long an experiment must run to get statistically significant results (we have one too, 🎉surprise 🎉). The most common scenario is that you FIRST enter the expected change yourself, and THEN you get a number on how much traffic you’ll need. But, you already know your traffic, so we’ve turned things around. Also, we want to see all (template) pages simultaneously, and show the difference in testing against various goals.

So, what we want to do is decrease the level of difficulty regarding determining how long to run an experiment and also how large an expected uplift can or must be (to be statistically significant).

The experiment is when the wheel meets realityThe calculator we present to you now therefore tells you how big the smallest change between the variations must be for us to discover it.

Moreover, with this tool you will get an overview of the entire site right away. Then you don’t have to evaluate how “doable” each experiment will be in advance. You only need to use the tool once (as long as your traffic and your site’s user behaviour doesn’t change dramatically).

What factors affect if an A/B test is meaningful?

There are several different aspects that affect the length of the experiment and how close the result is to the truth.

  1. How long the A/B test runs for (number of visitors exposed)
  2. How many variations that are included
  3. Your current conversion rate
  4. Your expected change in the conversion rate
  5. Which goal the experiment is measured against

Why is number 4 bold? Well, how big your expected uplift will be depends on how data driven, worked through and reliable your hypothesis is (that is, if the change affects user behaviour or not).

Many intended experiments will therefore not be feasible. The Experiment Feasibility Calculator is one of several tools that are used to prioritize and plan your backlog of improvement hypotheses. To see if an experiment is doable or not, you want to know right away.
Experiment Feasibility

The Feasibility Calculator visualizes your sweet spots

The Experiment Feasibility Calculator visualizes where you can A/B test on our site. Or rather, how far down into your funnel or on which type of page an experiment is feasible.

All you need to do is enter the data from your Google Analytics Account right into your spreadsheet, and the tool will do the rest.

Get the tool, the Experiment Feasibility Calculator

Or: Get the PRO-version with automatic Google Analytics connection here

You need:

  • Properly set up tracking in Google Analytics (or another analysis tool)
  • If you have an e-commerce site, Enhanced E-commerce is advantageous
  • Or possibly, create Custom Reports (we show you how in a tutorial below)
  • Set up Advances Segments (we show you how in a tutorial below)

*Custom reports and Advanced Segments are needed to get the data you want (you want to do this, and if your traffic and the behaviour on your site doesn’t change dramatically, you only need to do it once). Just look at the tutorial below.
When you have filled the document with your own data, the calculator will show the uplift the experiment will require on different parts of your site.

See the difference between different goals

(click to enlarge)
Feasibility calculator for A/B testing

Read the color codes like this

(click to enlarge)
Feasibility calculator for A/B testing

You will also see how test length (and number of variations) affect how feasible a test is:

Time for experiment

Get the tool, the Experiment Feasibility Calculator

Or: Get the PRO-version with automatic Google Analytics connection here

In order to cover the most common questions, we have recorded a demo that shows you step by step how to fill in the document and to set up the needed reports and segments.

Step by step, how to use the calculator:

In the demo, we use the Google Merchandise Store (a site where Google sells Google stuff, but they let partners use it as a GA demo account). Of course it’s possible to use an other analysis tool to find the requested data.

The tutorial is in Swedish.

We go trough the following steps in the tutorial:

  • Calculation for A/B testing Sitewide (01:05)
  • Calculation for A/B testing on Home page (04:00)
  • Calculation for A/B testing on Category pages (06:30)
  • Calculation for A/B testing on “Checkout-page” (11:55)
  • Calculation for A/B testing on “Checkout-complete” (11:55)
  • How to read and use the table (16:15)

Did you find out that you can’t A/B test? No worries.

This is how you optimize if you can’t A/B test

Feel free to ask us questions in the comment section below, it can help others too.

Läs även