How to run 2 split URL-tests towards the same landing page

Johan Nyberg

How to run 2 split URL-tests towards the same landing page

I encountered the problem that I wanted to start a new A/B-test (split test) for a client, but where the Control, ie. the page I wanted to challenge with my test, was used as a control in another A/B-test not yet reached the end of it’s testperiod.

In my example, I have used Visual Website Optimizer, but the principle applies even if you use a different testing tool such as Optimizely or Convert. How do you resolve this? To begin with I must explain the difference between an A/B-test and a split URL test.

A/B-test

A test type where you change one or more items on the same page. If you change more than one thing on the page, so it becomes a so-called multi-variate testing. One caveat: if you do not have extremely high traffic to the page, hold down the number of variations. The number of permutations takes off exponentially with the number of variations, and it may take time before your test goes into the goal. A good rule is to use a calculator for this purpose, which calculates how long your test is expected to take. Visual Website Optimizer, for example, his calculator here:
http://apps.conversionista.se/visual-test-duration-calculator/

Split URL test

A test where you have two different landing pages. You have the existing page, and a contender. For the most part you want to send 50% of traffic to each page. This type of test you want to run if you in your evaluation of the existing page has discovered a variety of deficiencies that may adversely affect the conversion. In this case, it is often easiest to build a whole new page, rather than trying to modify the existing one.

Ok, now you know the difference between an A/B-test and a split URL test. Now for the description of the two tests I want to run against the same landing page. And now you may have already figured out that both tests are split URL test, where one of the variations in the respective test runs against the same landing page:

Test 1: Test that is already up there

The test is already up and running, but have not yet reached statistical significance for conversion. However, it has already won in engagement (which is the opposite of bounce rate). Control index.html Variation: test1.html

Test 2: the new test that I want to start

The customer has just started a new banner campaign, and I have built a landing page with the same apperance and messages as the campaign. We usually talk about “keeping the scent” between search / ad and landing page, visitors are greeted by the same message / look of the landing page in the search results / advertisement. Control test2.html Variation: index.html Note that I use test2.html as Control. This is important. If I had used the index.html as Control for both tests had my testing tool would not have known which of the tests that I wanted to run. This opens up to another issue. If a visitor are exposed to Test 2 and 50% will be directed to the index.html. How can I make sure that Test 1 does not kick in and send 50% of the visitor to test1.html?

The solution is called cookies

All testing tool works in such a way where they put cookies in visitors taking part in a test. Th testing tool partly set a cookie that tells you which test the vistor are in, and it adds another cookie that tells which variation that they were exposed to.

This is done so that if the same visitor comes back to the site, they will always be presented with the same variation as they landed on the last time. To see the cookies that are created in your test, so I recommend you to install a plugin for Chrome called Edit this Cookie. Here you can see a screenshot where I’ve opened up the tool when the visitor first came to Test 2 on page test2.html and then forwarded to index.html. Edit this Cookie - Chrome-plugin
The cookie that we are looking for is called (based on that I use Visual Website Optimizer) _vis_opt_exp_3_split . The cookie has a number(3) in itself due to that Test 2 has a test-id of 3. And the cookie has a value of 2. This means that I, as visitors take part in the test with id 3, and has been exposed to variation 2 (index.html) in the test.

Next I go into Test 1 and add a segmentation rule for the test. In the middle segment (see screenshot), I choose: Cookie Value as a segment and _vis_opt_exp_3_split as a variable. I also adda segmentation rule of: not equal to (case insensitive) and finally in the field of value as I enter: 2. change cookie-value

This means that visitors have a cookie named _vis_opt_exp_3_split with the value 2 has been in Test 2 and sent to the Variation 2: index.html. We therefore exclude these visitors from Test 1, so that 50% of them aren’t sent to test1.html. Hope you kept up with me so far, it could be hard to get your mind straight but when you do, you see it’s not that hard. In closing, you get a link to A page that explains all the cookies used by Visual Website Optimizer.

Did this blogpost help you in your conversion work? Do you have any questions?

Write your comment below.

Get help from the strongest optimisation team in Sweden

Conversionista is open for business in The Netherlands.
Conversionista is open for business in The Netherlands. Read more.