Specialism · A/B testing
A/B testing: knowing instead of guessing.
Experience and best practice are often right, but not always. With an A/B test you measure whether a change really works better before rolling it out across the whole franchise’s site.
When
Time for A/B testing if…
- there’s debate about which version of a page works better
- you want to roll out a major change to all location pages
- your website has enough visitors for reliable figures
- changes have been made on gut feeling so far
What we do
From hypothesis to a decision you can back up.
- 01
Hypothesis
We set out what we’re changing, what we expect and why.
- 02
Set-up
We define the goal metric, the duration and the number of visitors needed.
- 03
Variants
We build the test version within the brand guidelines, alongside the current version as the control.
- 04
Duration
The test runs long enough to account for daily and weekly fluctuations.
- 05
Audit
We check whether the difference is statistically reliable, and what it does across all channels.
- 06
Decision
If the variant works, we roll it out. If it doesn’t, we’ve still learned something.
A/B testing across all channels
A better conversion rate mustn’t cost you elsewhere.
A change that raises the conversion rate on a page feels like a win. But a change can also have a negative effect on other channels. Think of a shorter page that converts better but is found less easily in Google.
That’s why, with every A/B test, we look not only at the conversion rate on the page itself, but also at the effect on everything else. Only if the bottom line is better do we roll out the change.
What an A/B test is
Two versions, one winner.
In an A/B test you compare two variants of a page, email or ad. Half the visitors see the current version (the control group), the other half the new version (the test group). Then you look at which version performs better against the goal you set in advance.
That can be about big differences, such as a different page layout, or something small, such as the text on a button. With a franchise you often test on a template that applies to all locations. A good result then works everywhere.
To be fair, not every website has enough traffic for a reliable test. If that’s the case, we’ll tell you and choose a different approach.
Part of
Every specialism works for the whole franchise brand.
This specialism is part of Hybrid and On-demand, used for the brand and for every location. Not a franchise, but still have a question? You can also hire us for this on its own.
Other specialisms
Frequently asked questions
About A/B testing.
How long should an A/B test run?
That depends on the number of visitors and the level of confidence you want. In any case, let a test run long enough to account for differences between days and weeks. Stopping too early gives a distorted picture.
How many visitors do I need?
You work that out in advance from the expected effect, the desired confidence level and the current conversion rate. There are calculators for it. We’ll work it out for you before the test starts.
What’s best to test?
Elements with a big influence on your goal: the text of a call to action, the layout of a page, the length of a form. Choose what ties in with what you want to achieve.
How do you analyse the results?
With statistical methods to determine whether the difference between the control and test groups is reliable, paying attention to confidence intervals. And we look at the effect on other channels.
A good conversation first, then a plan.
In one hour we discuss where you stand and which opportunities we see. If we are not the right fit, we will tell you.