A/B test
What is an A/B test?
An A/B test compares two versions of the same element — a headline, a button, an email, a landing page — by showing each to a random half of the audience and measuring which one produces more of the action you care about. It replaces an argument about taste with a number, and it only answers the question you actually set up.
How to run one
- Pick one number you want to raise, such as enquiries from a service page.
- Change one thing: the headline, the button text, the picture, the order of the sections.
- Split visitors at random into two groups. One sees the old version (A), the other the new one (B).
- Let the test run until each version has collected enough conversions, and for at least one full week so that weekdays and weekends even out.
- Look at the difference and check that it is statistically significant before naming a winner.
Example
A roofing repair page gets 4,000 visits a month and 80 enquiries, 2%. In version B the button “Send an enquiry” becomes “Free inspection within 3 days”. After four weeks A has 40 enquiries from 2,000 visits and B has 66 from 2,000. The gap is large enough to be unlikely to be chance, and B stays.
Had the page had 400 visits a month, the same test would have taken most of a year. Then it is more useful to ask five customers what held them back, or to watch session recordings.
Why it matters for a business
Without a test, changes are decided by taste or by whoever argues best in the meeting. A test tells you which version brings more enquiries from your own customers. The price is time: with little traffic a test is slow, and for very small differences it is useless. So test changes that can move the number visibly: the offer, the headline, the delivery charge, the length of the form.
What to ask
- Which number are we measuring, and how many conversions do we need for the result to hold?
- Does only one thing change between the two versions?
- Does the test run for at least one full week?
- How do we know the difference is not chance?
When an A/B test is worth running and how to read it is decided in our digital marketing work.