What is A/B testing?

In plain English

A/B testing compares two versions of something through a planned experiment. A website might show different versions of a page to allocated groups and observe a defined action. The purpose is to investigate a specific question under a controlled enough arrangement that the comparison is useful.

The design matters. If the versions appear to different audiences in different months, other conditions may explain the difference. An experiment should define allocation, measurement and the intended comparison before results are judged. GOV.UK’s explanation of A/B studies describes the comparative method; the general design principles are relevant beyond its example sector.

Testing does not guarantee a clear winner. A small sample, unreliable event or poorly chosen outcome can leave the question unresolved. A useful result may be that the available evidence is insufficient. Treating every observed fluctuation as an improvement makes later decisions less reliable.

Why it matters for a Jersey business

A Jersey business might want to know whether a clearer enquiry page helps suitable customers complete the form. A planned comparison can address that question, provided the traffic and implementation support it. For a small local audience, collecting enough relevant observations may take longer than an owner expects.

Some problems should simply be fixed. A broken telephone link, unreadable text or inaccurate opening time does not need an experiment to justify correction. Use testing where there is a real uncertainty between reasonable alternatives, and use ordinary usability review where a customer task is demonstrably difficult.

A realistic Jersey example

Imagine a Jersey training business considering two introductions to an event enquiry page. This is a hypothetical example. The team chooses a clearly defined completed-enquiry observation and checks that the measurement works in both versions. Visitors are allocated through the supported testing setup, and the review plan is recorded in advance.

If the evidence is limited, the owner does not announce that a tiny difference proves one introduction caused more bookings. They consider the uncertainty and review the wording with users as well. The experiment informs the decision within its limits, rather than supplying a guaranteed sales claim.

What to decide before running a test

Write one question. What are the alternatives, and what would a useful difference mean for the business? Choose a measure that reflects the question. A button click can show an interaction, while a completed enquiry or purchase is a different outcome. Label intermediate observations clearly.

Plan the allocation and review with someone who understands the method. Decide how repeat visitors, device differences and measurement interruptions are handled. Estimate whether the available audience can support the comparison before investing in a long experiment. Avoid deciding the stopping point solely because the latest numbers briefly favour a preferred version.

Verify the implementation. Check each version’s forms, links and recorded events. A reporting error can look like a design effect. Keep other material changes documented so the interpretation is not based on an unrecorded shift in audience or offers. If several things change together, acknowledge that the result may concern the combined experience.

Review uncertainty and practical value. A result should be interpreted alongside the experiment’s conditions, not only a percentage. GOV.UK’s test-and-learn guidance places experiments within a broader learning process. User feedback, task observation and real business records can provide useful context without being represented as the same type of evidence.

Keep a decision record. State what was tested, what the measurement showed, what remains uncertain and what action the team chose. This helps the business avoid repeating the same question later or claiming more from the result than the method supports.

Common myths

The version with one more action is proven better. Sample size, chance and implementation matter.

Every before-and-after comparison is an A/B experiment. Different methods have different limitations.

Testing replaces ordinary usability checks. Known faults and inaccessible controls still need direct attention.

Our take

Jersey Web’s view is to choose a question worth testing and make the evidence limits clear. A small business can learn useful things without turning every dashboard fluctuation into a confident claim. Match the method to the decision and keep the outcome measure honest.