A/B Testing Explained: Compare Website Variations Against a Clear Goal
How do you know whether a website change improves the result? An A/B test compares variations using a planned audience split and outcome measure.
1:47 · No narration: the video shows each step as text on screen, and every step is written out below, with buttons that jump the video to it.
Prefer YouTube? Watch this video on our YouTube channel (opens in a new tab).
The six points in the video
This is a silent video. The headings, explanations and labels below reproduce the information on screen.
-
Step 1: Start with a hypothesis
Explain what change might help and the outcome you expect.
On-screen labels: Observed problem; Proposed change; Expected effect.
-
Step 2: Define the variations
Keep the comparison focused enough to interpret.
On-screen labels: Original version; Test variation; Defined difference.
-
Step 3: Choose the outcome measure
Agree how success will be measured before starting.
On-screen labels: Primary outcome; Useful guardrail; Reliable measurement.
-
Step 4: Split the audience fairly
A planned allocation helps make the variations comparable.
On-screen labels: Defined allocation; Comparable groups; Consistent experience.
-
Step 5: Plan enough evidence
Sample size and run duration matter; a few visits can be misleading.
On-screen labels: Required sample; Planned duration; Low-traffic limits.
-
Step 6: Interpret the uncertainty
A small apparent difference does not automatically establish a winner.
On-screen labels: Observed difference; Statistical uncertainty; Practical importance.
A practical check for your business
Agree the comparison, primary outcome and evidence needed before starting a test. Use the observed result and its uncertainty to decide what has been learned, including when traffic is too limited.
Sources and further detail
See it in a diagram
Read the diagram as text
- Hypothesis: State the expected effect
- Comparison: Plan allocation and measure
- Evidence: Review the result and uncertainty
Check whether the test can answer the question. Consider the available traffic, sample requirements and practical importance.
An inconclusive result can still explain what the team has not established.
Diagram by Jersey Web. Open full size · Source and further detail (opens in a new tab)
Common questions
How do you know whether a website change improves the result?
An A/B test compares variations using a planned audience split and outcome measure. Treat an inconclusive result as inconclusive and choose the next useful investigation.
Go further
More video tips
All 205 videos →
CRM Explained: Organise Customer Records, Enquiries and Follow-Up Tasks
CRM, explained.
API Endpoints Explained: Where a Software Request Is Sent
API endpoints, explained.