difficulty 3/5
A/B testing
Randomly assigning subjects to two variants and comparing an outcome, so that the difference you measure can be attributed to the change rather than to who happened to see it.
Intuition
You change the button and conversions go up 3%. The question is not whether the number moved — it always moves. The question is whether it moved for the reason you think.
Two things could have produced that 3%. The change might work. Or this week's visitors might simply have been slightly keener than last week's, in the way that any two samples differ. Nothing in the 3% itself distinguishes these, and you cannot resolve it by staring at the number harder [2, §9.1].
Randomisation is what makes the difference interpretable. Assign each arriving visitor to A or B by coin flip and the two groups are, in expectation, alike in every respect except the change — including the respects you never thought to measure [3, §7.8]. Then the second question becomes answerable: how big a difference would two identical variants have shown, purely by chance? If the answer is "routinely bigger than 3%", you have learned nothing yet [2, §9.1].
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Statistical testing →
Sources
- NIST/SEMATECH e-Handbook of Statistical Methods — Product and Process Comparisonsstandard · retrieved 2026-09-05https://www.itl.nist.gov/div898/handbook/prc/prc.htm
- Think Stats — Ch. 9, Hypothesis Testing, 2nd ed.book · retrieved 2026-09-05https://greenteapress.com/thinkstats2/html/thinkstats2010.html
- Think Stats — Ch. 7, Relationships between variablesbook · retrieved 2026-10-04https://greenteapress.com/thinkstats2/html/thinkstats2008.html