difficulty 3/5
Multiple comparisons
Run twenty tests at the 5% level and you will usually find something. The corrections differ in what they promise, and the honest choice is which kind of mistake you would rather make.
Intuition
Hypothesis testing sets a price per test: at α = 0.05 you accept a 5% chance of calling noise a finding. Run one test and pay it once. Downey states the consequence in one line: "If you run one hypothesis test, the chance of a false positive is about 1 in 20, which might be acceptable. But if you run 20 tests, you should expect at least one false positive, most of the time" [1, §9.11].
NIST puts the warning in one sentence: "Doing pairwise comparison procedures over and over again for all possible pairs will not, in general, work. This is because the overall significance level is not as specified for a single pair comparison" [2, §7.4.7].
Every data scientist runs many tests, usually without noticing: a dashboard with thirty metrics, a feature-selection sweep, a model compared across ten segments, an A/B test read on every page of the funnel. The question this page answers is not whether to correct — it is which promise you want, because the two available promises cost very different amounts.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Statistical testing →
Sources
- Think Stats, 2nd ed.book · retrieved 2026-09-18https://greenteapress.com/thinkstats2/html/thinkstats2010.html
- NIST/SEMATECH e-Handbook — How can we make multiple comparisons?docs · retrieved 2026-09-18https://www.itl.nist.gov/div898/handbook/prc/section4/prc47.htm
- SciPy — scipy.stats.false_discovery_controldocs · retrieved 2026-09-18https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.false_discovery_control.html
- statsmodels — multipletestsdocs · retrieved 2026-09-18https://www.statsmodels.org/stable/generated/statsmodels.stats.multitest.multipletests.html