Statistics · 5 compared
Multiple-testing corrections
Test many things at the 5% level and something will pass by chance. Each correction makes a different promise about the mistakes you will make, and pays for it in findings.
Comparison
| Name What it does | Controls | Procedure | Findings |
|---|---|---|---|
| No correction Every test at α on its own. Right for one test; across a hundred, at least one false claim is close to certain. Full page → | each test only | p < α | the most, many of them false |
| Bonferroni Divide α by the number of tests. The chance of even one false claim stays below α. Full page → | FWER | p < α/m | the fewest |
| Holm Bonferroni applied step by step down the sorted p-values. The same promise, and never fewer findings. Full page → | FWER | step-down | at least Bonferroni's, always |
| Benjamini–Hochberg Bounds the share of findings that are false instead of the chance of any. A weaker promise, far more findings. Full page → | FDR | step-up | more than any FWER rule |
| Benjamini–Yekutieli Benjamini–Hochberg made safe for tests that depend on each other in any way, at the cost of findings. Full page → | FDR, any dependence | step-up, stricter | fewer than Benjamini–Hochberg |
On the cheat sheet: Statistical testing →
Why correct at all
One test at α = 0.05 risks one false positive in twenty. Twenty tests, and "you should expect at least one false positive, most of the time" [1, §9.11]. Every correction below spends some of your findings to buy that risk back; they differ in what they buy.
Two promises, not two strengths
FWER, the family-wise error rate, is the chance of making any false claim. Bonferroni, a "one-step correction", and Holm, a "step-down method using Bonferroni adjustments", control it [3, multipletests].
FDR, the false discovery rate, is "the expected proportion of rejected null hypotheses that are actually true" [2, description]: the share of your findings that are wrong. Benjamini–Hochberg and Benjamini–Yekutieli control it, and "false discovery control procedures tend to offer higher power than familywise error rate control procedures" [2, Notes].
Choose the promise first. Concluding, where one false claim is a disaster, wants FWER; screening, where a shortlist that is partly wrong is still useful, wants FDR.
Reading the rows
Holm over Bonferroni, always. Same promise, and never fewer findings: Holm starts from the threshold Bonferroni uses for every test, then relaxes it test by test.
Benjamini–Hochberg assumes the tests are independent, or have "positive regression dependencies" [2, Notes]. When they may depend on each other in any other way, Benjamini–Yekutieli "is guaranteed to control the FDR even when the p-values are not from independent tests", and is the more conservative of the two for it [2, method].
No row fixes a count you chose after looking. The correction is only as honest as m, the number of tests you actually ran; exploring on one part of the data and confirming on another needs no m at all [1, §9.11].
Sources
- Think Statsbook · retrieved 2026-09-18https://greenteapress.com/thinkstats2/html/thinkstats2010.html
- SciPy — scipy.stats.false_discovery_controldocs · retrieved 2026-09-18https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.false_discovery_control.html
- statsmodels — multipletestsdocs · retrieved 2026-09-18https://www.statsmodels.org/stable/generated/statsmodels.stats.multitest.multipletests.html