What does p say, and how many tests?

Five ways to judge many tests side by side, the pairs people confuse, and one run where no correction calls noise a finding 99% of the time.

prints on 2 A4 pagesfrom 6 pages and 1 cataloguefree to read · PDF with a free accountUpdated 24 Oct 2026

Download PDFTest yourself

The PDF and the Print button ask you to sign in; a free account is enough. Reading and Test yourself are open to everyone.

Side by side

The first columns are the Multiple-testing corrections catalogue’s; the rest come from each page.

Say what each one answers, what it cannot see and when to reach for it, then reveal its row.

Rule	Controls	Procedure	Findings	Reach for it when	Trap
No correction Reveal No correction	each test only	p < α	the most, many of them false	one test, chosen before you looked	Testing after exploring the same data.
Bonferroni Reveal Bonferroni	FWER	p < α/m	the fewest	never; Holm makes the same promise	Using Bonferroni.
Holm Reveal Holm	FWER	step-down	at least Bonferroni's, always	a conclusion where one false claim is a disaster	Correcting for the tests you report rather than the tests you ran.
Benjamini–Hochberg Reveal Benjamini–Hochberg	FDR	step-up	more than any FWER rule	screening, where a shortlist partly wrong is still useful	Expecting BH to control the chance of any error.
Benjamini–Yekutieli Reveal Benjamini–Yekutieli	FDR, any dependence	step-up, stricter	fewer than Benjamini–Hochberg	screening tests that may depend on each other	Applying BH to correlated tests without thinking.

A formula is printed only when it is shorter than the words; the others are said in words, and each page has its formula.

The “what is the difference between…” questions, in two lines each.

Hypothesis testing vs Confidence interval
A p-value says whether an effect can be told from chance; an interval says how big it plausibly is.
Report the interval next to the p-value; only the interval says whether the effect matters.
FWER vs FDR
FWER is the chance of even one false claim; FDR is the share of your claims that are false.
Concluding, control FWER with Holm; screening, control FDR with Benjamini–Hochberg.
Bootstrap vs Confidence interval
A formula interval needs the statistic's distribution worked out; the bootstrap reads it from resamples.
The formula for a mean on a decent sample; the bootstrap for a statistic with no formula, not for the tails.
Not significant vs No effect
Not significant says the test could not tell a difference from chance; it does not say there is none.
Read a non-result through its interval, and work out the power before the experiment, not after.

A hundred z-tests, ten with a real effect, judged by each rule over 4,000 replications.

Rule	Any false claim	Share false	Power	Found of 10
no correction	0.99	0.32	0.91	9.10
Bonferroni	0.04	0.01	0.38	3.80
Holm	0.04	0.01	0.39	3.90
Benjamini-Hochberg	0.28	0.05	0.61	6.10

“The first row is why the page exists.”

“The folklore about Holm is true and now checked.”

“But the interesting comparison is the other one.”

The run and these lines are the worked example on Multiple comparisons, as it printed:

How many tests did you run?
- One, fixed before looking Hypothesis testing
- Many, or you looked first Multiple comparisons
What should you report?
- How big the effect is Confidence interval
- A statistic with no formula Bootstrap
Can you assign the change at random?
- Yes, to independent units A/B testing

Each answer is from its page’s “Why & when”.

Each is the opening line of a pitfall; its page says why.