difficulty 3/5
Hypothesis testing
Assume there is no effect, work out how often chance alone would produce a result as striking as yours, and call it real if that is rare enough. The method is a proof by contradiction.
Intuition
You measure something and see a difference. Two explanations compete: there is a real effect, or you got a lucky sample. Nothing in the data itself can tell these apart, because a lucky sample looks exactly like a real effect — that is what makes it lucky.
Hypothesis testing gets at the question sideways. It does not try to show the effect is real. It assumes the opposite, works out how often chance alone would produce something as striking as what you saw, and lets you conclude that "often" and "almost never" mean different things.
The book this page leans on puts the logic exactly: "the goal of classical hypothesis testing is to answer the question, 'Given a sample and an apparent effect, what is the probability of seeing such an effect by chance?'" [1, §9.1]. And the shape of the argument is familiar from mathematics — "the logic of this process is similar to a proof by contradiction. To prove a mathematical statement, A, you assume temporarily that A is false" [1, §9.1].
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Statistical testing →
Sources
- Think Stats, 2nd ed.book · retrieved 2026-09-06https://greenteapress.com/thinkstats2/html/index.html
- NIST/SEMATECH e-Handbook of Statistical Methodsstandard · retrieved 2026-09-06https://www.itl.nist.gov/div898/handbook/prc/section1/prc12.htm