difficulty 3/5
Bias-variance tradeoff
The classical account of why a model can be too simple or too complex, and the modern result that breaks it — past the point of fitting the training data exactly, test error can start falling again.
Intuition
A model can fail in two opposite ways, and the classical account names both.
Too simple, and it cannot represent the pattern no matter how much data you give it — the errors it makes are systematic, the same in every direction, on every dataset. That is bias.
Too complex, and it fits the pattern and the noise. Hand it a different sample from the same population and it returns a visibly different model. Its errors are not systematic; they are unstable. That is variance.
The classical conclusion is that these pull against each other and you should find the point between them. The paper that this page is largely about states that conclusion carefully before dismantling it: "if ℋ is too small, all predictors in ℋ may under-fit the training data"; "if ℋ is too large, the empirical risk minimizer may over-fit spurious patterns"; and "the classical thinking is concerned with finding the 'sweet spot' between under-fitting and over-fitting" [1, §1].
That account is correct and incomplete, and knowing where it stops being correct is worth more than the account itself.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Validation and generalisation →
Sources
- Reconciling modern machine learning practice and the bias-variance trade-off (2019)paperhttps://ar5iv.labs.arxiv.org/html/1812.11118
- scikit-learn User Guide — Tuning the hyper-parameters of an estimatordocs · retrieved 2026-09-06https://scikit-learn.org/stable/modules/grid_search.html