difficulty 2/5
Overfitting
Fitting the training sample more closely than the process that generated it — so the model reproduces noise that will not recur, and does worse on new data the better it looks on old.
Intuition
A student who memorises last year's exam paper will score perfectly on last year's exam. That tells you nothing about whether they learned the subject, and the memorisation actively costs them on a paper they have not seen — time spent on the specific answers was time not spent on the general method.
Models do exactly this. The training sample is not the process you care about; it is one draw from it, carrying real structure and accidents of sampling. A model flexible enough will fit both, and it cannot tell you which is which, because from inside the training set they look identical. The accidents are the part that will not be there next time.
The uncomfortable consequence: the number that gets better is not the number that matters. Training error falls monotonically as you add capacity, right past the point where the model has started learning nonsense [1, §3.6.1.1].
Formal definition
How it works
Where it's used
Why & when
Worked example
Variants & extensions
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Validation and generalisation →
Sources
- Dive into Deep Learningbook · retrieved 2026-09-05https://d2l.ai/
- scikit-learn User Guide — Cross-validation: evaluating estimator performancedocs · retrieved 2026-09-05https://scikit-learn.org/stable/modules/cross_validation.html