difficulty 2/5
Cross-validation
Splitting the data k ways and letting each part be the test set once, so every sample is predicted by a model that never saw it — and the estimate uses all the data instead of a single lucky split.
Intuition
Scoring a model on the data it was fitted to measures memory, not learning. The library states it without hedging: "learning the parameters of a prediction function and testing it on the same data is a methodological mistake" [2, §3.1].
Holding out a test set fixes that, and immediately creates two new problems. The score now depends on which rows happened to land in the test set, and every row you hold out is a row you cannot learn from — a real cost when the dataset is small.
Cross-validation is the obvious compromise, and it is genuinely clever: rotate the held-out part. Split into k pieces, train k times, and each time score on the piece that was left out. Every sample gets predicted by a model that never saw it, and every sample also contributes to training in the other k−1 rounds. Nothing is wasted, and the score averages over k different splits instead of trusting one.
What it is not is a measurement of your model. That is the part almost everyone has backwards, and it is worth the section below.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Validation and generalisation →
Sources
- Cross-validation: what does it estimate and how well does it do it? (2023)paperhttps://ar5iv.labs.arxiv.org/html/2104.00673
- scikit-learn User Guide — Cross-validation: evaluating estimator performancedocs · retrieved 2026-09-06https://scikit-learn.org/stable/modules/cross_validation.html