Which split, and what leaks?
Seven ways to split data side by side, the pairs people confuse, and one run in which the same model scores differently on every split.
prints on 2 A4 pagesfrom 8 pages and 1 cataloguefree to read · PDF with a free accountUpdated 23 Oct 2026
The PDF and the Print button ask you to sign in; a free account is enough. Reading and Test yourself are open to everyone.
Side by side
The first columns are the Validation strategies catalogue’s; the rest come from each page.
A formula is printed only when it is shorter than the words; the others are said in words, and each page has its formula.
Easy to confuse
The “what is the difference between…” questions, in two lines each.
Holdout vs Cross-validation
0.75 vs 0.92 Test rows: 120
A holdout is one draw from a wide spread of possible scores; cross-validation averages k of them.
Cross-validation when one split would be noisy; a holdout for the final score, looked at once.
Validation set vs Test set
Both are rows the model was not fitted on. The validation set chooses features, settings and the epoch; the test set only reports.
Anything chosen on the test set makes its score optimistic. Look at it once, at the end.
K-fold vs Grouped k-fold
K-fold scatters one subject's rows across folds, so the model is scored on people it has met; grouped k-fold keeps each subject on one side.
Group whenever rows come in units. The random split's extra accuracy is recognition, not skill.
Overfitting is the model learning its training rows too well; leakage is the evaluation seeing what it should not.
Overfitting shows as a bad validation score; leakage hides itself, by making that score too good.
Early stopping vs Regularization
Both give up some fit to the training data to generalise better. One limits how long the model trains, the other how large its weights grow.
Early stopping is nearly free on any model trained in epochs; a penalty suits any linear model with many features.
One real run
The same model on the same data, split 400 different ways at four sizes of test set.
| Test rows | Mean | Std dev | Worst | Best | Spread falls |
|---|---|---|---|---|---|
| 120 | 0.83 | 0.03 | 0.75 | 0.92 | 0.17 |
| 240 | 0.83 | 0.02 | 0.77 | 0.89 | 0.12 |
| 420 | 0.83 | 0.02 | 0.79 | 0.87 | 0.08 |
| 600 | 0.83 | 0.01 | 0.80 | 0.87 | 0.07 |
The run and these lines are the worked example on Train-test split, as it printed: the same model and the same data, split 400 different ways
Which one?
Is one split steady enough?
- Yes, there is plenty of data Train-test split
- No, a score moves with the split Cross-validation
Are the rows independent?
- No, they come in groups Data leakage
- No, they are ordered in time Cross-validation
Are you also choosing the model?
- Yes, the search needs its own loop Hyperparameter tuning
Does the score look wrong?
- Too good to be true Data leakage
- Far below the training score Overfitting
Each answer is from its page’s “Why & when”.
All 39 traps on these pages
Each is the opening line of a pitfall; its page says why.
Train-test split
- Reporting one split as if it were the answer.
- Letting the test set choose anything.
- Preprocessing before splitting.
- Splitting rows when the data comes in groups.
- Splitting time series at random.
- Treating a reported test score as a true test score.
Cross-validation
- Preprocessing before splitting.
- Selecting a model on the CV score and then reporting that same score.
- Unstratified folds on an imbalanced target
- Shuffling dependent data
- Reading the standard deviation across folds as a standard error.
Data leakage
- Believing a pipeline is enough.
- Random splits on grouped data.
- Deduplicating after splitting
- Tuning on the test set.
- Resampling before the split.
Overfitting
- Preprocessing before splitting.
- Treating the validation score as an estimate of future performance.
- Assuming a small gap means a good model.
- Random splits on grouped or temporal data.
Bias-variance tradeoff
- Treating the U as a law.
- Diagnosing bias from a single number.
- Tuning capacity on the test set.
- Adding capacity to a high-bias model and expecting variance.
- Assuming the decomposition holds for your loss.
Hyperparameter tuning
- Reporting the best search score as model performance.
- Fitting preprocessing outside the search.
- Grid points on a linear scale for a multiplicative parameter.
- Searching parameters that do not interact, jointly.
- Treating the search budget as free.
Early stopping
- Stopping on the first rise.
- Halting without restoring.
- Reporting the validation number.
- Assuming the curve has one turn.
- Forgetting it interacts with the learning rate.
Regularization
- Unscaled features make the penalty arbitrary.
- Do not penalise the intercept.
- α must be chosen on held-out data.
- Regularized coefficients are biased on purpose.