difficulty 3/5
Data leakage
Any way information reaches the model that would not exist at prediction time. It does not make the model worse — it makes the score better, which is why it survives review and fails in production.
Intuition
Every other modelling mistake announces itself. Underfit and the score is bad; overfit and the validation score is bad. Leakage is the one failure that makes every number you look at better, which is why it survives peer review, dashboards and demos, and only shows up when the model meets real inputs.
The definition is short and worth memorising: leakage "occurs when information that would not be available at prediction time is used when building the model" [2, §12.2].
The trap is that the offending information rarely looks like the answer. It is the mean you subtracted, computed over all the rows. It is the feature you kept because it scored well — on the whole dataset. It is the patient who appears in both halves of the split under two different visit ids. None of that looks like cheating while you are doing it, and all of it is.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Validation and generalisation →On the cheat sheet: Missing data and outliers →
Sources
- Leakage and the Reproducibility Crisis in ML-based Science (2023)paperhttps://ar5iv.labs.arxiv.org/html/2207.07048
- scikit-learn User Guide — Common pitfalls and recommended practicesdocs · retrieved 2026-09-06https://scikit-learn.org/stable/common_pitfalls.html