difficulty 3/5
Missing data
Whether dropping or filling missing values biases your answer depends on why the values are missing — and on which answer you wanted. One of those two questions is usually unanswerable.
Intuition
Every real dataset has holes. The question is never whether to handle them — dropping the rows is a handling decision too — but whether the handling changes the answer, and in which direction.
That turns on something you usually cannot observe: why the values are missing. A sensor that fails at random and a sensor that fails when the reading is high leave exactly the same kind of gap in the file, and the second one will quietly bias every number you compute.
scikit-learn frames the basic trade-off plainly: discarding incomplete rows "comes at the price of losing data which may be valuable (even though incomplete)", and the alternative is to impute — "to infer them from the known part of the data" [2, §8.4]. What neither option can do is invent information that was never recorded.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Missing data and outliers →
Sources
- GAIN: Missing Data Imputation using Generative Adversarial Nets (2018)paperhttps://ar5iv.labs.arxiv.org/html/1806.02920
- scikit-learn User Guide — Imputation of missing valuesdocs · retrieved 2026-09-13https://scikit-learn.org/stable/modules/impute.html
- Think Stats, 2nd ed.book · retrieved 2026-09-13https://greenteapress.com/thinkstats2/html/index.html