difficulty 3/5
Model calibration
Making a model's stated confidence mean what it says — of the predictions it makes at 90% confidence, 90% should be right. Accuracy does not measure this, and modern networks fail it badly.
Intuition
A classifier that is right 74% of the time and says so is useful. One that is right 74% of the time while claiming 87% is dangerous in a specific way: every decision downstream that consumes the number — a threshold, an expected cost, a handover to a human — is being fed a lie.
The requirement has a one-sentence test: "given 100 predictions, each with confidence of 0.8, we expect that 80 should be correctly classified" [1, §2].
Accuracy does not measure this and neither does AUC, because both only care about ordering and thresholding. A model can rank every example perfectly and still state probabilities that are uniformly far too high. Calibration is a separate property, and it is the one that matters whenever the probability itself is used rather than just the argmax.
The uncomfortable finding is that progress made it worse: "modern neural networks, unlike those from a decade ago, are poorly calibrated" [1, abstract].
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Classification metrics →
Sources
- On Calibration of Modern Neural Networks (2017)paperhttps://ar5iv.labs.arxiv.org/html/1706.04599
- scikit-learn User Guide — Probability calibrationdocs · retrieved 2026-09-06https://scikit-learn.org/stable/modules/calibration.html