Which metric, and what does it hide?
Eight classification metrics side by side, the pairs people confuse, and one real run in which they disagree. Every name opens its full, sourced page.
prints on 2 A4 pagesfrom 10 pages and 1 cataloguefree to read · PDF with a free accountUpdated 21 Oct 2026
The PDF and the Print button ask you to sign in; a free account is enough. Reading and Test yourself are open to everyone.
Side by side
The first columns are the Classification metrics catalogue’s; the rest come from each page.
A formula is printed only when it is shorter than the words; the others are said in words, and each page has its formula.
Easy to confuse
The “what is the difference between…” questions, in two lines each.
Each alone can be gamed. Predict positive almost never and precision reaches 1.0; predict it for everything and recall does.
Report them together, or F1 when both errors cost about the same.
Both are about false alarms, but specificity divides by the real negatives and precision by the alarms raised.
Specificity survives a rebalanced test set; precision says how much of the queue is junk.
ROC AUC vs PR AUC
0.86 vs 0.12 Positives: 1%
ROC AUC divides false alarms by a huge pool of negatives, so they barely move it; PR AUC counts them against what was flagged.
When positives are rare, read the precision–recall side.
0.99 vs 0.03 Positives: 1%
Accuracy counts the true negatives and F1 leaves them out; with a rare class they are nearly all accuracy has.
With a rare class, accuracy is measuring the prevalence, not the model.
Accuracy checks the label; calibration checks what the stated confidence means.
A model that predicts the base rate for everything is perfectly calibrated and completely useless.
One real run
The same logistic regression on the same two classes; only the share of positives changes.
| Positives | Accuracy rises | Precision | Recall | F1 | ROC AUC | PR AUC falls |
|---|---|---|---|---|---|---|
| 50% | 0.79 | 0.79 | 0.79 | 0.79 | 0.87 | 0.87 |
| 10% | 0.91 | 0.68 | 0.26 | 0.38 | 0.87 | 0.51 |
| 1% | 0.99 | 0.55 | 0.02 | 0.03 | 0.86 | 0.12 |
| 0.1% | 0.999 | 0.00 | 0.00 | 0.00 | 0.83 | 0.03 |
“Accuracy rises as the problem gets harder.”
“ROC-AUC barely moves; PR-AUC collapses.”
The run and these lines are the worked example on Class imbalance, as it printed: identical class separability (1.6 sd apart), only the mix changes
Which one?
Do both errors cost about the same?
If not, which mistake hurts?
Will the threshold be chosen later?
Is the probability itself used?
- It is the product Cross-entropy
- It must mean what it says Model calibration
Each answer is from its page’s “Why & when”.
All 42 traps on these pages
Each is the opening line of a pitfall; its page says why.
Confusion matrix
- Check the orientation before reading anything into it.
- Raw counts hide the base rate.
- A matrix describes one threshold.
Accuracy
- Always compare against the base rate.
- Accuracy hides which error the model makes.
- It ignores confidence entirely.
Precision
- Precision alone is meaningless.
- It is undefined when nothing is flagged.
- It moves with the base rate.
Recall
- Recall alone is meaningless.
- "Sensitivity" and "recall" are the same thing.
- A recall target is a threshold decision, not a modelling one.
Specificity
- High specificity on rare-positive data is not an achievement.
- Specificity is not precision.
- 1 − specificity is the false positive rate.
F1-score
- Equal weighting is a decision you are making silently.
- F1 ignores true negatives.
- It is undefined only when there is nothing positive at all.
- Do not compare F1 across datasets.
ROC AUC
- Feed it scores, never labels.
- AUC flatters on imbalanced data.
- It says nothing about calibration.
- A single number hides curve shape.
Cross-entropy
- Reading a log loss without the base rate.
- Comparing log losses across datasets.
- Concluding from a worse log loss that the model is worse.
- Letting a probability reach 0 or 1.
- Averaging log loss over an imbalanced set and calling it a model summary.
- Mixing bits and nats.
- Optimising it and reporting accuracy.
Model calibration
- Fitting the temperature on training data.
- Judging calibration with too few samples per bin.
- Reading a Brier or log-loss improvement as a calibration improvement.
- Confusing calibration with accuracy.
- Assuming a bigger model is better calibrated.
- Recalibrating on data with a different class balance.
Class imbalance
- Reporting accuracy.
- Resampling before splitting.
- Resampling the test set.
- Forgetting that resampling breaks calibration.
- Reading ROC-AUC as if it were operational.
- Assuming the deployment prevalence equals the test prevalence.