Classification metrics

Six of them are the same four numbers

A binary classifier produces exactly four counts: true negatives, false positives, false negatives, true positives — in that fixed order in scikit-learn's confusion matrix [1, binary case]. The first six metrics in the table above are each a ratio of some subset of them. The last two are not, and have a section of their own below.

That is why the "cannot see" column is the most useful one here. Choosing a metric is choosing which of those four counts to discard, and the discarded ones are where the unpleasant surprises live.

The two axes that actually differ

Read the table down the middle and there are really two questions being asked.

"Of what the model flagged…" — precision, "the fraction of retrieved documents that are relevant" in the retrieval setting it comes from [3, §8.3]. Its denominator is the model's own behaviour, so it moves when the model becomes more or less eager, and it moves when the base rate changes even though the model has not.

"Of what was really there…" — recall and specificity. Their denominators come from the data, so they are invariant to prevalence. This is why medical testing reports sensitivity and specificity, while an operations team running an alert queue cares about precision — the metric to use "when it's very important for positive predictions to be accurate" [2, Choice of metric and tradeoffs].

Confusing the two is the single most common error in this area, because precision and specificity both sound like "how often are we wrong when we say yes".

Two that read the probabilities

Log loss and calibration never look at the four counts. They read the probability the model stated, before any threshold turned it into a label, which is why neither needs a threshold, and why both see what the other six cannot: whether a stated 0.9 was an honest 0.9.

Log loss is the negative log-likelihood of the true labels, which Guo et al. call "a standard measure of a probabilistic model’s quality", one "also referred to as the cross entropy loss in the context of deep learning" [4, §2, Negative log likelihood]. In expectation it is minimised only when the model's probabilities are the true ones [4, §2, Negative log likelihood], so being bolder than the evidence costs, and being right is not enough. What it cannot show is its own scale. Cross-entropy is "the summation of the entropy H(P) and the KL divergence between P and Q" [5, §22.11.5.1]: the entropy part is a floor no model goes under, it moves with the base rate, and a log loss read without that floor says nothing.

Calibration asks whether the stated confidence comes true: "given 100 predictions, each with confidence of 0.8, we expect that 80 should be correctly classified" [4, §2]. It cannot see whether the model separates the classes at all. A model that predicts the base rate for every case is perfectly calibrated, and tells you nothing about any one of them.

Nothing here handles asymmetric costs

Every count-based metric above weights the two error types by a fixed rule — accuracy and F1 weight them equally, precision and recall each ignore one. None of them knows that a missed fraud costs €400 and a blocked honest transaction costs one annoyed customer.

If the costs are genuinely different and known, the right move is not to hunt for a better metric from this list. It is to pick the decision threshold that minimises expected cost directly, and then report precision and recall at that threshold to describe what you chose.

Practical advice

Report the confusion matrix plus two derived numbers, never one. A single score always hides one of the four cells, and which one it hides is exactly what determines whether the model is fit for your problem. When the probability itself is used, add the log loss beside its base rate, and a calibration check.