difficulty 3/5
Cross-entropy
The loss every classifier on this site is trained with. It charges for being wrong and for being confident about it, and its floor is set by the base rate rather than by zero.
Intuition
Accuracy asks one question of each prediction: right or wrong. Cross-entropy asks two: were you right, and how sure were you?
That second question is the whole difference. A model that says 51% and is right has barely committed to anything. A model that says 99.9% and is wrong has made a claim it cannot take back, and the loss charges it 6.9 nats for that one row — as much as several thousand correct predictions at the same confidence.
The name comes from information theory, where it is the cost of coding data that
really comes from P using a model Q [2, §22.11.5.1]. In practice
it is a number you get out of log_loss or nn.CrossEntropyLoss, and the useful
thing to know is what it is measuring that accuracy is not, and what its
floor is.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Classification metrics →
Sources
- scikit-learn — Metrics and scoringdocs · retrieved 2026-09-18https://scikit-learn.org/stable/modules/model_evaluation.html
- Dive into Deep Learning — Information Theorybook · retrieved 2026-09-18https://d2l.ai/chapter_appendix-mathematics-for-deep-learning/information-theory.html
- Machine Learning Crash Course — Thresholds and the confusion matrixcourse · retrieved 2026-10-04https://developers.google.com/machine-learning/crash-course/classification/thresholding
- On Calibration of Modern Neural Networks (2017)paperhttps://ar5iv.labs.arxiv.org/html/1706.04599