difficulty 3/5
Entropy
The average surprise in a distribution, measured in bits. It is the floor on how short a code can be, and every loss on this site that says “log” is paying it.
Intuition
Entropy answers one question: how surprised should you expect to be?
d2l gets there through a deck of cards. "I see a card" tells you nothing — you knew that. "I see a heart" narrows 52 possibilities to 13. "This is the 3 of spades" names one of 52. And reading off the order of the whole shuffled deck picks one of 52! arrangements [1, §22.11.1]. Each statement is more surprising than the last, and the measure of information has to agree.
The quantity that makes this precise is −log p. d2l calls it exactly what it is: "we can interpret the self-information (−log(p)) as the amount of surprise we have at seeing a particular outcome. Indeed, as the probability approaches zero, the surprise becomes infinite. Similarly, we can interpret the entropy as the average amount of surprise from observing X" [1, §22.11.2.3].
That is the whole idea, and everything else on this page is a consequence. It is worth learning because it is not one topic among many: it is what cross-entropy is made of, what a decision tree's split criterion is, what perplexity exponentiates, and what a compressor is bounded by.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
Sources
- Dive into Deep Learning — Information Theorybook · retrieved 2026-09-18https://d2l.ai/chapter_appendix-mathematics-for-deep-learning/information-theory.html
- scikit-learn — Decision Treesdocs · retrieved 2026-09-18https://scikit-learn.org/stable/modules/tree.html