difficulty 3/5
Perplexity
The exponential of a model's average negative log-likelihood on held-out text — read as the effective number of choices it faces per token. It is per token, so it is not comparable across tokenizers.
Intuition
A language model puts a probability on a piece of text, and the obvious way to score one is to ask how much probability it gives to text it has not seen. The obvious way does not work: "shorter sequences are much more likely to occur than the longer ones, hence evaluating the model on Tolstoy's magnum opus War and Peace will inevitably produce a much smaller likelihood than, say, on Saint-Exupery's novella The Little Prince. What is missing is the equivalent of an average" [1, §9.3.2].
Averaging the log-likelihood per token gives cross-entropy, which is comparable across lengths. Perplexity is its exponential, and exists mostly so the number lands on a scale people can reason about: the effective number of choices the model thinks it has at each step.
d2l's way in is to ask how surprising the text is, comparing three continuations of "It is raining" — "outside", "banana tree", and "piouw;kcj pwepoiut". The first is fluent, the second at least spells words, the third is a model that "does not fit data properly" [1, §9.3.2]. Perplexity puts those three in order without anyone reading them.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: NLP tasks →
Sources
- Dive into Deep Learningbook · retrieved 2026-09-13https://d2l.ai/chapter_recurrent-neural-networks/language-model.html
- The Curious Case of Neural Text Degeneration (2020)paperhttps://ar5iv.labs.arxiv.org/html/1904.09751