difficulty 3/5
Decoding
The rule that turns a language model's next-token distribution into actual text. Choosing the most probable continuation at every step reliably produces the worst output.
Intuition
A language model does not produce text. It produces a probability distribution over the next token, and something else has to choose. That something else is the decoding strategy, and it turns out to matter as much as the model.
The obvious choice is to take the most likely token every time, or to search for the most likely sequence. Holtzman et al. open by pointing out that this does not work: "decoding strategies that optimize for output with high probability, such as beam search, lead to text that is incredibly degenerate, even when using state-of-the-art models such as GPT-2 Large" [1, §1].
They also note the other end is no better: "pure sampling — sampling directly from the probabilities predicted by the model — results in text that is incoherent and almost unrelated to the context", and the culprit is "the 'unreliable tail' ... composed of tens of thousands of candidate tokens with relatively low probability that are over-represented in the aggregate" [1, §1].
Good decoding is therefore a truncation problem: how much of the tail to throw away, and how to decide.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: NLP tasks →
Sources
- The Curious Case of Neural Text Degeneration (2020)paperhttps://ar5iv.labs.arxiv.org/html/1904.09751
- Dive into Deep Learningbook · retrieved 2026-09-13https://d2l.ai/chapter_recurrent-modern/beam-search.html