difficulty 3/5
Language model
A probability distribution over sequences of tokens, factorised so that each token is predicted from the ones before it. Counting works up to about two words of context and then runs out of data.
Intuition
A language model assigns a probability to a piece of text. That is the whole definition, and everything else — generation, pretraining, the objective GPT and BERT are trained on — follows from it.
The trick that makes it tractable is the chain rule. The probability of a sentence is the product of the probability of each word given the words before it [1, §9.3.1], so an object over an infinite space of sequences becomes a sequence of ordinary next-word predictions. Predicting one word at a time is a task you can supervise with any text at all, which is why this objective and not some other one is what large models are trained on.
The question the rest of this page answers is what "given the words before it" can mean in practice. Conditioning on all of them is what the factorisation asks for and what no counting method can do.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
Sources
- Dive into Deep Learningbook · retrieved 2026-09-13https://d2l.ai/chapter_recurrent-neural-networks/language-model.html
- The Curious Case of Neural Text Degeneration (2020)paperhttps://ar5iv.labs.arxiv.org/html/1904.09751