difficulty 3/5
Recurrent neural network
A network that carries a hidden state from one step of a sequence to the next, so one set of weights handles any length. The same recurrence destroys the gradient.
Intuition
An n-gram model buys context by widening its table, and the table grows as the vocabulary raised to the power n. d2l puts the cost plainly: "the number of model parameters would also increase exponentially with it, as we need to store |V|ⁿ numbers for a vocabulary set V" [1, §9.4]. Five words of context on a five-thousand-word vocabulary is 5000⁵, about 3 × 10¹⁸ numbers.
The recurrent answer is to stop storing histories and store a summary of the history instead. Keep a vector hₜ₋₁ that stands for everything seen so far, update it with each new token, and predict from it. d2l writes the swap as replacing P(xₜ | xₜ₋₁, …, x₁) with P(xₜ | hₜ₋₁), "where hₜ₋₁ is a hidden state that stores the sequence information up to time step t−1" [1, §9.4].
Nothing about that is approximate in principle: "for a sufficiently powerful function f ... the latent variable model is not an approximation. After all, hₜ may simply store all the data it has observed so far" [1, §9.4]. The whole of this page is about what happens in practice instead.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
Sources
- Dive into Deep Learningbook · retrieved 2026-09-13https://d2l.ai/chapter_recurrent-neural-networks/rnn.html
- Dive into Deep Learning — Backpropagation Through Timebook · retrieved 2026-09-13https://d2l.ai/chapter_recurrent-neural-networks/bptt.html
- Deep Learning — Sequence Modeling, chapter 10book · retrieved 2026-09-13https://www.deeplearningbook.org/contents/rnn.html