difficulty 3/5
Backpropagation
The chain rule applied backwards through a computational graph. One traversal produces the derivative with respect to every parameter, at roughly the cost of one forward pass.
Intuition
A network with sixty thousand parameters needs sixty thousand derivatives, one per parameter, at every step. The obvious method — nudge a parameter, see what happens to the loss, repeat — needs two forward passes per parameter. That is a hundred and twenty thousand evaluations for one update.
Backpropagation gets all sixty thousand from one traversal of the network in reverse. Not one per parameter; one, total. The worked example measures the difference at 73,432×, and the gap grows with the parameter count, which is why this algorithm is what makes deep learning possible rather than merely definable.
The mechanism is not exotic. It "traverses the network in reverse order, from the output to the input layer, according to the chain rule from calculus" [1, §5.3.3]. What makes it fast is the direction.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
Sources
- Dive into Deep Learningbook · retrieved 2026-09-12https://d2l.ai/chapter_multilayer-perceptrons/backprop.html
- Automatic Differentiation in Machine Learning: a Survey (2018)paperhttps://ar5iv.labs.arxiv.org/html/1502.05767