difficulty 3/5
SGD
Gradient descent that estimates the gradient from one example instead of the whole dataset — a much noisier direction, computed thousands of times more cheaply.
Intuition
Full-batch gradient descent reads a million rows to decide on one step. That is an enormous amount of work for a single decision, and most of it is redundant — the first thousand rows already point in roughly the right direction.
SGD takes the shortcut. Compute the gradient from one example, step immediately, move on. Any individual direction is badly wrong; averaged over thousands of steps it is right, and you have taken thousands of steps in the time full-batch took one. Its most important property is that "computation time per update does not grow with the number of training examples" [2, §8.3.1].
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Optimizers →
Sources
- Dive into Deep Learningbook · retrieved 2026-08-31https://d2l.ai/
- Deep Learning (2016)book · retrieved 2026-10-07https://www.deeplearningbook.org/contents/optimization.html