difficulty 3/5
Mini-batch gradient descent
Estimating the gradient from a small group of examples rather than one or all of them — the compromise that everything is actually trained with.
Intuition
Full-batch gives the exact direction at a cost that grows with the dataset. Single-example SGD gives a cheap direction with a lot of noise in it. Neither extreme is what anyone runs.
Mini-batch takes 32, or 128, or 256 examples, averages their gradients, and steps. Averaging cuts the noise; the batch is still small enough that steps come quickly. Most algorithms used for deep learning "fall somewhere in between, using more than one but fewer than all the training examples" [2, §8.1.3].
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Optimizers →
Sources
- Dive into Deep Learningbook · retrieved 2026-08-31https://d2l.ai/
- Deep Learning (2016)book · retrieved 2026-10-06https://www.deeplearningbook.org/contents/optimization.html
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour (2017)paperhttps://arxiv.org/abs/1706.02677
- Group Normalization (2018)paperhttps://arxiv.org/abs/1803.08494
- torch.utils.data.DataLoaderdocs · retrieved 2026-10-07https://pytorch.org/docs/stable/data.html