difficulty 4/5
Batch normalization
Re-centring and re-scaling each layer's inputs using the statistics of the current mini-batch — a layer that made much deeper networks trainable and much less sensitive to how they were initialised.
Intuition
Every layer in a network is trying to learn a function of its inputs. But those inputs are the outputs of the layer below, which is itself being updated — so the distribution arriving at any given layer keeps shifting underneath it. Each layer is chasing a moving target.
Batch normalization pins the target down. Before a layer sees its inputs, they are re-centred and re-scaled using the mean and variance of the current mini-batch. The weights below keep changing; the shape of what arrives stops changing.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
Sources
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015)paperhttps://ar5iv.labs.arxiv.org/html/1502.03167
- Dive into Deep Learningbook · retrieved 2026-10-07https://d2l.ai/chapter_convolutional-modern/batch-norm.html
- How Does Batch Normalization Help Optimization? (2018)paperhttps://arxiv.org/abs/1805.11604
- Group Normalization (2018)paperhttps://arxiv.org/abs/1803.08494
- Layer Normalization (2016)paperhttps://arxiv.org/abs/1607.06450
- Deep Residual Learning for Image Recognition (2015)paperhttps://arxiv.org/abs/1512.03385