difficulty 3/5
Adam
An optimiser that keeps a running average of both the gradient and its square, giving every parameter its own effective step size — the default choice for training neural networks.
Intuition
Plain SGD applies one learning rate to every parameter. But parameters are not alike: the loss is often highly sensitive to some directions in parameter space and insensitive to others, so a single η has to compromise between them [3, §8.5].
Adam gives each parameter its own effective step size, worked out from that parameter's own gradient history. A parameter with large gradients has them divided by a large running scale, one with small gradients by a small one, so rescaling one parameter's gradients leaves its steps unchanged: the method is "invariant to diagonal rescaling of the gradients" [1, abstract]. Its authors add that its hyper-parameters "typically require little tuning" [1, abstract], which is a large part of why it became the default. How far that holds is disputed; see the pitfalls.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Optimizers →
Sources
- Adam: A Method for Stochastic Optimization (2014)paperhttps://arxiv.org/abs/1412.6980
- Dive into Deep Learningbook · retrieved 2026-08-31https://d2l.ai/
- Deep Learning (2016)book · retrieved 2026-10-06https://www.deeplearningbook.org/contents/optimization.html
- Decoupled Weight Decay Regularization (2019)paperhttps://arxiv.org/abs/1711.05101
- The Marginal Value of Adaptive Gradient Methods in Machine Learning (2017)paperhttps://arxiv.org/abs/1705.08292
- tf.keras.optimizers.Adamdocs · retrieved 2026-10-06https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam