difficulty 2/5
Learning rate
How far each gradient step moves. On a quadratic the curvature sets a hard ceiling and the best value is exactly half of it; perhaps the most important hyperparameter to tune.
Intuition
Gradient descent computes a direction and then has to decide how far to go in it. The gradient answers the first question and says nothing about the second. The learning rate is the answer to the second, and it is the one number you supply.
The usual description is that too small is slow and too large is unstable. That is true and it undersells how sharp the boundary is. On a quadratic there is a specific value above which the algorithm does not merely become unstable — it provably moves away from the answer, every step, forever: the training error increases once the rate is "at least twice as large as its optimal value" [3, §11.4.1].
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Optimizers →
Sources
- Dive into Deep Learning — 12.3 Gradient Descentbook · retrieved 2026-09-20https://d2l.ai/chapter_optimization/gd.html
- Cyclical Learning Rates for Training Neural Networks (2017)paper · retrieved 2026-09-20https://ar5iv.labs.arxiv.org/html/1506.01186
- Deep Learning (2016)book · retrieved 2026-10-06https://www.deeplearningbook.org/