Which optimiser, which learning rate?
Six optimisers side by side, the pairs people confuse, and one problem run twice, where scaling the features matters more than the optimiser.
prints on 2 A4 pagesfrom 7 pages and 1 cataloguefree to read · PDF with a free accountUpdated 24 Oct 2026
The PDF and the Print button ask you to sign in; a free account is enough. Reading and Test yourself are open to everyone.
Side by side
The first columns are the Optimizers catalogue’s; the rest come from each page.
A formula is printed only when it is shorter than the words; the others are said in words, and each page has its formula.
Easy to confuse
The “what is the difference between…” questions, in two lines each.
Gradient descent reads every row for one exact step; SGD takes one noisy step per row.
SGD, or its mini-batch form, once the data is large; full gradients for small, smooth problems.
SGD moves every parameter by one learning rate; Adam scales each by its own gradient history.
Adam to get going on any scaling; SGD with momentum when test accuracy is what is measured.
Both search along a line on the full data; L-BFGS also keeps a few gradient pairs to model the curvature.
L-BFGS for most smooth problems; conjugate gradient when even that memory is too much.
Learning rate vs Batch size
A bigger batch averages away gradient noise, so it can take a larger step; the two are tuned together.
Change one, revisit the other.
One real run
One logistic regression run with each optimiser, on features scaled 1 to 1,000 and standardised.
| Optimiser | Passes, raw | LRs, raw | Passes, std | LRs, std |
|---|---|---|---|---|
| grad-descent | never | 0 | 5 | 3 |
| sgd | never | 0 | 1 | 6 |
| mini-batch | never | 0 | 1 | 6 |
| adam | 5 | 4 | 1 | 5 |
| conj-gradient | 328 | n/a | 7 | n/a |
| l-bfgs | 48 | n/a | 4 | n/a |
“On badly scaled features, one learning rate cannot serve every direction.”
“Standardise, and the plain methods catch up.”
“A line search needs no learning rate, only full passes.”
The run and these lines are the worked example on Adam, as it printed: logistic regression, 4000 rows, 10 features; passes until within 0.01 of the minimum
Which one?
How large is the data?
- Large, or growing Mini-batch gradient descent · SGD
- Small, and the loss smooth BFGS
Do you need it working quickly?
- Yes, with little tuning Adam
Is memory the limit?
- Yes, even for L-BFGS Conjugate gradient
Each answer is from its page’s “Why & when”.
All 29 traps on these pages
Each is the opening line of a pitfall; its page says why.
Gradient descent
- The learning rate is the first thing to check when training fails.
- Unscaled features make the surface a ravine.
- A fixed learning rate is a compromise.
- Convergence is to a local minimum.
SGD
- SGD does not converge to a point at a fixed learning rate.
- The loss curve will be jagged.
- Sorted data is dangerous.
- Pure one-example SGD wastes hardware.
Mini-batch gradient descent
- Changing batch size silently changes the effective learning rate.
- The last batch is smaller.
- Batch size interacts with normalisation.
- Powers of two are a runtime question, not a statistical one.
Adam
- Adam is not always better than SGD.
- Use AdamW for weight decay.
- Bias correction matters early.
- Epsilon is not only numerical hygiene.
Conjugate gradient
- It needs an accurate gradient.
- Restart it periodically.
- Conditioning still matters.
- The line search costs function evaluations.
BFGS
- Use L-BFGS above a few thousand parameters.
- Give it an analytic gradient.
- No learning rate is a feature with a cost.
- Smoothness is what the theory assumes, not a hard requirement.
Learning rate
- Reading divergence as a bug.
- Assuming there is one right value.
- Tuning it on a linear grid.
- Changing the batch size and leaving the rate alone.
- Expecting an adaptive optimiser to make the choice.