Optimizers

We email you a link and a six-digit code. Either one signs you in; there is no password.Enter your email. We send a link and a six-digit code to confirm it — no password to remember. The account is free.

You come back to this page, right where you were.

Already have an account? The same email simply signs you in.

Comparison

Name What it adds	Data per step	State kept	Tolerates noisy gradients
Gradient descent The base rule. Step against the gradient of the full training set, scaled by a fixed learning rate. Full page →	whole dataset	none	n/a — exact
SGD Estimates the gradient from one example. Far worse per step, far more steps, and the noise helps escape poor minima. Full page →	1 example	none	Yes — by design
Mini-batch Averages over a small batch. Cuts the noise and matches how parallel hardware actually works. Full page →	32–512	none	Yes
Adam Keeps running averages of the gradient and its square, giving every parameter its own effective step size. Full page →	mini-batch	2 values / parameter	Yes — built for it
Conjugate gradient Chooses each direction so it does not undo earlier progress. Kills the zig-zag with almost no memory. Full page →	whole dataset	1 direction	Poorly; mini-batch variants exist
BFGS / L-BFGS Infers curvature from how the gradient changed between steps, then aims each step using it. Full page →	whole dataset	n×n, or ~10 pairs	No

Two axes, not a ranking

The table is sorted by neither speed nor quality, because neither orders this family. What separates these methods is two independent choices.

How much data does one step look at? All of it gives an exact gradient at enormous cost per step. One example gives a cheap, noisy one. A mini-batch sits between, and that is where almost all deep learning lives [2, §12.5].

How much is remembered between steps? Plain descent remembers nothing. Adam keeps two numbers per parameter. Conjugate gradient keeps one direction. BFGS keeps an entire curvature matrix — and L-BFGS exists because that matrix stops fitting in memory long before modern models stop growing.

The split that actually matters

Read the last column and the family divides cleanly in two.

Methods that tolerate noise — SGD, mini-batch, Adam — are what neural networks are trained with. They assume every gradient is an estimate, and are built to make progress anyway.

Methods that do not — conjugate gradient, BFGS — are faster and better aimed when gradients are exact, and unreliable when they are not. Both infer something about the landscape from the relationship between successive gradients, and mini-batch noise corrupts that inference: second-order updates "typically require much larger batch sizes, like 10,000" [1, §8.1.3].

This is why methods that are standard in scientific computing are rare in deep learning, despite converging in far fewer iterations. Rare, not absent: practitioners report reasonable results with nonlinear conjugate gradients, and mini-batch versions have been used successfully [1, §8.6.2].

Choosing

Training a neural network → Adam, or AdamW if you want weight decay. Tune SGD with momentum only if a benchmark number justifies the effort.
Fitting a smooth model with exact gradients → L-BFGS. It is already the default solver for logistic regression in most libraries.
Very large, sparse, deterministic problem → conjugate gradient, where the memory cost of a curvature estimate is unaffordable.
Understanding what any of them do → plain gradient descent. Everything else in the table is a modification of that one update rule.

What none of them fix

An optimiser searches the landscape it is given. If the loss surface is a ravine because features are on wildly different scales, every method here struggles — Adam least, but none of them escape it. Scaling the inputs changes the landscape itself, and usually helps more than changing the optimiser.