difficulty 4/5
BFGS
An optimiser that builds an approximation of the curvature from the gradients it has already seen, using it to take far better-aimed steps than gradient descent.
Intuition
The gradient tells you which way is downhill. It does not tell you how the slope is changing — whether you are on a gentle bowl where a long step is safe, or a sharp trough where the same step overshoots. That information is curvature, and it lives in the second derivatives.
Newton's method uses curvature directly, but it has to invert a matrix of second derivatives with one row per parameter, at O(k³) cost, on every iteration, so "only networks with a very small number of parameters can be practically trained via Newton's method" [2, §8.6.1]. BFGS gets much of the benefit for a fraction of the cost by inferring curvature from how the gradient has changed between the steps it has already taken [2, §8.6.3].
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Optimizers →
Sources
- scipy.optimize.minimizedocs · retrieved 2026-08-31https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html
- Deep Learning (2016)book · retrieved 2026-10-06https://www.deeplearningbook.org/contents/optimization.html
- minimize(method='L-BFGS-B')docs · retrieved 2026-10-06https://docs.scipy.org/doc/scipy/reference/optimize.minimize-lbfgsb.html
- sklearn.linear_model.LogisticRegressiondocs · retrieved 2026-10-06https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html