difficulty 4/5
Conjugate gradient
An optimiser that chooses each search direction so it does not undo the progress of the previous ones — the fix for gradient descent zig-zagging down a narrow valley.
Intuition
Gradient descent in a long narrow valley behaves badly in a specific way: each step points mostly across the valley rather than along it, so the path zig-zags between the walls and creeps toward the bottom. Worse, each step partially undoes the progress of the one before: with exact line searches each new direction is orthogonal to the last, and "by following the gradient at the end of each line search we are, in a sense, undoing progress we have already made in the direction of the previous line search" [2, §8.6.2].
Conjugate gradient stops that. Every new direction is chosen to be conjugate to the previous one — "it will not undo progress made in that direction" [2, §8.6.2]. No progress is repeated, so no zig-zag.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Optimizers →
Sources
- scipy.optimize.minimizedocs · retrieved 2026-08-31https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html
- Deep Learning (2016)book · retrieved 2026-10-06https://www.deeplearningbook.org/contents/optimization.html
- scipy.sparse.linalg.cgdocs · retrieved 2026-10-06https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.cg.html