Gradient

Intuition

Stand on a hillside in fog. You cannot see the summit, but you can feel which way the ground rises most steeply under your feet, and how sharply.

That is the gradient: a direction and a magnitude. The direction is the way the function increases fastest from where you are standing; the magnitude is how fast. Everything in gradient-based optimisation follows from having that information locally, without ever seeing the whole landscape.

Formal definition

For a function of several variables, the gradient is the vector whose entries are the partial derivatives with respect to each variable — each one measuring how the output changes when that single input is nudged, holding the others fixed [2, §4.3].

The result is a vector in the same space as the input. A function of 10 million parameters has a gradient with 10 million components, which is exactly why training a large network is possible at all: one evaluation gives you a direction for every parameter simultaneously.

How it works

Two properties do all the work:

It points uphill. The gradient gives the direction of steepest ascent, so minimising means moving in the direction of -∇f. This is the source of the minus sign that appears in every update rule in this section [2, §4.3].

Its length measures steepness. On a steep slope the gradient is long; on a plateau it is near zero. That is why gradient descent with a fixed learning rate takes long steps on steep ground and short steps near a minimum: each step is the learning rate times the gradient, so nobody schedules that — the geometry does it [2, §4.3].

Where the gradient is exactly zero the function is flat: a minimum, a maximum, or a saddle point. The gradient alone cannot tell you which; the second derivative can [2, §4.3.1].

Where it's used

• Training every neural network. Backpropagation is an efficient method for computing exactly this vector [1, §5.3.3].
• Fitting many other models. Logistic regression and gradient boosting are fitted with gradients too.
• Sensitivity analysis. The partial derivative for one input says how much the output responds to it.
• Adversarial examples. Take the gradient of the loss with respect to the input rather than the parameters, and a tiny step in the direction of its sign can change the prediction [2, §7.13].

Why & when

Gradients are the reason large-scale machine learning works. They give a search direction at a cost comparable to one function evaluation, no matter how many parameters there are — which is what makes optimising millions of them tractable [2, §6.5.3].

They are not always available. A gradient needs the function to be differentiable. Discrete choices, hard thresholds and non-differentiable losses have no gradient at the point that matters, which is why decision trees are fitted by search rather than descent [3, §1.10.7], and why a network trained by gradients uses a sigmoid where a hard threshold would leave it with a gradient of zero [2, §6.2.2.2].

Worked example

import numpy as np

def f(x):
    # a simple bowl, steeper along the second axis
    return x[0]**2 + 5 * x[1]**2

def grad(x):
    return np.array([2 * x[0], 10 * x[1]])

point = np.array([3.0, 1.0])
g = grad(point)

print(g)                     # [ 6. 10.] — steepest-ascent direction
print(np.linalg.norm(g))     # 11.66  — how steep it is here
print(-g / np.linalg.norm(g))  # unit vector pointing downhill

The second component is larger even though its coordinate is smaller, because the function rises five times faster along that axis. That imbalance is exactly what makes badly scaled problems hard to optimise — the gradient points mostly across the valley rather than along it.

Pitfalls

The gradient is local, and only local. It describes the surface at one point. It says nothing about whether a minimum lies ahead or how far away it is.

Zero gradient does not mean minimum. Saddle points also have zero gradient, and in high dimensions they are more likely than minima [1, §12.1.2.2].

Scale changes the direction. Rescale one feature and the gradient rotates. This is not a numerical artefact — it is why feature scaling changes how fast gradient descent converges, and sometimes whether it converges at all.

A large gradient is not necessarily a large step. How far you actually move depends on the learning rate, which is a separate decision entirely.