Deep learning · 5 compared
Activation functions
Without a non-linearity between layers, a deep network collapses into a single linear one — these are the functions that stop that happening, and what each costs in return.
Comparison
| Name What it does | Output range | Peak gradient | Saturates? |
|---|---|---|---|
| Step The perceptron's original. Hard threshold, no gradient anywhere — which is exactly why networks could not be trained with it. Full page → | {0, 1} | 0 (undefined at 0) | Everywhere |
| Sigmoid The smooth step. Bounded, reads as a probability, and its gradient never exceeds 0.25. Full page → | (0, 1) | 0.25 | Both tails |
| Tanh A rescaled sigmoid, zero-centred. Four times the peak gradient, same saturation problem. Full page → | (−1, 1) | 1 | Both tails |
| ReLU Pass positives, zero negatives. Gradient of exactly 1 above zero, so depth stops attenuating it. Full page → | [0, ∞) | 1 | Negative side only |
| Leaky ReLU ReLU with a shallow negative slope, so a unit that goes negative still receives gradient and can recover. Full page → | (−∞, ∞) | 1 | No |
On the cheat sheet: Activation functions →
Why there has to be one at all
Stack two linear layers and you have not built anything. D2L states the reason in one line: "an affine function of an affine function is itself an affine function" [1, §5.1.1.3]. A hundred linear layers collapse algebraically into one, and the network can represent nothing a single layer could not.
The activation is the only thing standing between a deep network and that collapse. Every row below is a different answer to "what non-linearity, at what cost?"
Read the gradient column
The history of this table is the history of one number.
Step has no usable gradient at all, which is why the perceptron could not be trained by descent. Sigmoid fixed that but capped the gradient at 0.25 — and in a deep network gradients multiply layer by layer, so 0.25 raised to the tenth power is about one in a million. Tanh raised the ceiling to 1, then saturated in the tails the same way.
ReLU broke the pattern rather than improving on it. Its gradient is exactly 1 for every positive input and does not decay anywhere, so depth stops attenuating the signal. D2L: its derivatives are well behaved — they are either zero or pass the argument straight through [1, §5.1.2.1].
The trade nobody mentions
ReLU's zero for negative inputs is not free. A unit whose input is negative for every example gets no gradient and cannot recover — it is dead for the rest of training. Leaky ReLU fixes that with a shallow slope — generalisations of ReLU "receive gradient everywhere" [2, §6.3.1] — and pays for it by giving up ReLU's sparse, exactly-zero activations.
That is the shape of every row here: nothing is strictly better, each choice trades one property for another.
Choosing
- Hidden layers → ReLU. The default, and the thing anything else has to beat.
- Measured dead units → leaky ReLU. Count zero activations first; do not switch on suspicion.
- Recurrent cells → tanh, where the bounded output stops a fed-back state from exploding.
- Binary output layer → sigmoid, where a (0, 1) value is genuinely wanted.
- Multi-class output → softmax, which generalises sigmoid across classes.
Newer functions — GELU, Swish, Mish — are all proposed relative to ReLU and win by small, task-dependent margins. GELU is standard in transformers. None of them changes the argument above; they are refinements of the ReLU idea rather than departures from it.
Sources
- Dive into Deep Learningbook · retrieved 2026-08-31https://d2l.ai/
- Deep Learning (2016)bookhttps://www.deeplearningbook.org/