Which activation, and where?
Five activations side by side, the pairs people confuse, and one deep network trained with each, where one learns nothing and one loses a fifth of its units.
prints on 2 A4 pagesfrom 6 pages and 1 cataloguefree to read · PDF with a free accountUpdated 24 Oct 2026
The PDF and the Print button ask you to sign in; a free account is enough. Reading and Test yourself are open to everyone.
Side by side
The first columns are the Activation functions catalogue’s; the rest come from each page.
A formula is printed only when it is shorter than the words; the others are said in words, and each page has its formula.
Easy to confuse
The “what is the difference between…” questions, in two lines each.
Tanh is sigmoid rescaled to (−1, 1), zero-centred with four times the peak gradient; both saturate in the tails.
Tanh inside a network, if a bounded activation is wanted at all; sigmoid at a binary output.
ReLU vs Leaky ReLU
Leaky ReLU keeps a small slope below zero, so no unit stops receiving gradient; ReLU's exact zeros make it sparse.
ReLU by default; leaky ReLU once dead units are measured.
Sigmoid gives each output its own probability; softmax makes the outputs one distribution that sums to 1.
Softmax when exactly one class is true; a sigmoid per label when several can be.
Saturated unit vs Dead unit
A saturated sigmoid passes back a gradient that is tiny but never zero; a dead ReLU unit passes back none, for every input.
Saturation comes with depth and is fixed by the activation; dead units often come from too large a learning rate.
One real run
The same ten-layer network trained on two circles with each activation, and once with none.
| Activation | Gradient kept | Dead | Test acc. |
|---|---|---|---|
| none | 0.79 | 0.00 | 0.46 |
| step | 0 | 0.12 | 0.48 |
| sigmoid | 1.6e-6 | 0.00 | 0.50 |
| tanh | 0.64 | 0.00 | 0.99 |
| relu | 0.87 | 0.20 | 0.99 |
| leaky-relu | 0.87 | 0.00 | 0.99 |
“Without a non-linearity, ten layers are one straight line.”
“Sigmoid's gradient does not survive the depth.”
“A fifth of ReLU's units die; leaky ReLU loses none.”
The run and these lines are the worked example on ReLU, as it printed: 10 hidden layers of 32 units, two circles, mean of 5 seeds
Which one?
Where does it sit?
What does the output answer?
Are many units dead?
- Yes, and counted Leaky ReLU
Each answer is from its page’s “Why & when”.
All 23 traps on these pages
Each is the opening line of a pitfall; its page says why.
Perceptron
- Scale your features.
- It only converges if the data is linearly separable.
- The output is not a confidence.
Sigmoid
- Use a numerically stable implementation.
- Outputs are not zero-centred.
- Do not stack sigmoids deep.
- Sigmoid output is not calibrated by default.
Tanh
- It still saturates.
- Zero-centred inputs matter too.
- The derivative has a cheap closed form.
- Being better than sigmoid is a low bar.
ReLU
- Dying ReLU.
- Unbounded output.
- Not zero-centred.
- Watch your learning rate.
Leaky ReLU
- a is a hyperparameter you probably will not tune.
- It gives up sparsity.
- It is not reliably better.
- Diagnose before switching.
Softmax
- Taking the log of the output.
- Reading confidence off it.
- Applying it twice.
- Using it for multi-label problems.