difficulty 2/5
ReLU
Pass positive values through unchanged and clamp everything negative to zero — an activation so simple it is barely a function, and the default choice in modern networks because of it.
Intuition
Sigmoid and tanh spend enormous effort producing a smooth, bounded, S-shaped curve — and that smoothness is precisely what kills the gradient in their tails.
ReLU does almost nothing instead. If the input is positive, return it. If it is negative, return zero. No exponentials, no bounds, no saturation on the positive side. It looks too crude to work, and D2L calls it "significantly more amenable to optimization than the sigmoid or the tanh function" [1, §5.1.3].
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Activation functions →
Sources
- Dive into Deep Learningbook · retrieved 2026-08-31https://d2l.ai/
- Deep Learning (2016)bookhttps://www.deeplearningbook.org/
- Deep Sparse Rectifier Neural Networks (2011)paperhttps://proceedings.mlr.press/v15/glorot11a.html