difficulty 3/5
Leaky ReLU
ReLU with a small non-zero slope for negative inputs instead of a flat zero — so a unit that goes negative still receives a gradient and can come back.
Intuition
ReLU's one real failure mode is that zero is an absorbing state. A unit whose input is negative for every example outputs zero, receives zero gradient, and can therefore never be updated back into usefulness by its own gradient [3, §6.3.1].
Leaky ReLU makes the negative side a shallow slope instead of a flat line. The unit still mostly ignores negative inputs, but a small gradient keeps flowing — enough for it to recover.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Activation functions →
Sources
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015)paperhttps://arxiv.org/abs/1502.01852
- Dive into Deep Learningbook · retrieved 2026-08-31https://d2l.ai/
- Deep Learning (2016)book · retrieved 2026-10-07https://www.deeplearningbook.org/contents/mlp.html