difficulty 2/5
Softmax
Turns a vector of scores into a probability distribution by exponentiating and normalising — the standard last step of a multi-class classifier, and the standard place for a numerical overflow.
Intuition
A classifier's last layer produces one number per class. Nothing about those numbers makes them probabilities: they can be negative, they can all be large, and there is no reason for them to sum to anything in particular.
Softmax fixes both problems in one step, and the two halves of its name are the two halves of the fix. Exponentiating makes every number positive and keeps the order they were already in. Dividing by the total makes them sum to one. What comes out is a distribution over the classes [1, §4.1.1.2].
The name is misleading in a way worth knowing early. It is not a soft version
of max — it does not approximate the largest value. It is a soft version of
argmax: a smooth stand-in for the one-hot vector that would put all its mass
on the winner: "a 'softened' version of the arg max"
[2, §6.2.2.3].
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Activation functions →
Sources
- Dive into Deep Learning — 4.1 Softmax Regressionbook · retrieved 2026-09-20https://d2l.ai/chapter_linear-classification/softmax-regression.html
- Deep Learning — 4.1 Overflow and Underflow (2016)book · retrieved 2026-09-20https://www.deeplearningbook.org/