difficulty 4/5
Transformer
A stack of identical layers, each self-attention followed by a position-wise feed-forward network, both wrapped in a residual connection and layer normalisation, with word order supplied separately.
Intuition
Attention is a mechanism, not a model. It computes a weighted average over positions; it does not learn anything on its own, it has no depth, and it has no notion of order. The Transformer is what you get when you build an architecture whose only mixing operation is attention, and then supply everything attention lacks.
Each of those additions answers a specific deficiency. Attention has no order, so add a positional encoding. Attention is linear in its values, so put a small neural network after it. Stacking either of those many times is hard to train, so wrap each in a residual connection and normalise.
The result is an architecture with no recurrence and no convolution, which matters for an unglamorous reason: every position can be computed at the same time. The model did not become more expressive than an LSTM so much as it became parallel, and parallel is what let it be trained on far more text. The paper reports exactly that pairing: better quality "while being more parallelizable and requiring significantly less time to train" [1, abstract].
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
Sources
- Attention Is All You Need (2017)paperhttps://ar5iv.labs.arxiv.org/html/1706.03762
- Dive into Deep Learningbook · retrieved 2026-09-05https://d2l.ai/
- Layer Normalization (2016)paperhttps://arxiv.org/abs/1607.06450