Attention

Intuition

To interpret it in "the trophy did not fit in the suitcase because it was too big", a model has to look at trophy rather than suitcase. Which word to look at is not fixed by position — swap two adjectives and the answer changes — so it cannot be wired in. It has to be computed from the content.

Attention is that computation. Each position emits a query — what am I looking for — and every position emits a key — what I have. Compare them, turn the comparisons into weights that sum to one, and take a weighted average of the values. A word that matches the query strongly contributes most.

The reason this reshaped the field is not expressiveness but distance. In a recurrent network, information from token 1 reaches token 500 by passing through 499 steps — and, as that page measures, a gradient only a hundred steps back can already arrive at around 10⁻²³ of its strength. With attention it is one step, for every pair, always.

Attention is older than the transformer, and was invented to fix something else. It appears in Bahdanau et al.'s 2015 translation model, a recurrent encoder-decoder, in the paragraph explaining why the decoder should read a weighted mixture of the encoder's per-word states rather than a single summary: "Intuitively, this implements a mechanism of attention in the decoder. The decoder decides parts of the source sentence to pay attention to. By letting the decoder have an attention mechanism, we relieve the encoder from the burden of having to encode all information in the source sentence into a fixed-length vector" [1, §3.1]. The transformer's contribution two years later was to notice that once you have this, the recurrence it was bolted onto is no longer earning its place. See seq2seq for the bottleneck it was answering.

Formal definition

How it works

Where it's used

Why & when

Worked example

Pitfalls