difficulty 3/5
Sequence-to-sequence
An encoder reads a variable-length input and a decoder writes a variable-length output. Squeezing that into one fixed vector is the bottleneck attention was invented to remove.
Intuition
Translation, summarisation and speech transcription share a shape that classification does not have: the input and the output are both sequences, they are different lengths, and no token of one lines up with a token of the other. d2l calls these problems ones "where inputs and outputs each consist of variable-length unaligned sequences", and says the response is an encoder-decoder architecture [2, §10.7].
Split the work. One network reads the input and produces a representation; a second network writes the output from it, one token at a time. The second half is already familiar — "if we ignore the encoder, the decoder in a sequence-to-sequence architecture behaves just like a normal language model" [2, §10.7].
Everything interesting is in what passes between them. The original answer was one fixed-length vector, and Bahdanau et al. open by naming that as the problem: "we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture" [1, Abstract].
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: NLP tasks →
Sources
- Neural Machine Translation by Jointly Learning to Align and Translate (2015)paperhttps://ar5iv.labs.arxiv.org/html/1409.0473
- Dive into Deep Learning — Sequence-to-Sequence Learningbook · retrieved 2026-09-13https://d2l.ai/chapter_recurrent-modern/seq2seq.html