difficulty 4/5
Vision transformer
A transformer applied to an image cut into fixed-size patches, with no convolution anywhere. It has far less built-in knowledge about images than a CNN, and it has to buy the difference with data.
Intuition
A transformer consumes a sequence of tokens. An image is not a sequence, and the obvious fix — one token per pixel — is impossible, because attention costs the square of the sequence length and a 224×224 image has fifty thousand pixels.
The Vision Transformer's answer is almost embarrassingly direct: cut the image into fixed squares, flatten each one, project it to the model dimension, and call that a token. Dosovitskiy et al. describe the result as showing "that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks" [1, Abstract].
What makes the page interesting is not that it works. It is when it works. A CNN is built knowing several true things about images; a ViT is not told them, and has to discover them. That trade is the whole story, and it has a price denominated in training data.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
Sources
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2021)paperhttps://ar5iv.labs.arxiv.org/html/2010.11929
- Dive into Deep Learning — Transformers for Visionbook · retrieved 2026-09-14https://d2l.ai/chapter_attention-mechanisms-and-transformers/vision-transformer.html
- Learning Transferable Visual Models From Natural Language Supervision (2021)paperhttps://arxiv.org/abs/2103.00020