Computer vision · 5 compared
Computer vision architectures
The lineage of image-recognition networks from AlexNet to the Vision Transformer, and what each one changed about how images get processed.
Comparison
| Name What it changed | Depth | Input unit |
|---|---|---|
| AlexNet 2012 Showed a deep convolutional network trained on GPUs could win ImageNet outright, making depth the default answer. | 5 convolutional layers | pixels |
| VGG 2014 Replaced large filters with stacks of 3×3 convolutions, showing depth mattered more than filter size. | 16–19 weight layers | pixels |
| GoogLeNet (Inception) 2014 Ran several filter sizes in parallel inside one block, widening the network while holding compute flat. | 22 layers | pixels |
| ResNet 2015 Added skip connections so layers learn a residual correction, which made very deep networks trainable at all. | up to 152 layers | pixels |
| ViT 2020 Dropped convolution entirely, cutting the image into patches and feeding them to a transformer as a sequence. | transformer encoder | image patches (16×16 in the title) |
Lineage
- 2012AlexNet Showed a deep convolutional network trained on GPUs could win ImageNet outright, making depth the default answer. ImageNet Classification with Deep Convolutional Neural Networks led to VGG
- 2014VGG Replaced large filters with stacks of 3×3 convolutions, showing depth mattered more than filter size. Very Deep Convolutional Networks for Large-Scale Image Recognition led to ResNet
- 2014GoogLeNet (Inception) Ran several filter sizes in parallel inside one block, widening the network while holding compute flat. Going Deeper with Convolutions led to ResNet
- 2015ResNet Added skip connections so layers learn a residual correction, which made very deep networks trainable at all. Deep Residual Learning for Image Recognition led to ViT
- 2020ViT Dropped convolution entirely, cutting the image into patches and feeding them to a transformer as a sequence. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale the line ends here
What this catalogue is for
Eight years separate the first and last entry below, and almost everything changed except the task. Read in order, they are a single argument working itself out: make the network deeper, then find out what stops you going deeper, then fix that, then question whether convolution was needed at all.
Convolutional networks are the family these belong to, and the shared idea is that spatial structure should be built into the model rather than learned from scratch [1, §9.4]. Underneath, convolutional networks "are simply neural networks that use convolution in place of general matrix multiplication in at least one of their layers" [1, ch. 9, introduction].
How to read the table
Depth is the number the paper itself reports, not a derived count — the architectures measure themselves differently and normalising them would invent precision that isn't there. Input unit is what the first layer actually consumes, which is where the last entry breaks the pattern.
Deliberately absent: accuracy numbers. ImageNet results depend on crop, ensemble and evaluation protocol, and the figures quoted across the web frequently do not match the papers. A table that puts them side by side implies a comparison that was never run under equal conditions.
The two turning points
Everything up to 2014 says the same thing in different ways: deeper is better. That claim ran into a wall — past a certain depth, adding layers made networks harder to optimise, not better. ResNet is the fix. By reformulating layers as learning "residual functions with reference to the layer inputs, instead of learning unreferenced functions", it made depths like 152 layers trainable [2, abstract].
ViT is the second break, and a sharper one. It abandons the family premise. Rather than building spatial structure into the model, it splits the image into fixed patches and hands them to a transformer as a sequence of tokens, showing "a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks" [3, abstract].
That is why the lineage below ends rather than continues: the last arrow points out of the family.
What to take from it
If you are choosing an architecture today, this history is context, not a shortlist — you would start from a pretrained model. What it is good for is recognising the pattern: each entry removed one hand-designed constraint. Larger filters, then depth limits, then convolution itself. The direction of travel is consistently toward letting scale and data decide what the earlier generation had specified by hand.
Sources
- Deep Learning (2016)bookhttps://www.deeplearningbook.org/
- Deep Residual Learning for Image Recognition (2015)paperhttps://arxiv.org/abs/1512.03385
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020)paperhttps://arxiv.org/abs/2010.11929