difficulty 3/5
Transfer learning
Taking a network trained on a large dataset, replacing only its output layer and training on your much smaller one — the features are inherited, the classifier is all you learn from scratch.
Intuition
You have 800 photographs of two kinds of thing and you want a classifier. A deep network has tens of millions of parameters. Fitting one to 800 examples is not a difficult problem, it is an impossible one.
But most of what such a network knows is not about your two classes. The early layers found edges; the middle ones found textures and parts. None of that is specific to whatever you are classifying — an edge is an edge in a photograph of anything. Networks trained on natural images learn first-layer features "similar to Gabor filters and color blobs", whatever the dataset or task [3, abstract]. Only the last layer, which maps features to your categories, is genuinely yours.
So keep the rest. Download a network someone trained on a million images, discard its final layer, bolt on one shaped for your problem, and train. The 800 examples now only have to specify the last step, and there is enough of them to do that.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
Sources
- Dive into Deep Learningbook · retrieved 2026-09-05https://d2l.ai/
- Deep Residual Learning for Image Recognition (2015)paperhttps://arxiv.org/abs/1512.03385
- How transferable are features in deep neural networks? (2014)paperhttps://arxiv.org/abs/1411.1792
- Do Better ImageNet Models Transfer Better? (2019)paperhttps://arxiv.org/abs/1805.08974
- Domain Adaptive Transfer Learning with Specialist Models (2018)paperhttps://arxiv.org/abs/1811.07056
- Transfusion: Understanding Transfer Learning for Medical Imaging (2019)paperhttps://arxiv.org/abs/1902.07208