difficulty 4/5
Self-supervised learning
Build the labels out of the data itself — two augmented views of one image are a positive pair — and the representation transfers as well as a supervised one.
Intuition
Transfer learning needs a labelled dataset to pretrain on. CLIP needs captions. Self-supervised learning needs neither: the supervision is manufactured from the image itself.
The construction is almost embarrassingly simple. Take one image, augment it twice, and declare the two results a positive pair. Everything else in the batch is a negative. Train the network to pull the pair together and push the rest apart.
What is surprising is how far that gets you. "A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy … matching the performance of a supervised ResNet-50" [1, Abstract].
The interesting question is therefore not whether it works but what stops it cheating. A task you invented can usually be solved by a trick you did not intend, and most of SimCLR's paper is about closing those doors.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
Sources
- A Simple Framework for Contrastive Learning of Visual Representations (2020)paperhttps://ar5iv.labs.arxiv.org/html/2002.05709
- Learning Transferable Visual Models From Natural Language Supervision (2021)paperhttps://ar5iv.labs.arxiv.org/html/2103.00020