difficulty 4/5
CLIP
Train an image encoder and a text encoder together so a picture and its caption land in the same place. The label set becomes something you write, not something you train for.
Intuition
Every classifier on this site so far has a fixed list of answers decided before training. Radford et al. open on exactly that: "state-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept" [1, Abstract].
CLIP's move is to make the label a sentence. Train an image encoder and a text encoder jointly so that a photo and its caption land close together, and the class list stops being a property of the weights. Afterwards, "natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks" [1, Abstract].
The supervision is free in the sense that matters: captions already exist alongside images, at internet scale, without anyone being paid to draw boxes.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: Vision tasks →
Sources
- Learning Transferable Visual Models From Natural Language Supervision (2021)paperhttps://ar5iv.labs.arxiv.org/html/2103.00020
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2021)paperhttps://ar5iv.labs.arxiv.org/html/2010.11929
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019)paperhttps://ar5iv.labs.arxiv.org/html/1908.10084