difficulty 3/5
Word embeddings
Dense vectors for words, learned from the company they keep — so that words used in similar contexts end up near each other, without anyone defining what similar means.
Intuition
A token id is a number with no structure. Token 4,812 and token 4,813 are as unrelated as any other pair, so a model given ids has to learn everything about every word from scratch, and learns nothing about a word it has seen twice.
An embedding replaces the id with a vector, and the vectors are arranged so that distance means something. The arrangement is not designed — it is inferred from a single observation about language: words that mean similar things get used in similar places. Drank and ate are surrounded by the same kinds of word. So are cat and dog. Count what surrounds each word across a large corpus and that similarity becomes geometry — word2vec was evaluated expecting that "similar words tend to be close to each other" [1, §1.1].
What is surprising is how much structure comes free. The relationships are not only clustered but partly linear — enough that some analogies can be done with arithmetic.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
Sources
- Efficient Estimation of Word Representations in Vector Space (2013)paperhttps://ar5iv.labs.arxiv.org/html/1301.3781
- Dive into Deep Learningbook · retrieved 2026-09-05https://d2l.ai/
- Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings (2016)paperhttps://arxiv.org/abs/1607.06520