difficulty 2/5
Tokenization
Cutting text into the units a model actually consumes — where splitting on words leaves every unseen word as a hole, and subword methods earn their vocabulary from frequency instead.
Intuition
Before a model sees anything, something has to decide what the pieces are. Split on spaces and you have a word vocabulary — which sounds natural and fails immediately, because language does not have a finite word list. Every proper noun, typo, compound and inflection you never saw at training time becomes the same thing: unknown.
Split on characters instead and nothing is ever unknown, but you have thrown away every unit of meaning larger than a letter and made every sequence five times longer.
Subword tokenization is the compromise, and its trick is that nobody chooses the pieces. Start from characters, count which adjacent pair occurs most in the corpus, glue that pair together, repeat [1, §3.2]. Frequent words end up as single tokens because they earned it; rare ones end up as several pieces that are themselves frequent. Nothing is ever unknown, and common text stays short.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
Sources
- Neural Machine Translation of Rare Words with Subword Units (2016)paperhttps://ar5iv.labs.arxiv.org/html/1508.07909
- Dive into Deep Learningbook · retrieved 2026-09-05https://d2l.ai/
- SentencePiece: A simple and language independent subword tokenizer and detokenizer (2018)paperhttps://ar5iv.labs.arxiv.org/html/1808.06226