difficulty 2/5
TF-IDF
A weight that multiplies how often a term appears in one document by how rare it is elsewhere, so a word appearing everywhere contributes nothing and a word appearing in few documents dominates.
Intuition
A bag of words treats every column as equally informative, and it plainly is not. In a corpus about cars, the word auto appears in nearly every document and therefore separates none of them; the word carburettor appears in six and tells you a great deal about those six. The textbook uses the same example: a collection on the auto industry "is likely to have the term auto in almost every document" [1, §6.2.1].
So weight each term by how unusual it is. The word that occurs many times in this document and rarely elsewhere is the one that characterises it. The word that occurs everywhere characterises nothing, whatever its count.
That is the entire idea, and it is old, simple, and still the thing to try before anything expensive. It is also the reason stop-word lists went out of fashion: a weighting scheme that already assigns the a weight of zero does not need a hand-maintained list telling it to ignore the.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
Sources
- Introduction to Information Retrieval (2008)bookhttps://nlp.stanford.edu/IR-book/html/htmledition/irbook.html
- scikit-learn User Guide — Feature extraction from textdocs · retrieved 2026-09-06https://scikit-learn.org/stable/modules/feature_extraction.html