difficulty 3/5
BLEU
A precision score over n-gram overlap with a reference translation, times a penalty for being too short. A parameterised family, not a single number.
Intuition
Generated text has no single right answer, so it cannot be scored by equality. BLEU's answer is to compare against a human reference and count how much of the candidate's vocabulary and phrasing appears in it — "in principle, for any n-gram in the predicted sequence, BLEU evaluates whether this n-gram appears in the target sequence" [1, §10.7.8].
Two instincts make it work at all. Counting precision rather than recall means a system is not punished for omitting one of the many valid ways to say something. Counting longer n-grams as well as single words means word order partly survives the comparison: unigrams check whether you used the right words, four-grams check whether you put them in a plausible sequence.
It has held up for over twenty years on the strength of three practical virtues Post lists: "BLEU's relative language independence, its ease of computation, and its reasonable correlation with human judgments" [2, §1]. The rest of this page is about the conditions on that third one, which are stronger than most people using the metric realise.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: NLP tasks →
Sources
- Dive into Deep Learning — Sequence-to-Sequence Learningbook · retrieved 2026-09-13https://d2l.ai/chapter_recurrent-modern/seq2seq.html
- A Call for Clarity in Reporting BLEU Scores (2018)paperhttps://ar5iv.labs.arxiv.org/html/1804.08771
- BERTScore: Evaluating Text Generation with BERT (2020)paperhttps://ar5iv.labs.arxiv.org/html/1904.09675
- Tangled up in BLEU — Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics (2020)paperhttps://ar5iv.labs.arxiv.org/html/2006.06264