Text metrics

We email you a link and a six-digit code. Either one signs you in; there is no password.Enter your email. We send a link and a six-digit code to confirm it — no password to remember. The account is free.

You come back to this page, right where you were.

Already have an account? The same email simply signs you in.

Comparison

Name What it measures	Compares	Precision or recall	AUC on word-swapped pairs	Breaks when
BLEU Clipped n-gram precision against a reference, times a brevity penalty, averaged geometrically over n = 1 to 4. The machine-translation default for two decades, and a family rather than one number. Full page →	word n-grams	precision	0.527	the reference is processed differently
ROUGE-n BLEU's arithmetic run the other way — recall of reference n-grams rather than precision of candidate ones. Built for summarisation, where omitting the important sentence is the failure that matters.	word n-grams	recall	—	a good summary is worded differently
ROUGE-L Longest common subsequence instead of fixed-length n-grams, so matching words count even with other words inserted between them. More forgiving of reordering; no better at meaning.	longest common subsequence	recall	0.536	the same as ROUGE-n
METEOR Unigram precision and recall, but allowed to back off from exact matching to stems, synonyms and a paraphrase table — so running may match run. Needs those resources, so it covers few languages fully.	words, stems, synonyms	both	0.532	your language has no resources
chrF / chrF++ Character n-grams rather than word n-grams, which makes it tokenizer-independent and far kinder to morphology. chrF++ adds word bigrams. Cheap, and consistently better correlated than BLEU.	character n-grams	both	0.608	you need an interpretable number
TER and edit distance Count the edits needed to turn the candidate into the reference, normalised by reference length. Answers how much work is left for a human, which is sometimes the better question.	edit operations	neither	0.650 (CharacTER)	a fluent rewrite costs many edits
BERTScore Match tokens by cosine similarity between contextual embeddings instead of by string equality, then take precision, recall and F1 over the best matches. Sees paraphrase; costs a forward pass. Full page →	contextual embeddings	both	0.693	the encoder never saw your domain
Perplexity The only row needing no reference — it scores the model on held-out text, not an output. How well the distribution was learned is not how good the generation is. Full page →	the model against held-out text	neither	—	you compare across tokenizers
Exact match and token F1 For tasks whose answer is a span rather than a sentence. Exact match is unforgiving and unambiguous; token F1 softens it enough to survive an article or a trailing word.	spans	both	—	the answer has several correct forms
Human evaluation The thing every other row is a proxy for. Expensive, noisy, and the only measurement that is not a guess about what a reader would say.	what a reader thinks	neither	—	annotators are underpaid or unbriefed

d2l asks the question this table answers: "we can evaluate a predicted sequence by comparing it with the target sequence (the ground truth). But what precisely is the appropriate measure for comparing similarity between two sequences?" [4, §10.7.6]. There is no single answer, which is why there are ten rows.

Scoring generated text means comparing it to something. The rows differ in what that something is — reference n-grams, reference characters, reference embeddings, edit operations, a held-out corpus, a person — and every limitation below follows from that choice rather than from any implementation detail.

The first six rows are all counting surface overlap

Zhang et al. write the common core: for reference x and candidate x̂, the exact match precision and recall are the share of the candidate's n-grams found in the reference and the share of the reference's found in the candidate [1, §2.1]. Everything in the top half of the table is a variation on those two fractions.

BLEU takes the precision side, with three modifications: "each n-gram in the reference can be matched at most once", counts are accumulated across the corpus rather than per sentence, and "very short candidates are discouraged using a brevity penalty" [1, §2.1]. See BLEU for what that costs.

ROUGE takes the recall side. "ROUGE-n computes Exact-Rₙ (usually n = 1, 2), while ROUGE-L is a variant of Exact-R₁ with the numerator replaced by the length of the longest common subsequence" [1, §2.1]. The swap from precision to recall is the whole reason it belongs to summarisation: a summary that omits the point should be punished, and a summary that adds a little should not.

METEOR loosens what counts as a match, "allowing backing-off from exact unigram matching to matching word stems, synonyms, and paraphrases. For example, running may match run if no exact match is possible" [1, §2.1]. That buys real robustness and costs portability — it "requires external resources", so "only five languages are supported with the full feature set, and eleven are partially supported" [1, §2.1].

chrF changes the unit rather than the arithmetic: it "compares character n-grams in the reference and candidate sentences", and chrF++ "extends chrF to include word bigram matching" [1, §2.1]. Working in characters sidesteps the tokenization problem that makes BLEU numbers incomparable between papers [2, §2.2].

Edit distance asks a different question altogether — TER "normalizes edit distance by the number of reference words" [1, §2.2] — and is worth keeping when the downstream job is post-editing, because then the number of edits is the quantity of interest.

What none of them can see

Every metric here is given pairs of sentences and asked, without training, whether they mean the same thing — where the hard negatives were produced by swapping words, so “flights from New York to Florida” has to be told apart from “flights from Florida to New York”. The three metrics built on word n-grams land between 0.527 and 0.536, which is to say within four points of a coin flip. Comparing characters instead does better, and comparing contextual embeddings better still, but nothing here is close to solved. The useful reading is not the ranking — it is that the metrics an entire field reports are, on this particular ability, barely distinguishable from guessing.

The figure is one task: given two sentences and no training, say whether they are paraphrases, where the hard negatives were built by swapping words so that Flights from New York to Florida has to be told apart from Flights from Florida to New York [1, §6].

The three word-n-gram metrics land at 0.527, 0.532 and 0.536 [1, §6]. Chance is 0.5. Whatever those metrics are measuring, it is very nearly orthogonal to whether the sentence still says the same thing after its arguments are exchanged — which is exactly the property Zhang et al. predict from the definitions, since "given a small window of size two, BLEU will only mildly penalize swapping of cause and effect clauses" [1, §1].

Two details make the figure more interesting rather than less. First, moving to characters or to embeddings genuinely helps — 0.608 for chrF++, 0.693 for BERTScore [1, §6] — so this is a solvable problem and not a law of nature. Second, and more sharply: a BERT classifier fine-tuned on the paraphrase task itself scores 0.351 on these adversarial pairs [1, §6], well below chance. Supervision on the ordinary distribution does not confer the ability either; it can actively mislead.

Correlation with human judgement is not a property of a metric

It is a property of a metric and the set of systems you compute it over, and both papers on the subject say so from different directions.

Across tasks: on the 2015 COCO captioning challenge, BLEU's Pearson correlation with the two human measures is −0.019 and −0.005, while METEOR reaches 0.606 and SPICE 0.759 [1, §5]. The same metric that is serviceable for translation is worthless here, and nothing about the metric changed.

Across system quality: Mathur et al. report that with a large cohort of translation systems, correlations run "as high as r = 0.9", but "if considering only the few best systems, the correlation reduced markedly", with "negative correlations in some instances" among the top four [3, §1]. Their summary is that metrics "can only be relied upon for making coarse distinctions between poor and good translation outputs, but not for assessing similar quality outputs" [3, §1] — which is the comparison almost every paper and every A/B test is actually making.

And the headline correlations are fragile in a specific way: outlier systems "can have a disproportionate effect on the computed correlation", to the point that "in the worst case scenario, outliers introduce a high correlation when there is no association between metric and human scores for the rest of the systems" [3, §6].

How to use the table

Report at least two rows from different families. A word-n-gram metric and a character or embedding one fail differently, and agreement between them is worth something that either alone is not.

Match the mode to the failure you care about. Precision for translation, where saying something wrong is the risk; recall for summarisation, where omitting something is; edit distance when a human will fix the output anyway.

Fix the preprocessing, and publish it. This is the single largest source of non-comparable numbers in the literature [2, Abstract], and it is free to fix.

Do not tune against a metric you have not validated on your own task. The COCO row is the cautionary case [1, §5].

Keep humans in the loop for anything that matters. Every automatic row is an understudy, and the figure above shows how far from the part some of them are. Note also what the bottom row's "Breaks when" says: human evaluation is not automatically the ground truth either — it is only as good as the briefing and the incentives behind it.

And remember the one row that is not a comparison. Perplexity scores the model rather than the output, which makes it the right instrument during training and the wrong one for judging generated text — where, as decoding shows, output with lower perplexity than human text is worse, not better.