Which NLP task, and scored how?
Seven language tasks side by side, what each returns and how it is scored, and one run where the text a model rates most probable is its worst.
prints on 2 A4 pagesfrom 8 pages and 1 cataloguefree to read · PDF with a free accountUpdated 24 Oct 2026
The PDF and the Print button ask you to sign in; a free account is enough. Reading and Test yourself are open to everyone.
Side by side
The first columns are the NLP tasks catalogue’s; the rest come from each page.
A formula is printed only when it is shorter than the words; the others are said in words, and each page has its formula.
Easy to confuse
The “what is the difference between…” questions, in two lines each.
Token accuracy vs Span F1
Token accuracy counts tokens, most of them outside any entity; span F1 counts whole entities found exactly.
Span F1 for tagging, always; token accuracy flatters a model that finds nothing.
Bi-encoder vs Cross-encoder
A bi-encoder embeds each text once, so search is a lookup; a cross-encoder reads each pair together, slower and finer.
A bi-encoder to retrieve, a cross-encoder to re-rank the few it returns.
Greedy vs Sampling
Greedy takes the likeliest token every step and soon loops; sampling draws from the distribution and reads like text.
Sampling with top-p for open-ended text; beam or greedy only for short, closed answers.
BLEU vs Perplexity
BLEU compares an output with reference texts; perplexity measures how well a model predicts held-out text.
BLEU for translation, beside people; perplexity to compare language models, never on their own output.
One real run
One trigram model decoding 300 tokens with each strategy, against held-out human text.
| Strategy | Repeated 4-grams | Distinct tokens | Self-perplexity |
|---|---|---|---|
| held-out human text | 0% | 55.7% | 881.60 |
| greedy | 96.6% | 3% | 16.40 |
| sampling, T = 1.0 | 0% | 68.3% | 64.80 |
| top-k, k = 50 | 1.3% | 63.7% | 63.30 |
| top-p, p = 0.9 | 0% | 66% | 67.40 |
“Greedy decoding produces the most probable text and the worst text.”
“The human column is the reference nobody prints.”
“Do not read the sampling rows against each other.”
The run and these lines are the worked example on Decoding, as it printed: a trigram model of 132,188 words, decoded 300 tokens per strategy
Which one?
How big is the answer?
- A label for the whole text Bag of words
- Spans or a label per token Named entity recognition · BERT
- New text Sequence-to-sequence · Decoding
Do you rank or group texts?
- Yes, by meaning Sentence embeddings
How will generated text be scored?
- Against reference texts BLEU
- By how well a model predicts it Perplexity
Each answer is from its page’s “Why & when”.
All 48 traps on these pages
Each is the opening line of a pitfall; its page says why.
Bag of words
- Fitting the vocabulary on the test set.
- Assuming a high count means importance.
- Reflexively removing stop words.
- Comparing documents by raw dot product.
- Reading positional independence as a claim about language.
BERT
- Using the leading classification vector as a sentence embedding without fine-tuning.
- Expecting BERT to generate text.
- Forgetting the 512-token limit.
- Mixing tokenizers.
- Reading "bidirectional" as two directions concatenated.
Named entity recognition
- Reporting token accuracy.
- Comparing F1 across datasets with different entity lengths.
- Decoding each token independently.
- Assuming an ablation transfers across languages.
- Ignoring the word-versus-sub-token seam.
- Treating partial matches as successes.
- Assuming more tag scheme is better.
Sentence embeddings
- Using raw BERT output as a sentence vector.
- Comparing vectors from different models or different pooling.
- Forgetting to length-normalise.
- Using a cross-encoder for search.
- Treating high cosine as equivalence.
- Benchmarking on a domain you are not in.
Sequence-to-sequence
- Passing a single vector between encoder and decoder.
- Assuming attention is a transformer idea.
- Evaluating only on short sentences.
- Ignoring the train-test mismatch from teacher forcing.
- Reading attention weights as an explanation of the model.
- Forgetting the end-of-sequence token.
Decoding
- Using beam search for open-ended generation.
- Reporting generated text without the decoding settings.
- Treating a low temperature as a safety measure.
- Picking k once and reusing it everywhere.
- Tuning decoding on perplexity.
- Expecting a bigger model to fix repetition.
BLEU
- Comparing BLEU across papers.
- Comparing BLEU across test sets or languages.
- Reading a one-point gain as progress.
- Using it on short outputs without smoothing.
- Using it outside machine translation without checking it works there.
- Assuming a high score means correct meaning.
- Quoting a correlation without saying which systems it was computed over.
Perplexity
- Comparing perplexities across tokenizers.
- Quoting it without the vocabulary size.
- Minimising the perplexity of generated text.
- Expecting it to catch rare catastrophic errors.
- Reading it as accuracy or truthfulness.
- Comparing across corpora.