Which vision task, and scored how?
Six vision tasks side by side, what each one returns and how it is scored, and one run where sloppier boxes barely move one score and wipe out another.
prints on 2 A4 pagesfrom 7 pages and 1 cataloguefree to read · PDF with a free accountUpdated 24 Oct 2026
The PDF and the Print button ask you to sign in; a free account is enough. Reading and Test yourself are open to everyone.
Side by side
The first columns are the Vision tasks catalogue’s; the rest come from each page.
A formula is printed only when it is shorter than the words; the others are said in words, and each page has its formula.
Easy to confuse
The “what is the difference between…” questions, in two lines each.
Semantic segmentation vs Instance segmentation
Semantic segmentation labels every pixel by class; instance segmentation also tells two touching objects apart.
Semantic when you need area by class; instance when you need to count or follow objects.
Object detection vs Instance segmentation
Detection returns a box per object; instance segmentation returns its exact mask, at more labelling cost.
A box when position is the question; a mask when shape or area is. Do not compare their scores.
A classifier is trained on your labels; CLIP classifies with no labels by matching the image to class names as text.
CLIP to start with no labels; a trained classifier once labels exist and accuracy matters.
IoU vs Mean average precision
IoU scores how well one box overlaps its target; mAP turns every box into one score at an IoU threshold.
Report mAP with its threshold; mAP@0.5 alone cannot see sloppy boxes.
One real run
One detector whose boxes get sloppier, scored at three IoU thresholds.
| Box jitter | mAP@0.5 | mAP@0.75 | mAP@[.5:.95] |
|---|---|---|---|
| 0.02 | 0.88 | 0.86 | 0.78 |
| 0.05 | 0.88 | 0.82 | 0.62 |
| 0.10 | 0.84 | 0.21 | 0.38 |
| 0.15 | 0.73 | 0.04 | 0.23 |
| 0.20 | 0.55 | 0.01 | 0.14 |
“At a jitter of 0.10 the two metrics disagree about whether the detector works.”
“Across the whole sweep the difference is total.”
“The COCO average sits in between, which is the point of it.”
The run and these lines are the worked example on Mean average precision, as it printed: one detector, five levels of box sloppiness — how each metric reacts
Which one?
What must the answer point at?
- The whole image Image classification
- Each object Object detection · Instance segmentation
- Every pixel Semantic segmentation
Is anything unusual about the data?
- No labels, only text CLIP
- The answer is a new image Diffusion model
Each answer is from its page’s “Why & when”.
All 49 traps on these pages
Each is the opening line of a pitfall; its page says why.
Image classification
- Assuming one label per image is a property of the world.
- Quoting top-5 as if it were accuracy.
- Comparing top-1 on one paper to top-5 on another.
- Reading a leaderboard without its intervals.
- Treating overlapping intervals as proof of a tie.
- Assuming benchmark gains still transfer.
- Forgetting the label set is a design decision.
Object detection
- Inheriting anchor shapes from a paper.
- Reporting accuracy.
- Training a dense detector with plain cross entropy.
- Forgetting that NMS is a hyperparameter.
- Blaming the backbone for missed small objects.
- Comparing detectors at different IoU thresholds.
- Assuming more anchors is strictly better.
Semantic segmentation
- Reporting pixel accuracy.
- Reporting mean IoU without the per-class breakdown.
- Confusing the three segmentation tasks.
- Expecting semantic segmentation to separate touching instances.
- Building an encoder-decoder with no skip connections.
- Forgetting that border pixels are lost to every convolution.
- Treating boundary error and missing classes as the same failure.
Instance segmentation
- Using a per-pixel softmax over classes.
- Assuming the mask head needs to know the class.
- Cropping regions with a quantizing operation.
- Judging alignment by AP50.
- Comparing mask AP to box AP.
- Trusting the benchmark on crowds.
- Forgetting that the backbone was chosen for classification.
CLIP
- Reading contrastive accuracy without the chance line.
- Reproducing CLIP at a small batch.
- Using bare class names.
- Treating prompt templates as a hack.
- Expecting fine-grained or counting ability.
- Assuming zero-shot beats a supervised baseline.
- Forgetting the training data is uncurated internet text.
Diffusion model
- Simulating the forward chain during training.
- Optimising the full variational bound.
- Forgetting to condition on t.
- Assuming more steps is strictly better.
- Comparing FID across resolutions, datasets or sample counts.
- Expecting GAN-speed sampling.
- Treating the noise schedule as a detail.
Mean average precision
- Writing "mAP" without a threshold.
- Reporting AP50 alone.
- Comparing a COCO number to a VOC number.
- Crediting the architecture for a training-recipe change.
- Assuming more anchors raises AP.
- Forgetting that AP ignores calibration.
- Using the trapezoidal area instead of the step sum.