Vision tasks

Name What the task is	Answer attached to	Returns	Scored by	Costs
Image classification One image in, one label out. The oldest shape in vision and the only one whose metric nobody argues about — as long as you accept that there is exactly one right answer per image. Full page →	the whole image	a label	accuracy, top-5	61 h per 1,000 images
Object detection Every instance found and drawn around. The answer is now a set of variable size, so scoring needs a rule for what counts as the same object — which is where IoU and a threshold enter. Full page →	a box	boxes and labels	mAP@[.5:.95]	92 h per 1,000 images, for a point
Semantic segmentation A class per pixel, with instances not separated. The only rung that can label things with no clear boundaries — sky, road, grass — because it never has to say how many there are. Full page →	a pixel	a class map	mean IoU	every pixel, but no instances
Instance segmentation A mask per object rather than a class per pixel. Both the detection question and the segmentation question at once, and the most expensive annotation in this table by a factor of four. Full page →	an instance	a mask per object	mask AP	259 h per 1,000 images
Image–text retrieval A caption in, a ranked list of images out, or the reverse. The label set becomes something you write rather than something you train for, and the supervision is already on the web. Full page →	an image–text pair	a ranking	recall@k, top-1	nothing — the caption exists
Image generation Noise in, an image out, with no correct answer to compare against. Every metric here scores a distribution rather than a prediction, which is a different kind of claim. Full page →	nothing	a new image	FID, IS, human	no annotation at all
3-D shape understanding A point cloud in, a label or a per-point label out. The same ladder of granularity, on data with no grid — which breaks every convolution in this table. Full page →	a point	a label, or one per point	accuracy, mean IoU	sensor plus manual cleanup

Read the Costs column first. The architecture is nearly interchangeable across these rows — d2l makes that the organising idea of its whole vision chapter, noting that layerwise representations "have been successfully used in various computer vision tasks such as object detection, semantic segmentation, and style transfer" [4, ch. 14, introduction]. What is not interchangeable is what you had to pay a human to produce, and that decides the metric, the failure mode and whether the task is tractable at all.

The rows are one ladder, and COCO drew it before anyone built the models

The first four rows are not four subjects. They are four heights of the same question — where is it? — and the COCO paper lays them out in exactly that order in its opening figure: previous datasets "focused on (a) image classification, (b) object bounding box localization or (c) semantic pixel-level segmentation", and COCO adds "(d) segmenting individual object instances" [1, §I].

Each rung answers the one below it and adds something. Classification asks whether the object is present: the task "requires binary labels indicating whether objects are present in an image" [1, §II]. Detection adds a place — "stating that an object belonging to a specified class is present, and localizing it in the image" [1, §II]. Semantic segmentation drops the instance and takes the pixel: it "requires that each pixel of an image be labeled as belonging to a category, such as sky, chair, floor, street" [1, §II]. Instance segmentation asks for both.

The third rung is the one people put in the wrong place. It is not "detection, but more precise" — it is a different question, and its advantage is exactly what it gave up. Dropping instances "enables the labeling of objects for which individual instances are hard to define, such as grass, streets, or walls" [1, §II]. A detector cannot label the sky, because there is no such thing as one sky.

The ladder is priced, and the price is not linear

COCO reports two of its three annotation stages per dataset and the third per instance, which hides what the three numbers say together. On a common unit of worker hours per 1,000 images, a label costs 61, adding a located instance takes it to 92, and a mask takes it to 259 — so a mask is 4.2 times a label, and segmentation alone is 65% of the total. The bottom row is not a fourth rung but the alternative: CLIP trained on 400 million image–text pairs collected from the internet, where the caption was written by whoever posted the picture. That is not free, but its cost is crawling and filtering rather than worker hours, and it is why the row sits at zero on this axis. Note that COCO never annotated bounding boxes at all — Section VI derives them from the masks — so the middle rung is a point on each instance, not a box.

COCO's annotation pipeline has three stages, and the paper gives a cost for each one. Category labeling — which categories are in this image — "took ∼20k worker hours to complete" [1, §IV-A]. Instance spotting, a cross on each object, ran to "∼10k worker hours" [1, §IV-B]. Then segmentation: "Segmenting 2,500,000 object instances is an extremely time consuming task requiring over 22 worker hours per 1,000 segmentations" [1, §IV-C].

Two of those are reported per dataset and the third per instance, so the comparison they are making is not visible until you put them on one unit. The worked example does that, and the answer is that a mask costs 4.2 times a label, with segmentation alone accounting for 65% of a fully masked image.

Two details worth having. First, COCO never annotated boxes at all — the detection experiments "obtain tight-fitting bounding boxes from the annotated segmentation masks" [1, §VI], so the middle rung is a point on each instance rather than a drawn box. Second, humans find the top rung hard too: workers had to pass a training task before they were allowed to segment, and "approximately 1 in 3 workers passed the training stage" [1, §IV-C].

The benchmark declines to score the hardest case, and says so

Worth knowing before you trust a number from this table. COCO segmented every instance in an image up to about ten or fifteen; past that, "the remaining instances were marked as 'crowds' using a single (possibly multi-part) segment", and "for the purpose of evaluation, areas marked as crowds will be ignored and not affect a detector's score" [1, §IV-C].

So the dense-crowd case — the one a real deployment hits in a train station — is not in the score at all. This is not a flaw in COCO, which is explicit about it. It is a reminder that the metric was built by people who had to stop somewhere, and the place they stopped is the place your application probably starts.

The same caution applies at the top of the ladder, where the label itself is uncertain. The paper found that "even dedicated experts often disagree on object presence" [1, §IV-D], which is why eight crowd workers were used per image rather than one — the union of eight achieved better recall than any single expert [1, §IV-D].

Every rung up loses the metric a little

The pattern that runs down the table: the finer the answer, the more the score becomes an opinion.

Classification accuracy is uncontested — there is one label and either you got it or you did not. Detection needs a rule for same object, and mAP@[.5:.95] is a specific opinion about how much localisation precision matters and how much weight rare classes deserve. Segmentation adds a second: mean IoU averages over classes, not over pixels, so a class occupying 0.1% of the image counts as much as the road.

At the bottom of the table the metric stops having a ground truth at all. DDPM reports "an Inception score of 9.46 and a state-of-the-art FID score of 3.17" [3, Abstract] — both compare a distribution of generated images against a distribution of real ones. There is no per-image right answer to check, which is a different kind of claim from "91.2% top-1" even though it is printed in the same font. Worse, as FID sets out, the number you compute from a finite sample is not the number the metric is defined to be, and the gap depends on the model being measured.

COCO showed this coming. Even holding the detection correct, segmentation quality is separately hard: the paper benchmarks "given that the detector reports a correct bounding box, how well does the predicted segmentation of that object match the ground truth segmentation?" and finds that predicting the mask "is difficult even for correct detections" [1, §VI]. Each rung is its own problem, not a polish on the one below.

The row that refuses to climb the ladder

The bottom of the annotation figure is the interesting part. CLIP's argument starts from exactly the cost this table has been pricing: "State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept" [2, Abstract].

The escape is to stop paying for labels. CLIP trains on "400 million (image, text) pairs collected from the internet" [2, Abstract], where the caption was written by whoever posted the picture, and the result "match[es] the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on" [2, Abstract].

That is why the row sits at zero on the figure's axis. It is not free — crawling and filtering 400 million pairs has its own cost, and the paper is candid that this changes what the supervision is rather than removing it. But it is not worker hours, and the whole shape of the field after 2021 follows from that substitution. The observation driving it was that "the aggregate supervision accessible to modern pre-training methods within web-scale collections of text surpasses that of high-quality crowd-labeled NLP datasets" [2, §1] — and that computer vision, unlike NLP, had not yet made the swap.

Worked example

Putting COCO's three annotation stages on a single unit, because the paper reports two of them per dataset and the third per instance.

Python 50 lines Show the code Hide the code

"""What each rung of the vision task ladder costs to annotate."""

IMAGES = 328_000
INSTANCES = 2_500_000
CATEGORY_HOURS = 20_000
SPOTTING_HOURS = 10_000
SEG_HOURS_PER_1K = 22
STATED_TOTAL = 70_000

per_image = INSTANCES / IMAGES
print(f"  COCO: {IMAGES:,} images, {INSTANCES:,} instances, {per_image:.1f} per image")
print(f"  (the paper states 7.7; the rounded counts above give {per_image:.2f})\n")

# Put every stage on the same unit: worker hours per 1,000 images.
seg_hours = SEG_HOURS_PER_1K * INSTANCES / 1000
stages = [
    ("a label per image", "category labeling", CATEGORY_HOURS, 8),
    ("a point per instance", "instance spotting", SPOTTING_HOURS, 8),
    ("a mask per instance", "instance segmentation", seg_hours, 1),
]

print("  what each rung costs, on a common unit")
print(f"{'the output':>22}{'stage':>24}{'workers':>9}{'hours':>11}{'h / 1k images':>15}")
cumulative = 0.0
rows = []
for output, stage, hours, workers in stages:
    cumulative += hours
    per_1k = hours / IMAGES * 1000
    rows.append((output, per_1k, cumulative / IMAGES * 1000))
    print(f"{output:>22}{stage:>24}{workers:>9}{hours:>11,.0f}{per_1k:>15.1f}")

print("\n  and the ladder is cumulative: a mask needs the label and the point first")
print(f"{'to get':>22}{'you also pay for':>34}{'h / 1k images':>15}{'vs a label':>12}")
# Not "a box": COCO never annotated boxes. Section VI takes them from the
# masks, so a box costs either a point or a whole mask depending on where you
# stop, and the ladder is honest only if it names what was actually paid for.
labels = ["a label", "a located instance", "a mask"]
needs = ["nothing else", "the label", "the label and the point"]
base = rows[0][2]
for name, need, (_, _, cum) in zip(labels, needs, rows):
    print(f"{name:>22}{need:>34}{cum:>15.1f}{cum / base:>11.1f}x")

seg_share = rows[2][1] / rows[2][2] * 100
print(f"\n    segmentation alone is {seg_share:.0f}% of the cost of a fully masked image")

print("\n  the one number that does not reconcile")
print(f"    22 h / 1,000 masks x {INSTANCES:,} instances = {seg_hours:,.0f} worker hours")
print(f"    plus {CATEGORY_HOURS:,} + {SPOTTING_HOURS:,} = {cumulative:,.0f} hours in total,")
print(f"    against the {STATED_TOTAL:,}+ the paper states in its discussion.")
print("    The rate is the reliable figure; the total is stated as a lower bound.")

Output

  COCO: 328,000 images, 2,500,000 instances, 7.6 per image
  (the paper states 7.7; the rounded counts above give 7.62)

  what each rung costs, on a common unit
            the output                   stage  workers      hours  h / 1k images
     a label per image       category labeling        8     20,000           61.0
  a point per instance       instance spotting        8     10,000           30.5
   a mask per instance   instance segmentation        1     55,000          167.7

  and the ladder is cumulative: a mask needs the label and the point first
                to get                  you also pay for  h / 1k images  vs a label
               a label                      nothing else           61.0        1.0x
    a located instance                         the label           91.5        1.5x
                a mask           the label and the point          259.1        4.2x

    segmentation alone is 65% of the cost of a fully masked image

  the one number that does not reconcile
    22 h / 1,000 masks x 2,500,000 instances = 55,000 worker hours
    plus 20,000 + 10,000 = 85,000 hours in total,
    against the 70,000+ the paper states in its discussion.
    The rate is the reliable figure; the total is stated as a lower bound.

The last block is a real discrepancy and it is worth leaving in. Multiplying the paper's own segmentation rate by its own instance count gives 55,000 hours, which with the two earlier stages comes to 85,000 — against the "over 70,000 worker hours" the discussion reports [1, §VIII]. Both numbers are the paper's. The per-1,000 rate is the one to use, because it is stated as a rate and the total is stated as a lower bound; the ratios in this table depend only on the rates, so they stand either way.

A practical reading of the table. Pick the lowest rung that answers your question. Every step up multiplies annotation cost, weakens the metric, and adds a failure mode the rung below did not have. The exception is the retrieval row, which is not higher on the ladder at all — it is a way of not paying for it, and it is worth checking whether your problem can be phrased that way before you start labelling anything.