Image classification

Intuition

One image goes in, one label comes out. It is the smallest question you can ask about a picture, and for a decade it was the question the whole field was organised around — because it was the one with a benchmark.

That benchmark made an assumption that is easy to miss and impossible to undo once you have built on it: each image has exactly one right answer. ILSVRC images are "manually labeled with the presence of one of 1000 object categories", and "each image contains one ground truth label" [1, §2.1].

Almost everything strange about how image classification is measured follows from that one decision. Top-5 accuracy exists because of it. The gap between benchmark progress and real progress opened because of it. It is worth understanding the assumption before the architecture.

Formal definition

A classifier maps an image to a score for each of K classes, normally through a softmax, and is trained against the recorded label with cross-entropy. What differs from ordinary classification is only how it is scored.

Top-1 error penalises the model "if their highest-confidence output label cᵢ₁ did not match ground truth class Cᵢ" [1, §4.1].

Top-5 error lets the model hedge. "Concretely, each image i has a single class label Cᵢ. An algorithm is allowed to return 5 labels cᵢ₁, … cᵢ₅, and is considered correct if cᵢⱼ = Cᵢ for some j" [1, §4.1]. The error is the fraction of images on which every one of the five is wrong.

And the reason for top-5 is the label, not the model. "The scale of ILSVRC classification task (1000 categories and more than a million of images) makes it very expensive to label every instance of every object in every image. Therefore, on this dataset only one object category is labeled in each image. This creates ambiguity in evaluation. For example, an image might be labeled as a 'strawberry' but contain both a strawberry and an apple. Then an algorithm would not know which one of the two objects to name" [1, §4.1].

Top-5 is a patch for a labelling budget. It is not a claim about what the task is.

How it works

The patch turned out not to matter for ranking, and the paper says so. After trying top-1, top-5 and a hierarchical cost that charges more for confusing a dog with a ship than with another dog: "in practice we found that all three measures of error (top-5, top-1, and hierarchical) produced the same ordering of results. Thus, since ILSVRC2012 we have been exclusively using the top-5 metric which is the simplest and most suitable to the dataset" [1, §4.1].

This is the honest version of the story. Three metrics encoding three different opinions about what a mistake costs, and all three agreed on who won. That is worth knowing both ways round: the choice of metric was not doing damage, and it was not doing much work either.

What did matter was the resolution of the benchmark. The challenge published 99.9% bootstrap confidence intervals for every top submission [1, §6.2], and the worked example below shows a plain binomial reproduces them to within 0.013 percentage points. So the width of those intervals is not a fact about the models. It is a fact about having 100,000 test images.

The label problem came back later, and larger. Beyer et al. re-annotated the ImageNet validation set and named three separate defects: many images "contain multiple, similarly prominent objects, limiting the relevance of a single label"; the annotation pipeline's one-class-at-a-time question produces "systematic inaccuracies"; and "some ImageNet classes are inherently ambiguous, drawing distinctions between essentially identical groups of images" [2, §2].

Their relabelling assigned 57,553 labels to 46,837 images [2, §3.3] — about 1.23 valid labels per image where the benchmark records one. And measured against those labels, the return on benchmark progress halved: regressing original accuracy onto the new ReaL accuracy gave a slope of 0.86 for the first half of their model pool and 0.51 for the second half, a difference significant at p < 0.001 [2, §4].

The endpoint is the part worth sitting with. "Several models already surpass the ReaL accuracy obtained by the original ImageNet labels" [2, §4]. The models are now better than the labels at saying what is in the picture.

Where it's used

• Pretraining, which is what ImageNet classification is mostly for now — a backbone trained on the task and then reused for detection, segmentation and everything else [3, §14.2].
• Medical and industrial screening, where a single-label-per-image framing genuinely holds and the classes are decided in advance.
• Content moderation and tagging, usually as multi-label rather than multi-class, which is the same network with a different output layer and loss.
• Quality gates in a pipeline, where a cheap classifier decides whether the expensive model runs at all.
• As the bottom rung of the vision ladder — see vision tasks for what the rungs above cost.

Why & when

Use it when the answer really is one label. The task is cheap to annotate, the metric is uncontested, and every rung above it costs more.

Use multi-label when the image can contain several things. Independent sigmoids and a per-class loss, not a softmax. Forcing a genuinely multi-object image through a softmax is the exact mistake ImageNet institutionalised [2, §2].

Report top-1 unless you have a reason not to. Top-5 was designed around a particular dataset's labelling budget [1, §4.1]; on your data it usually just makes the number look better.

Compute the resolution of your test set before you compare two models. One line of arithmetic, and it tells you whether the comparison can succeed at all.

Treat sub-point differences as unproven. Five ILSVRC entries spanning two competition years and 0.54 points are mutually inseparable by the challenge's own intervals.

Check whether your labels are still the limit. Beyer et al.'s result — that good models now beat the original labels — is a warning that applies to any benchmark old enough to have been optimised against [2, §4].

Worked example

The challenge published confidence intervals for its own leaderboard. Two questions follow: where do those intervals come from, and what do they say about the rankings everyone quoted.

"""How finely can a classification benchmark actually separate two models?"""

import math

N_TEST = 100_000  # Sec. 6.4.1: "full test set of 100,000 images"
Z_999 = 3.290527  # two-sided 99.9%
Z_95 = 1.959964

# year, codename, top-5 error %, reported 99.9% interval — all from Table 5
TABLE_5 = [
    (2014, "GoogLeNet", 6.66, (6.40, 6.92)),
    (2014, "VGG", 7.32, (7.05, 7.60)),
    (2014, "MSRA", 8.06, (7.78, 8.34)),
    (2014, "AHoward", 8.11, (7.83, 8.39)),
    (2014, "DeeperVision", 9.51, (9.21, 9.82)),
    (2013, "Clarifai*", 11.20, (10.87, 11.53)),
    (2014, "CASIAWS*", 11.36, (11.03, 11.69)),
    (2014, "Trimps*", 11.46, (11.13, 11.80)),
    (2014, "Adobe*", 11.58, (11.25, 11.91)),
    (2013, "Clarifai", 11.74, (11.41, 12.08)),
    (2013, "NUS", 12.95, (12.60, 13.30)),
    (2013, "ZF", 13.51, (13.14, 13.87)),
    (2013, "AHoward", 13.55, (13.20, 13.91)),
    (2013, "OverFeat", 14.18, (13.83, 14.54)),
    (2014, "Orange*", 14.80, (14.43, 15.17)),
    (2012, "SuperVision*", 15.32, (14.94, 15.69)),
    (2012, "SuperVision", 16.42, (16.04, 16.80)),
    (2012, "ISI", 26.17, (25.71, 26.65)),
    (2012, "VGG", 26.98, (26.53, 27.43)),
    (2012, "XRCE", 27.06, (26.60, 27.52)),
    (2012, "UvA", 29.58, (29.09, 30.04)),
]


def binomial_halfwidth(err_pct: float, n: int, z: float) -> float:
    p = err_pct / 100.0
    return 100.0 * z * math.sqrt(p * (1 - p) / n)


print("  does a plain binomial reproduce the paper's bootstrap intervals?")
print(f"{'entry':>14}{'error':>8}{'reported +/-':>14}{'binomial +/-':>14}{'diff (pp)':>11}")
for _, name, err, (lo, hi) in TABLE_5[:6] + TABLE_5[-2:]:
    reported = (hi - lo) / 2
    binom = binomial_halfwidth(err, N_TEST, Z_999)
    print(f"{name:>14}{err:>8.2f}{reported:>14.3f}{binom:>14.3f}{reported - binom:>11.3f}")

# --- what that resolution is, at a few test-set sizes ------------------------
print("\n  the smallest gap a benchmark can see, at 7% error")
print(f"{'test images':>14}{'99.9% +/-':>12}{'95% +/-':>10}{'gap needed (99.9%)':>21}")
for n in (1_000, 5_000, 10_000, 50_000, 100_000, 1_000_000):
    h999 = binomial_halfwidth(7.0, n, Z_999)
    h95 = binomial_halfwidth(7.0, n, Z_95)
    print(f"{n:>14,}{h999:>12.3f}{h95:>10.3f}{2 * h999:>21.3f}")

# --- which submissions are actually separable -------------------------------
pairs, separable, ties = 0, 0, []
for i in range(len(TABLE_5)):
    for j in range(i + 1, len(TABLE_5)):
        pairs += 1
        lo_i, hi_i = TABLE_5[i][3]
        lo_j, hi_j = TABLE_5[j][3]
        if hi_i < lo_j or hi_j < lo_i:
            separable += 1
        else:
            ties.append((TABLE_5[i], TABLE_5[j]))

print(f"\n  of the {pairs} pairs in Table 5, {separable} have non-overlapping intervals")

# The longest run of entries that are mutually inseparable: a real tie group.
order = sorted(TABLE_5, key=lambda r: r[2])
best_run, run = [], [order[0]]
for entry in order[1:]:
    if all(not (e[3][1] < entry[3][0] or entry[3][1] < e[3][0]) for e in run):
        run.append(entry)
    else:
        if len(run) > len(best_run):
            best_run = run
        run = [entry]
if len(run) > len(best_run):
    best_run = run
span = best_run[-1][2] - best_run[0][2]
print(f"  and {len(ties)} do not. Largest mutually inseparable group: {len(best_run)} entries")
print(f"    {', '.join(e[1] for e in best_run)}")
print(f"    spanning {best_run[0][2]:.2f}% to {best_run[-1][2]:.2f}% error, a range of {span:.2f} pp")

  does a plain binomial reproduce the paper's bootstrap intervals?
         entry   error  reported +/-  binomial +/-  diff (pp)
     GoogLeNet    6.66         0.260         0.259      0.001
           VGG    7.32         0.275         0.271      0.004
          MSRA    8.06         0.280         0.283     -0.003
       AHoward    8.11         0.280         0.284     -0.004
  DeeperVision    9.51         0.305         0.305     -0.000
     Clarifai*   11.20         0.330         0.328      0.002
          XRCE   27.06         0.460         0.462     -0.002
           UvA   29.58         0.475         0.475      0.000

  the smallest gap a benchmark can see, at 7% error
   test images   99.9% +/-   95% +/-   gap needed (99.9%)
         1,000       2.655     1.581                5.310
         5,000       1.187     0.707                2.375
        10,000       0.840     0.500                1.679
        50,000       0.375     0.224                0.751
       100,000       0.265     0.158                0.531
     1,000,000       0.084     0.050                0.168

  of the 210 pairs in Table 5, 189 have non-overlapping intervals
  and 21 do not. Largest mutually inseparable group: 5 entries
    Clarifai*, CASIAWS*, Trimps*, Adobe*, Clarifai
    spanning 11.20% to 11.74% error, a range of 0.54 pp

The bootstrap was not necessary. Across all 21 entries the published interval and a one-line binomial disagree by at most 0.013 percentage points. The challenge ran "from 20,000 until convergence" bootstrap rounds [1, §6.2] and landed exactly where √(p(1−p)/N) says it would. That is not a criticism — it is a check that the intervals mean what they appear to mean, and it means you can compute the resolution of any accuracy benchmark without running anything.

The resolution table is the part to keep. At 7% error, a 10,000-image test set cannot conservatively see a difference smaller than 1.68 points; at 100,000 it is 0.53; at a million it is 0.17. Most held-out sets in practice are nearer the top of that table than the bottom, which is why so many reported improvements are unfalsifiable rather than false.

And the leaderboard has ties in it. Of 210 pairs, 21 overlap. The largest mutually inseparable group is five entries — Clarifai†, CASIAWS†, Trimps†, Adobe† and Clarifai — spanning 11.20% to 11.74%, across two competition years, all reported in rank order.

This does not contradict the paper, which claims only that "the difference between the winning method and the runner-up each year is significant even at the 99.9% level" [1, §6.2]. That claim holds. The mid-table is where the ordering stops meaning anything, and the paper never said otherwise.

One caution in the other direction, because the test is conservative. Non-overlapping marginal intervals prove a difference; overlapping ones do not prove a tie. Both models are scored on the same images, so a paired test — which looks only at the images where they disagree — has considerably less variance than two independent intervals suggest. The honest reading of the five-way group is "not established by this table", not "identical".

Pitfalls

Assuming one label per image is a property of the world. It is a property of an annotation budget [1, §4.1], and Beyer et al. measured the cost: about 1.23 valid labels per validation image [2, §3.3].

Quoting top-5 as if it were accuracy. It is accuracy given five guesses, on a dataset designed around that allowance.

Comparing top-1 on one paper to top-5 on another. They differ by several points on the same model.

Reading a leaderboard without its intervals. Twenty-one of the 210 pairs here are not separable, including five consecutive entries.

Treating overlapping intervals as proof of a tie. The models share a test set; the paired comparison is stronger than the marginal one.

Assuming benchmark gains still transfer. The slope from ImageNet accuracy to ReaL accuracy fell from 0.86 to 0.51 across the model pool [2, §4], so a point is worth about half what it used to be.

Forgetting the label set is a design decision. Some ImageNet classes draw "distinctions between essentially identical groups of images" [2, §2], and no amount of model capacity fixes a class boundary that was never real.