Data augmentation

Intuition

Your training set has 800 photographs. Every one of them was taken from some particular angle, in some particular light, with the subject at some particular size. The network has no way to know which of those properties are the subject and which are the circumstances, so it will happily use any of them that predicts the label [1, §14.1].

Augmentation removes that option by contradiction. Show the same photograph flipped, cropped, brightened — each time with the same label — and "the subject faces left" stops being a usable rule, because sometimes it does not. What survives repeated contradiction is whatever is actually invariant, which is the thing you wanted the model to learn [3, §7.4].

It is not more data in any information-theoretic sense. Every pixel came from photographs you already had. What it adds is a statement about which transformations should not change the answer — a piece of domain knowledge, injected as examples rather than as architecture [3, §7.4].

Formal definition

D2L: "image augmentation generates similar but distinct training examples after a series of random changes to the training images, thereby expanding the size of the training set" [1, §14.1].

Two mechanisms are offered, and they are different. One is the expansion of the training set. The other is the interesting one: "random tweaks of training examples allow models to rely less on certain attributes, thereby improving their generalization ability" [1, §14.1].

Rely less on certain attributes is the whole argument. The point is not volume, it is the removal of shortcuts.

The standard operations are flipping, cropping and colour: horizontal and vertical flipping with left-right most common, random resized cropping with varying aspect ratios, and adjustments to "brightness, contrast, saturation, and hue" [1, §14.1.1].

How it works

Randomness is the mechanism, not an implementation detail. "Most image augmentation methods have a certain degree of randomness" [1, §14.1.1] — a fixed flip applied to every image would just produce a mirrored dataset, teaching nothing. It is the fact that the model sees a different version each epoch that stops it keying on any one of them.

It applies at training time only. D2L is unambiguous: "in order to obtain definitive results during prediction, we usually only apply image augmentation to training examples, and do not use image augmentation with random operations during prediction" [1, §14.1.2]. Leaving it on at inference makes predictions non-reproducible — run the same image twice, get two answers.

In practice it is a composed pipeline attached to the dataset. Torchvision's canonical classification recipe is exactly the set above — a random resized crop, a horizontal flip with probability 0.5, then normalisation [2, Transforming images] — and the same transforms can be applied to bounding boxes and masks alongside the image [2, Transforming images], which matters because a flipped image with unflipped boxes is silently corrupt training data.

Where it's used

• Every vision training run of consequence, to the point that the augmentation recipe is part of how architectures are compared: two algorithms are only comparable under the same augmentation scheme [3, §7.4].
• Small-data regimes, alongside transfer learning — the two attack the same shortage from different directions.
• Detection and segmentation, where the transform must be applied to the targets as well as the input [2, Transforming images].
• Audio, as time stretching, pitch shifting and noise injection — the same idea, different invariances.
• Text, more cautiously, since most transformations of a sentence are not label-preserving.

Why & when

Reach for it whenever the model is larger than the dataset justifies, which for deep networks is the normal case.

Choose transformations by asking what should not change the label. This is the entire design decision, and it is domain knowledge, not a hyperparameter.

Do not apply a transformation your test data will never contain. Vertical flips help on satellite imagery, where there is no up; on street photographs they teach the model to recognise things that do not occur.

Turn it off for evaluation [1, §14.1.2]. This is the most common bug in the area and it makes results irreproducible rather than merely worse.

Watch for it breaking the label. A horizontal flip changes a 6 into something that is not a 6, and turns a photograph of a left hand into a right one. Character recognition has to tell "b" from "d" and "6" from "9", so flips and 180° rotations are out [3, §7.4].

Worked example

The property that makes augmentation legitimate, checked directly: the array changes and the label does not.

import numpy as np

img = np.arange(36).reshape(6, 6)
row = 2
views = {
    "original":      img,
    "h-flip":        np.fliplr(img),
    "crop + resize": img[1:4, 1:4].repeat(2, 0).repeat(2, 1),
    "brightness":    np.rint(np.clip(img * 1.25, 0, 35)).astype(int),
}
for name, a in views.items():
    print(f"  {name:<15} row {row}: {a[row].tolist()}")

print(f"\n  every view differs from the original: "
      f"{all(not np.array_equal(img, a) for n, a in views.items() if n != 'original')}")
print("  the label attached to all four        : unchanged")

  original        row 2: [12, 13, 14, 15, 16, 17]
  h-flip          row 2: [17, 16, 15, 14, 13, 12]
  crop + resize   row 2: [13, 13, 14, 14, 15, 15]
  brightness      row 2: [15, 16, 18, 19, 20, 21]

  every view differs from the original: True
  the label attached to all four        : unchanged

Four different arrays, one label. Any rule the network might learn that depends on the exact pixel values — "position 0 of row 2 is 12" — is contradicted by three of the four. A rule that depends on the relationships between values survives all of them.

Notice what each transformation destroys. The flip reverses order, so left-right position becomes unusable. The crop changes scale and duplicates values, so absolute size becomes unusable. The brightness shift moves every value by a multiplicative factor, so absolute intensity becomes unusable. Each one is a specific shortcut being closed, chosen because the thing it closes off is genuinely irrelevant to the label.

That is why the choice of transformations is domain knowledge rather than a knob. Adding a vertical flip here would close off "which way up it is" — useful if that is irrelevant to your problem, and destructive if it is not.

Pitfalls

Leaving augmentation on at inference. Predictions stop being reproducible [1, §14.1.2], and a metric computed under it is measuring a randomised model.

Augmenting the image and not the label. In detection and segmentation the targets must move with the pixels; torchvision passes both through the same transform for exactly this reason [2, Transforming images].

Transformations that change the class. Flips on digits and text, aggressive hue shifts where colour is the label, crops that remove the object entirely.

Augmenting before the train/validation split. Augmented copies of the same source image landing on both sides is leakage, and the validation score becomes meaningless.

Treating it as a substitute for data. It reduces reliance on incidental attributes [1, §14.1]; it does not add subjects, poses or conditions your collection never contained.