Dropout

Intuition

A unit that can be removed at any moment cannot be relied on. Dropout removes units at random during training, so no unit can assume any particular other unit will be there, and the layer has to spread its work rather than route it through a few indispensable paths.

The name describes half of it. The other half — dividing the survivors by what fraction survived — is what makes the trick usable, and it is the half that gets left out of explanations.

Formal definition

"The method is called dropout because we literally drop out some neurons during training. Throughout training, on each iteration, standard dropout consists of zeroing out some fraction of the nodes in each layer before calculating the subsequent layer" [1, §5.6].

The zeroing alone would shrink every layer it touches. The fix is to inject the noise "in an unbiased manner so that the expected value of each layer — while fixing the others — equals the value it would have taken absent noise" [1, §5.6], which is the division in the formula above.

The consequence is the part worth memorising: "dropout is used only during training: it replaces an activation h with a random variable with expected value h" [1, §5.6.4]. At test time, "we do not drop out any nodes and thus do not need to normalize" [1, §5.6.1].

How it works

Training with dropout does not train one network. Each mask defines a different subnetwork, and dropout is best read as "an inexpensive approximation to training and evaluating a bagged ensemble of exponentially many neural networks" [2, §7.12].

That framing raises the obvious question, which the worked example answers by measurement: the ordinary deterministic forward pass is supposed to stand in for averaging that ensemble, and it does not do so exactly. It is exact wherever dropout is followed by a linear map, and inexact as soon as a nonlinearity sits between two dropped layers.

Where it's used

Fully connected layers, overwhelmingly, and most of all the wide ones near the end of a network where the parameters are concentrated. It is applied far less to convolutional layers, whose weight sharing already constrains them.

It has a second life at test time, deliberately left on: "some researchers use dropout at test time as a heuristic for estimating the uncertainty of neural network predictions: if the predictions agree across many different dropout outputs, then we might say that the network is more confident" [1, §5.6.1]. The figure below is that spread, drawn.

Why & when

Use it where a layer is wide relative to the data it is fit on. Skip it where something else already does the job — the measurement below could not distinguish any rate from 0.1 to 0.7 from no dropout at all, in a model that was already stopping early on the same validation split.

That is the honest summary: dropout is one member of a family [2, §7.12], and the family members substitute for each other more than they compound.

Worked example

All in scripts/dropout.py.

What the division is for

def dropout(x, p, invert, gen):
    """Standard dropout. invert=False is the version without the rescaling."""
    if p == 0.0:
        return x
    keep = (gen.uniform(size=x.shape) >= p).astype(float)
    return x * keep / (1 - p) if invert else x * keep

  a layer of activations, mean 3.00 by construction
       p    no dropout   masked only    masked / (1-p)   std / mean
    0.00        3.0012        3.0012            3.0012       0.7077
    0.10        3.0012        2.7017            3.0024       0.8169
    0.25        3.0012        2.2504            3.0024       0.9998
    0.50        3.0012        1.4989            3.0024       1.4149
    0.75        3.0012        0.7498            3.0030       2.2394

    The middle column is the layer the next layer would receive if the
    rescaling were skipped: smaller by exactly (1 - p), so every
    downstream weight is now calibrated to a signal that vanishes at
    test time. The right column is unbiased at every rate -- and the
    last column is the price, which is variance, rising with p.

The middle column is the bug you get for free by implementing the name. Every downstream weight would be fit against a signal that is (1 − p) too small, and then at test time the signal arrives at full strength.

The model that ships is not the model that trained

  deterministic output vs the mean over 20,000 sampled masks
            dropout is followed by     p  deterministic   mask mean      gap  in s.e.
                  a linear readout   0.2         1.5093      1.5099   0.0006      0.2
                  a linear readout   0.5         1.5093      1.5131   0.0038      0.6
          three more linear layers   0.2        -0.5224     -0.5252   0.0027      0.3
          three more linear layers   0.5        -0.5224     -0.5216   0.0008      0.0
      a linear readout, after tanh   0.2        -0.0634     -0.0640   0.0006      0.3
      a linear readout, after tanh   0.5        -0.0634     -0.0659   0.0026      0.6
            three more tanh layers   0.2         0.2082      0.1590   0.0492     17.3
            three more tanh layers   0.5         0.2082      0.0860   0.1222     20.7

    The last column is what makes the table readable: a gap of a couple
    of standard errors is the sampling. The last two rows are not sampling.
    Wherever dropout is followed by a linear map the rule is exact, and
    the arithmetic says why -- the mask has mean 1 and is independent of
    what it multiplies, so the average passes straight through. Put a
    nonlinearity BETWEEN two dropped layers and it stops passing through.
    The model you ship is then a cheap approximation of an ensemble you
    never build: usually a good one, never the thing you trained.

What rate, and does it matter

The usual advice is a number. The measurement says the number matters much less than the thing it is usually measured against.

  test loss by dropout rate, 24 seeds, the same data per seed
      p    fixed 120 epochs    stopped on validation    vs p=0   in s.e.
    0.0              0.8640                   0.4855                    
    0.1              0.9378                   0.4955   +0.0100       1.7
    0.2              0.9888                   0.4926   +0.0071       1.5
    0.3              0.9555                   0.4872   +0.0017       0.4
    0.5              0.7845                   0.4893   +0.0038       0.6
    0.7              0.6278                   0.4847   -0.0008      -0.1
    0.9              0.9932                   0.6125   +0.1270       9.9

    The two columns disagree about almost everything, and the left one
    is wrong. Dropout slows convergence, so a fixed epoch budget charges
    that to the rate: it makes 0.1 look harmful and 0.7 look like a
    discovery, and neither survives letting each rate stop where its own
    validation loss is lowest.

    What survives is narrow. Nothing from 0.1 to 0.7 is distinguishable
    from no dropout at all here, and 0.9 is clearly worse. This model is
    already regularized -- it stops early, on the same validation split.
    A second regularizer doing the same job has nothing left to do.

Pitfalls

Implementing the name and not the formula. Zeroing without dividing shifts every layer's scale by (1 − p) between training and inference. Frameworks do the division at training time for you; hand-rolled dropout is where this goes wrong.

Leaving it on at inference. The test-time pass is deterministic [1, §5.6.1], so predictions that change between identical calls mean it is still active. That is a real technique, but only when chosen.

Comparing rates at a fixed epoch budget. Dropout slows convergence, so a fixed budget charges that to the rate. Above, it makes 0.1 look harmful and 0.7 look like a discovery, and neither survives letting each rate stop where its own validation loss is lowest — see early stopping.

Stacking it on a model that is already regularized. Dropout overlaps with batch normalization, which injects noise of its own and whose paper reports it can, in some cases, remove the need for dropout. Adding both is not adding twice, and the table above is a small version of the same result.

Expecting the deterministic pass to equal the ensemble. It does not, once anything nonlinear sits between two dropped layers — 20.7 standard errors of disagreement above. The approximation is good, and it is still an approximation.