Residual connection

Intuition

Adding layers to a network should never make it worse. The paper makes that precise: "there exists a solution by construction to the deeper model: the added layers are identity mapping, and the other layers are copied from the learned shallower model", so "a deeper model should produce no higher training error than its shallower counterpart" [1, §1].

Reality disagreed. "With the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error" [1, §1].

That last clause is the diagnosis. Higher training error rules out overfitting and points squarely at optimisation: the good solution existed and the optimiser could not find it.

The paper's reading is that solvers "might have difficulties in approximating identity mappings by multiple nonlinear layers" [1, §3.1]. Nothing about a convolution followed by a nonlinearity makes "output exactly what you were given" a natural setting.

So make it the default. Wire the input around the block and add it back, and the block's job changes from produce the right output to produce the correction. If no correction is needed, the weights go to zero — which is the easiest thing gradient descent ever does.

Formal definition

How it works

Where it's used

Why & when

Worked example

Pitfalls