Track
Deep learning, end to end
Networks, optimisation, regularisation and what to do when training fails.
Steps in order
- 01PerceptronThe simplest neural unit — it multiplies each input by a weight, adds them up with a bias, and fires 1 if the total clears a threshold and 0 if it doesn't.Free
- 02DropoutZero a random fraction of activations during training and divide the rest by what survived. The model that ships is a cheap stand-in for an ensemble it never builds.Free
- 03SigmoidAn S-shaped activation that squashes any real number into (0, 1) — the historical default for neural networks, now largely displaced from hidden layers because it strangles the gradient.
- 04ReLUPass positive values through unchanged and clamp everything negative to zero — an activation so simple it is barely a function, and the default choice in modern networks because of it.
- 05SoftmaxTurns a vector of scores into a probability distribution by exponentiating and normalising — the standard last step of a multi-class classifier, and the standard place for a numerical overflow.
- 06TanhA rescaled sigmoid that outputs between −1 and 1 instead of 0 and 1 — zero-centred, which makes it better behaved than sigmoid, though it saturates just the same.
- 07BackpropagationThe chain rule applied backwards through a computational graph. One traversal produces the derivative with respect to every parameter, at roughly the cost of one forward pass.
- 08ConvolutionSliding a small learned kernel across an input and taking a weighted sum at each position — the same weights everywhere, which is how a layer can look at an image without a parameter per pixel.
- 09Depthwise separable convolutionSplit a convolution into filtering each channel and then mixing channels. Roughly nine times cheaper for a 3×3 kernel, and what is left is mostly matrix multiplication.
- 10Leaky ReLUReLU with a small non-zero slope for negative inputs instead of a flat zero — so a unit that goes negative still receives a gradient and can come back.
- 11PoolingReplacing each window of a feature map with a single summary — usually its maximum — to shrink the grid and make the representation less sensitive to exactly where a feature sat.
- 12Receptive fieldThe region of the input a single unit can possibly depend on — which grows only linearly with depth, and whose influence is concentrated in a Gaussian centre far smaller than the region itself.
- 13Recurrent neural networkA network that carries a hidden state from one step of a sequence to the next, so one set of weights handles any length. The same recurrence destroys the gradient.
- 14Transposed convolutionA convolution run backwards — it scatters the kernel instead of gathering it, turning a small feature map into a larger one. The name is literal, not a metaphor.
- 15Batch normalizationRe-centring and re-scaling each layer's inputs using the statistics of the current mini-batch — a layer that made much deeper networks trainable and much less sensitive to how they were initialised.
- 16Residual connectionAdding a block's input to its output so the block only has to learn the difference — which made networks of a hundred layers and more trainable, and gave gradients a path that skips the weights.