All concepts

Filter by topic and difficulty. The filters live in the URL, so a filtered view is a link you can send someone.

A/B testingStatistics · 3/5Randomly assigning subjects to two variants and comparing an outcome, so that the difference you measure can be attributed to the change rather than to who happened to see it.
AccuracyMetrics & losses · 1/5FreeThe fraction of predictions a classifier got right — correct predictions divided by total predictions.
AdamOptimization · 3/5An optimiser that keeps a running average of both the gradient and its square, giving every parameter its own effective step size — the default choice for training neural networks.
AttentionNLP · 4/5Letting every position compute a weighted average of every other, with the weights produced by comparing a query against keys — a learned, content-based routing of information.
AutocorrelationStatistics · 3/5The correlation of a series with a shifted copy of itself. It is how you find a cycle you were not told about, and how you discover that your sample size is smaller than you thought.
BackpropagationDeep learning · 3/5The chain rule applied backwards through a computational graph. One traversal produces the derivative with respect to every parameter, at roughly the cost of one forward pass.
Bag of wordsNLP · 2/5FreeText as a vector of word counts with the order thrown away — crude enough that two sentences meaning opposite things collide, useful enough that it was the default representation for thirty years.
BaggingMachine learning · 3/5Fitting the same unstable model to many bootstrap resamples of one dataset and averaging the results — variance falls, bias does not, and nothing about the model itself has to change.
Batch normalizationDeep learning · 4/5Re-centring and re-scaling each layer's inputs using the statistics of the current mini-batch — a layer that made much deeper networks trainable and much less sensitive to how they were initialised.
Bayes' theoremStatistics · 2/5FreeA rule for turning the probability of the evidence given a hypothesis into the probability of the hypothesis given the evidence. The two are not the same, and the gap between them is the prior.
Bayesian linear regressionMachine learning · 4/5Linear regression that returns a distribution over coefficients rather than single values — so every prediction carries an honest uncertainty, and the regularization strength is learned, not tuned.
BERTNLP · 4/5A transformer encoder pretrained by hiding 15% of the tokens and predicting them from both sides at once, then fine-tuned on a task with one extra output layer and no architectural surgery.
BFGSOptimization · 4/5An optimiser that builds an approximation of the curvature from the gradients it has already seen, using it to take far better-aimed steps than gradient descent.
Bias-variance tradeoffMachine learning · 3/5The classical account of why a model can be too simple or too complex, and the modern result that breaks it — past the point of fitting the training data exactly, test error can start falling again.
BLEUMetrics & losses · 3/5A precision score over n-gram overlap with a reference translation, times a penalty for being too short. A parameterised family, not a single number.
BoostingMachine learning · 4/5Fitting weak learners in sequence, each one on a version of the data reweighted to emphasise what the previous ones got wrong, and combining them into a weighted vote.
BootstrapStatistics · 3/5Resample your own data with replacement, recompute the statistic, and read the spread. It gives you an interval for statistics that have no formula — and quietly fails for some that do.
Central limit theoremStatistics · 3/5Add up enough independent values from almost any distribution and the sum is approximately normal. Almost is doing real work — the theorem has conditions, and heavy-tailed data breaks them.
Class imbalanceMachine learning · 3/5One class is rare, and every default in the toolchain assumes it is not. The accuracy looks excellent, the model predicts the rare class never, and most of the standard fixes only move the threshold.
CLIPComputer vision · 4/5Train an image encoder and a text encoder together so a picture and its caption land in the same place. The label set becomes something you write, not something you train for.
Confidence intervalStatistics · 3/5A range built so that, across repeated experiments, a stated fraction of such ranges contain the true value. The fraction is a property of the method, not of the interval in front of you.
ConfoundingStatistics · 3/5A third variable that causes both the thing you changed and the thing you measured. Adjusting for the right one rescues an estimate; adjusting for the wrong one invents an effect that is not there.
Confusion matrixMetrics & losses · 1/5A table counting how a classifier's predictions line up against the truth — correct and incorrect, broken down by which class was predicted and which was actual.
Conjugate gradientOptimization · 4/5An optimiser that chooses each search direction so it does not undo the progress of the previous ones — the fix for gradient descent zig-zagging down a narrow valley.
ConvolutionDeep learning · 3/5Sliding a small learned kernel across an input and taking a weighted sum at each position — the same weights everywhere, which is how a layer can look at an image without a parameter per pixel.
Convolutional neural networkComputer vision · 3/5A stack that alternates convolution and downsampling so that features grow from edges to objects while the grid shrinks — the architecture that made image models tractable.
CorrelationStatistics · 2/5FreeOne number for how tightly two variables sit on a straight line. It says nothing about the slope of that line, and near zero it does not mean the two are unrelated.
Cross-entropyMetrics & losses · 3/5The loss every classifier on this site is trained with. It charges for being wrong and for being confident about it, and its floor is set by the base rate rather than by zero.
Cross-validationMachine learning · 2/5Splitting the data k ways and letting each part be the test set once, so every sample is predicted by a model that never saw it — and the estimate uses all the data instead of a single lucky split.
Data augmentationComputer vision · 2/5FreeApplying random label-preserving transformations to training examples, so the model sees each one many ways and cannot rely on the accidents of how it happened to be captured.
Data leakageMachine learning · 3/5Any way information reaches the model that would not exist at prediction time. It does not make the model worse — it makes the score better, which is why it survives review and fails in production.
Decision treeMachine learning · 2/5A model that predicts by asking a sequence of yes/no questions about the features, splitting the data at each answer until it reaches a leaf holding a prediction.
DecodingNLP · 3/5The rule that turns a language model's next-token distribution into actual text. Choosing the most probable continuation at every step reliably produces the worst output.
Depthwise separable convolutionDeep learning · 3/5Split a convolution into filtering each channel and then mixing channels. Roughly nine times cheaper for a 3×3 kernel, and what is left is mostly matrix multiplication.
Diffusion modelComputer vision · 4/5Destroy an image with noise in a thousand fixed steps, then train a network to undo one step at a time. Sampling runs the chain backwards from pure noise.
Distance-weighted KNNMachine learning · 2/5KNN where each neighbour's vote counts in inverse proportion to how far away it is — so a point sitting right next to the query outweighs one at the edge of the neighbourhood.
DropoutDeep learning · 2/5FreeZero a random fraction of activations during training and divide the rest by what survived. The model that ships is a cheap stand-in for an ensemble it never builds.
Early stoppingOptimization · 2/5FreeStop training when held-out loss stops improving, and hand back the best weights rather than the last ones. It is regularization by limiting epochs instead of weights.
Eigenvectors and eigenvaluesMathematics · 3/5The directions a matrix does not turn, and how much it stretches each of them. Almost every method that reduces, ranks or propagates through a linear map is a question about these.
Elastic NetMachine learning · 4/5Linear regression penalised by a weighted blend of the L1 and L2 terms — it selects features like Lasso while keeping correlated groups together like Ridge.
EntropyMathematics · 3/5The average surprise in a distribution, measured in bits. It is the floor on how short a code can be, and every loss on this site that says “log” is paying it.
Expectation and varianceMathematics · 2/5FreeThe two numbers that summarise a random quantity — where it sits and how far it moves. Each carries a condition people forget, and one of them does not always exist.
F1-scoreMetrics & losses · 2/5The harmonic mean of precision and recall — one number that is only high when a model is both trustworthy when it fires and thorough in what it finds.
Feature engineeringMachine learning · 2/5Deciding what the columns are — rescaling, encoding, combining and deriving inputs so that the structure a model can represent is the structure that is actually there.
Feature importanceMachine learning · 3/5Ranking inputs by how much a fitted model relies on them — where the default measure for tree ensembles is biased toward high-cardinality features and the honest one is measured on held-out data.
Fréchet Inception DistanceMetrics & losses · 4/5Fit a Gaussian to Inception features for real and generated images, then measure the distance between them. The number you compute is not the number you want.
Gaussian discriminant analysisMachine learning · 4/5A classifier that models what each class's data looks like — fitting a Gaussian per class — and then uses Bayes' rule to turn that into a prediction.
Generalized linear modelsStatistics · 4/5One framework that covers linear, logistic and Poisson regression at once — a linear predictor, a link function connecting it to the response, and a distribution for the noise.
Generative adversarial networkComputer vision · 4/5Two networks in a game — one invents samples, the other tries to spot them. The generator never sees the data, only gradients that come back through its opponent.
GradientMathematics · 2/5FreeThe vector of a function's partial derivatives — it points in the direction of steepest increase, and its length says how steep that increase is.
Gradient boostingMachine learning · 4/5Boosting generalised to any differentiable loss — each new tree is fitted to the negative gradient of the loss at the current predictions, and added with a shrinkage factor.
Gradient descentOptimization · 2/5FreeRepeatedly step in the direction the loss decreases fastest, by an amount set by the learning rate — the algorithm underneath almost all model training.
HallucinationNLP · 3/5Generated text that the source does not support — either contradicting it or going beyond it. The second kind is not automatically wrong, and no overlap metric can detect the first.
Huber lossMetrics & losses · 3/5A loss that behaves like MSE for small errors and like MAE for large ones — smooth enough to optimise easily, robust enough that outliers stop dragging the fit.
Hyperparameter tuningMachine learning · 3/5Searching the settings a model cannot learn for itself. Which search you use matters less than noticing that most settings do not matter, and that a grid spends its budget as if they all did.
Hypothesis testingStatistics · 3/5Assume there is no effect, work out how often chance alone would produce a result as striking as yours, and call it real if that is rare enough. The method is a proof by contradiction.
Image classificationComputer vision · 2/5FreeOne image in, one label out. The simplest shape in vision, and the benchmark whose single-label assumption shaped everything built on top of it.
Instance segmentationComputer vision · 4/5A mask per object, not a class per pixel. It is detection and segmentation at once, and the thing that made it work was removing two roundings.
Instruction tuningNLP · 4/5Fine-tuning a pretrained model on tasks phrased as instructions, and then on human preferences between its own outputs. What turns a text completer into something that answers the question you asked.
Intersection over unionMetrics & losses · 2/5The area two regions share divided by the area they cover between them. It is read as "how correct is this box", and it is stricter and scale-blind in ways that number hides.
Jupyter NotebookTooling · 1/5FreeA document that interleaves executable code, prose and its own outputs, run cell by cell against a live kernel — the standard tool for exploratory analysis, and a known hazard for reproducibility.
K-meansMachine learning · 2/5Pick k centres, assign every point to the nearest, move each centre to the mean of what it caught, repeat. It always converges and it can converge somewhere arbitrarily bad.
KNNMachine learning · 2/5Classify a new point by looking at the k closest training examples and taking their majority vote — a model that stores the data instead of learning from it.
Language modelNLP · 3/5A probability distribution over sequences of tokens, factorised so that each token is predicted from the ones before it. Counting works up to about two words of context and then runs out of data.
Lasso regressionMachine learning · 3/5Linear regression with a penalty on the sum of absolute coefficients — it drives weak coefficients to exactly zero, so fitting the model also selects the features.
Leaky ReLUDeep learning · 3/5ReLU with a small non-zero slope for negative inputs instead of a flat zero — so a unit that goes negative still receives a gradient and can come back.
Learning rateOptimization · 2/5How far each gradient step moves. On a quadratic the curvature sets a hard ceiling and the best value is exactly half of it; perhaps the most important hyperparameter to tune.
Linear regressionMachine learning · 2/5FreeA model that predicts a numeric outcome as a weighted sum of the input features, fitted by choosing the weights that make the squared prediction errors as small as possible.
Logistic regressionMachine learning · 3/5A linear model for classification — it feeds a weighted sum through a sigmoid to produce a calibrated probability rather than a raw score, despite the "regression" in its name.
Machine learning workflowTooling · 2/5FreeThe end-to-end loop a model lives in — framing, data, features, training, evaluation, serving and the feedback from production that changes all of them.
MAEMetrics & losses · 1/5The average size of the errors, ignoring their direction — the plain, honest answer to "how far off is this model typically?"
Manifold learningMachine learning · 3/5Non-linear projections that keep neighbours together at the cost of everything else. They make wonderful pictures, and almost nothing you can measure off those pictures is real.
MAPEMetrics & losses · 2/5The average error expressed as a percentage of the true value — scale-free and intuitive, and unusable when the target can approach zero.
Maximum likelihoodStatistics · 3/5Pick the parameters that make the observed data as probable as possible. Nearly every loss function in machine learning is this rule with a minus sign and a logarithm in front of it.
Mean average precisionMetrics & losses · 3/5The area under a precision-recall curve, averaged over classes — and in detection, also over IoU thresholds. The threshold you report it at decides whether it can see localisation quality at all.
Mini-batch gradient descentOptimization · 3/5Estimating the gradient from a small group of examples rather than one or all of them — the compromise that everything is actually trained with.
Missing dataMachine learning · 3/5Whether dropping or filling missing values biases your answer depends on why the values are missing — and on which answer you wanted. One of those two questions is usually unanswerable.
Model calibrationMetrics & losses · 3/5Making a model's stated confidence mean what it says — of the predictions it makes at 90% confidence, 90% should be right. Accuracy does not measure this, and modern networks fail it badly.
MSEMetrics & losses · 1/5FreeThe average of the squared differences between predicted and actual values — the standard way to measure how wrong a regression model's predictions are.
MSLEMetrics & losses · 3/5MSE computed on the logarithms of the values — it measures the ratio between prediction and truth rather than the difference, so being wrong by a factor counts the same at any scale.
Multiple comparisonsStatistics · 3/5Run twenty tests at the 5% level and you will usually find something. The corrections differ in what they promise, and the honest choice is which kind of mistake you would rather make.
Named entity recognitionNLP · 3/5A label for every token, read back as spans — people, places, organisations. The output is structured, so one wrong token invalidates a whole entity and accuracy stops being the right metric.
Natural language processingNLP · 2/5FreeThe study of how computers handle human language — turning text into representations a model can work with, then using those for classification, inference, translation and question answering.
Object detectionComputer vision · 3/5Finding every object in an image and drawing a box around each. The output is a variable-length set, which is what makes it harder than classification and why the machinery looks the way it does.
OutliersStatistics · 2/5Values far from the rest, which may be mistakes or may be the finding. Detecting them with a mean and a standard deviation fails in a specific way — the more of them there are, the fewer you find.
OverfittingMachine learning · 2/5Fitting the training sample more closely than the process that generated it — so the model reproduces noise that will not recur, and does worse on new data the better it looks on old.
PCAMachine learning · 3/5A rotation of the axes onto the directions of greatest spread. It makes the data uncorrelated and rank-orders the new axes, so you can drop the last ones and know exactly what that costs.
PerceptronDeep learning · 2/5FreeThe simplest neural unit — it multiplies each input by a weight, adds them up with a bias, and fires 1 if the total clears a threshold and 0 if it doesn't.
PerplexityMetrics & losses · 3/5The exponential of a model's average negative log-likelihood on held-out text — read as the effective number of choices it faces per token. It is per token, so it is not comparable across tokenizers.
PointNetComputer vision · 4/5A neural network that consumes a raw 3D point cloud directly, respecting the fact that the points are a set — reorder the file and the answer must not change.
Polynomial regressionMachine learning · 2/5Fitting a curve by adding powers and products of the original features, then running ordinary linear regression on the expanded set — still a linear model, now with a curved shape.
PoolingDeep learning · 3/5Replacing each window of a feature map with a single summary — usually its maximum — to shrink the grid and make the representation less sensitive to exactly where a feature sat.
PrecisionMetrics & losses · 1/5Of everything the model flagged as positive, the fraction that really was positive — the measure of how much you can trust a positive prediction.
PromptingNLP · 3/5Specifying a task in the model's own input, with an instruction and perhaps a few examples and no weight updates. Good at saying what the task is; poor at supplying an ability.
R²Metrics & losses · 2/5How much better a model predicts than simply always guessing the mean — the one regression score that is comparable across different datasets.
Random forestMachine learning · 3/5Bagged decision trees with a second source of randomness — each split may only consider a random subset of the features, which decorrelates the trees and lets averaging do far more work.
RecallMetrics & losses · 1/5Of everything that really was positive, the fraction the model actually found — the measure of how much it misses.
Receptive fieldDeep learning · 3/5The region of the input a single unit can possibly depend on — which grows only linearly with depth, and whose influence is concentrated in a Gaussian centre far smaller than the region itself.
Recurrent neural networkDeep learning · 3/5A network that carries a hidden state from one step of a sequence to the next, so one set of weights handles any length. The same recurrence destroys the gradient.
RegularizationMachine learning · 3/5Adding a penalty on coefficient size to a model's objective, so it trades a little fit on the training data for far better behaviour on data it has not seen.
ReLUDeep learning · 2/5Pass positive values through unchanged and clamp everything negative to zero — an activation so simple it is barely a function, and the default choice in modern networks because of it.
Residual connectionDeep learning · 4/5Adding a block's input to its output so the block only has to learn the difference — which made networks of a hundred layers and more trainable, and gave gradients a path that skips the weights.
Retrieval-augmented generationNLP · 3/5Retrieve passages from a document store, put them in the model's input, and generate from those. Moves facts out of the weights and into an index you can inspect, correct and replace.
Ridge regressionMachine learning · 3/5Linear regression with a penalty on the sum of squared coefficients — it shrinks every coefficient toward zero without ever removing one, which stabilises models with correlated features.
RMSEMetrics & losses · 1/5The square root of the MSE — the same error measure, brought back into the units of the thing being predicted so a human can read it.
ROC AUCMetrics & losses · 3/5The area under the curve traced by a classifier's true and false positive rates as the decision threshold sweeps — one number summarising performance at every threshold at once.
Self-supervised learningComputer vision · 4/5Build the labels out of the data itself — two augmented views of one image are a positive pair — and the representation transfers as well as a supervised one.
Semantic segmentationComputer vision · 3/5A class label for every pixel. The output is the same size as the input, which forces an encoder-decoder shape and makes pixel accuracy a useless way to score it.
Sentence embeddingsNLP · 3/5One fixed-length vector per sentence, arranged so that cosine similarity means semantic similarity. It makes search, clustering and deduplication linear in the collection instead of quadratic.
Sequence-to-sequenceNLP · 3/5An encoder reads a variable-length input and a decoder writes a variable-length output. Squeezing that into one fixed vector is the bottleneck attention was invented to remove.
SGDOptimization · 3/5Gradient descent that estimates the gradient from one example instead of the whole dataset — a much noisier direction, computed thousands of times more cheaply.
SigmoidDeep learning · 2/5An S-shaped activation that squashes any real number into (0, 1) — the historical default for neural networks, now largely displaced from hidden layers because it strangles the gradient.
SoftmaxDeep learning · 2/5Turns a vector of scores into a probability distribution by exponentiating and normalising — the standard last step of a multi-class classifier, and the standard place for a numerical overflow.
SpecificityMetrics & losses · 2/5Of everything that really was negative, the fraction the model correctly left alone — recall measured on the negative class.
StationarityStatistics · 3/5The assumption that a series' mean, variance and correlation structure do not change over time. Nearly every time-series method needs it, and ordinary regression fails loudly without it.
SVMMachine learning · 4/5A classifier that places its boundary to leave the widest possible gap between the classes, with only the points on the edge of that gap deciding where it goes.
TanhDeep learning · 2/5A rescaled sigmoid that outputs between −1 and 1 instead of 0 and 1 — zero-centred, which makes it better behaved than sigmoid, though it saturates just the same.
TF-IDFNLP · 2/5A weight that multiplies how often a term appears in one document by how rare it is elsewhere, so a word appearing everywhere contributes nothing and a word appearing in few documents dominates.
Time series decompositionStatistics · 2/5Splitting a series into trend, a repeating seasonal pattern, and what is left. Three readable things in place of one unreadable one, and the basis of most forecasting methods.
TokenizationNLP · 2/5Cutting text into the units a model actually consumes — where splitting on words leaves every unseen word as a hole, and subword methods earn their vocabulary from frequency instead.
Train-test splitMachine learning · 1/5FreeWithhold rows from training so the score you report was measured on data the model never saw. The split has to come first, be large enough, and respect how the rows arrive.
Transfer learningComputer vision · 3/5Taking a network trained on a large dataset, replacing only its output layer and training on your much smaller one — the features are inherited, the classifier is all you learn from scratch.
TransformerNLP · 4/5A stack of identical layers, each self-attention followed by a position-wise feed-forward network, both wrapped in a residual connection and layer normalisation, with word order supplied separately.
Transposed convolutionDeep learning · 3/5A convolution run backwards — it scatters the kernel instead of gathering it, turning a small feature map into a larger one. The name is literal, not a metaphor.
Vision transformerComputer vision · 4/5A transformer applied to an image cut into fixed-size patches, with no convolution anywhere. It has far less built-in knowledge about images than a CNN, and it has to buy the difference with data.
Word embeddingsNLP · 3/5Dense vectors for words, learned from the company they keep — so that words used in similar contexts end up near each other, without anyone defining what similar means.