Track
Machine-learning models explained
Linear models, trees, ensembles, clustering — how each one fails.
Steps in order
- 01Train-test splitWithhold rows from training so the score you report was measured on data the model never saw. The split has to come first, be large enough, and respect how the rows arrive.Free
- 02Linear regressionA model that predicts a numeric outcome as a weighted sum of the input features, fitted by choosing the weights that make the squared prediction errors as small as possible.Free
- 03OverfittingFitting the training sample more closely than the process that generated it — so the model reproduces noise that will not recur, and does worse on new data the better it looks on old.
- 04Cross-validationSplitting the data k ways and letting each part be the test set once, so every sample is predicted by a model that never saw it — and the estimate uses all the data instead of a single lucky split.
- 05Decision treeA model that predicts by asking a sequence of yes/no questions about the features, splitting the data at each answer until it reaches a leaf holding a prediction.
- 06Feature engineeringDeciding what the columns are — rescaling, encoding, combining and deriving inputs so that the structure a model can represent is the structure that is actually there.
- 07K-meansPick k centres, assign every point to the nearest, move each centre to the mean of what it caught, repeat. It always converges and it can converge somewhere arbitrarily bad.
- 08KNNClassify a new point by looking at the k closest training examples and taking their majority vote — a model that stores the data instead of learning from it.
- 09Distance-weighted KNNKNN where each neighbour's vote counts in inverse proportion to how far away it is — so a point sitting right next to the query outweighs one at the edge of the neighbourhood.
- 10Polynomial regressionFitting a curve by adding powers and products of the original features, then running ordinary linear regression on the expanded set — still a linear model, now with a curved shape.
- 11RegularizationAdding a penalty on coefficient size to a model's objective, so it trades a little fit on the training data for far better behaviour on data it has not seen.
- 12BaggingFitting the same unstable model to many bootstrap resamples of one dataset and averaging the results — variance falls, bias does not, and nothing about the model itself has to change.
- 13Random forestBagged decision trees with a second source of randomness — each split may only consider a random subset of the features, which decorrelates the trees and lets averaging do far more work.
- 14Bias-variance tradeoffThe classical account of why a model can be too simple or too complex, and the modern result that breaks it — past the point of fitting the training data exactly, test error can start falling again.
- 15Class imbalanceOne class is rare, and every default in the toolchain assumes it is not. The accuracy looks excellent, the model predicts the rare class never, and most of the standard fixes only move the threshold.
- 16Data leakageAny way information reaches the model that would not exist at prediction time. It does not make the model worse — it makes the score better, which is why it survives review and fails in production.
- 17Feature importanceRanking inputs by how much a fitted model relies on them — where the default measure for tree ensembles is biased toward high-cardinality features and the honest one is measured on held-out data.
- 18Hyperparameter tuningSearching the settings a model cannot learn for itself. Which search you use matters less than noticing that most settings do not matter, and that a grid spends its budget as if they all did.
- 19Lasso regressionLinear regression with a penalty on the sum of absolute coefficients — it drives weak coefficients to exactly zero, so fitting the model also selects the features.
- 20Logistic regressionA linear model for classification — it feeds a weighted sum through a sigmoid to produce a calibrated probability rather than a raw score, despite the "regression" in its name.
- 21Missing dataWhether dropping or filling missing values biases your answer depends on why the values are missing — and on which answer you wanted. One of those two questions is usually unanswerable.
- 22PCAA rotation of the axes onto the directions of greatest spread. It makes the data uncorrelated and rank-orders the new axes, so you can drop the last ones and know exactly what that costs.
- 23Manifold learningNon-linear projections that keep neighbours together at the cost of everything else. They make wonderful pictures, and almost nothing you can measure off those pictures is real.
- 24Ridge regressionLinear regression with a penalty on the sum of squared coefficients — it shrinks every coefficient toward zero without ever removing one, which stabilises models with correlated features.
- 25Bayesian linear regressionLinear regression that returns a distribution over coefficients rather than single values — so every prediction carries an honest uncertainty, and the regularization strength is learned, not tuned.
- 26BoostingFitting weak learners in sequence, each one on a version of the data reweighted to emphasise what the previous ones got wrong, and combining them into a weighted vote.
- 27Elastic NetLinear regression penalised by a weighted blend of the L1 and L2 terms — it selects features like Lasso while keeping correlated groups together like Ridge.
- 28Gaussian discriminant analysisA classifier that models what each class's data looks like — fitting a Gaussian per class — and then uses Bayes' rule to turn that into a prediction.
- 29Gradient boostingBoosting generalised to any differentiable loss — each new tree is fitted to the negative gradient of the loss at the current predictions, and added with a shrinkage factor.
- 30SVMA classifier that places its boundary to leave the widest possible gap between the classes, with only the points on the edge of that gap deciding where it goes.