Validation strategies

We email you a link and a six-digit code. Either one signs you in; there is no password.Enter your email. We send a link and a six-digit code to confirm it — no password to remember. The account is free.

You come back to this page, right where you were.

Already have an account? The same email simply signs you in.

Comparison

Name How it splits	Model fits	Each row is tested	Respects	Breaks when
Holdout One split, one score. Cheap, and the only strategy that gives an untouched estimate of the model you are actually shipping — provided you look at it once.	1	once, or never	nothing by default	the dataset is small
K-fold Rotate the held-out part k ways so every row is predicted by a model that did not see it. The default, and the baseline the rest of this table varies. Full page →	k	exactly once	nothing by default	rows are not exchangeable
Stratified k-fold K-fold that keeps each fold's class balance equal to the whole set's. Costs nothing extra and is the safer default on classification tasks. Full page →	k	exactly once	class balance	rows are grouped or ordered
Grouped k-fold Keeps every row belonging to the same unit — patient, user, session, device — on one side of the split, so the model cannot recognise a subject it has already met. Full page →	k	exactly once	group boundaries	the groups are unknown
Time series split Each training set is a prefix and each test block follows it, so the model is never fitted on data from after the point it is scored at. Full page →	k	at most once	temporal order	you shuffle it
Leave-one-out K-fold taken to its limit, one row held out at a time. Nearly unbiased and maximally correlated between folds, which is the worst case for estimating its own variance. Full page →	n	exactly once	nothing by default	n is large, or you want an interval
Nested cross-validation An inner loop chooses the model and an outer loop scores what the inner loop chose, so the tuning never sees the rows that grade it. Full page →	k × m	exactly once, per loop	the selection boundary	the budget will not stretch

Read the Breaks when column before the others. Every row in this table produces a number whatever data you give it, and a splitter that does not match your data does not complain — it returns a higher score. That asymmetry is what makes this a table worth memorising rather than looking up.

The first three rows are one method with guards added

Holdout, k-fold and stratified k-fold differ only in how much structure they promise to preserve. K-fold is "a conceptually simple improvement over a one-time train-test split" [2, §1.2], and stratification is k-fold with one extra guarantee: each fold "contains approximately the same percentage of samples of each target class as the complete set" [1, §3.1.2.2].

There is no reason to prefer plain k-fold to stratified k-fold on a classification problem. It costs the same, and it rules out a fold that happens to miss a class.

Five-fold cross-validation on ten samples, with the real assignment KFold produces. Each row trains on eight samples and is scored on the two it never saw, and the held-out block walks the length of the data. The bar underneath is the reason the whole scheme exists: after five rounds every sample has been predicted exactly once by a model that was not fitted on it, so the estimate uses all the data without any model ever grading its own work.

The next three rows exist because rows are not independent

This is where the table earns its keep. KFold assumes any row may sit opposite any other, and three common kinds of data violate that.

Grouped data. When several rows come from one subject, a random split puts the same subject on both sides. Grouped folds ensure "that the same group is not represented in both testing and training sets", which matters exactly when "the model is flexible enough to learn from highly person specific features" [1, §3.1.2.4]. The consequence of getting this wrong is not subtle: it is leakage, and the survey's category [L3.2] describes the "extreme (but unfortunately common) case" where "train and test samples come from the same people or units" [3, §2.4].

Ordered data. For time series, "successive training sets are supersets of those that come before them" [1, §3.1.2.6], because the test set "should not contain any data from a date before the training set" [3, §2.4].

The same ten samples split for time series. Every training set is a prefix of the data and every test block sits immediately after it, so the model is never fitted on a point that comes after the one it is being scored on. Two things follow, and both are visible: the early splits train on very little, and samples 0 and 1 are never tested at all. Shuffling would fix both and destroy the guarantee.

Shuffling deserves its own warning, because it is the default in so many places and is sometimes exactly wrong. For rows ordered by time, "shuffling the data will likely lead to a model that is overfit and an inflated validation score: it will be tested on samples that are artificially similar (close in time) to training samples" [1, §3.1.3].

What none of them give you

An estimate of your model. Every row of this table estimates the average error of the procedure across datasets, not the error of the fit you have. The distinction is proved rather than asserted: cross-validation "provides little information" about the error of the model at hand, and "should be viewed as an estimate of Err rather than of Err_XY" [2, §3.1].

A usable confidence interval. The fold errors are correlated because every row is used for both training and testing, so the usual variance estimate is too small and the intervals are too narrow [2, §1]. It is not a matter of care: there is "no unbiased estimator of the standard error of the CV point estimate based on one instance of CV" [2, §1.2]. Nested cross-validation is the paper's remedy [2, §4.3], and the last row of the table is where it sits.

Which to reach for

Start with stratified k-fold at k = 5. It is the right default for almost every classification problem with independent rows.

Switch to grouped the moment anything repeats — and check, rather than assume, because the units that repeat are often not the ones named in the schema.

Switch to time series whenever the prediction is about the future, and accept that the early folds will train on very little.

Add nested when you are tuning and reporting the same number, which is otherwise the maximum of many noisy estimates [1, §3.1.4].

Keep a holdout you look at once, at the end, whatever else you do. It is the only line in this table that estimates the thing a stakeholder thinks the number means.