Train-test split

Intuition

A model's error on the data it was fitted to is not evidence. It can be driven to zero by memorising, and memorising is the one thing you are trying to detect [1, §3.6.1.1].

So rows are withheld. The model never sees them, and its error on them stands in for its error on data that does not exist yet [1, §3.6.1]. Everything difficult about splitting follows from taking that sentence literally — never sees them is a stronger requirement than it looks [1, §3.6.3], and the number that comes back is noisier than it looks.

Formal definition

The quantity of interest cannot be computed. In practice "we must estimate the generalization error by applying our model to an independent test set constituted of a random selection of examples … that were withheld from our training set" [1, §3.6.1].

That estimate rests on an assumption worth stating out loud: "we assume that the training data and the test data are drawn independently from identical distributions … commonly called the IID assumption" [1, §3.6.1]. Part four of the worked example is what happens when it quietly fails.

A third split appears as soon as you compare models. "Error on the holdout data, i.e., validation set, is called the validation error" [1, §3.6.1], and the division of labour is that the validation set chooses and the test set reports.

How it works

The rule is simple and constantly broken: "In principle, we should not touch our test set until after we have chosen all our hyperparameters" [1, §3.6.3]. The reason is that a test set can be overfitted like anything else, and the failure is silent — "If we overfit our training data, there is always the evaluation on test data to keep us honest. But if we overfit the test data, how would we ever know?" [1, §3.6.3].

Hence three splits: "The common practice for addressing the problem of training on the test set is to split our data three ways, incorporating a validation set in addition to the training and test datasets" [1, §3.6.3].

Where it's used

Before anything else, and once per project. Every score on this site that is described as held-out came from one, and cross-validation is the same idea applied k times over so that every row gets a turn on the outside.

In code it is one call — train_test_split, or a cv argument that does the splitting for you [2, §3.1]. The mechanics are never the problem. The four measurements below are.

Why & when

Always, with the size chosen deliberately rather than by habit. The 80/20 in every tutorial is a trade: a bigger test set makes the number you report steadier, and a smaller one makes the model you report it for slightly better. Part one prices that trade.

Split by group whenever rows are not independent, and split by time whenever the model will be used on the future. Both are the IID assumption [1, §3.6.1] failing in a way a random split cannot see.

Worked example

All in scripts/datasplit.py.

A single split is one draw

  the same model and the same data, split 400 different ways
   test rows   mean accuracy   std dev    worst    best   spread
         120           0.831     0.032    0.750   0.917    0.167
         240           0.832     0.021    0.767   0.887    0.121
         420           0.832     0.016    0.788   0.871    0.083
         600           0.833     0.012    0.800   0.873    0.073

    Nothing changed between those 400 runs except which rows landed on
    which side. A single number from a single split is a draw from that
    column, and the smaller the test set the wider the draw.

Nothing about the model changed across those 400 runs. At 120 test rows the same model scores 0.750 or 0.917 depending on the shuffle, and that spread is wider than most reported improvements.

Choosing on the test set spends it

The validation/test distinction sounds like bookkeeping. It is worth this much:

        best, best_s = None, -1.0
        for _ in range(k):
            cols = g.choice(x.shape[1], size=6, replace=False)
            wb = fit_logistic(x[tr][:, cols], y[tr])
            s = accuracy(x[te][:, cols], y[te], wb)
            if s > best_s:
                best_s, best = s, (cols, wb)

  pick the best of k independent models on one test set, then re-measure
  models compared   chosen score   on fresh data   optimism
                1          0.576           0.572     +0.004
                3          0.622           0.607     +0.015
               10          0.647           0.623     +0.025
               30          0.663           0.629     +0.034
              100          0.672           0.633     +0.039

    The chosen model's score on the set that chose it is not an estimate
    of anything. It is the maximum of k noisy numbers, and the maximum of
    k draws rises with k whether or not any model is better. The last row
    is a model that looks 3.9 points better than it is.

The maximum of k noisy numbers grows with k whether or not any model is better. This is the same effect as multiple comparisons, arriving through a door nobody watches.

Anything fitted before the split has seen everything

  a target that is pure coin-flip noise, 600 rows, 4,000 features
            feature selection done   test accuracy
       before the split (all rows)           0.701
inside the split (train rows only)           0.501
                         the truth           0.500

    There is no signal in this data: y is a coin flip independent of every
    one of the 4,000 columns. Selecting features while the test rows were
    visible manufactures +0.201 of accuracy out of nothing. Doing the
    identical selection on the training rows alone reports chance, which
    is the correct answer. The split has to come first.

Rows that arrive in groups

  60 subjects, 20 rows each: the same rows split two ways
                 split   test accuracy            what it measures
           random rows           0.820        subjects it has seen
        whole subjects           0.761         subjects it has not

    +0.059 of the random split's accuracy is recognising subjects it
    already trained on. If rows arrive in groups -- patients, users,
    sessions, documents -- the split has to be over groups, or the score
    answers a question nobody asked.

Pitfalls

Reporting one split as if it were the answer. It is a draw from a distribution whose width you can measure, and at small test sizes that width is larger than the effect being claimed. Report a spread, or use cross-validation.

Letting the test set choose anything. Model, features, threshold, stopping epoch, or just "which run to write up" — each one spends it, and the resulting number is optimistic by an amount that grows with how many options were compared.

Preprocessing before splitting. Scaling, imputing, feature selection, resampling, target encoding: fit on the training rows only. Above, feature selection done a moment too early manufactured 0.201 of accuracy from a target that was a coin flip.

Splitting rows when the data comes in groups. Patients, users, sessions, sentences from one document. A random split puts the same group on both sides and the model scores itself on subjects it has already met.

Splitting time series at random. The same failure with the clock added: the model is handed the future to learn from and then asked to predict the past. Split at a date.

Treating a reported test score as a true test score. Even careful sources do not always have one — d2l says of its own experiments that they are "really working with what should rightly be called training data and validation data, with no true test sets" [1, §3.6.3]. That is candour, not sloppiness, and it is worth the same candour about your own numbers.