Training, Validation, and Test Sets

Why a model has to be judged on data it never saw, and how to divide data so that it can be. Parameters and hyperparameters, the separate jobs of training, validation and test data, how much the score of one holdout split varies and why, why the winner of a comparison looks better than it is, the test set used once, leakage and the wrong way to select inputs reproduced by simulation, stratified splits, and the split in scikit-learn.

A regression model is fitted to 442 diabetes patients and predicts their disease progression with a mean squared error of 2,860. That number describes how well the model fits the patients it was built from. A doctor using the model wants to know something else: how far off it will be for the next patient, who was not among the 442. The two numbers differ, and the difference grows with the flexibility of the model. The Bias-Variance Tradeoff showed a degree-12 polynomial whose training error was about a third of the irreducible noise variance and whose expected error on new data was nearly eight times it.

The remedy is simple to state. Keep some of the data away from the model while it is built, and measure its error on that data afterward. This article covers how to do that: the three roles data can play, how large each part should be, how noisy the resulting estimates are, and the ways in which held-out data stops being held out without anyone noticing. The examples use the diabetes study of Efron, Hastie, Johnstone and Tibshirani (2004) from the Regression articles and the breast cancer biopsies from Binary Classification Metrics.

Parameters, hyperparameters and decisions

Fitting a model means estimating its parameters from data: the intercept and slopes of a linear regression, the split points of a regression tree, the weights of a neural network. The fitting algorithm chooses them to make some loss small on the data it is given.

Many methods also have settings the fitting algorithm does not choose. The number of neighbors $k$ in nearest-neighbor regression, the degree of a polynomial, the depth of a tree and the strength of a penalty are hyperparameters (also called tuning parameters). Changing one changes the flexibility of the model, and the training error alone cannot set it, because the training error always prefers the most flexible option: $k = 1$, the highest degree, the deepest tree.

Beyond hyperparameters, an analysis makes many other decisions: which inputs to include, how to transform them, whether to use a linear model or a forest. All of these are choices among candidate models, and Hastie, Tibshirani and Friedman (2009, §7.2) separate the two tasks that held-out data serves:

  • Model selection: estimating the performance of different models in order to choose one.
  • Model assessment: having chosen a final model, estimating its prediction error on new data.

The two tasks need separate data, for a reason the rest of this article demonstrates: a score used to choose a model is no longer an honest estimate of that model's error.

Three roles for data

When data are plentiful, the cleanest arrangement divides them at random into three parts (Hastie et al., 2009, §7.2):

  • The training set is used to fit each candidate model, that is, to estimate its parameters.
  • The validation set is used to estimate the prediction error of each candidate, so that one can be chosen: a value of $k$, a set of inputs, a type of model.
  • The test set is used once, at the end, to estimate the error of the chosen model.

Hastie et al. (2009, §7.2) put the rule for the test set vividly: it "should be kept in a 'vault,' and be brought out only at the end of the data analysis". They suggest 50% for training and 25% each for validation and testing as a typical division, while noting that no general rule exists, since the right sizes depend on how noisy the data are and how complex the models.

A worked three-way split

The 442 diabetes patients are split at random (seed 742) into 265 for training, 89 for validation and 88 for testing, 60%, 20% and 20%. The model is nearest-neighbor regression on two inputs, body mass index (BMI) and the serum measurement S5, standardized with the means and standard deviations of the training part: a new patient's progression is predicted by the average of the $k$ training patients closest to it in those two measurements. The hyperparameter $k$ is chosen on the validation set.

$k$ Validation MSE Test MSE
1 7,932.8 6,197.1
5 4,315.8 3,844.5
10 3,898.2 3,486.4
20 3,760.5 3,219.7
50 3,980.4 3,347.2
100 4,219.0 3,479.4

Over all values from 1 to 100, the validation error is lowest at $k = 20$, 3,760.5, so $k = 20$ is chosen. The table shows the test column only to make a point; in a real analysis it would stay sealed until this choice was made. Two observations follow.

First, the two columns differ by several hundred at every $k$. The validation and test sets are both random samples of under 90 patients, and this validation set happens to contain patients who are harder to predict. Neither number is wrong. Each is an estimate with a large sampling error, a point taken up below.

Second, once $k$ is chosen, the training and validation parts can be combined, the model refitted on all 354 patients with $k = 20$, and the test set used once. Its test MSE is 3,431.2, a root mean squared error (RMSE) of 58.6 progression units. That is the number to report. It is slightly worse than the 3,219.7 of the model fitted to 265 patients, which a larger training set would not usually produce; with 88 test patients, a difference of this size is within the noise.

How noisy is one holdout estimate?

A test MSE is an average of squared errors over the test cases. If the test set holds $m$ cases whose squared errors $e_1^2, \dots, e_m^2$ have standard deviation $s$, then the standard error of their mean is approximately

$$\operatorname{SE}(\widehat{\mathrm{MSE}}) \approx \frac{s}{\sqrt{m}}.$$

For one 80/20 split of the diabetes data, a linear regression on all ten inputs has a test MSE of 2,745.3 on 88 patients. The squared errors have a standard deviation of 3,460.8, larger than their mean, because a few patients are predicted very badly and squaring magnifies their errors. The standard error is $3{,}460.8 / \sqrt{88} = 368.9$, about 13% of the estimate.

The formula captures only part of the uncertainty: the variation from which cases happened to land in the test set. A second source is which cases happened to land in the training set, since a different training sample gives a different fitted model. Repeating the whole split many times shows both together. The table summarizes 2,000 random splits at four test fractions.

Test fraction Test cases Training cases Mean test MSE SD across splits 10th to 90th percentile
10% 44 398 3,006.2 562.7 2,302 to 3,782
20% 88 354 3,031.3 382.1 2,555 to 3,526
33% 147 295 3,034.2 277.8 2,683 to 3,396
50% 221 221 3,080.8 205.1 2,819 to 3,358
Test MSE of the same linear regression over 2,000 random splits of the 442 diabetes patients. With 44 test patients the estimate ranges from below 2,000 to above 4,500; with 221 it is much tighter, but its center has moved up, because each model was fitted to only 221 patients.
Test MSE of the same linear regression over 2,000 random splits of the 442 diabetes patients. With 44 test patients the estimate ranges from below 2,000 to above 4,500; with 221 it is much tighter, but its center has moved up, because each model was fitted to only 221 patients.

Two effects pull in opposite directions. A larger test set makes the estimate less variable: the standard deviation falls from 563 to 205 as the test set grows from 44 to 221 patients, roughly in proportion to $1/\sqrt{m}$. A larger test set also means a smaller training set, and a model fitted to fewer cases is worse, so the estimate describes a worse model than the one that will be used: the mean rises from 3,006 to 3,081. The second effect is small here, because a linear model with eleven coefficients is already well estimated from 221 patients, but for flexible models it can dominate. This is the tradeoff behind every choice of split ratio, and behind the choice of the number of folds in cross-validation, which uses every case for testing once and so softens it.

On a dataset of this size, the arithmetic is sobering. A single 80/20 split pins down a model's error only to within about 13%, one standard error. With 100,000 cases the same split would give 20,000 test cases and a standard error about fifteen times smaller, and the question of how to divide the data mostly disappears. Splitting is a small-data problem.

The winner's curse

A validation score that is used to choose among models is an optimistic estimate of the chosen model's error. The reason is selection. Each candidate's validation error is its true error plus noise. The candidate with the smallest validation error tends to be one whose noise happened to be favorable, and that luck does not carry over to new data.

The nearest-neighbor example shows the effect when it is repeated over 1,000 random 60/20/20 splits. In each split, $k$ is chosen as the value with the smallest validation MSE, and the chosen model is then scored on the test set.

Quantity, averaged over 1,000 splits MSE
Validation MSE of the chosen $k$ 3,197.2
Test MSE of the chosen $k$ 3,416.9

The validation score of the winner understates its error on new data by 219.7, or 6.9%. The chosen $k$ itself wanders: its median is 21, but the middle 80% of choices run from 9 to 67, because the validation curve is flat near its minimum and noise decides where the lowest point falls.

The optimism grows with the number of candidates compared. Choosing one value of $k$ from a smooth curve is a mild search. A harsher one compares all 1,023 non-empty subsets of the ten diabetes inputs. Suppose an analyst does this the wrong way, fitting each subset by least squares on a training set of 265 patients and choosing the subset with the lowest error on the 88-patient test set. To see what that costs, a third group of 89 patients is kept untouched, and the whole procedure is repeated over 1,000 random splits.

Model Mean MSE on the set used to choose Mean MSE on untouched data
Subset chosen by test error 2,891.6 3,159.5
All ten inputs, no choice made 3,058.1 3,048.5
Each point is one of 1,000 random splits. The input subset with the lowest error on the test set (horizontal axis) is scored on untouched patients (vertical axis). Most points lie above the diagonal: the error that made the subset look best does not hold up.
Each point is one of 1,000 random splits. The input subset with the lowest error on the test set (horizontal axis) is scored on untouched patients (vertical axis). Most points lie above the diagonal: the error that made the subset look best does not hold up.

On the data used to choose it, the winning subset looks 5% better than the full model. On untouched data it is 4% worse, and it beats the full model in only 38.9% of the splits. The reported error of the winner, 2,891.6, understates its true error by 267.9, or 9.3%. The full model, which involved no choice, is scored honestly by either set: 3,058.1 against 3,048.5.

This is why the test set is used once. Hastie et al. (2009, §7.2) state the consequence directly: if the test set is used repeatedly to choose the model with the smallest test error, the chosen model's test error "will underestimate the true test error, sometimes substantially." Once a test set has influenced a decision, it has become a validation set, and an honest estimate needs fresh data. The influence does not have to be formal. Looking at test results, adjusting the model and looking again amounts to the same search, carried out by hand.

The same effect operates at the scale of a research field. When many groups tune their methods against the same public benchmark test set, the best published results on that benchmark contain some of the selection optimism measured here. Cawley and Talbot (2010) show that overfitting in model selection can produce differences in performance of the same size as the differences between learning algorithms that comparisons are meant to detect.

Leakage: when held-out data is not held out

Leakage is any route by which information from the held-out cases reaches the model before it is evaluated. The split may be made correctly and the evaluation still contaminated, because some step before the split looked at all the data.

The wrong way to select inputs

Hastie et al. (2009, §7.10.2) describe a mistake that they "have seen ... committed many times in published papers in top rank journals". A study has 50 cases in two classes and 5,000 measured inputs, as in a gene-expression experiment. The analysis keeps the 100 inputs most correlated with the class, builds a classifier on those 100, and estimates its error by cross-validation. The flaw is the order of the steps: the 100 inputs were chosen using all 50 cases, including the ones each fold later holds out.

The consequences can be measured when the truth is known. In the simulation below (100 datasets, seed 742), all 5,000 inputs are pure noise, standard normal and independent of the class, so no classifier can do better than an error rate of 0.5. The classifier is the 1-nearest-neighbor rule on the selected inputs, and the error is estimated by 5-fold Cross-Validation.

Procedure Mean cross-validated error 10th to 90th percentile
Inputs selected once, from all 50 cases 0.013 0.00 to 0.04
Inputs selected again inside each training fold 0.531 0.42 to 0.64
Cross-validated error of a 1-nearest-neighbor classifier on inputs that carry no information about the class (true error 0.5), over 100 simulated datasets. Choosing the inputs before splitting into folds reports almost perfect accuracy; choosing them inside each fold reports the truth.
Cross-validated error of a 1-nearest-neighbor classifier on inputs that carry no information about the class (true error 0.5), over 100 simulated datasets. Choosing the inputs before splitting into folds reports almost perfect accuracy; choosing them inside each fold reports the truth.

Done the wrong way, cross-validation reports an error of 1.3% for a classifier that is guessing. Hastie et al. (2009) report 3% in their own simulation of the same setting. The mechanism is visible in the held-out folds. Among 5,000 noise inputs, the 100 most correlated with the class across all 50 cases are correlated with it partly because of the cases that end up in each held-out fold. Measured on the 10 held-out cases of a fold, the selected inputs' correlation with the class averaged 0.369, with the sign that made them look useful in selection, rather than the 0 that pure noise should give; Hastie et al. report about 0.28. When the inputs are chosen afresh from the 40 training cases of each fold, the correlation on the held-out cases averages 0.000.

The rule that follows is general: every step that uses the outcome must be repeated inside each training part, and nothing about the held-out cases may influence the model before it predicts them (Hastie et al., 2009, §7.10.2). Ambroise and McLachlan (2002) showed that published error rates of near zero for gene-expression classifiers came from exactly this selection bias, and that correcting for it gave substantially higher error estimates. The course wiki's Feature Selection applies the same rule to the selection of words for text classification.

Hastie et al. (2009) note one qualification. A screening step that does not use the outcome, such as keeping the 1,000 inputs with the highest variance across all cases, can be done before splitting, because it gives no input an unfair advantage in predicting the labels.

Other common forms of leakage

  • Preprocessing fitted on all the data. Standardizing inputs, filling in missing values or fitting a dimension-reduction step on the full dataset lets the held-out cases shape the transformation. When the step does not use the outcome, the effect is usually small, but fitting each step on the training part only, as a scikit-learn pipeline does, removes the question.
  • Duplicates and related cases. If the same patient, document or image appears in both parts, or several records come from the same person, the test set measures recall of familiar cases. Splitting by group (all of a patient's records on one side) prevents it.
  • Time. A model meant to forecast the future must be tested on cases that come after its training cases. A random split of time-ordered data lets the model learn from the future.
  • Inputs that encode the outcome. A variable recorded after the outcome, or as a consequence of it, can make a model look excellent and be useless when deployed, because it will not be available at prediction time.

Making the parts alike

Random assignment makes the training, validation and test sets samples from the same population, which is what entitles the test error to describe new cases from that population. With small sets, chance can still make them unlike each other. The breast cancer data have 212 malignant masses among 569 (37.3%). Over 1,000 random 90/10 splits, the share of malignant masses in the 57-case test set ranged from 17.5% to 54.4%, and in the middle 80% of splits from 29.8% to 43.9%. A stratified split draws each class separately in proportion to its size, so every test set had 21 malignant masses, 36.8%. Stratifying by the outcome is the default for classification with scikit-learn's cross-validation helpers, and is worth requesting explicitly in train_test_split.

Random assignment cannot fix a sample that is unlike the population where the model will be used. If the 442 patients were all from one clinic, any split of them estimates performance at that clinic. The precision of a classifier, which depends on how common the positive class is, is the clearest example: Binary Classification Metrics shows how a test set with an unrealistic prevalence misstates it.

Choosing the sizes

There is no correct ratio, but the considerations are clear from the experiments above.

  • The test set needs to be large enough that its standard error, roughly $s/\sqrt{m}$, is small compared with the differences that matter. With a few hundred cases this takes a large share of the data; with hundreds of thousands, a small share is plenty.
  • The training set should be large enough that the model fitted to it is close to the one that would be fitted to all the data. For a simple model this is a modest number; for a flexible one it may be most of the data.
  • The validation set must support the comparisons made on it. The more candidates are compared, the more selection optimism it absorbs, and the larger it needs to be.

Common choices such as 60/20/20 or 70/15/15 are reasonable starting points for datasets of moderate size. When the data are too few for three useful parts, which is often, the validation step is done instead by resampling the training data: by Cross-Validation or the bootstrap, or by an analytic estimate such as AIC or BIC, as described in Model Selection (Hastie et al., 2009, §7.2). A test set is still kept back for the final assessment whenever the data allow it. When there are no decisions to make, because the model has no hyperparameters and no inputs to choose, a training and test split is enough.

More training data does not raise the risk of overfitting. Overfitting is a model fitting the noise of its particular sample, and the variance part of the error, which measures it, shrinks as the sample grows (The Bias-Variance Tradeoff shows the effect of quadrupling the sample). The cost of a larger training set is paid elsewhere: in a smaller, noisier test set and in longer fitting times.

In Python

scikit-learn's train_test_split divides arrays into two parts at random. A three-way split takes two calls, and the second call's test_size is a fraction of what remains: to get 20% of the original as validation from the 80% left after removing the test set, ask for $0.2/0.8 = 0.25$.

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsRegressor
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_diabetes(return_X_y=True, scaled=False)
X = X[:, [2, 8]]                                   # BMI, S5

X_rest, X_test, y_rest, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
X_train, X_val, y_train, y_val = train_test_split(X_rest, y_rest, test_size=0.25, random_state=0)
print(len(y_train), len(y_val), len(y_test))      # 264 89 89

val_mse = {}
for k in range(1, 101):
    model = make_pipeline(StandardScaler(), KNeighborsRegressor(n_neighbors=k))
    model.fit(X_train, y_train)
    val_mse[k] = np.mean((model.predict(X_val) - y_val) ** 2)
k_best = min(val_mse, key=val_mse.get)

final = make_pipeline(StandardScaler(), KNeighborsRegressor(n_neighbors=k_best))
final.fit(X_rest, y_rest)                          # training + validation
print(k_best, np.mean((final.predict(X_test) - y_test) ** 2))   # 7 3818.4: the test set, used once

The random_state argument fixes the random split so that the analysis can be repeated exactly; any integer will do, and reporting it is part of reporting the analysis. The split differs from the one in the tables above, and here the validation set prefers $k = 7$ and the test MSE is 3,818.4: a reminder of how much a single split varies. Putting the scaler inside the pipeline means it is fitted to the training data only. For classification, train_test_split(X, y, stratify=y) makes a stratified split. The numbers in this article are computed by tools/experiments/data-science/training-validation-and-test-sets.py in the site's repository.

References

  • Ambroise, C., and McLachlan, G. J. (2002). Selection bias in gene extraction on the basis of microarray gene-expression data. Proceedings of the National Academy of Sciences, 99(10), 6562–6566. record
  • Cawley, G. C., and Talbot, N. L. C. (2010). On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research, 11, 2079–2107. record
  • Efron, B., Hastie, T., Johnstone, I., and Tibshirani, R. (2004). Least angle regression. The Annals of Statistics, 32(2), 407–499. record · 12 pages
  • Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer. record · 23 pages