Stacking

Combining different kinds of model by fitting a meta model to their predictions. Wolpert's stacked generalization and Breiman's stacked regressions, why the meta model must be fitted to out-of-fold predictions and what happens when it is not (a random forest given all the weight on the diabetes data), the out-of-fold procedure step by step, non-negative weights, a seeded experiment on Friedman's problem where a stack beats every member, and two real datasets where it does not.

Four quite different models are fitted to the diabetes study of 442 patients (Efron, Hastie, Johnstone and Tibshirani, 2004): least squares on the ten baseline measurements, 20-nearest-neighbor regression, a random forest and gradient boosting with stumps. Under 10-fold cross-validation (the folds of Multiple Linear Regression) their $R^2$ values are 0.484, 0.446, 0.443 and 0.448. The usual next step is to keep the best, least squares, and discard the rest. But the four make different assumptions and different errors, and Introduction to Ensemble Models showed that a combination of models that disagree can beat each of them. Stacking chooses the combination by learning it: a second model, the meta model, is fitted to predict the outcome from the first models' predictions.

The idea is due to Wolpert (1992), who called it stacked generalization. Breiman (1996) made it practical for regression by restricting the meta model to a linear combination with non-negative weights, and called the result stacked regressions. Done naively, stacking fails in a predictable way, and the remedy, fitting the meta model to cross-validated predictions, is the heart of the method.

Two levels of models

Wolpert's vocabulary is still used. The original models are the level-zero models, or base models. Their predictions for the training cases form a new dataset, the level-one data: one row per case, one column per base model, with the case's outcome alongside. The meta model, or level-one model, is fitted to the level-one data. To predict a new case, each base model makes its prediction, and the meta model combines them.

For regression the meta model is usually a linear regression, so the stack predicts

$$\hat{y}(x) = a_0 + \sum_{k=1}^{K} a_k \hat{f}_k(x),$$

where $\hat{f}_1, \dots, \hat{f}_K$ are the $K$ base models and $a_0, a_1, \dots, a_K$ are coefficients the meta model estimates. For classification it is usually a logistic regression on the base models' predicted probabilities. Any model can serve, but the meta model sees only $K$ inputs, which are highly correlated because every base model is predicting the same thing, so a simple one is the safe choice.

Stacking generalizes a familiar practice. Choosing the base model with the lowest cross-validated error is also a rule for combining models, one that gives weight 1 to a single model and 0 to the others. Wolpert (1992) described cross-validation as using a "crude winner-takes-all" strategy and showed that it is a special case of stacking; Hastie, Tibshirani and Friedman (2009, §8.8) make the same connection. Stacking replaces the choice with estimated weights.

The leak: fitting the meta model to in-sample predictions

The obvious way to build the level-one data is to fit each base model to the training set and record its predictions for the same training cases. It does not work, and the diabetes data show why. Here are the four models' $R^2$ on the cases they were fitted to (averaged over the ten training folds), beside their cross-validated $R^2$:

Base model $R^2$ on its own training data 10-fold CV $R^2$
least squares 0.519 0.484
20-nearest neighbors 0.504 0.446
random forest 0.926 0.443
boosted stumps 0.579 0.448

The random forest fits its own training data far better than any other model, because its deep trees have seen those cases, and predicts new cases no better than the others. A meta model fitted to in-sample predictions sees only the left-hand column. Fitted to all 442 patients with non-negative weights, it puts all the weight on the forest (1.196) and none on the other three. Without the constraint, its coefficients are $-0.028$, $-0.173$, $1.629$ and $-0.426$: the forest gets more than all the weight and the other models are subtracted. Both stacks predict new patients worse than least squares alone, with cross-validated $R^2$ of 0.432 and 0.406.

This is a form of data leakage: information about the outcomes the meta model is fitted to has already been used by the base models, so the level-one data make the most flexible model look best. Breiman (1996) named it as the first problem with fitting combination weights by least squares on the training set, which "will overfit the data". Hastie et al. (2009, §8.8) give a sharper example. If the base models are the best regressions using 1, 2, ..., $M$ of $M$ inputs, a regression of $y$ on their in-sample predictions puts all the weight on the largest model, since it always fits the training data best. The models have not been put "on the same footing" with respect to their complexity.

Out-of-fold predictions

The remedy is to build the level-one data from predictions each base model makes for cases it did not see. Wolpert (1992) built them from a partition of the learning set, training on one part and guessing the other, and Breiman (1996) took leave-one-out predictions: for case $i$, the prediction of model $k$ fitted to all cases but $i$, written $\hat{f}_k^{(-i)}(x_i)$. The meta model's weights then solve

$$\min_{a} \sum_{i=1}^{n} \Bigl( y_i - \sum_{k=1}^{K} a_k \hat{f}_k^{(-i)}(x_i) \Bigr)^2,$$

the stacking criterion of Hastie et al. (2009, eq. 8.59). Each prediction in the sum is honest, so each base model is judged by how well it predicts new cases, which is what the stack will ask of it.

Leave-one-out requires fitting each base model $n$ times. $V$-fold Cross-Validation is the usual substitute, and Breiman (1996) found in simulations that, to his surprise, level-one data from 10-fold cross-validation gave better stacks than leave-one-out data, at a tenth of the cost. The procedure is:

  1. Split the training cases into $V$ folds.
  2. For each fold $v$, fit every base model to the other $V - 1$ folds and predict the cases in fold $v$. After $V$ rounds every case has one out-of-fold prediction from each base model; together they form the level-one data.
  3. Fit the meta model to the level-one data and the outcomes.
  4. Refit every base model to the whole training set. These are the models used for prediction; the meta model combines their predictions for new cases.
Building the level-one data with five folds. In each round one fold (orange) is held out, every base model is fitted to the other four, and their predictions for the held-out cases fill that fold's rows. After five rounds each training case has a prediction from each model that did not see it, and the meta model is fitted to these.
Building the level-one data with five folds. In each round one fold (orange) is held out, every base model is fitted to the other four, and their predictions for the held-out cases fill that fold's rows. After five rounds each training case has a prediction from each model that did not see it, and the meta model is fitted to these.

Each base model is fitted $V + 1$ times, and the meta model once. On the diabetes data, with five folds, the out-of-fold predictions rank the models honestly: their $R^2$ is 0.492 for least squares, 0.454 for nearest neighbors, 0.444 for the forest and 0.459 for boosting, close to the cross-validated values above rather than to the in-sample ones. The non-negative meta model now gives least squares a weight of 0.588, nearest neighbors 0.413, boosting 0.098 and the forest 0.

Meta-model weights (non-negative least squares) fitted to in-sample predictions (solid) and to out-of-fold predictions (hatched). Left: the diabetes data. Right: Friedman's problem, medians over 30 training sets. In-sample predictions make the random forest, which fits its own training data far better than it predicts new data, look like the only model worth using.
Meta-model weights (non-negative least squares) fitted to in-sample predictions (solid) and to out-of-fold predictions (hatched). Left: the diabetes data. Right: Friedman's problem, medians over 30 training sets. In-sample predictions make the random forest, which fits its own training data far better than it predicts new data, look like the only model worth using.

Evaluating a stack honestly needs one more level of separation. The weights were fitted using every training case, so the stack's error must be measured on cases outside the whole procedure. The numbers here come from nested cross-validation: the outer 10-fold split holds out a tenth of the patients, the entire stacking procedure, inner 5-fold split included, runs on the other nine tenths, and the held-out tenth is predicted at the end.

Constraining the weights

The base models' out-of-fold predictions on the diabetes data have correlations between 0.915 and 0.962, and least-squares coefficients of highly correlated inputs are unstable (see Multiple Linear Regression). Unconstrained, the meta model fitted to out-of-fold predictions gives the forest a negative weight, $-0.244$, offset by larger positive weights on the others. Breiman (1996) identified this as the second problem with stacking. He tried ridge regression, which helped but not consistently, and found that requiring the weights to be non-negative gave "consistently good results": in his experiments the constrained stack almost always predicted better than the single best model.

His explanation starts from a stronger constraint. If the weights are non-negative and add up to 1, the stack's prediction at every input lies between the smallest and largest of the base models' predictions there, so the stack can only interpolate among its members, never extrapolate beyond them. Hastie et al. (2009, §8.8) add that such weights can be read as posterior probabilities of the models being right, in the spirit of Bayesian model averaging. Breiman imposed only non-negativity. He showed that when some of the base models predict well, non-negative weights fitted without the sum constraint add up to nearly 1 anyway, and in his simulations the results were almost identical with or without it. His combination has no intercept; with one, as here, the non-negative weights on the diabetes data add up to 1.100 and the intercept is $-13.87$.

Under nested cross-validation:

Meta model In-sample level-one data Out-of-fold level-one data
least squares 0.406 0.471
non-negative least squares 0.432 0.478
ridge (RidgeCV) 0.406 0.471

The out-of-fold stacks all beat the in-sample ones, and the non-negative one is best. The ridge meta model, scikit-learn's default, chose the largest penalty on its default grid, which is still negligible next to predictions measured in the hundreds, and so matched least squares. None of the stacks beats least squares alone (0.484), and the plain average of the four models (0.479) is about as good as the best stack. On these data the relationship appears close to linear: the linear model already captures most of what the four models can predict from 442 patients, and the other models' disagreements with it are mostly noise. Stacking cannot create information the base models do not have.

When stacking pays

Stacking helps when the base models are good at different things. Friedman's (1991) first test problem, used throughout this topic, makes that easy to arrange:

$$y = 10 \sin(\pi x_1 x_2) + 20 (x_3 - 0.5)^2 + 10 x_4 + 5 x_5 + \varepsilon, \qquad \varepsilon \sim N(0, 1),$$

with ten inputs uniform on $[0, 1]$, five of them irrelevant. Gradient boosting with stumps fits a sum of functions of one input each (see Boosting); it handles the quadratic and linear terms well but cannot represent the interaction of $x_1$ and $x_2$ in the sine. A random forest can represent the interaction but approximates the smooth terms with steps. Four base models, least squares, 10-nearest neighbors, a random forest and boosted stumps, were fitted to 30 training sets of 300 cases (seed 5), and stacked with a non-negative meta model on 5-fold out-of-fold predictions. Expected test errors at 2,000 test points, including the noise variance of 1, are:

Model Median test MSE Interquartile range
least squares 7.099 7.016 to 7.276
10-nearest neighbors 9.627 9.307 to 9.991
random forest 5.486 5.284 to 5.625
boosted stumps 3.924 3.764 to 4.098
average of the four 5.074 4.931 to 5.276
stack, in-sample level-one data 4.993 4.804 to 5.204
stack, out-of-fold level-one data 3.548 3.464 to 3.727

The out-of-fold stack beats its best member, the boosted stumps, in all 30 training sets. Its median weights are 0.75 on the boosted stumps, 0.32 on the forest, 0.11 on nearest neighbors and 0 on least squares: mostly the additive model, corrected by the model that can see the interaction. The plain average does worse than the boosted stumps, because it gives equal weight to two much weaker models, and the in-sample stack again puts all its weight on the forest (median 1.11). Breiman (1996) observed the same pattern on a larger scale: the largest gains came from stacking dissimilar models, subset regressions with ridge regressions, and "the more similar the predictors, the less advantage there is in stacking."

There is a limit to read from the experiment too. Gradient boosting with depth-3 trees, which can model the interaction itself, reaches a median test error of 3.447 on the same training sets, slightly better than the stack. A stack is only as good as the best combination of the models given to it, and a single better model outside the set can beat it.

Stacking classifiers

For classification the level-one data are usually the base models' predicted probabilities, and the meta model a logistic regression. On the breast cancer data (Wolberg, Mangasarian, Street and Street, 1993; 10-fold stratified cross-validation, the folds of Random Forests), with logistic regression, Gaussian naive Bayes, 5-nearest neighbors and a random forest as base models:

Model Accuracy Log loss
logistic regression 0.977 0.074
naive Bayes 0.938 0.597
5-nearest neighbors 0.965 0.312
random forest 0.961 0.165
average of the probabilities 0.965 0.093
stack, in-sample level-one data 0.974 0.077
stack, out-of-fold level-one data 0.975 0.076

Log loss measures the quality of the predicted probabilities; lower is better. Naive Bayes, whose independence assumption is badly violated by 30 strongly correlated measurements, makes extreme probability predictions, some of them confidently wrong, and including it in a plain average raises the average's log loss to 0.093, worse than logistic regression alone. Both stacks learn to give naive Bayes little weight and recover most of the loss. Neither beats logistic regression alone, which is already close to the limit of what these data allow: about 13 of 569 tumors misclassified. The in-sample stack is not much worse than the out-of-fold one here, but it still leans on the random forest, with an average meta-model coefficient of 4.17 against 2.25 out of fold.

Practical points

  • Choose dissimilar base models. Models that make the same errors give the meta model nothing to work with. A linear model, a nearest-neighbor model and a tree ensemble disagree in useful ways; three linear models with different penalties mostly do not.
  • Keep the meta model simple and constrained. Its inputs are few and highly correlated. Non-negative linear weights for regression and a regularized logistic regression for classification are robust defaults.
  • Never fit the meta model to in-sample predictions, and evaluate the whole stack on data the procedure never touched.
  • Include the original inputs only with care. Passing the inputs to the meta model alongside the predictions lets it combine the models differently in different regions, at the cost of a more complex meta model fitted to correlated, partly redundant columns. Wolpert (1992) allowed for level-one inputs of any kind, and Hastie et al. (2009, §8.8) note that the weights can depend on the input.
  • Expect modest gains. A stack is never better than the information in its members. When one model is clearly best and the others add only noise, the stack will roughly match it, at the cost of $V + 1$ fits of every base model and a model that is harder to explain.

In Python

from sklearn.datasets import load_diabetes
from sklearn.ensemble import GradientBoostingRegressor, RandomForestRegressor, StackingRegressor
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import KFold, cross_val_score
from sklearn.neighbors import KNeighborsRegressor
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_diabetes(return_X_y=True)
stack = StackingRegressor(
    estimators=[('linear', LinearRegression()),
                ('knn', make_pipeline(StandardScaler(), KNeighborsRegressor(20))),
                ('forest', RandomForestRegressor(500, max_features=3, random_state=0)),
                ('stumps', GradientBoostingRegressor(n_estimators=200, learning_rate=0.05,
                                                     max_depth=1, random_state=0))],
    final_estimator=LinearRegression(positive=True),     # non-negative weights
    cv=KFold(5, shuffle=True, random_state=1))
outer = KFold(10, shuffle=True, random_state=0)
print(cross_val_score(stack, X, y, cv=outer, scoring='r2').mean())   # nested CV

StackingRegressor and StackingClassifier carry out the out-of-fold procedure: the meta model (final_estimator) is fitted to predictions from cross_val_predict with the folds given by cv (5-fold by default), and the base models are refitted to the full data for prediction. The default meta model is RidgeCV for regression and LogisticRegression for classification. For classifiers, stack_method chooses what is stacked (predicted probabilities by default, when available), and passthrough=True adds the original inputs to the level-one data. The option cv="prefit", which takes base models already fitted, fits the meta model to their predictions on the training set, and the documentation warns of a very high risk of overfitting if those models were trained on the same data: the in-sample stack of this article. The numbers here are computed by tools/experiments/data-science/stacking.py in the site's repository.

References

  • Breiman, L. (1996). Stacked regressions. Machine Learning, 24(1), 49–64. record · 2 pages
  • Efron, B., Hastie, T., Johnstone, I., and Tibshirani, R. (2004). Least angle regression. The Annals of Statistics, 32(2), 407–499. record · 12 pages
  • Friedman, J. H. (1991). Multivariate adaptive regression splines. The Annals of Statistics, 19(1), 1–67. record · 4 pages
  • Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer. record · 23 pages
  • Wolberg, W., Mangasarian, O., Street, N., and Street, W. (1993). Breast Cancer Wisconsin (Diagnostic) [Data set]. UCI Machine Learning Repository. https://doi.org/10.24432/C5DW2B record · 14 pages
  • Wolpert, D. H. (1992). Stacked generalization. Neural Networks, 5(2), 241–259. Read in the Los Alamos National Laboratory technical report version (LA-UR-90-3460), which quotations follow. record · 2 pages