The Bias-Variance Tradeoff

Why a model that fits its training data better can predict new data worse. Underfitting and overfitting, the decomposition of expected squared error into squared bias, variance and noise derived step by step, a seeded simulation of polynomial fits that measures each term and checks it against the exact values, why training error misleads, how more data shifts the balance, nearest-neighbor regression on the diabetes study, and where the classical picture stops: classification loss and double descent.

Twenty measurements are taken at evenly spaced inputs between 0 and 1. The outcome follows a smooth curve, $\sin(5x)$, plus random noise with standard deviation 0.3. A straight line through the points misses the curve: it rises where the data fall. A polynomial of degree 12 passes close to almost every point, and between them it swings in ways the curve never does. A cubic does neither. If the experiment were repeated with fresh noise, the straight line would come out nearly the same each time, consistently wrong in the same places. The degree-12 polynomial would come out different each time, right on average but erratic in any one fit.

Those two failures have names. The line underfits: its form is too rigid to follow the pattern. The degree-12 polynomial overfits: it is flexible enough to follow the noise as well as the pattern. This article makes the two failures precise. The expected error of a prediction splits exactly into three parts, squared bias, variance and irreducible noise, and making a model more flexible usually lowers the first while raising the second. That is the bias-variance tradeoff, and it is the reason the rest of this topic, from loss functions to Cross-Validation and model selection, insists on measuring a model on data it was not fitted to.

Flexibility, underfitting and overfitting

The flexibility or complexity of a model is the variety of shapes it can take. A straight line, $\hat{y} = b_0 + b_1 x$, can only tilt and shift. A polynomial of degree $d$ has $d + 1$ coefficients and can bend $d - 1$ times. A regression with more inputs, more interaction terms or more polynomial terms is more flexible, as are a nearest-neighbor predictor that averages fewer neighbors, a regression tree with more splits and a neural network with more units (James, Witten, Hastie and Tibshirani, 2021, §2.1.3).

An underfit model is too rigid for the pattern in the data. It predicts poorly on the data it was fitted to and on new data alike, so the problem shows up in the training error. An overfit model is flexible enough to bend toward the particular noise in its training sample. It predicts its own training data well and new data poorly, and the training error hides the problem. Multiple Linear Regression showed a case on the diabetes data: a degree-3 polynomial in all ten inputs reached a training $R^2$ of 0.798 and a cross-validated $R^2$ of $-14.8$.

Polynomials fitted to 20 noisy points from sin(5x) (dashed). Each thin blue curve is the fit to a different simulated training set; the orange curve is the average over 10,000 sets. Degree 1 is stable but systematically wrong: high bias, low variance. Degree 3 is right on average and fairly stable. Degree 12 is right on average but each fit swings wildly, most of all near the ends: low bias, high variance.
Polynomials fitted to 20 noisy points from sin(5x) (dashed). Each thin blue curve is the fit to a different simulated training set; the orange curve is the average over 10,000 sets. Degree 1 is stable but systematically wrong: high bias, low variance. Degree 3 is right on average and fairly stable. Degree 12 is right on average but each fit swings wildly, most of all near the ends: low bias, high variance.

The figure shows the two failures by repeating the experiment. Each thin curve is a fit to one simulated training set. For degree 1 the curves lie close together, but their average misses the true curve: the error is systematic. For degree 12 the average sits on the true curve, but individual fits scatter far from it: the error is random, and it depends on which sample happened to be drawn. The degree-3 fits are both close together and centered on the truth.

Deriving the decomposition

The picture can be made exact for squared error. Suppose the outcome at input $x$ is

$$Y = f(x) + \varepsilon,$$

where $f$ is the true regression function, the mean outcome at $x$, and $\varepsilon$ is noise with mean 0 and variance $\sigma^2$, independent of everything else. A training set $\mathcal{D}$ is drawn, a method is applied to it, and the result is a fitted function $\hat{f}$. Because $\mathcal{D}$ is random, so is $\hat{f}$: a different training set gives a different fit. Now fix a test input $x_0$ and draw a new outcome there, $Y_0 = f(x_0) + \varepsilon_0$, with noise $\varepsilon_0$ independent of the training set. The expected test error at $x_0$ is

$$\operatorname{Err}(x_0) = \mathbb{E}\bigl[(Y_0 - \hat{f}(x_0))^2\bigr],$$

averaged over both the training set and the new noise. It answers the question: if this method were used again and again on fresh training samples, how far off would its prediction at $x_0$ be on average, in squared units?

Write $f_0 = f(x_0)$ and $\hat{f}_0 = \hat{f}(x_0)$, and let $\bar{f}_0 = \mathbb{E}[\hat{f}_0]$ be the average prediction over training sets: the orange curve in the figure. The derivation has two steps, each adding and subtracting a quantity so that the cross term vanishes.

Step 1: separate the noise. Since $Y_0 = f_0 + \varepsilon_0$,

$$\mathbb{E}\bigl[(Y_0 - \hat{f}_0)^2\bigr] = \mathbb{E}\bigl[\varepsilon_0^2\bigr] + 2\,\mathbb{E}\bigl[\varepsilon_0\,(f_0 - \hat{f}_0)\bigr] + \mathbb{E}\bigl[(f_0 - \hat{f}_0)^2\bigr].$$

The first term is $\sigma^2$. In the second, $\varepsilon_0$ is independent of the training set and therefore of $\hat{f}_0$, so the expectation factors into $\mathbb{E}[\varepsilon_0]\,\mathbb{E}[f_0 - \hat{f}_0] = 0$. What remains is $\sigma^2 + \mathbb{E}[(f_0 - \hat{f}_0)^2]$: the noise, plus the expected squared distance between the fit and the truth.

Step 2: separate the systematic part from the random part. Add and subtract the average prediction:

$$\mathbb{E}\bigl[(f_0 - \hat{f}_0)^2\bigr] = \mathbb{E}\bigl[\bigl((f_0 - \bar{f}_0) + (\bar{f}_0 - \hat{f}_0)\bigr)^2\bigr] = (f_0 - \bar{f}_0)^2 + 2\,(f_0 - \bar{f}_0)\,\mathbb{E}\bigl[\bar{f}_0 - \hat{f}_0\bigr] + \mathbb{E}\bigl[(\hat{f}_0 - \bar{f}_0)^2\bigr].$$

The first term is a constant, since neither $f_0$ nor $\bar{f}_0$ depends on the particular training set. The middle term is zero, because $\mathbb{E}[\hat{f}_0] = \bar{f}_0$ by definition. Putting the steps together,

$$\operatorname{Err}(x_0) = \underbrace{\sigma^2}_{\text{irreducible error}} + \underbrace{\bigl(\bar{f}_0 - f_0\bigr)^2}_{\text{squared bias}} + \underbrace{\mathbb{E}\bigl[(\hat{f}_0 - \bar{f}_0)^2\bigr]}_{\text{variance}}.$$

This is the decomposition in Hastie, Tibshirani and Friedman (2009, eq. 7.9) and James et al. (2021, eq. 2.7), who state it without proof. Each term has a direct meaning:

  • The irreducible error $\sigma^2$ is the variation of outcomes around their mean at $x_0$. No method can remove it, because even the true $f$ would make it. It sets a floor under the expected test error.
  • The bias $\bar{f}_0 - f_0$ is how far the method's average prediction is from the truth: a systematic error that more training samples of the same size would not average away. It comes from the method's assumptions, such as a straight line where the truth is curved.
  • The variance is how much the prediction at $x_0$ moves from one training set to another. It comes from the method's sensitivity to the particular sample.

Three points about the result deserve care. First, it is a statement about an expectation, over all the training sets the method might have seen and all the noise at $x_0$. A single prediction from a single fitted model has an error, but that error does not split into a bias part and a variance part; only the average of squared errors does. Second, the parts add because the cross terms vanish, not because they are three independent sources of error in any physical sense. Third, the result is for squared error. For other losses, such as the 0-1 loss of classification, a decomposition of this kind does not hold in the same additive form, a point taken up at the end of the article.

Averaging $\operatorname{Err}(x_0)$ over the test inputs $x_0$ gives the expected test error of the method as a whole, and the squared bias and variance averaged in the same way (James et al., 2021, §2.2.2).

Measuring each term by simulation

None of the three terms can be computed from a single real dataset, because the true $f$ and the distribution of training sets are unknown. In a simulation both are known, so each term can be measured directly. The setting is the one from the opening:

  • true function $f(x) = \sin(5x)$ on $[0, 1]$;
  • $n = 20$ training inputs fixed at $x_i = (i - 0.5)/20$, so 0.025, 0.075, up to 0.975;
  • noise standard deviation $\sigma = 0.3$, so the irreducible error is $\sigma^2 = 0.09$;
  • 10,000 training sets, each with fresh noise (seed 742);
  • polynomials of degree 0 to 12 fitted to each set by least squares;
  • the error measured at 201 test inputs evenly spaced on $[0, 1]$.

For each degree and each test input, the 10,000 fitted values give a mean, which is compared with $f(x_0)$ to give the bias, and a variance. Averaging over the test inputs gives the table. As an independent check, each fitted model was also tested against a fresh noisy outcome at every test input, and the average squared error recorded.

Degree Squared bias Variance $\sigma^2$ Sum Measured test MSE Training MSE
0 0.5072 0.0045 0.09 0.6017 0.6022 0.5930
1 0.1756 0.0091 0.09 0.2747 0.2749 0.2527
2 0.0533 0.0138 0.09 0.1570 0.1573 0.1273
3 0.0036 0.0186 0.09 0.1122 0.1125 0.0750
4 0.0004 0.0237 0.09 0.1141 0.1143 0.0677
5 0.0000 0.0292 0.09 0.1192 0.1194 0.0629
6 0.0000 0.0356 0.09 0.1256 0.1258 0.0583
8 0.0000 0.0548 0.09 0.1448 0.1449 0.0493
10 0.0000 0.1149 0.09 0.2049 0.2048 0.0405
11 0.0000 0.2270 0.09 0.3170 0.3169 0.0361
12 0.0002 0.6172 0.09 0.7074 0.7073 0.0316

The sum of the three terms matches the directly measured test error to within 0.0005 at every degree, as the derivation requires. (Degrees 7 and 9 follow the same pattern and are omitted; the script prints all thirteen.)

The decomposition measured over 10,000 simulated training sets. Squared bias falls quickly and is near zero from degree 4 on; variance rises slowly, then steeply after degree 9. Their sum plus the noise variance, the expected test error, is U-shaped with its minimum at degree 3. Training MSE falls at every degree and passes below the noise level, a warning that it rewards fitting noise.
The decomposition measured over 10,000 simulated training sets. Squared bias falls quickly and is near zero from degree 4 on; variance rises slowly, then steeply after degree 9. Their sum plus the noise variance, the expected test error, is U-shaped with its minimum at degree 3. Training MSE falls at every degree and passes below the noise level, a warning that it rewards fitting noise.

The table and figure show the tradeoff in its classical form. Squared bias falls from 0.507 for a constant to 0.004 for a cubic, and is effectively zero from degree 5 on: $\sin(5x)$ on $[0, 1]$ is well approximated by a low-degree polynomial. Variance rises with every added coefficient, slowly at first and then steeply. The expected test error is lowest at degree 3, 0.1122, and degree 12 is 6.3 times worse, 0.7074, even though its bias is essentially zero. Almost all of degree 12's error is variance. The minimum is not where the two terms are equal: at degree 3 the variance, 0.0186, is five times the squared bias, 0.0036. What is minimized is their sum, and where it bottoms out depends on how fast each term changes.

Every number in the table can also be computed exactly, without simulation. A least-squares fit is linear in the outcomes: the fitted value at $x_0$ is $\hat{f}(x_0) = \mathbf{h}(x_0)^{\mathsf{T}} \mathbf{y}$ for a weight vector $\mathbf{h}(x_0)$ that depends only on the inputs. Its mean is therefore $\mathbf{h}(x_0)^{\mathsf{T}} \mathbf{f}$, where $\mathbf{f}$ holds the true values $f(x_i)$, and its variance is $\sigma^2 \lVert \mathbf{h}(x_0) \rVert^2$ (Hastie et al., 2009, eq. 7.11). The exact totals agree with the simulated ones to within 0.0033, the largest gap being at degree 11, where the variance is large and its simulated estimate least precise.

One test input

The averages hide how the terms behave at a single point. At $x_0 = 0.9$, where $f(0.9) = -0.9775$:

Degree Mean prediction Bias Standard deviation Bias² + variance + 0.09 Measured
1 −0.6609 +0.3166 0.1149 0.2035 0.2041
3 −0.9705 +0.0071 0.1357 0.1085 0.1103
9 −0.9756 +0.0019 0.2228 0.1396 0.1427

The straight line predicts $-0.66$ on average, a bias of $+0.32$: near the right end the line runs above the falling curve, every time. Its predictions vary little from sample to sample (standard deviation 0.115). The cubic and the degree-9 polynomial are both nearly unbiased there, and differ in their spread: 0.136 against 0.223. Each row's sum agrees with the directly measured squared error to within 0.0031, a difference consistent with the sampling error of a simulation with 10,000 sets.

Where the variance comes from

For a least-squares fit with $p$ coefficients and noise variance $\sigma^2$, the variance of the fitted values averaged over the $n$ training inputs is exactly $\sigma^2 p / n$ (Hastie et al., 2009, eq. 7.12). Each coefficient adds the same amount, $\sigma^2 / n = 0.0045$ here. The simulation confirms it: at the training inputs the average variance is 0.0045 for degree 0, 0.0180 for degree 3 and 0.0585 for degree 12, exactly $0.0045 \times (d + 1)$.

Over the whole interval the variance is larger, 0.6172 for degree 12 against 0.0585 at the training inputs. A high-degree polynomial is pinned near the data points but free to swing between them, and most of all beyond the outermost points, 0.025 and 0.975. The fits in the figure show the swings near the ends. Variance is therefore a property of where predictions are made as well as of the model: a model can be stable where data are dense and erratic where they are sparse, which is another face of the danger of extrapolation.

Why training error misleads

The last column of the table is the training MSE: the average squared residual on the 20 points each model was fitted to. It falls at every degree, from 0.593 to 0.032, and from degree 3 on it is below the noise variance of 0.09, the smallest expected squared error any predictor can achieve on new outcomes. A fitted model always looks better on its own sample than it is, and a degree-12 polynomial with 13 coefficients and 20 points is partly memorizing its sample.

How much training error understates test error can be quantified for least squares. When a model with $p$ coefficients has no bias, its expected training MSE is $\sigma^2 (1 - p/n)$: for degree 12, $0.09 \times (1 - 13/20) = 0.0315$, matching the simulated 0.0316. Its expected error at the same inputs with fresh noise is $\sigma^2 (1 + p/n)$. The gap, $2 p \sigma^2 / n$, is the optimism of the training error (Hastie et al., 2009, eq. 7.24), and it grows in proportion to the number of coefficients. Mallows' $C_p$ and AIC estimate the optimism and add it back to the training error (Hastie et al., 2009, §7.5), and BIC penalizes the number of coefficients more heavily (§7.7); such criteria are tools of model selection. The more direct correction is to measure error on data the model never saw, by a held-out test set or Cross-Validation.

More data shifts the balance

The tradeoff is not fixed. Variance depends on the amount of data: $\sigma^2 p / n$ falls as $n$ grows. Bias depends mainly on the form of the model. With more data, then, more flexible models become affordable. Repeating the exact calculation with 80 training inputs instead of 20:

Expected test error against polynomial degree with 20 training points (green) and 80 (pink); circles mark each minimum. With 80 points the variance is much smaller, the curve is almost flat beyond degree 3, the best degree moves from 3 to 4, and degree 12 costs little: 0.106 against 0.707.
Expected test error against polynomial degree with 20 training points (green) and 80 (pink); circles mark each minimum. With 80 points the variance is much smaller, the curve is almost flat beyond degree 3, the best degree moves from 3 to 4, and degree 12 costs little: 0.106 against 0.707.

With $n = 80$ the minimum moves from degree 3 to degree 4 (0.0962), and degree 12 is only slightly worse (0.1063), against 0.7074 with $n = 20$. The squared bias at each degree is almost unchanged (0.0532 against 0.0533 at degree 2); the gain is all in variance. In general the best complexity grows with the amount of data, which is why a model that overfits a small study can be the right choice for a large one, and why rules of thumb such as "at most one coefficient per ten observations" are only rough guides.

A real example: nearest neighbors on the diabetes data

The simulation needed a known truth. On real data the terms cannot be separated, but the tradeoff still shows in the test error. The diabetes study used throughout the Regression topic records, for 442 patients, ten baseline measurements and a measure of disease progression one year later (Efron, Hastie, Johnstone and Tibshirani, 2004). Here progression is predicted from two of them, BMI and the serum measurement S5, by k-nearest-neighbor regression: to predict for a new patient, find the $k$ training patients closest in (standardized) BMI and S5, and average their progression. The method is described in K-Nearest Neighbors; here it matters only that $k$ controls flexibility. With $k = 1$ each prediction copies a single patient, and with $k$ equal to the whole training set every prediction is the overall mean.

For nearest neighbors the decomposition takes a simple form. Treat the training inputs as fixed and let the noise be the only randomness. The prediction at $x_0$ is the mean of $k$ outcomes, each with variance $\sigma^2$, so its variance is $\sigma^2 / k$. Its bias is the gap between $f(x_0)$ and the average of $f$ at the $k$ neighbors, which grows as $k$ grows and the neighbors reach farther away (Hastie et al., 2009, eq. 7.10):

$$\operatorname{Err}(x_0) = \sigma^2 + \Bigl[f(x_0) - \frac{1}{k}\sum_{\ell=1}^{k} f(x_{(\ell)})\Bigr]^2 + \frac{\sigma^2}{k}.$$

The 442 patients were split at random into 295 for training and 147 for testing, and the split repeated 100 times (seed 742). The table gives the mean squared errors averaged over the 100 splits, in squared units of progression.

$k$ Training MSE Test MSE Test RMSE
1 44.6 5,959.3 77.2
2 1,492.1 4,535.9 67.3
5 2,450.3 3,670.2 60.6
10 2,843.1 3,436.8 58.6
20 3,020.5 3,306.0 57.5
30 3,119.0 3,296.3 57.4
50 3,243.0 3,328.4 57.7
100 3,415.5 3,426.7 58.5
150 3,721.8 3,712.6 60.9
Nearest-neighbor regression of diabetes progression on BMI and S5, averaged over 100 random train/test splits. Flexibility increases to the right. Training MSE falls to nearly zero at k = 1, while test MSE is lowest at k = 28 and rises steeply for small k. The dotted green line, a least-squares plane on the same two inputs, beats the best k.
Nearest-neighbor regression of diabetes progression on BMI and S5, averaged over 100 random train/test splits. Flexibility increases to the right. Training MSE falls to nearly zero at k = 1, while test MSE is lowest at k = 28 and rises steeply for small k. The dotted green line, a least-squares plane on the same two inputs, beats the best k.

With $k = 1$ the training error is almost zero, because each training patient is its own nearest neighbor. It is not exactly zero only because 10 pairs of patients share the same BMI and S5. The test error at $k = 1$, 5,959, is about as bad as predicting the training mean for everyone (5,935). As $k$ grows the test error falls to its minimum at $k = 28$, 3,295.9, an RMSE of 57.4, then rises slowly as the neighborhoods grow too wide. Training and test error meet near $k = 100$, where the model is too smooth to fit even its own data closely.

The least-squares plane in the same two inputs has a mean test MSE of 3,223.1, lower than the best nearest-neighbor fit. This is the tradeoff again, between methods rather than within one. A plane has three coefficients and low variance; if the true relationship is close to linear, its bias is small too, and a far more flexible method pays in variance without gaining enough in bias (James et al., 2021, §3.5). Which side of the tradeoff a method falls on depends on the truth, not only on the method.

A single split would have been less informative. Across the 100 splits, the test MSE at $k = 28$ ranges from 2,954 to 3,754 between the 10th and 90th percentiles. Differences between nearby values of $k$ are much smaller than that spread, so the curve locates the best $k$ only roughly. Choosing a value with this uncertainty in mind is the business of cross-validation and model selection.

Ways to move along the tradeoff

Every modeling decision that changes flexibility moves a model along the curve:

  • The form of the model: polynomial degree, interaction terms, the number of inputs. Each added coefficient adds variance, and adds value only if it removes enough bias.
  • Smoothing parameters: $k$ in nearest neighbors, the depth of a tree, the number of hidden units in a network.
  • Regularization: ridge regression and the lasso keep all the inputs but shrink their coefficients toward zero. A shrunken fit is biased, deliberately, and has lower variance; if the decrease in variance exceeds the increase in squared bias, its test error is lower (Hastie et al., 2009, §7.3). The default penalty in scikit-learn's logistic regression, noted in Logistic Regression, is an example.
  • Averaging: bagging and random forests average many flexible models fitted to resampled data, which reduces variance with little increase in bias. Boosting builds a flexible model from many rigid ones, reducing bias step by step.
  • More data, which lowers variance at every level of flexibility, as the simulation showed.

The tradeoff explains why parsimony works. Among models with similar test error, the simpler one is usually more stable. The one-standard-error rule turns that into a procedure: choose the simplest model whose estimated test error is within one standard error of the best (Hastie et al., 2009, §7.10). Parsimony is a consequence of the tradeoff, not a principle that overrides it: when the truth is complex and the data plentiful, the more flexible model is the better one.

Where the classical picture stops

Classification. The decomposition above is for squared error. For a classifier judged by its error rate, the 0-1 loss, the parts do not add. Hastie et al. (2009, §7.3.1) give a simple reason: if the true probability of class 1 at some input is 0.9 and a method's average estimate is 0.6, the bias in the probability is large, yet the classification is correct, because 0.6 is still on the right side of 0.5. Estimation errors that leave a prediction on the right side of the decision boundary cost nothing, so bias and variance interact rather than add, and the best tuning for a classifier can differ from the best tuning for its probability estimates. The loss functions in Loss Functions for Classification that act on probabilities behave more like squared error. The course wiki's Vector Space Classification discusses the tradeoff for text classifiers, comparing linear classifiers with nearest neighbors.

Double descent. The U-shaped curve assumes that flexibility can only grow by adding parameters a model must fit exactly. When the number of parameters exceeds the number of observations, a least-squares fit can pass through every training point in infinitely many ways, and a method that picks the smoothest of them, such as the minimum-norm solution, can predict well again. Test error then rises to a peak near the point where the model can just interpolate the data and falls a second time beyond it. James et al. (2021, §10.8) demonstrate this double descent with spline fits to 20 points, and Belkin, Hsu, Ma and Mandal (2019) documented it for neural networks, random forests and other models, arguing that it reconciles the classical tradeoff with the practice of training very large models to zero training error. James et al. (2021, §10.8) explain why it does not contradict the tradeoff: the number of parameters stops measuring flexibility once a model interpolates, and their minimum-norm fit with 42 basis functions has lower variance than the one with 20. They add that a ridge penalty chosen well would have avoided the second peak and given lower test error, and that whether fitting to zero training error is advisable depends on how strong the signal is relative to the noise. The library's note on Superposition, Memorization, and Double Descent describes a related observation in a toy neural network.

Not a law of nature. "Reducing bias increases variance" describes what usually happens when flexibility is adjusted within one family of models with the data held fixed. Bias and variance can fall together, with more data, better inputs or a model family closer to the truth. Geman, Bienenstock and Doursat (1992) framed the "bias/variance dilemma" for neural networks as nonparametric estimators: a model flexible enough to learn anything from enough data needs, for any fixed amount of data, assumptions that introduce bias. That is the lasting point. The choice of model is a choice of what errors to risk, and only data the model has not seen can show how the choice worked out.

In Python

The simulation needs only NumPy. A minimal version, measuring squared bias and variance for one degree:

import numpy as np

rng = np.random.default_rng(742)
f = lambda x: np.sin(5 * x)
x = (np.arange(20) + 0.5) / 20                # training inputs
grid = np.linspace(0, 1, 201)                 # test inputs
degree, sigma, R = 3, 0.3, 10_000

preds = np.empty((R, grid.size))
for r in range(R):
    y = f(x) + rng.normal(0, sigma, x.size)
    coef = np.polynomial.legendre.legfit(2 * x - 1, y, degree)
    preds[r] = np.polynomial.legendre.legval(2 * grid - 1, coef)

bias2 = np.mean((preds.mean(axis=0) - f(grid)) ** 2)
variance = np.mean(preds.var(axis=0))
print(bias2, variance, bias2 + variance + sigma ** 2)   # about 0.0036, 0.0186, 0.112

The polynomial is fitted in the Legendre basis on $[-1, 1]$, which gives the same fitted values as powers of $x$ but avoids the numerical trouble that powers cause at high degrees. The loop draws different noise from the article's script, which generates all 10,000 training sets at once, so the printed values agree with the table to about three decimals. For nearest neighbors, scikit-learn's KNeighborsRegressor(n_neighbors=k) fits the model, and a StandardScaler in a pipeline puts the inputs on a common scale. The numbers in this article are computed by tools/experiments/data-science/bias-variance-tradeoff.py in the site's repository.

References

  • Belkin, M., Hsu, D., Ma, S., and Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences, 116(32), 15849–15854. record
  • Efron, B., Hastie, T., Johnstone, I., and Tibshirani, R. (2004). Least angle regression. The Annals of Statistics, 32(2), 407–499. record · 12 pages
  • Geman, S., Bienenstock, E., and Doursat, R. (1992). Neural networks and the bias/variance dilemma. Neural Computation, 4(1), 1–58. record
  • Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer. record · 23 pages
  • James, G., Witten, D., Hastie, T., and Tibshirani, R. (2021). An Introduction to Statistical Learning with Applications in R (2nd ed.). Springer. record · 30 pages