In the diabetes study of Efron, Hastie, Johnstone and Tibshirani (2004), body mass index (BMI) alone explains 34.4% of the variation in disease progression after one year, as worked out in Simple Linear Regression. The study measured nine other variables at baseline, and some of them are related to progression too: average blood pressure (BP) has a correlation of 0.441 with progression, and the serum measurement S5 has 0.566. S5 is described in the scikit-learn documentation as possibly the logarithm of the serum triglycerides level. These inputs are also related to each other: BMI and BP have a correlation of 0.395, BMI and S5 0.446. Multiple linear regression uses several inputs at once. It predicts better than any one of them alone, and it answers a different question from a set of simple regressions: what each input is associated with when the others are taken into account.
The model
With $p$ inputs $x_1, \dots, x_p$, the model is
$$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_p x_p + \varepsilon,$$
and the fitted equation is $\hat{y} = b_0 + b_1 x_1 + \cdots + b_p x_p$. With two inputs the fitted equation describes a plane above the $(x_1, x_2)$ plane, and with more it describes a hyperplane, which cannot be drawn but behaves the same way. Fitted to BMI, BP and S5, least squares gives
$$\widehat{\text{progression}} = -334.88 + 6.500\,\text{BMI} + 0.903\,\text{BP} + 49.58\,\text{S5}.$$
A patient with a BMI of 30, a BP of 100 and an S5 of 5.0 is predicted to have progression $-334.88 + 6.500 \times 30 + 0.903 \times 100 + 49.58 \times 5.0 = 198.3$. Every input needs a value; a prediction from this model cannot leave one out.
Matrix form
Writing the model for all $n$ observations at once makes the algebra compact (Gelman, Hill and Vehtari, 2020, §10.7). Stack the outputs in a vector $\mathbf{y}$ of length $n$, and the inputs in an $n \times (p+1)$ design matrix $X$ whose first column is all ones, for the intercept, and whose other columns are the inputs. With the coefficients in a vector $\mathbf{b}$, the fitted values are $\hat{\mathbf{y}} = X\mathbf{b}$, and
$$\mathrm{RSS}(\mathbf{b}) = (\mathbf{y} - X\mathbf{b})^{\mathsf{T}}(\mathbf{y} - X\mathbf{b}).$$
Deriving the least-squares coefficients
Expanding the product,
$$\mathrm{RSS}(\mathbf{b}) = \mathbf{y}^{\mathsf{T}}\mathbf{y} - 2\,\mathbf{b}^{\mathsf{T}} X^{\mathsf{T}}\mathbf{y} + \mathbf{b}^{\mathsf{T}} X^{\mathsf{T}} X\,\mathbf{b}.$$
Its gradient with respect to $\mathbf{b}$ is $-2X^{\mathsf{T}}\mathbf{y} + 2X^{\mathsf{T}}X\mathbf{b}$, and setting that to zero gives the normal equations
$$X^{\mathsf{T}} X\, \mathbf{b} = X^{\mathsf{T}} \mathbf{y}.$$
If the columns of $X$ are linearly independent, $X^{\mathsf{T}}X$ is invertible and positive definite, the stationary point is the unique minimum, and
$$\mathbf{b} = (X^{\mathsf{T}} X)^{-1} X^{\mathsf{T}} \mathbf{y}.$$
With $p = 1$ these are exactly the two equations derived in Simple Linear Regression. The same equations, for scoring documents by weighted fields, appear in Zones and Weighted Zone Scoring.
The normal equations can be rewritten as $X^{\mathsf{T}}(\mathbf{y} - X\mathbf{b}) = X^{\mathsf{T}}\mathbf{e} = \mathbf{0}$: the residual vector is orthogonal to every column of $X$. Geometrically, $\hat{\mathbf{y}}$ is the orthogonal projection of $\mathbf{y}$ onto the space spanned by the columns, the closest point to $\mathbf{y}$ that any choice of coefficients can reach. Because the column of ones is among them, the residuals sum to zero, as in simple regression.
For the model with BMI, BP and S5, the matrices are
$$X^{\mathsf{T}}X = \begin{pmatrix} 442 & 11{,}658.1 & 41{,}834.0 & 2{,}051.5 \\ 11{,}658.1 & 316{,}099.9 & 1{,}114{,}060.2 & 54{,}564.1 \\ 41{,}834.0 & 1{,}114{,}060.2 & 4{,}043{,}826.5 & 195{,}422.5 \\ 2{,}051.5 & 54{,}564.1 & 195{,}422.5 & 9{,}642.2 \end{pmatrix}, \qquad X^{\mathsf{T}}\mathbf{y} = \begin{pmatrix} 67{,}243.0 \\ 1{,}861{,}676.5 \\ 6{,}571{,}949.8 \\ 322{,}152.6 \end{pmatrix}.$$
The first row holds $n$ and the sums of the three inputs; the other entries are sums of products. Solving the system gives the coefficients above, $(-334.8812, 6.5001, 0.9030, 49.5771)$, the same values scikit-learn returns. In practice software does not form the inverse explicitly. It solves the system with a matrix factorization, which is faster and more accurate when the columns are nearly dependent.
Holding the other inputs constant
The coefficient of BMI changes as inputs are added:
| Model | BMI coefficient (SE) | BP coefficient | S5 coefficient | $R^2$ | Adjusted $R^2$ |
|---|---|---|---|---|---|
| BMI | 10.233 (0.674) | 0.3439 | 0.3424 | ||
| BMI + BP | 8.519 (0.705) | 1.385 | 0.3960 | 0.3932 | |
| BMI + BP + S5 | 6.500 (0.697) | 0.903 | 49.58 | 0.4801 | 0.4765 |
Each multiple-regression coefficient describes the difference in predicted progression between patients who differ by one unit in that input and are the same in the other inputs in the model. Among patients with the same BP and S5, a one-unit difference in BMI goes with a difference of 6.50 in predicted progression. Among all patients, ignoring BP and S5, it goes with 10.23. The two numbers answer different questions, and both are correct.
The simple slope is larger because patients with higher BMI tend also to have higher BP, and higher BP goes with higher progression. Comparing patients by BMI alone, the comparison picks up some of the BP difference too. This is exact, not just a story. Fit the two-input model $\hat{y} = b_0 + b_1 x_1 + b_2 x_2$, and let $\delta$ be the slope of the simple regression of $x_2$ on $x_1$. The simple slope of $y$ on $x_1$ is $\sum (x_{1i} - \bar{x}_1)\,y_i / S_{x_1x_1}$. Substituting $y_i = b_0 + b_1 x_{1i} + b_2 x_{2i} + e_i$ and using the normal equation $\sum (x_{1i} - \bar{x}_1)\,e_i = 0$ gives
$$b_1^{\text{simple}} = b_1 + b_2\,\delta.$$
For the diabetes data, the slope of BP on BMI is $\delta = 1.2379$, and $8.5190 + 1.3847 \times 1.2379 = 10.2331$, the simple slope exactly. The difference between the simple and multiple slopes is the other input's coefficient times how strongly the two inputs move together. This is the omitted-variable effect: leaving out an input that is related to both the output and an included input shifts the included input's coefficient.
Gelman, Hill and Vehtari (2020, §10.2) recommend reading coefficients as comparisons between units rather than as the effect of changing one input in one unit, because the data alone describe comparisons between patients, not what would happen if one patient's BMI changed. They also point out that "holding the others constant" is sometimes impossible: if a model contains both $x$ and $x^2$, one cannot change $x$ with $x^2$ held fixed. The interaction and polynomial sections below meet this directly.
R squared and adjusted R squared
$R^2 = 1 - \mathrm{RSS}/\mathrm{TSS}$ keeps its meaning from simple regression, the share of variation in the output accounted for by the model, and with several inputs it equals the squared correlation between $y$ and $\hat{y}$ (James, Witten, Hastie and Tibshirani, 2021, §3.2.2). But it can never fall when an input is added, because least squares could always set the new coefficient to zero and keep the old RSS; James et al. (2021, §3.2.2) note that $R^2$ always increases when variables are added, even weakly related ones. The adjusted $R^2$ charges for each coefficient:
$$R^2_{\text{adj}} = 1 - \frac{\mathrm{RSS}/(n - p - 1)}{\mathrm{TSS}/(n - 1)},$$
which falls when an added input reduces RSS too little to pay for its extra coefficient (James et al., 2021, §6.1.3). With all ten baseline variables, $R^2 = 0.5177$ and the adjusted value is 0.5066, against 0.4801 and 0.4765 for the three-input model. Adding seven inputs buys about three points of explained variation. Adjusted $R^2$ is a rough guide; the honest test of a model's predictions is on data it was not fitted to, discussed under overfitting below.
Categorical inputs
Inputs need not be numerical. The diabetes data record sex as 1 or 2; the documentation does not say which code means which. A categorical input with two levels enters the model as an indicator (or dummy) variable that is 1 for one level and 0 for the other. The level coded 0 is the baseline, and the indicator's coefficient is the difference between the two levels. For a variable with $k$ levels, $k - 1$ indicators are used, one for each level except the baseline (James et al., 2021, §3.3.1). Including an indicator for every level would make the columns linearly dependent, since they would sum to the column of ones, and the normal equations would have no unique solution. In machine learning, building indicators for a categorical variable is called one-hot encoding (James et al., 2021, §3.3.1). The term refers to the inputs. A two-class output, as in Logistic Regression, is simply coded 0 and 1.
With an indicator for sex 2 as the only input, the fit is $\hat{y} = 149.02 + 6.65 \times \text{indicator}$: the 207 patients coded 2 average 155.67 and the 235 coded 1 average 149.02. The difference, with a standard error of 7.35, is indistinguishable from zero ($p = 0.37$). Adding the indicator to the model with BMI, BP and S5 changes its coefficient to $-13.04$ (standard error 5.46, $p = 0.017$): among patients with the same BMI, BP and S5, those coded 2 are predicted to progress 13 units less. The sign reversed. Patients coded 2 have higher average BP (98.2 against 91.5), BMI and S5, which raise their predicted progression; once those differences are accounted for, what remains is lower.
Collinearity
When inputs are strongly correlated, the data contain little information about each one's separate contribution. Two of the diabetes serum measurements, S1 (total cholesterol) and S2 (LDL cholesterol), have a correlation of 0.897. In the model with all ten inputs, S1's coefficient is $-1.090$ with standard error 0.573. Added alone to the model with BMI, BP and S5, its coefficient is $-0.284$ with standard error 0.089. The standard error is six times larger in the full model, because there S2, S3 and S4 (HDL and the ratio of total cholesterol to HDL) can largely stand in for S1.
The variance inflation factor measures this. Regress input $j$ on all the other inputs, call that fit's $R^2_j$, and set
$$\mathrm{VIF}_j = \frac{1}{1 - R^2_j}.$$
It is the factor by which the variance of $b_j$ is inflated compared with a model where input $j$ is uncorrelated with the rest; James et al. (2021, §3.3.3) give values above 5 or 10 as a common sign of a problem. In the full diabetes model, the VIFs are 59.2 for S1, 39.2 for S2, 15.4 for S3, 10.1 for S5 and 8.9 for S4, and between 1.2 and 1.5 for age, sex, BMI, BP and S6. Collinearity does not harm predictions within the range of the data. It makes individual coefficients unstable and hard to interpret, so a coefficient with a large VIF should not be read as that input's separate effect. Remedies are to drop or combine redundant inputs, or to use regularized methods such as ridge regression, which shrink the coefficients toward zero and reduce their variance (James et al., 2021, §6.2).
Interactions
The model so far is additive: the BMI slope is the same at every BP. An interaction term, the product of two inputs, lets one input's slope depend on another. Adding $\text{BMI} \times \text{BP}$:
$$\widehat{\text{progression}} = 66.45 - 1.617\,\text{BMI} - 1.489\,\text{BP} + 0.1068\,\text{BMI} \times \text{BP}.$$
Collecting the BMI terms, the BMI slope at a given BP is $-1.617 + 0.1068 \times \text{BP}$: 6.93 at a BP of 80, 8.53 at 95 and 10.14 at 110. The values 80 and 110 lie close to the 10th and 90th percentiles of BP in the data, 78.0 and 113.0. Higher BMI goes with more progression in all of them, and more steeply among patients with higher blood pressure. The interaction coefficient has standard error 0.0442 ($p = 0.016$), and $R^2$ rises from 0.3960 to 0.4040.
The individual coefficients of BMI and BP in this model, $-1.617$ and $-1.489$, look alarming but describe slopes at a BP or BMI of zero, far outside the data, and should not be read alone. Centering the inputs at their means before forming the product makes the main coefficients describe slopes at average values, which is easier to interpret (Gelman et al., 2020, §12.2). It is standard to keep both inputs in a model whenever their product is included, even if their own coefficients are not significant (the hierarchical principle; James et al., 2021, §3.3.2).
Polynomial terms
A straight line cannot follow a curve, but a model that is linear in its coefficients can. Adding a squared input gives
$$\hat{y} = b_0 + b_1 x + b_2 x^2,$$
which is a parabola in $x$. It is still a linear regression, because $\hat{y}$ is a linear combination of the columns $1$, $x$ and $x^2$ with unknown coefficients, and the normal equations solve for them as before. The second of Anscombe's (1973) datasets, which a straight line fits with $R^2 = 0.666$, is fitted by a quadratic with $R^2 = 1.00000$ to five decimals:
$$\hat{y} = -5.9957 + 2.7808\,x - 0.12671\,x^2.$$
With several inputs, a polynomial of degree $d$ includes every product of inputs whose powers add up to at most $d$. For two inputs and degree 2 the columns are $1, x_1, x_2, x_1^2, x_1 x_2, x_2^2$: the squares bend the surface and the product is an interaction. The number of columns, including the constant, is $\binom{p + d}{d}$. With 3 inputs that is 10 for degree 2 and 20 for degree 3; with 10 inputs it is 66 and 286.
More columns, worse predictions
Each added column lowers the RSS on the data the model is fitted to, so training $R^2$ always improves. Whether predictions improve for new patients is a separate question, which cross-validation answers by fitting on part of the data and measuring on the rest. Here the 442 patients are split into ten folds at random (with a fixed seed); each fold in turn is held out, the model is fitted to the other nine, and $R^2$ is computed on the held-out fold. The table reports the mean of the ten held-out values. The method is explained in Cross-Validation, and for classifiers in Evaluating Classifiers.
| Inputs | Degree | Columns (without constant) | Training $R^2$ | Cross-validated $R^2$ |
|---|---|---|---|---|
| BMI, BP, S5 | 1 | 3 | 0.480 | 0.459 |
| BMI, BP, S5 | 2 | 9 | 0.495 | 0.458 |
| BMI, BP, S5 | 3 | 19 | 0.514 | 0.448 |
| all ten | 1 | 10 | 0.518 | 0.484 |
| all ten | 2 | 65 | 0.592 | 0.408 |
| all ten | 3 | 285 | 0.798 | −14.792 |
With three inputs, the quadratic and cubic terms raise training $R^2$ slightly and leave cross-validated $R^2$ flat or lower: the relationships here have little curvature. With all ten inputs, degree 2 already predicts worse than the plain linear model, and degree 3, with 285 columns fitted to about 398 patients in each training split, fits the training data far better ($R^2 = 0.798$) and new patients catastrophically. Its held-out $R^2$ averages $-14.8$ over the ten folds (median $-3.74$). A held-out $R^2$ of $-14.8$ means squared errors 15.8 times as large as those of simply predicting the held-out fold's mean: $R^2$ computed on new data has no lower bound. This is overfitting: a model flexible enough to fit the noise in its training data predicts new data worse, the subject of The Bias-Variance Tradeoff. (Sex takes only the values 1 and 2, so its square equals $3 \times \text{sex} - 2$, a combination of columns already present, and some of the 285 columns are redundant; least squares still finds a solution, but it is no better behaved.)
Choosing which inputs and terms to include is model selection, covered in Model Selection. For choosing among many candidate inputs in text classification, the course wiki's Feature Selection describes the standard filters.
In Python
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import KFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
data = load_diabetes(scaled=False)
X, y = data.data, data.target
X3 = X[:, [2, 3, 8]] # BMI, BP, S5
model = LinearRegression().fit(X3, y)
print(model.intercept_, model.coef_) # -334.88, [6.500 0.903 49.58]
print(model.predict([[30, 100, 5.0]])) # 198.3
folds = KFold(10, shuffle=True, random_state=0)
for degree in (1, 2, 3):
poly = make_pipeline(PolynomialFeatures(degree, include_bias=False), LinearRegression())
print(degree, cross_val_score(poly, X, y, cv=folds, scoring="r2").mean())
PolynomialFeatures builds the columns of powers and products, and LinearRegression fits a linear model to whatever columns it receives; it neither knows nor needs to know that some are squares of others. Wrapping the two in a pipeline ensures that, during cross-validation, the columns for each held-out fold are built in the same way as for the training folds. The numbers in this article are computed by tools/experiments/data-science/multiple-linear-regression.py in the site's repository.
References
- Anscombe, F. J. (1973). Graphs in statistical analysis. The American Statistician, 27(1), 17–21. record · 4 pages
- Efron, B., Hastie, T., Johnstone, I., and Tibshirani, R. (2004). Least angle regression. The Annals of Statistics, 32(2), 407–499. record · 12 pages
- Gelman, A., Hill, J., and Vehtari, A. (2020). Regression and Other Stories. Cambridge University Press. record · 6 pages
- James, G., Witten, D., Hastie, T., and Tibshirani, R. (2021). An Introduction to Statistical Learning with Applications in R (2nd ed.). Springer. record · 30 pages