Introduction to Regression described the relationship between body mass index (BMI) and disease progression in a diabetes study of 442 patients (Efron, Hastie, Johnstone and Tibshirani, 2004) as positive, roughly straight and moderate in strength. Simple linear regression turns that description into a formula: one input $x$, one output $y$, and a straight line
$$\hat{y} = b_0 + b_1 x,$$
where $b_0$ is the intercept, the predicted output when $x = 0$, and $b_1$ is the slope, the difference in predicted output between two cases whose inputs differ by one unit. Infinitely many lines could be drawn through the scatter plot. This article derives which one is best under the standard criterion, works the calculation by hand on five patients, then applies it to all 442 and asks how much the line explains and how precisely its slope is known.
The least-squares criterion
A line makes a prediction $\hat{y}_i = b_0 + b_1 x_i$ for each observation, and misses by the residual $e_i = y_i - \hat{y}_i$. A good line has small residuals, but there are many ways to combine $n$ residuals into one measure of size. Adding them does not work, because positive and negative residuals cancel, and many poor lines have residuals summing to zero. The standard choice adds their squares, the residual sum of squares
$$\mathrm{RSS}(b_0, b_1) = \sum_{i=1}^{n} e_i^2 = \sum_{i=1}^{n} \bigl(y_i - b_0 - b_1 x_i\bigr)^2,$$
and the least-squares line is the pair $(b_0, b_1)$ that makes RSS as small as possible (James, Witten, Hastie and Tibshirani, 2021, §3.1.1, where RSS is eq. 3.3). The same quantity is sometimes called the sum of squared errors, SSE.
Squaring has three consequences. It makes the problem easy to solve exactly, because RSS is a smooth bowl-shaped function of the two coefficients with a single lowest point, found by calculus below. It is the right criterion when the errors follow a normal distribution, in the sense that least squares then gives the maximum likelihood estimates (Gelman, Hill and Vehtari, 2020, §8.1). And it weights large residuals heavily: a residual of 6 contributes 36, four times as much as a residual of 3. The least-squares line therefore works hard to avoid large misses, which makes it sensitive to outliers, as the third of Anscombe's datasets shows. Minimizing the sum of absolute residuals instead gives a line that is more robust to outliers but has no closed-form solution.
The method itself is old. Legendre published it in 1805 and Gauss in 1809, and Gauss claimed to have used it earlier. That priority dispute is, in Stigler's (1981) words, the most famous in the history of statistics; Stigler argues, "though not conclusively", that Gauss probably had the method before Legendre but did not succeed in communicating it.
Deriving the line
The minimum of RSS is where both partial derivatives are zero. Differentiating with respect to the intercept,
$$\frac{\partial\,\mathrm{RSS}}{\partial b_0} = -2 \sum_{i=1}^{n} \bigl(y_i - b_0 - b_1 x_i\bigr) = 0,$$
and with respect to the slope,
$$\frac{\partial\,\mathrm{RSS}}{\partial b_1} = -2 \sum_{i=1}^{n} x_i \bigl(y_i - b_0 - b_1 x_i\bigr) = 0.$$
Dropping the factor $-2$, these are two linear equations in the two unknowns, called the normal equations:
$$\sum_{i} e_i = 0, \qquad \sum_{i} x_i\, e_i = 0.$$
The first says the residuals sum to zero. Writing it out, $\sum y_i = n b_0 + b_1 \sum x_i$, and dividing by $n$ gives
$$\bar{y} = b_0 + b_1 \bar{x}, \qquad\text{so}\qquad b_0 = \bar{y} - b_1 \bar{x},$$
where $\bar{x}$ and $\bar{y}$ are the sample means. The least-squares line always passes through the point of means $(\bar{x}, \bar{y})$. Substituting this $b_0$ into the second equation turns each residual into $e_i = (y_i - \bar{y}) - b_1 (x_i - \bar{x})$. Because the residuals sum to zero, $\sum x_i e_i = \sum (x_i - \bar{x}) e_i$, and the second equation becomes
$$\sum_{i} (x_i - \bar{x})\bigl[(y_i - \bar{y}) - b_1 (x_i - \bar{x})\bigr] = 0.$$
Solving for $b_1$:
$$b_1 = \frac{\sum_{i}(x_i - \bar{x})(y_i - \bar{y})}{\sum_{i}(x_i - \bar{x})^2} = \frac{S_{xy}}{S_{xx}},$$
where $S_{xx}$ is the sum of squared deviations of $x$ from its mean and $S_{xy}$ is the sum of products of the deviations of $x$ and $y$. This is equation 3.4 of James et al. (2021). The second derivatives are $2n$, $2\sum x_i^2$ and $2\sum x_i$, which form a positive definite matrix whenever the $x_i$ are not all equal, so this stationary point is the unique minimum. If all the $x_i$ were equal, $S_{xx}$ would be zero and no slope could be estimated, which makes sense: data at a single input value say nothing about how the output changes with the input.
The two normal equations have a geometric reading worth remembering. The residuals are uncorrelated with the input: whatever the line has not explained shows no straight-line relationship with $x$. If it did, tilting the line further would reduce RSS.
Five patients by hand
The first five patients in the dataset have these BMI values and progression measures:
| Patient | BMI $x$ | Progression $y$ | $x - \bar{x}$ | $y - \bar{y}$ | $(x - \bar{x})(y - \bar{y})$ | $(x - \bar{x})^2$ |
|---|---|---|---|---|---|---|
| 1 | 32.1 | 151 | 5.6 | 9.4 | 52.64 | 31.36 |
| 2 | 21.6 | 75 | −4.9 | −66.6 | 326.34 | 24.01 |
| 3 | 30.5 | 141 | 4.0 | −0.6 | −2.40 | 16.00 |
| 4 | 25.3 | 206 | −1.2 | 64.4 | −77.28 | 1.44 |
| 5 | 23.0 | 135 | −3.5 | −6.6 | 23.10 | 12.25 |
| Sum | 132.5 | 708 | 0 | 0 | 322.40 | 85.06 |
The means are $\bar{x} = 132.5/5 = 26.50$ and $\bar{y} = 708/5 = 141.60$. The slope and intercept are
$$b_1 = \frac{322.40}{85.06} = 3.7903, \qquad b_0 = 141.60 - 3.7903 \times 26.50 = 41.158.$$
The fitted values and residuals:
| Patient | Fitted $\hat{y}$ | Residual $e$ | $e^2$ |
|---|---|---|---|
| 1 | 162.83 | −11.83 | 139.8 |
| 2 | 123.03 | −48.03 | 2,306.7 |
| 3 | 156.76 | −15.76 | 248.4 |
| 4 | 137.05 | 68.95 | 4,753.9 |
| 5 | 128.33 | 6.67 | 44.4 |
The residuals sum to zero and so does the sum of each residual times its BMI, to rounding, as the normal equations require. RSS is 7,493.22. Values are rounded in the tables; the squares are computed from unrounded residuals.
Five patients make a noisy sample. Patient 4 has a near-average BMI but the highest progression, and the line, which must keep that large residual in check, tilts less than it would without that patient. The slope of 3.79 from five patients is far from the 10.23 found from all 442 below. Estimates from small samples vary, and a later section measures how much.
All 442 patients
The same formulas applied to the full study give $\bar{x} = 26.376$, $\bar{y} = 152.133$, and
$$\widehat{\text{progression}} = -117.773 + 10.2331 \times \text{BMI}.$$
The slope says that patients whose BMI differs by one unit differ in predicted progression by 10.23 units. The careful reading is a comparison between patients, not a statement about what would happen to a patient whose BMI changed: the data are observational, and patients with higher BMI may differ in other ways too (Gelman, Hill and Vehtari, 2020, §10.2). Multiple Linear Regression shows that the BMI slope falls once blood pressure and a serum measurement are taken into account.
The intercept of $-117.77$ is the predicted progression at a BMI of zero, which is physically impossible and far below the smallest BMI in the data, 18.0. The intercept is needed to place the line, but here it has no meaning of its own: interpreting it is an extrapolation, in the sense of Introduction to Regression. Centering the input, regressing on $x - \bar{x}$ instead of $x$, would make the intercept equal to $\bar{y}$, the predicted value for a patient of average BMI, without changing the slope.
The bowl in the figure is the least-squares criterion seen directly. Lines through the means with slopes of 8 or 12 have RSS of 1,762,510 and 1,746,455, both above the minimum of 1,719,582.
Correlation and the slope
The correlation coefficient measures the direction and strength of a straight-line relationship without regard to units. With sample standard deviations $s_x = \sqrt{S_{xx}/(n-1)}$ and $s_y = \sqrt{S_{yy}/(n-1)}$, where $S_{yy} = \sum (y_i - \bar{y})^2$,
$$r = \frac{1}{n - 1} \sum_{i=1}^{n} \left(\frac{x_i - \bar{x}}{s_x}\right)\left(\frac{y_i - \bar{y}}{s_y}\right) = \frac{S_{xy}}{\sqrt{S_{xx}\, S_{yy}}}.$$
Each term is the product of the two variables' z-scores, their distances from the mean in units of standard deviation. Points in the upper-right and lower-left quadrants around the means contribute positive products, points in the other two quadrants negative ones, and $r$ is their average, which by the Cauchy-Schwarz inequality lies between $-1$ and $+1$. It equals $\pm 1$ only when all points lie exactly on a line. A value near zero means no straight-line relationship, which is not the same as no relationship: points on a symmetric parabola can have $r = 0$ exactly.
Comparing the formula for $r$ with $b_1 = S_{xy}/S_{xx}$ shows that
$$b_1 = r \,\frac{s_y}{s_x}.$$
The slope is the correlation rescaled from standard-deviation units into the units of the data. For the diabetes data, $r = 0.5865$, $s_x = 4.4181$ and $s_y = 77.0930$, and $0.5865 \times 77.0930 / 4.4181 = 10.233$, the slope found above.
Regression to the mean
Rewriting the line with $b_0 = \bar{y} - b_1 \bar{x}$ and $b_1 = r\, s_y/s_x$ gives
$$\frac{\hat{y} - \bar{y}}{s_y} = r\,\frac{x - \bar{x}}{s_x}.$$
In standardized units, the prediction is $r$ times the input. A patient one standard deviation above the mean BMI (a BMI of 30.79) is predicted to be $r = 0.586$ standard deviations above the mean progression, a prediction of 197.3, not a full standard deviation. Because $|r| < 1$ unless the fit is perfect, predictions are always closer to the mean, in standardized units, than the inputs they come from. This is Galton's regression toward mediocrity, described in Introduction to Regression. It requires no causal mechanism, only imperfect correlation. The same reasoning explains why a patient selected for an extreme first measurement will tend to be less extreme on the second, even with no treatment at all.
The asymmetry also means the regression of $y$ on $x$ is a different line from the regression of $x$ on $y$. Predicting BMI from progression gives $\widehat{\text{BMI}} = 21.263 + 0.03361 \times \text{progression}$; drawn in the same axes, with progression vertical, that line has slope $1/0.03361 = 29.75$, much steeper than 10.23. Each line minimizes squared errors in its own output, and they coincide only when $r = \pm 1$.
How much the line explains
The total sum of squares, $\mathrm{TSS} = \sum (y_i - \bar{y})^2$, is the RSS of the horizontal line at the mean, the model that ignores the input. It splits into two parts. Writing $y_i - \bar{y} = (\hat{y}_i - \bar{y}) + e_i$ and squaring,
$$\sum_i (y_i - \bar{y})^2 = \sum_i (\hat{y}_i - \bar{y})^2 + \sum_i e_i^2 + 2 \sum_i (\hat{y}_i - \bar{y})\, e_i.$$
The cross term is zero: $\hat{y}_i - \bar{y} = b_1 (x_i - \bar{x})$, and $\sum (x_i - \bar{x}) e_i = 0$ by the normal equations. So
$$\mathrm{TSS} = \mathrm{ESS} + \mathrm{RSS},$$
where $\mathrm{ESS} = \sum (\hat{y}_i - \bar{y})^2$ is the explained sum of squares, the variation of the fitted values around the mean. The coefficient of determination is the explained share,
$$R^2 = \frac{\mathrm{ESS}}{\mathrm{TSS}} = 1 - \frac{\mathrm{RSS}}{\mathrm{TSS}}$$
(James et al., 2021, eq. 3.17). For the diabetes data, TSS is 2,621,009, ESS is 901,427 and RSS is 1,719,582, which add up as they should, and $R^2 = 0.3439$: BMI accounts for about 34% of the variation in progression.
For simple linear regression, $R^2$ is the square of the correlation. Substituting $\hat{y}_i - \bar{y} = b_1(x_i - \bar{x})$ gives $\mathrm{ESS} = b_1^2 S_{xx} = S_{xy}^2 / S_{xx}$, so
$$R^2 = \frac{S_{xy}^2}{S_{xx}\, S_{yy}} = r^2.$$
Here $0.5865^2 = 0.3439$, matching. The identity holds only with one input; with several, $R^2$ is the squared correlation between $y$ and the fitted values $\hat{y}$ (James et al., 2021, §3.2.2).
Two cautions apply. First, $R^2$ lies between 0 and 1 for a least-squares fit with an intercept, evaluated on the data it was fitted to. Evaluated on new data, $1 - \mathrm{RSS}/\mathrm{TSS}$ can be negative when a model predicts worse than the new data's mean, and Multiple Linear Regression shows an example. Second, a low $R^2$ does not mean the slope is unimportant or poorly estimated. BMI explains a third of the variation in progression, and yet, as the next section shows, its slope is known quite precisely. What $R^2$ measures is how well individual outcomes can be predicted, and individuals vary for many reasons besides BMI.
How precisely the slope is known
The five-patient slope of 3.79 and the full-study slope of 10.23 differ because each sample gives a different estimate. To say how far an estimate might be from the truth, regression needs a model of how the data arose. The standard one is
$$Y = \beta_0 + \beta_1 X + \varepsilon,$$
with true but unknown coefficients $\beta_0$ and $\beta_1$, and errors $\varepsilon$ that have mean zero, a common variance $\sigma^2$, and are uncorrelated with each other (James et al., 2021, §3.1.2). Under that model, the least-squares estimates are unbiased, and their standard errors are
$$\mathrm{SE}(b_1) = \frac{\sigma}{\sqrt{S_{xx}}}, \qquad \mathrm{SE}(b_0) = \sigma \sqrt{\frac{1}{n} + \frac{\bar{x}^2}{S_{xx}}}$$
(James et al., 2021, eq. 3.8). The slope is known more precisely when the errors are small, when there are more observations, and when the inputs are spread out: a line through points spread over a wide range is harder to tilt than one through a narrow cluster.
The error standard deviation $\sigma$ is estimated by the residual standard error, $\mathrm{RSE} = \sqrt{\mathrm{RSS}/(n-2)}$, dividing by $n - 2$ because two coefficients were estimated. For the diabetes data, RSE is 62.52 on 440 degrees of freedom: a typical patient's progression is about 63 units from the line. Then
$$\mathrm{SE}(b_1) = \frac{62.52}{\sqrt{S_{xx}}} = 0.6738.$$
If the errors are also normally distributed, $(b_1 - \beta_1)/\mathrm{SE}(b_1)$ has a $t$ distribution with $n - 2$ degrees of freedom, and a 95% confidence interval for the slope is $b_1 \pm t_{0.975,\,n-2}\,\mathrm{SE}(b_1)$. With $t_{0.975,\,440} = 1.9654$, the interval is $[8.909, 11.557]$. To test whether BMI is related to progression at all, the null hypothesis $\beta_1 = 0$ gives $t = 10.2331/0.6738 = 15.19$, and a $p$-value of about $3.5 \times 10^{-42}$ (eq. 3.14 of James et al., 2021). The same calculations appear for comparing retrieval systems in Comparing Systems with Significance Tests. The evidence that the slope is positive is overwhelming, even though $R^2$ is only 0.34.
Predicting the mean and predicting a patient
The line can answer two different questions about patients with a BMI of 30. The predicted value is $-117.773 + 10.2331 \times 30 = 189.2$ either way, but the uncertainty differs.
- The average progression of all such patients. Its uncertainty comes only from not knowing the true line. The standard error is $\mathrm{RSE}\sqrt{1/n + (x_0 - \bar{x})^2/S_{xx}}$, and the 95% confidence interval is $[181.7, 196.8]$.
- The progression of one new patient. Now the patient's own error $\varepsilon$ adds to the uncertainty about the line, and the standard error is $\mathrm{RSE}\sqrt{1 + 1/n + (x_0 - \bar{x})^2/S_{xx}}$. The 95% prediction interval is $[66.1, 312.3]$.
At a BMI of 30 the prediction interval is 246.2 units wide and the confidence interval only 15.1 units, because individual patients scatter far more than the line is uncertain. As a rough check of the model, 426 of the 442 patients (96.4%) lie within their own 95% prediction intervals. Both intervals widen as $x_0$ moves away from $\bar{x}$, and both depend on assumptions about the errors that Linear Regression Assumptions examines. The least-squares line itself requires none of them: it is the line with the smallest RSS whatever the errors are like. The assumptions matter for the standard errors, intervals and tests.
When a straight line is appropriate
A practical order of work:
- Plot the data. If the points follow a curve, a straight line will be wrong in a systematic way, whatever the value of $r$. Multiple Linear Regression shows how to fit curves with the same machinery.
- Compute $r$ and fit the line. A correlation near zero with a straight form means the input says little about the output; the line is valid but not useful for prediction.
- Plot the residuals against the fitted values. They should scatter evenly around zero with no curve or funnel.
- Report the slope with its interval, and $R^2$, and say over what range of inputs the line was fitted.
In Python
scikit-learn fits the line; scipy.stats.linregress also returns $r$, the standard error of the slope and its $p$-value.
from scipy import stats
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
data = load_diabetes(scaled=False)
bmi, y = data.data[:, 2], data.target
model = LinearRegression().fit(bmi.reshape(-1, 1), y)
print(model.intercept_, model.coef_[0]) # -117.77, 10.233
print(model.score(bmi.reshape(-1, 1), y)) # R^2 = 0.3439
fit = stats.linregress(bmi, y)
print(fit.rvalue, fit.stderr, fit.pvalue) # 0.5865, 0.6738, about 3.5e-42
model.score returns $R^2$ on whatever data it is given, which is the training data here. The numbers in this article, including the intervals, are computed by tools/experiments/data-science/simple-linear-regression.py in the site's repository.
References
- Efron, B., Hastie, T., Johnstone, I., and Tibshirani, R. (2004). Least angle regression. The Annals of Statistics, 32(2), 407–499. record · 12 pages
- Gelman, A., Hill, J., and Vehtari, A. (2020). Regression and Other Stories. Cambridge University Press. record · 6 pages
- James, G., Witten, D., Hastie, T., and Tibshirani, R. (2021). An Introduction to Statistical Learning with Applications in R (2nd ed.). Springer. record · 30 pages
- Stigler, S. M. (1981). Gauss and the invention of least squares. The Annals of Statistics, 9(3), 465–474. record