Introduction to Regression

What regression is for and the vocabulary every regression method shares. Inputs and outputs, describing a scatter plot by direction, form and strength, Anscombe's quartet, the model as a function plus error, predictions and residuals, comparing a model with the mean, and why predictions outside the data are unreliable, worked on a diabetes study of 442 patients.

In a study published by Efron, Hastie, Johnstone and Tibshirani (2004), 442 diabetes patients were measured at the start on ten variables: age, sex, body mass index (BMI), average blood pressure and six blood serum measurements. One year later, the investigators recorded a single number for each patient, a quantitative measure of how far the disease had progressed. The data are distributed with scikit-learn, so anyone can reproduce what follows. A natural first question is how progression after a year relates to BMI at the start. Do patients with higher BMI tend to progress further? By how much, on average? And if a new patient arrives with a BMI of 30, what progression should we expect?

Regression is the family of methods that answers questions of this kind: it models how a numerical outcome depends on one or more inputs, and uses that model to describe the dependence or to predict the outcome for new cases. This article sets out the vocabulary that every later article in the Data Science regression topic builds on. Simple linear regression then derives the best straight line, Linear Regression Assumptions asks when its error bars can be trusted, Multiple Linear Regression adds inputs, and Logistic Regression handles outcomes that are yes or no.

Where the word comes from

The name is historical and slightly misleading. Francis Galton (1886) compared the heights of adult children with the average height of their two parents, which he called the "mid-parentage", after scaling women's heights to be comparable with men's. He found that children of unusually tall parents were tall, but less unusual than their parents, and that children of short parents were short, but less so. In his words, the height deviation of the offspring "is, on the average, two-thirds of the height-deviate of its mid-parentage", and he called the effect "regression towards mediocrity". The tendency is now called regression to the mean. It appears whenever two measurements are imperfectly correlated, and Simple Linear Regression shows exactly why. The line Galton drew to summarize the relationship came to be called a regression line, and the name spread to the whole family of methods, including many that have nothing to do with returning toward an average.

Inputs and outputs

A regression has one output, the quantity being explained or predicted, and one or more inputs that it is explained by. Each has several names, used interchangeably in practice. James, Witten, Hastie and Tibshirani (2021, §2.1) list predictors, independent variables, features or just variables for the inputs, and response or dependent variable for the output. The diabetes study has one output, progression after a year, and ten candidate inputs.

The roles are set by the question. In a designed experiment the investigator sets the input, for example a dose of fertilizer or a drug, and observes the output, so the roles are fixed by the design. In an observational study like the diabetes data, nothing was manipulated. BMI is treated as the input because the question is about predicting progression from what is known at the start, not because anyone believes progression could alter a BMI measured a year earlier. Calling a variable "independent" does not make it a cause. A regression of progression on BMI describes how progression differs, on average, between patients whose BMI differs. Whether changing one patient's BMI would change that patient's progression is a causal question that the data alone cannot settle (Gelman, Hill and Vehtari, 2020, §10.2).

By convention the input goes on the horizontal axis of a scatter plot and the output on the vertical axis, because the model will predict vertical positions from horizontal ones. The convention matters more than it seems. The line that best predicts progression from BMI is not the same as the line that best predicts BMI from progression, as Simple Linear Regression shows, so swapping the axes changes the answer.

Regression problems are distinguished from classification problems by the type of output. When the output is a number, such as progression, blood pressure or price, the task is regression. When it is a category, such as whether a tumor is malignant or which digit an image shows, the task is classification (James et al., 2021, §2.1.5). The boundary is not sharp. Logistic regression predicts a category by first modeling a probability, which is a number, and the article on it treats it as a member of the regression family.

Describing a scatter plot

Before fitting anything, the first step is to plot the output against each input and describe what the plot shows. Three properties cover most of what matters, and a fourth catches what the other three miss.

Progression after one year against three baseline measurements for the 442 patients, each with its least-squares line. BMI shows a moderate positive relationship (r = +0.586), HDL cholesterol a weaker negative one (r = -0.395), and age only a weak one (r = +0.188), even though age ranges over sixty years.
Progression after one year against three baseline measurements for the 442 patients, each with its least-squares line. BMI shows a moderate positive relationship (r = +0.586), HDL cholesterol a weaker negative one (r = -0.395), and age only a weak one (r = +0.188), even though age ranges over sixty years.

Direction says whether the output tends to rise or fall as the input increases. Progression rises with BMI, so the relationship is positive. It falls with HDL cholesterol (the variable s3 in the dataset, often called "good" cholesterol), so that relationship is negative. A plot with no upward or downward tendency has no direction.

Form says what shape the trend follows. The BMI cloud has no visible bend, so a straight line is a reasonable summary of its trend. Other data follow a curve, such as a rise that levels off. The form decides what kind of model is appropriate: a straight line fitted to curved data will be wrong in a systematic way, too high in some regions and too low in others, no matter how carefully it is fitted.

Strength says how tightly the points cluster around the trend. The BMI relationship is moderate: the trend is clear, but patients with the same BMI differ widely in progression. For the age plot, the trend is so weak relative to the scatter that knowing a patient's age says little about their progression. The correlation coefficient $r$, defined and derived in Simple Linear Regression, measures the direction and strength of a straight-line relationship on a scale from $-1$ to $+1$. Across the ten diabetes inputs, $r$ runs from $-0.395$ for HDL to $+0.586$ for BMI, and the weakest relationship is with sex, at $+0.043$.

Unusual points are observations that do not follow the pattern of the rest. They may be recording errors, cases of a different kind, or genuine but rare values. They deserve investigation, because some of them can move a fitted model a long way.

Why the plot comes first

Anscombe (1973) made the case for plotting with four small datasets, each of eleven points, that share nearly every summary statistic. In all four the inputs have mean 9.00 and variance 11.00, the outputs have mean 7.50 and variance about 4.12, the correlation is 0.816 (0.817 for the fourth set, to three decimals), and the least-squares line is $\hat{y} = 3.00 + 0.500\,x$. The numbers were recomputed for this article from Anscombe's published table.

Anscombe's quartet. All four datasets have the same means, variances, correlation and least-squares line (orange), yet only the first is described well by that line. The second is a smooth curve, the third is a line with one outlier, and in the fourth a single point determines the slope.
Anscombe's quartet. All four datasets have the same means, variances, correlation and least-squares line (orange), yet only the first is described well by that line. The second is a smooth curve, the third is a line with one outlier, and in the fourth a single point determines the slope.

Only the first set is the kind of data the summary suggests. The second set follows a smooth curve exactly, so its form is not linear. The third lies on a straight line except for one point, which tilts the fitted line away from the other ten. In the fourth set, ten points share the same input value and the eleventh sits far to the right; the slope of the fitted line is determined entirely by that one point. A correlation of 0.816 describes all four equally, and describes three of them badly. The lesson is the reason for the order of work in every later article: plot, describe, then model.

The regression model

A regression model is a rule $f$ that takes the input values and returns a prediction. With one input $x$, the statistical model behind most regression methods is

$$Y = f(X) + \varepsilon,$$

where $f$ is a fixed but unknown function and $\varepsilon$ is a random error term with mean zero, independent of $X$ (James et al., 2021, eq. 2.1). The function $f$ is the systematic part, the average outcome at each input value. The error collects everything else that affects the outcome: other variables that were not measured, measurement error, and variation that is simply unpredictable. Regression estimates $f$ from data. The estimate is written $\hat{f}$, and its value at an input $x$ is the predicted value or fitted value

$$\hat{y} = \hat{f}(x).$$

The hat marks an estimate, distinguishing the model's prediction $\hat{y}$ from the observed value $y$. For the diabetes data, the least-squares line derived in Simple Linear Regression is

$$\widehat{\text{progression}} = -117.77 + 10.233 \times \text{BMI}.$$

The first patient in the dataset has a BMI of 32.1, so the model predicts $-117.77 + 10.233 \times 32.1 = 210.7$. That patient's observed progression was 151.

Residuals

The residual is the observed value minus the predicted value:

$$e_i = y_i - \hat{y}_i.$$

For the first patient, $e_1 = 151 - 210.7 = -59.7$. The sign has a fixed meaning. A negative residual means the observation fell below the model's prediction, so the model overpredicted, as here. A positive residual means the model underpredicted. On a scatter plot, a residual is the vertical distance from a point to the fitted line, measured in the units of the output. It is vertical, not the shortest distance to the line, because the model predicts the output from the input, and the error is in the prediction of the output.

The least-squares line for progression on BMI. Green segments are the residuals of 14 patients chosen at random: the vertical gaps between observed and predicted progression. The dotted blue line is the mean-only model, which ignores BMI. The shaded regions lie outside the range of BMI in the data, where the dashed continuation of the line is an extrapolation.
The least-squares line for progression on BMI. Green segments are the residuals of 14 patients chosen at random: the vertical gaps between observed and predicted progression. The dotted blue line is the mean-only model, which ignores BMI. The shaded regions lie outside the range of BMI in the data, where the dashed continuation of the line is an extrapolation.

The residuals of the 442 patients range from $-164.9$ to $+154.9$, and their absolute values average 51.8. A residual is also the most direct evidence about the error term $\varepsilon$, but it is not the same thing. The error is the deviation from the true, unknown $f$; the residual is the deviation from the estimate $\hat{f}$. The distinction matters in Linear Regression Assumptions, because the assumptions are stated about the errors and checked with the residuals.

How good is the model?

A model is only useful if it predicts better than not using the input at all. The natural baseline ignores BMI and predicts the overall mean progression, 152.1, for every patient. Its residuals are the deviations from the mean, and the sum of their squares is 2,621,009. The line's squared residuals sum to 1,719,582, so the line removes

$$1 - \frac{1{,}719{,}582}{2{,}621{,}009} = 0.344$$

of the squared prediction error, about a third. Measured by absolute rather than squared residuals, the average miss falls from 65.8 for the mean to 51.8 for the line. The first of these, the share of squared variation removed, is the coefficient of determination $R^2$, derived properly in Simple Linear Regression. A value of 0.344 says that BMI explains a real but limited share of why patients progress differently. Most of the variation comes from elsewhere, and Multiple Linear Regression shows how much more the other baseline variables explain.

Small residuals on average are not enough. The residuals should also show no pattern. If they are mostly positive at both ends of the input range and negative in the middle, the model is missing a curve; if their spread grows with the input, its uncertainty is not uniform. Plotting the residuals against the fitted values is therefore part of every regression analysis, and Linear Regression Assumptions works through what each pattern means.

Extrapolation

A fitted model can be evaluated at any input, but it is supported by data only where data exist. Predicting for inputs inside the observed range is interpolation. Predicting for inputs outside it is extrapolation, and the model then reports a number with no evidence behind it.

The BMI values in the diabetes study run from 18.0 to 42.2. Within that range the line is an average of what was observed. Outside it, the line simply continues. At a BMI of 60 it predicts a progression of 496.2, well above the largest value in the data, 346. At the other end, the line crosses zero at a BMI of 11.5 and predicts negative progression below that, although the smallest observed value is 25. Nothing in the data says whether the true relationship continues to rise steeply for very high BMI, flattens, or changes in some other way, because no such patients were measured. The straight line is an assumption about those regions, not a finding.

Two practical rules follow. First, report the range of the inputs used to fit a model, and flag any prediction outside it. Second, with several inputs, check the combination as well as each input separately. A patient whose BMI and blood pressure are each within the observed ranges may still have a combination of the two that no patient in the data had, and a prediction for that patient is an extrapolation even though no single input is out of range.

Extrapolation can have consequences far beyond a wrong number in a table. On the night before the Challenger launch in January 1986, the forecast temperature at launch was 31 °F, while none of the 23 earlier flights in the record had launched below 53 °F (Draper, 1993). Logistic Regression analyzes those flights and shows what the data could and could not say about a launch that cold.

The regression family

The methods in this topic share the structure set out above: a model of the outcome as a function of inputs, fitted to data by making some measure of the residuals small, judged against a simple baseline, and trusted only within the data's range.

  • Simple linear regression uses one input and a straight line, fitted by least squares.
  • Multiple linear regression uses several inputs at once. With powers and products of the inputs as extra columns, the same machinery fits curves and interactions.
  • Logistic regression models the probability of a yes-or-no outcome, so its predictions stay between 0 and 1.

Many other methods fit the same pattern with more flexible functions $f$: regression trees split the inputs into regions, neural networks compose simple nonlinear functions, and Symbolic Regression searches over formulas themselves. More flexible models can follow curves that a straight line cannot, at the cost of needing more data and being harder to interpret, a trade-off taken up in The Bias-Variance Tradeoff.

In Python

scikit-learn ships the diabetes data, and its LinearRegression class fits a least-squares line. load_diabetes(scaled=False) returns the variables in their original units; by default the inputs are centered and rescaled.

from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression

data = load_diabetes(scaled=False)
X = data.data[:, [2]]   # BMI, kept two-dimensional: one column
y = data.target         # progression after one year

model = LinearRegression().fit(X, y)
print(model.intercept_, model.coef_[0])   # about -117.77 and 10.233
y_hat = model.predict(X)                  # fitted values
residuals = y - y_hat
print(y_hat[0], residuals[0])             # about 210.7 and -59.7

The input X must be two-dimensional, with one row per patient and one column per input, even when there is a single input; selecting with [:, [2]] instead of [:, 2] keeps the column dimension. The numbers in this article are computed by tools/experiments/data-science/introduction-to-regression.py in the site's repository.

References

  • Anscombe, F. J. (1973). Graphs in statistical analysis. The American Statistician, 27(1), 17–21. record · 4 pages
  • Draper, D. (1993). Challenger USA Space Shuttle O-Ring [Data set]. UCI Machine Learning Repository. https://doi.org/10.24432/C5PW2T record · 4 pages
  • Efron, B., Hastie, T., Johnstone, I., and Tibshirani, R. (2004). Least angle regression. The Annals of Statistics, 32(2), 407–499. record · 12 pages
  • Galton, F. (1886). Regression towards mediocrity in hereditary stature. The Journal of the Anthropological Institute of Great Britain and Ireland, 15, 246–263. record
  • Gelman, A., Hill, J., and Vehtari, A. (2020). Regression and Other Stories. Cambridge University Press. record · 6 pages
  • James, G., Witten, D., Hastie, T., and Tibshirani, R. (2021). An Introduction to Statistical Learning with Applications in R (2nd ed.). Springer. record · 30 pages