The unit's hand-sized sample is twelve NHANES adults. The least-squares fit of systolic blood pressure on age, sex and body mass index (BMI) to those twelve says that pressure rises by 0.187 mm Hg per year of age and by 1.379 mm Hg per unit of BMI. One of the twelve is a 39-year-old woman with a BMI of 45.1 and a pressure of 116 mm Hg. Fit the same model without her and the age slope becomes $-0.155$, pressure now falling with age, while the BMI coefficient more than doubles to 3.017. Her BMI is the highest of the twelve, but her residual in the full fit is not remarkable by the usual yardstick. How can a single fit reveal which observations control it, and which of them disagree with the rest, without refitting the model once for every observation?
The answer rests on three quantities. The leverage $h_{ii}$ of observation $i$ measures how far its inputs lie from the center of the inputs, and how strongly its own response pulls its own fitted value. A studentized residual puts each residual on the scale of its own standard deviation, which differs from case to case. Influence measures how much the fit changes when the observation is removed; Cook's distance is the standard summary. Each of the three is a function of one fit, by exact identities proved below.
The route is as follows. First the residuals: their covariance matrix, and why it makes leverage the central quantity. Then leverage itself: its bounds, its form as a Mahalanobis distance, and what a leverage of 1 means. Then a single algebraic fact, the Sherman–Morrison formula, which gives every leave-one-out quantity from the full fit. With it come the externally studentized residual, its exact $t$ distribution under the normal linear model and the outlier test built on it, and Cook's distance with its reading as a confidence region. The twelve adults show each quantity at work, the 5,176 adults of the unit's population show what changes when $n$ is large, and a small constructed example shows where single-case diagnostics fail. The article assumes The Geometry of Least Squares for the hat matrix and The Normal Linear Model: t and F Tests for Coefficients for the $t$ test of a coefficient. Linear Regression Assumptions introduced leverage and studentized residuals for one input on the diabetes data; this article proves what that page states and goes on to deletion, influence and testing.
Notation and the running data
The notation is the unit's. The model is $\mathbf{y} = X\boldsymbol\beta + \boldsymbol\varepsilon$ with $X$ an $n \times p$ matrix of full column rank $p$ whose first column is $\mathbf{1}$, so the intercept is counted in $p$. (The Data Science pages count $p$ inputs and $p + 1$ columns; here the twelve adults have $p = 4$: the intercept, age, an indicator of female sex, and BMI.) The $i$th row of $X$ is $\mathbf{x}_i^{\mathsf T}$, and $M = X^{\mathsf T}X$. The least-squares estimate is $\hat{\boldsymbol\beta} = M^{-1}X^{\mathsf T}\mathbf{y}$, the hat matrix $H = XM^{-1}X^{\mathsf T}$ with entries $h_{ij} = \mathbf{x}_i^{\mathsf T}M^{-1}\mathbf{x}_j$, the fitted values $\hat{\mathbf{y}} = H\mathbf{y}$, the residuals $\mathbf{e} = (I - H)\mathbf{y}$, $\mathrm{RSS} = \lVert\mathbf{e}\rVert^2$ and $s^2 = \mathrm{RSS}/(n - p)$. The Gauss–Markov conditions are $E[\boldsymbol\varepsilon] = \mathbf{0}$ and $\operatorname{Cov}(\boldsymbol\varepsilon) = \sigma^2 I$; the normal linear model adds $\boldsymbol\varepsilon \sim N_n(\mathbf{0}, \sigma^2 I)$.
From The Geometry of Least Squares this article uses: $H$ is symmetric and idempotent ($H^2 = H$), it is the orthogonal projection onto the column space $\mathcal{C}(X)$, its trace is $p$, so $\sum_i h_{ii} = p$, and $0 \le h_{ii} \le 1$, with $h_{ii} \ge 1/n$ when the model has an intercept (its Theorem 2). A subscript $(i)$ marks a quantity computed with observation $i$ deleted: $\hat{\boldsymbol\beta}_{(i)}$, $\mathrm{RSS}_{(i)}$, and $s_{(i)}^2 = \mathrm{RSS}_{(i)}/(n - p - 1)$, since the deleted fit has $n - 1$ observations and $p$ coefficients.
The running data are the unit's (lmdata.py in the experiment folder). The twelve adults are drawn at random from the population with a fixed seed; with all twelve, the fit is
$$\widehat{\text{sbp}} = 71.77 + 0.187\,\text{age} - 8.67\,\text{female} + 1.379\,\text{BMI},$$
with $s = 11.44$ mm Hg. The population is the 5,176 NHANES 2017–2018 adults with a systolic pressure and a BMI, whose fit is $\widehat{\text{sbp}} = 90.05 + 0.517\,\text{age} - 1.92\,\text{female} + 0.385\,\text{BMI}$. All numbers on this page come from tools/experiments/statistics/regression-diagnostics.py; the twelve adults are computed in exact rational arithmetic, so every identity below is checked with equality, not to rounding.
Residuals are not the errors
The assumptions of the model concern the errors $\boldsymbol\varepsilon$, which are never observed; the residuals $\mathbf{e}$ are what a fit provides. The two differ in a precise way.
Proposition 1. $\mathbf{e} = (I - H)\boldsymbol\varepsilon$. Under the Gauss–Markov conditions, $E[\mathbf{e}] = \mathbf{0}$ and $\operatorname{Cov}(\mathbf{e}) = \sigma^2(I - H)$; in particular $\operatorname{Var}(e_i) = \sigma^2(1 - h_{ii})$ and $\operatorname{Cov}(e_i, e_j) = -\sigma^2 h_{ij}$ for $i \ne j$. Also $\operatorname{Cov}(\hat{\mathbf{y}}) = \sigma^2 H$, so $\operatorname{Var}(\hat y_i) = \sigma^2 h_{ii}$.
Proof. $HX = X$ because $H$ projects onto $\mathcal{C}(X)$, so $(I - H)X\boldsymbol\beta = \mathbf{0}$ and $\mathbf{e} = (I - H)(X\boldsymbol\beta + \boldsymbol\varepsilon) = (I - H)\boldsymbol\varepsilon$, whose mean is $\mathbf{0}$. The covariance of $\mathbf{e}$ is part 2 of Theorem 1 of The Gauss–Markov Theorem. For the fitted values, $\hat{\mathbf{y}} = H\mathbf{y} = X\boldsymbol\beta + H\boldsymbol\varepsilon$, and for a fixed matrix $A$, $\operatorname{Cov}(A\boldsymbol\varepsilon) = A\operatorname{Cov}(\boldsymbol\varepsilon)A^{\mathsf T}$; with $A = H$ this is $\sigma^2HH^{\mathsf T} = \sigma^2H^2 = \sigma^2H$. $\square$
Two consequences matter for everything that follows. The residuals have unequal variances even when the errors have equal ones, and the cases with high leverage have the smallest residual variance: the fit is drawn toward them. And the residuals are correlated, with $n$ residuals confined to the $(n - p)$-dimensional space $\mathcal{C}(X)^\perp$. A residual plot inspects the errors through this filter.
Example (a misreading of small residuals). It is natural to read a small residual as a sign that a case fits the model well. For the woman of the opening, $e = -16.60$ mm Hg against $s = 11.44$, a residual 1.45 standard deviations from zero on the naive scale, unremarkable among twelve. But her leverage is 0.573, so her residual has standard deviation $\sigma\sqrt{1 - 0.573}$, about $0.65\sigma$: the fit has already moved most of the way toward her. A small residual at a high-leverage point says as much about the fit as about the point.
Leverage
Leverage has three readings, and the algebra shows they are the same number. It is the weight of $y_i$ in its own fitted value: $\hat y_i = \sum_j h_{ij}y_j$, so $h_{ii} = \partial\hat y_i/\partial y_i$. It is the variance of the fitted value in units of $\sigma^2$ (Proposition 1). And, when the model has an intercept, it is a distance of $\mathbf{x}_i$ from the center of the inputs, which is the content of the next result.
Proposition 2. For every $i$:
- $h_{ii} = \sum_j h_{ij}^2$, so $\sum_{j \ne i} h_{ij}^2 = h_{ii}(1 - h_{ii})$.
- If $h_{ii} = 1$, then $h_{ij} = 0$ for every $j \ne i$, $\hat y_i = y_i$ and $e_i = 0$ whatever the data.
- If $\mathbf{1} \in \mathcal{C}(X)$, then $h_{ii} \ge 1/n$ (proved in The Geometry of Least Squares).
- If the first column of $X$ is $\mathbf{1}$, write $\mathbf{z}_i$ for the other $p - 1$ entries of row $i$, $\bar{\mathbf{z}}$ for their mean over the $n$ rows, and $Z_c$ for the $n \times (p - 1)$ matrix with rows $(\mathbf{z}_i - \bar{\mathbf{z}})^{\mathsf T}$. Then $$h_{ii} = \frac1n + (\mathbf{z}_i - \bar{\mathbf{z}})^{\mathsf T}(Z_c^{\mathsf T}Z_c)^{-1}(\mathbf{z}_i - \bar{\mathbf{z}}).$$
Proof. (1) $H = H^2 = HH^{\mathsf T}$, and the $(i, i)$ entry of $HH^{\mathsf T}$ is $\sum_j h_{ij}^2$. Subtracting $h_{ii}^2$ from both sides gives the second form. (2) If $h_{ii} = 1$, the second form of (1) gives $\sum_{j \ne i} h_{ij}^2 = 0$, so every $h_{ij}$ with $j \ne i$ vanishes and $\hat y_i = h_{ii}y_i = y_i$. (3) is part 5 of Theorem 2 of The Geometry of Least Squares. (4) The columns of $Z_c$ are the centered inputs, each orthogonal to $\mathbf{1}$, and $\mathcal{C}(X) = \operatorname{span}(\mathbf{1}) \oplus \mathcal{C}(Z_c)$, because subtracting a multiple of $\mathbf{1}$ from a column does not change the column space. The projection onto an orthogonal sum of subspaces is the sum of the two projections, so $H = P_{\mathbf 1} + Z_c(Z_c^{\mathsf T}Z_c)^{-1}Z_c^{\mathsf T}$; the $(i, i)$ entry of the second term is the quadratic form in the statement. ($Z_c$ has full column rank because $X$ does.) $\square$
Part 4 makes leverage a geometric quantity. $Z_c^{\mathsf T}Z_c/(n - 1)$ is the sample covariance matrix $S$ of the inputs, so $h_{ii} = 1/n + d_i^2/(n - 1)$, where $d_i^2 = (\mathbf{z}_i - \bar{\mathbf{z}})^{\mathsf T}S^{-1}(\mathbf{z}_i - \bar{\mathbf{z}})$ is the squared Mahalanobis distance of case $i$ from the center of the inputs. Detecting Outliers uses the same distance to flag unusual combinations of measurements. Leverage does not involve $y$ at all: it says where a case sits, not whether it fits. With one input, part 4 reduces to the formula $h_{ii} = 1/n + (x_i - \bar x)^2/S_{xx}$ that Linear Regression Assumptions states.
Since the leverages add to $p$, their average is $p/n$. Faraway (2002, §7.1) gives as a rule of thumb that leverages above $2p/n$ deserve a closer look. For the twelve adults, $p/n = 0.333$ and $2p/n = 0.667$; no adult exceeds it. The two largest leverages are 0.573, the 39-year-old woman with BMI 45.1, and 0.561, a 67-year-old woman with BMI 36.0. Both are among the four women of the sample, who make the female indicator a column with few ones; the first has the highest BMI of the twelve, and the second is the oldest of the four women. High leverage alone does not make a case influential: the 67-year-old woman's Cook's distance, defined below, is 0.021, against 1.65 for the 39-year-old.
Studentized residuals
Proposition 1 says how to put the residuals on a common scale: divide each by an estimate of its own standard deviation. Two estimates of $\sigma$ are available, and they give two kinds of studentized residual.
The internally studentized residual uses $s$ from the full fit: $$r_i = \frac{e_i}{s\sqrt{1 - h_{ii}}}.$$ It is not $t$-distributed, because $e_i$ is part of $s$. In fact it is bounded.
Proposition 3. $e_i^2 \le (1 - h_{ii})\,\mathrm{RSS}$, so $r_i^2 \le n - p$.
Proof. Let $\mathbf{u}_i$ be the $i$th unit vector. Then $e_i = \mathbf{u}_i^{\mathsf T}\mathbf{e} = \mathbf{u}_i^{\mathsf T}(I - H)\mathbf{e}$, because $\mathbf{e}$ already lies in $\mathcal{C}(X)^\perp$, and this equals $\big((I - H)\mathbf{u}_i\big)^{\mathsf T}\mathbf{e}$. By the Cauchy–Schwarz inequality, $e_i^2 \le \lVert(I - H)\mathbf{u}_i\rVert^2\lVert\mathbf{e}\rVert^2$, and $\lVert(I - H)\mathbf{u}_i\rVert^2 = \mathbf{u}_i^{\mathsf T}(I - H)\mathbf{u}_i = 1 - h_{ii}$. Dividing by $s^2(1 - h_{ii}) = \mathrm{RSS}(1 - h_{ii})/(n - p)$ gives the second inequality. $\square$
With twelve adults and four coefficients, no internally studentized residual can exceed $\sqrt{8} = 2.83$ in absolute value, however wild the observation. Worked problem 3 shows that the bound is reached exactly when a single case departs from an otherwise perfect fit.
The externally studentized residual estimates $\sigma$ without case $i$: $$t_i = \frac{e_i}{s_{(i)}\sqrt{1 - h_{ii}}}.$$ Deleting the case keeps a gross error from inflating its own yardstick. The next section shows that $s_{(i)}$ needs no refit, and the one after shows that $t_i$ has an exact $t$ distribution.
Deleting one case without refitting
Here is the plan. Deleting row $i$ changes $M = X^{\mathsf T}X$ by the rank-one matrix $\mathbf{x}_i\mathbf{x}_i^{\mathsf T}$, and the inverse of a matrix changed by a rank-one term has a closed form. Everything about the deleted fit then follows from $e_i$, $h_{ii}$ and $M^{-1}\mathbf{x}_i$.
Lemma (Sherman–Morrison). Let $M$ be invertible and $\mathbf{u}$, $\mathbf{v}$ vectors with $1 + \mathbf{v}^{\mathsf T}M^{-1}\mathbf{u} \ne 0$. Then $$(M + \mathbf{u}\mathbf{v}^{\mathsf T})^{-1} = M^{-1} - \frac{M^{-1}\mathbf{u}\mathbf{v}^{\mathsf T}M^{-1}}{1 + \mathbf{v}^{\mathsf T}M^{-1}\mathbf{u}}.$$
Proof. Write $c = \mathbf{v}^{\mathsf T}M^{-1}\mathbf{u}$, a scalar. Multiply $M + \mathbf{u}\mathbf{v}^{\mathsf T}$ by the proposed inverse: $$(M + \mathbf{u}\mathbf{v}^{\mathsf T})\Big(M^{-1} - \frac{M^{-1}\mathbf{u}\mathbf{v}^{\mathsf T}M^{-1}}{1 + c}\Big) = I + \mathbf{u}\mathbf{v}^{\mathsf T}M^{-1} - \frac{\mathbf{u}\mathbf{v}^{\mathsf T}M^{-1} + \mathbf{u}\,(\mathbf{v}^{\mathsf T}M^{-1}\mathbf{u})\,\mathbf{v}^{\mathsf T}M^{-1}}{1 + c} = I + \mathbf{u}\mathbf{v}^{\mathsf T}M^{-1}\Big(1 - \frac{1 + c}{1 + c}\Big) = I.$$ A one-sided inverse of a square matrix is the inverse. $\square$
The formula carries the names of Sherman and Morrison (1950), whose paper, by its title, treats the adjustment of an inverse when one element of the matrix changes. Cross-Validation applies it, without proof, to obtain leave-one-out prediction errors from one fit; here it is proved and pushed further. With $\mathbf{u} = -\mathbf{x}_i$ and $\mathbf{v} = \mathbf{x}_i$, $c = -h_{ii}$, so for $h_{ii} < 1$ $$\big(M - \mathbf{x}_i\mathbf{x}_i^{\mathsf T}\big)^{-1} = M^{-1} + \frac{M^{-1}\mathbf{x}_i\mathbf{x}_i^{\mathsf T}M^{-1}}{1 - h_{ii}}. \tag{1}$$
Theorem 1 (deletion identities). Suppose $h_{ii} < 1$. Then
- $\hat{\boldsymbol\beta} - \hat{\boldsymbol\beta}_{(i)} = \dfrac{M^{-1}\mathbf{x}_i\,e_i}{1 - h_{ii}}$;
- $y_i - \mathbf{x}_i^{\mathsf T}\hat{\boldsymbol\beta}_{(i)} = \dfrac{e_i}{1 - h_{ii}}$ (the deleted residual);
- $\mathrm{RSS}_{(i)} = \mathrm{RSS} - \dfrac{e_i^2}{1 - h_{ii}}$, so $(n - p - 1)\,s_{(i)}^2 = (n - p)\,s^2 - \dfrac{e_i^2}{1 - h_{ii}}$;
- $t_i = r_i\sqrt{\dfrac{n - p - 1}{n - p - r_i^2}}$, an increasing function of $r_i$ for $r_i^2 < n - p$, that is, whenever $s_{(i)} > 0$.
Proof. (1) The deleted fit is $\hat{\boldsymbol\beta}_{(i)} = (M - \mathbf{x}_i\mathbf{x}_i^{\mathsf T})^{-1}(X^{\mathsf T}\mathbf{y} - \mathbf{x}_iy_i)$. Expand with (1), writing $\mathbf{g} = M^{-1}\mathbf{x}_i$ and using $M^{-1}X^{\mathsf T}\mathbf{y} = \hat{\boldsymbol\beta}$, $\mathbf{x}_i^{\mathsf T}\hat{\boldsymbol\beta} = \hat y_i$ and $\mathbf{x}_i^{\mathsf T}\mathbf{g} = h_{ii}$: $$\hat{\boldsymbol\beta}_{(i)} = \hat{\boldsymbol\beta} - \mathbf{g}y_i + \frac{\mathbf{g}\,(\hat y_i - h_{ii}y_i)}{1 - h_{ii}} = \hat{\boldsymbol\beta} + \frac{\mathbf{g}\,\big[-y_i(1 - h_{ii}) + \hat y_i - h_{ii}y_i\big]}{1 - h_{ii}} = \hat{\boldsymbol\beta} - \frac{\mathbf{g}\,(y_i - \hat y_i)}{1 - h_{ii}}.$$ (2) Multiply (1) by $\mathbf{x}_i^{\mathsf T}$: $\mathbf{x}_i^{\mathsf T}\hat{\boldsymbol\beta}_{(i)} = \hat y_i - h_{ii}e_i/(1 - h_{ii})$, so $y_i - \mathbf{x}_i^{\mathsf T}\hat{\boldsymbol\beta}_{(i)} = e_i + h_{ii}e_i/(1 - h_{ii}) = e_i/(1 - h_{ii})$. (3) For $j \ne i$, the deleted fit's residual is $y_j - \mathbf{x}_j^{\mathsf T}\hat{\boldsymbol\beta}_{(i)} = e_j + h_{ji}e_i/(1 - h_{ii})$, by part 1 multiplied by $\mathbf{x}_j^{\mathsf T}$. Square and add over $j \ne i$, writing $a = e_i/(1 - h_{ii})$: $$\mathrm{RSS}_{(i)} = \sum_{j \ne i}e_j^2 + 2a\sum_{j \ne i}h_{ij}e_j + a^2\sum_{j \ne i}h_{ij}^2.$$ Now $\sum_j h_{ij}e_j = (H\mathbf{e})_i = 0$, because $\mathbf{e} \perp \mathcal{C}(X)$, so $\sum_{j \ne i}h_{ij}e_j = -h_{ii}e_i$; and $\sum_{j \ne i}h_{ij}^2 = h_{ii}(1 - h_{ii})$ by Proposition 2. Hence $$\mathrm{RSS}_{(i)} = \mathrm{RSS} - e_i^2 - \frac{2h_{ii}e_i^2}{1 - h_{ii}} + \frac{h_{ii}e_i^2}{1 - h_{ii}} = \mathrm{RSS} - \frac{e_i^2(1 - h_{ii}) + h_{ii}e_i^2}{1 - h_{ii}} = \mathrm{RSS} - \frac{e_i^2}{1 - h_{ii}}.$$ (4) Divide part 3 by $(n - p)s^2$: $s_{(i)}^2/s^2 = (n - p - r_i^2)/(n - p - 1)$, since $e_i^2/[(1 - h_{ii})s^2] = r_i^2$. Then $t_i = r_i\,s/s_{(i)}$ gives the formula. Its derivative in $r_i$ on $-\sqrt{n - p} < r_i < \sqrt{n - p}$ is $\sqrt{n - p - 1}\,(n - p)/(n - p - r_i^2)^{3/2} > 0$. $\square$
Part 2 is the leave-one-out identity of Cross-Validation, and part 3 says that the deleted case's contribution to the residual sum of squares is its squared deleted residual times $1 - h_{ii}$. Part 4 shows that the two studentized residuals rank the cases identically; they differ in scale, and $|t_i| \to \infty$ as $r_i^2 \to n - p$. The script checks all four parts for each of the twelve adults against an explicit refit without the case, in exact rational arithmetic, and the equalities hold exactly.
The distribution of the externally studentized residual
Here is the idea in one sentence: $t_i$ is the ordinary $t$ statistic for one extra coefficient, the coefficient of an indicator that singles out case $i$. That turns its distribution into a fact about the normal linear model.
The mean-shift outlier model for case $i$ is $$\mathbf{y} = X\boldsymbol\beta + \gamma\,\mathbf{u}_i + \boldsymbol\varepsilon,$$ in which case $i$ alone has its mean shifted by $\gamma$. Under $\gamma = 0$ it is the original model.
Theorem 2. Let $h_{ii} < 1$. In the mean-shift model fitted by least squares, $\hat\gamma = e_i/(1 - h_{ii})$, the estimate of $\boldsymbol\beta$ is $\hat{\boldsymbol\beta}_{(i)}$, the residual sum of squares is $\mathrm{RSS}_{(i)}$, and the $t$ statistic for $\gamma$ equals $t_i$. Under the normal linear model, $t_i$ has the $t$ distribution with $n - p - 1$ degrees of freedom.
Proof. The augmented design $[X, \mathbf{u}_i]$ has full column rank $p + 1$, because $\mathbf{u}_i \notin \mathcal{C}(X)$: otherwise $(I - H)\mathbf{u}_i = \mathbf{0}$ and $1 - h_{ii} = \mathbf{u}_i^{\mathsf T}(I - H)\mathbf{u}_i = 0$. Any choice of $\boldsymbol\beta$ can be paired with $\gamma = y_i - \mathbf{x}_i^{\mathsf T}\boldsymbol\beta$, which makes the residual of case $i$ zero. So minimizing the augmented sum of squares over $\gamma$ leaves $\sum_{j \ne i}(y_j - \mathbf{x}_j^{\mathsf T}\boldsymbol\beta)^2$ to be minimized over $\boldsymbol\beta$, which is the deleted fit: the minimizer is $\hat{\boldsymbol\beta}_{(i)}$, the minimum is $\mathrm{RSS}_{(i)}$, and $\hat\gamma = y_i - \mathbf{x}_i^{\mathsf T}\hat{\boldsymbol\beta}_{(i)} = e_i/(1 - h_{ii})$ by Theorem 1. For the standard error, the $t$ statistic of the last coefficient uses the last diagonal entry of the inverse of the augmented cross-product matrix $$[X, \mathbf{u}_i]^{\mathsf T}[X, \mathbf{u}_i] = \begin{pmatrix} M & \mathbf{x}_i \\ \mathbf{x}_i^{\mathsf T} & 1 \end{pmatrix}.$$ For a block matrix with invertible upper-left block $M$, the lower-right entry of the inverse is the reciprocal of the Schur complement $1 - \mathbf{x}_i^{\mathsf T}M^{-1}\mathbf{x}_i = 1 - h_{ii}$ (multiply the matrix by the candidate inverse to check). So the standard error of $\hat\gamma$ is $\hat\sigma/\sqrt{1 - h_{ii}}$, where the augmented model's variance estimate is $\hat\sigma^2 = \mathrm{RSS}_{(i)}/(n - p - 1) = s_{(i)}^2$. The same $1 - h_{ii}$ appears in the Frisch–Waugh–Lovell theorem of The Geometry of Least Squares, which writes $\hat\gamma$ as the regression of $\mathbf{y}$ on $(I - H)\mathbf{u}_i$, whose squared length is $1 - h_{ii}$. The $t$ statistic is $$\frac{e_i/(1 - h_{ii})}{s_{(i)}/\sqrt{1 - h_{ii}}} = \frac{e_i}{s_{(i)}\sqrt{1 - h_{ii}}} = t_i.$$ Under the normal linear model with $\gamma = 0$, the $t$ statistic of a single coefficient in a model with $p + 1$ coefficients has the $t$ distribution with $n - (p + 1)$ degrees of freedom, as Theorem 2 of The Normal Linear Model proves. $\square$
Two corollaries follow. Since $r_i^2/(n - p) = t_i^2/(n - p - 1 + t_i^2)$ by Theorem 1, and $t_i^2$ has the $F_{1,\,n-p-1}$ distribution, $r_i^2/(n - p)$ has the beta distribution with parameters $\tfrac12$ and $(n - p - 1)/2$ (the beta form of $F$ in Sampling Distributions from the Normal). That is the exact distribution of the internally studentized residual, and it is bounded, as Proposition 3 requires. And the deleted residual $e_i/(1 - h_{ii})$ is the estimate of the shift $\gamma$: the size of an outlier is measured by the deleted residual, not by the ordinary one.
The Bonferroni outlier test. Testing every case for a shift means $n$ tests, and the largest $|t_i|$ is what attracts attention. Faraway (2002, §7.3) applies the Bonferroni correction: declare case $i$ an outlier when $|t_i|$ exceeds $c = t_{1 - \alpha/(2n),\,n-p-1}$, the upper $\alpha/(2n)$ point of the $t$ distribution. By Theorem 2 and the union bound of Multiple Comparisons and the Bonferroni Correction, $$P\Big(\max_i |t_i| > c\Big) \le \sum_{i=1}^n P(|t_i| > c) = n\cdot\frac{\alpha}{n} = \alpha$$ when no case is shifted, whatever the dependence among the $t_i$. For the twelve adults at $\alpha = 0.05$, $c = 4.17$.
Influence and Cook's distance
A case can have high leverage and fit perfectly well, or fit badly where its leverage is small; in neither case does deleting it change much. Influence is the change itself. Cook (1977) proposed measuring it by the distance between $\hat{\boldsymbol\beta}$ and $\hat{\boldsymbol\beta}_{(i)}$ in the metric of the confidence ellipsoids for $\boldsymbol\beta$, and showed that the measure combines the studentized residual with the variances of the residual and the fitted value. Cook's distance is $$D_i = \frac{(\hat{\boldsymbol\beta} - \hat{\boldsymbol\beta}_{(i)})^{\mathsf T}X^{\mathsf T}X\,(\hat{\boldsymbol\beta} - \hat{\boldsymbol\beta}_{(i)})}{p\,s^2}.$$
Theorem 3. $$D_i = \frac{\lVert\hat{\mathbf{y}} - X\hat{\boldsymbol\beta}_{(i)}\rVert^2}{p\,s^2} = \frac{r_i^2}{p}\cdot\frac{h_{ii}}{1 - h_{ii}}.$$
Proof. The first form holds because $(\hat{\boldsymbol\beta} - \hat{\boldsymbol\beta}_{(i)})^{\mathsf T}X^{\mathsf T}X(\hat{\boldsymbol\beta} - \hat{\boldsymbol\beta}_{(i)}) = \lVert X(\hat{\boldsymbol\beta} - \hat{\boldsymbol\beta}_{(i)})\rVert^2$: it is the squared change in all $n$ fitted values. For the second, Theorem 1 gives $\hat{\boldsymbol\beta} - \hat{\boldsymbol\beta}_{(i)} = M^{-1}\mathbf{x}_ie_i/(1 - h_{ii})$, so the numerator is $$\frac{e_i^2}{(1 - h_{ii})^2}\,\mathbf{x}_i^{\mathsf T}M^{-1}MM^{-1}\mathbf{x}_i = \frac{e_i^2\,h_{ii}}{(1 - h_{ii})^2}.$$ Dividing by $ps^2$ and writing $e_i^2/[s^2(1 - h_{ii})] = r_i^2$ gives the result. $\square$
The factorization is Cook's point: $r_i^2$ measures lack of fit and $h_{ii}/(1 - h_{ii})$, the ratio $\operatorname{Var}(\hat y_i)/\operatorname{Var}(e_i)$ from Proposition 1, measures potential. A case needs both to be influential.
Reading $D_i$ as a confidence region. Under the normal linear model, the $100(1 - \alpha)\%$ confidence region for $\boldsymbol\beta$ is the ellipsoid $$\big\{\boldsymbol\beta : (\boldsymbol\beta - \hat{\boldsymbol\beta})^{\mathsf T}X^{\mathsf T}X(\boldsymbol\beta - \hat{\boldsymbol\beta}) \le p\,s^2\,F_{1-\alpha;\,p,\,n-p}\big\}$$ (Theorem 5 of The Normal Linear Model states it at the 95% level, and its proof holds at any level), with $F_{1-\alpha;\,p,\,n-p}$ the upper $\alpha$ point of the $F$ distribution. Comparing with the definition, $\hat{\boldsymbol\beta}_{(i)}$ lies exactly on the boundary of the region whose level $1 - \alpha$ satisfies $F_{1-\alpha;\,p,\,n-p} = D_i$. So $D_i$ converts into a confidence level: the share of the $F_{p,\,n-p}$ distribution below $D_i$. A $D_i$ near the median of $F_{p,\,n-p}$ means that deleting one case moves the estimate to the edge of a 50% confidence region, a large change for one case among many. Cook (1977) gives this reading, with the same 50% example, and calls the levels "descriptive levels of significance". This is a yardstick for comparison, not a test: $D_i$ does not have an $F$ distribution.
Scaled changes in one fitted value and one coefficient. Two companions of $D_i$ standardize the change by $s_{(i)}$. The change in case $i$'s own fitted value, $\hat y_i - \mathbf{x}_i^{\mathsf T}\hat{\boldsymbol\beta}_{(i)} = h_{ii}e_i/(1 - h_{ii})$ by Theorem 1, divided by $s_{(i)}\sqrt{h_{ii}}$, is
$$\frac{h_{ii}e_i}{(1 - h_{ii})\,s_{(i)}\sqrt{h_{ii}}} = t_i\sqrt{\frac{h_{ii}}{1 - h_{ii}}},$$
the externally studentized version of $\sqrt{pD_i}$. The change in coefficient $j$, the $j$th entry of $M^{-1}\mathbf{x}_ie_i/(1 - h_{ii})$, divided by its standard error $s_{(i)}\sqrt{(M^{-1})_{jj}}$, says how many standard errors one case moves each coefficient. R's influence.measures reports these as DFFITs and DFBETAs, together with Cook's distances and leverages (R Core Team, n.d.). The script checks the first identity for every one of the twelve adults.
The twelve adults, case by case
Back to the 39-year-old woman. Her leverage is $h = 0.573$ and her residual $e = -16.60$, so her internally studentized residual is $r = -16.60/(11.44\sqrt{1 - 0.573}) = -2.22$, close to the bound $\sqrt 8 = 2.83$. With her deleted, $s_{(i)} = 7.58$: she alone accounts for a large part of the residual variation. Her externally studentized residual is $t = -3.35$.
Her Cook's distance is $D = (2.22^2/4)\times 0.573/0.427 = 1.65$. The median of the $F_{4,8}$ distribution is 0.91, and 1.65 lies at about its 75th percentile ($F_{4,8}$ distribution function 0.747 at 1.65): deleting her moves $\hat{\boldsymbol\beta}$ to the edge of a 75% confidence region. The coefficients confirm it. Refitted without her,
$$\widehat{\text{sbp}} = 42.88 - 0.155\,\text{age} - 18.04\,\text{female} + 3.017\,\text{BMI},$$
and the differences $\hat{\boldsymbol\beta} - \hat{\boldsymbol\beta}_{(i)}$ (full fit minus deleted fit), in standard errors computed with the deleted $s_{(i)}$, are 2.36 for age, $-2.82$ for BMI and 1.13 for the female indicator: deleting her lowers the age slope and raises the BMI coefficient. The next largest Cook's distance among the twelve is 0.36, for a 77-year-old man with BMI 30.0 and a pressure of 112.7, whose leverage is 0.355.
Is she an outlier? Her unadjusted two-sided $p$-value from the $t_7$ distribution is 0.0123; multiplied by $n = 12$ for the Bonferroni correction it becomes 0.147, and $|t| = 3.35$ is below the critical value 4.17. The test does not call her an outlier. A case with high leverage pulls the fit toward itself and so hides part of its own discrepancy: by Proposition 1 her residual has variance only $(1 - 0.573)\sigma^2$, and the deleted residual $e/(1 - h)$, the estimated shift of Theorem 2, is $-38.9$ mm Hg against $e = -16.60$. The diagnostics separate two questions. Is she unusual given the others? Not significantly. Does the fit depend on her? Heavily. With twelve adults, the honest report is the fit with and without her: the sign of the age slope is not determined by these data.
Five thousand adults
What changes with $n$? The average leverage is $p/n = 0.00077$ and the rule-of-thumb threshold $2p/n = 0.00155$. There are 166 adults above $2p/n$, and 62 above $3p/n$; with thousands of cases, a fixed multiple of the average flags many ordinary ones. The largest leverage is 0.0110, an adult with a BMI of 84.4, 14.2 times the average. That adult's externally studentized residual is $-1.75$ and Cook's distance 0.0085 (the largest leverage, but a modest residual).
The largest Cook's distance is 0.0091, for a 66-year-old woman with BMI 58.7 and pressure 201 mm Hg, who has leverage 0.0034 and $t = 3.27$ (the largest Cook's distance among the 5,176). The median of $F_{4,\,5172}$ is 0.84, so no case comes close to moving the estimate to the edge of even a small confidence region; deleting her moves no coefficient by more than 0.18 of its standard error. In a sample this large, no single adult can control the fit, which is what the formula $D_i = (r_i^2/p)\,h_{ii}/(1 - h_{ii})$ predicts when every $h_{ii}$ is small.
The outlier test tells a different story. The Bonferroni critical value at $\alpha = 0.05$ is 4.43, and 10 adults exceed it, all of them above the fitted plane, with pressures from 200.7 to 238.0 mm Hg; the largest $t_i$ is 6.08. The test assumes normal errors, and the residuals are not normal. Their distribution is skewed to the right (skewness 0.88), and 63 internally studentized residuals lie beyond 3 in absolute value, 61 of them positive, where a normal error distribution would produce about 14 beyond 3. So the flags mix two things that the test cannot separate: a right tail heavier than the normal one, and whatever is unusual about the ten adults themselves.
A seeded simulation shows how far each can be checked. It keeps the population's design matrix $X$ and generates 200 response vectors per seed, for 30 seeds (16,000,000 + $s$). The errors are drawn in two ways:
- With normal errors, the share of datasets in which the Bonferroni test flags at least one case had median 4.5% across the 30 seeds (interquartile range 4.0% to 5.5%). The union bound guarantees at most 5%; if the 5,176 tests were independent the level would be $1 - (1 - 0.05/n)^n = 4.88\%$, and the observed shares are consistent with that.
- With errors resampled from all 5,176 standardized residuals, the test flagged at least one case in 100% of the datasets, and the mean number flagged per dataset had median 10.0 across seeds (interquartile range 9.8 to 10.1). This check is circular: the pool contains the ten flagged residuals, and each simulated dataset draws about ten values from them, so it reproduces ten flags whatever the ten adults are. Leaving the ten out of the pool would not help, because it biases the count the other way: no resampled error could then lie beyond the eleventh largest residual, so such a scheme undercounts by construction, whatever the ten are. Neither resampling scheme can separate a right tail heavier than the normal from individually unusual adults. What the data do show is the heavy right tail itself (61 of the 63 residuals beyond 3 are positive, against about 14 beyond 3 expected under normality). Whether the ten are also individually unusual needs their records: their pressures, 200.7 to 238.0 mm Hg, are high but within the range blood pressure readings can take, so the flags are a reason to look at those records, not a finding that they are errors. Refitting without the ten flagged adults gives an age slope of 0.513 (from 0.517), a female coefficient of $-1.93$ (from $-1.92$) and a BMI coefficient of 0.379 (from 0.385); no coefficient moves by more than 0.31 of its standard error. Whatever they are, they do not drive the fit. The skew of the residuals is a question for the model (a transformation of the response, or intervals that do not assume normality, as Linear Regression Assumptions discusses), not for case deletion.
When one deletion is not enough: masking
Every diagnostic above deletes one case at a time. Two similar outliers defeat that strategy, because each one, when deleted, leaves the other to hold the fit in place. This is masking, the regression form of the effect Detecting Outliers shows for z-scores; Faraway (2002, §7.3) notes that two or more outliers next to each other can hide each other.
A small constructed example shows it in exact arithmetic. Ten points lie near the line $y = x$, at $x = 1, \dots, 10$ with $y = x + \tfrac12$ at odd $x$ and $y = x - \tfrac12$ at even $x$, and two identical points sit at $(20, 4)$. The ten points alone give a slope of 0.970. All twelve points give slope 0.101 and intercept 4.45: the pair has flattened the line almost completely. Yet each of the two has leverage 0.421, externally studentized residual $-1.17$ and Cook's distance 0.48, unremarkable figures; the largest $|t_i|$ among the other ten is 1.67, larger than theirs. Deleting either one alone leaves the other, and the slope with either one deleted is only 0.220. Only deleting both reveals the problem: the line through the ten points would predict 19.56 at $x = 20$, and the observed 4 lies 28.3 residual standard deviations below that prediction.
The remedies change what is deleted or how much each case counts. They include deletion of groups of cases chosen by leverage and position, plots of the data and of partial residuals, and robust fitting methods that downweight discrepant cases from the start. They are beyond this article, and the lesson for single-case diagnostics is a limit on what they can see. A clean set of diagnostics is evidence that no single case controls the fit, not that no group of cases does.
What is proved here, what is cited, what the experiment shows
Proved on this page: the residuals as $(I - H)\boldsymbol\varepsilon$ and the covariance of the fitted values (Proposition 1, with the covariance of the residuals taken from The Gauss–Markov Theorem); the properties of leverage, including its Mahalanobis form (Proposition 2); the bound $r_i^2 \le n - p$ (Proposition 3); the Sherman–Morrison formula and the deletion identities for coefficients, deleted residuals, residual sums of squares and studentized residuals (Theorem 1); the identification of $t_i$ with the mean-shift $t$ statistic (Theorem 2); the forms of Cook's distance and of the scaled change in a fitted value (Theorem 3); and the Bonferroni bound on the outlier test.
Taken from other pages of the site: the projection properties of $H$, the bounds on leverage and the Frisch–Waugh–Lovell theorem (The Geometry of Least Squares); the $t$ distribution of a coefficient's $t$ statistic and the $F$ confidence region (The Normal Linear Model); the beta form of $F$ (Sampling Distributions from the Normal); the union bound (Multiple Comparisons and the Bonferroni Correction).
Cited: Cook's (1977) proposal of the distance and its reading against the percentage points of $F$; the $2p/n$ rule of thumb and the Bonferroni outlier procedure as Faraway (2002) presents them; the names under which R reports the scaled changes.
Computed: every number for the twelve adults, in exact arithmetic, with each identity checked against an explicit refit; the population diagnostics; and a seeded simulation in two versions, whose results are observations about this design and these error distributions: the Bonferroni test's level near its nominal value under normal errors, and about ten flags per dataset under errors resampled from all the residuals, a check that is circular and cannot separate a heavy tail from unusual individuals.
Worked problems
These problems were written for this article; each is solved with the results proved above, and the script checks every number.
Problem 1 (leverage by hand). Five observations have a single input $x = 1, 2, 3, 4, 10$ and an intercept. Find every leverage, and check that they add to $p$.
Solution. With an intercept and one input, part 4 of Proposition 2 gives $h_{ii} = 1/n + (x_i - \bar x)^2/S_{xx}$. Here $\bar x = 20/5 = 4$ and $S_{xx} = 9 + 4 + 1 + 0 + 36 = 50$. So $h = 1/5 + 9/50 = 19/50 = 0.380$ at $x = 1$; $1/5 + 4/50 = 7/25$ at $x = 2$; $1/5 + 1/50 = 11/50$ at $x = 3$; $1/5$ at $x = 4$; and $1/5 + 36/50 = 23/25 = 0.920$ at $x = 10$. The sum is $(19 + 14 + 11 + 10 + 46)/50 = 100/50 = 2 = p$. The point at $x = 10$ carries 92% of the weight of its own fitted value, and its residual has variance $\sigma^2(1 - 0.92) = 0.08\,\sigma^2$.
Problem 2 (every deletion quantity from one fit). Add the responses $y = 2, 3, 5, 4, 7$ to the inputs of Problem 1. Find the least-squares line, then the effect of deleting the point at $x = 10$ without refitting: the deleted residual, the coefficients without it, $\mathrm{RSS}_{(5)}$, both studentized residuals and Cook's distance. Check the deleted coefficients by a direct fit.
Solution. With $\bar y = 21/5$, $S_{xy} = \sum(x_i - \bar x)(y_i - \bar y) = 25$, so the slope is $25/50 = 1/2$ and the intercept $21/5 - 2 = 11/5$. The fitted values are $2.7, 3.2, 3.7, 4.2, 7.2$, the residuals $-0.7, -0.2, 1.3, -0.2, -0.2$, and $\mathrm{RSS} = 23/10$, so $s^2 = (23/10)/3 = 23/30$.
For the fifth point, $e_5 = -1/5$ and $h_{55} = 23/25$, so by Theorem 1 the deleted residual is $(-1/5)/(2/25) = -5/2$: the line through the other four points predicts 9.5 at $x = 10$, and the observed 7 is 2.5 below. The coefficient change is $M^{-1}\mathbf{x}_5\,e_5/(1 - h_{55})$; with $M = \begin{pmatrix} 5 & 20 \\ 20 & 130 \end{pmatrix}$, $M^{-1} = \frac{1}{250}\begin{pmatrix} 130 & -20 \\ -20 & 5 \end{pmatrix}$ and $M^{-1}\mathbf{x}_5 = \frac{1}{250}(130 - 200,\ -20 + 50) = (-0.28,\ 0.12)$. Multiplying by $-5/2$ gives $\hat{\boldsymbol\beta} - \hat{\boldsymbol\beta}_{(5)} = (0.7,\ -0.3)$, so $\hat{\boldsymbol\beta}_{(5)} = (11/5 - 7/10,\ 1/2 + 3/10) = (3/2,\ 4/5)$. A direct fit to the first four points agrees: there $\bar x = 5/2$, $\bar y = 7/2$, $S_{xy} = 4$, $S_{xx} = 5$, slope $4/5$ and intercept $7/2 - 2 = 3/2$.
Next, $\mathrm{RSS}_{(5)} = 23/10 - (1/25)/(2/25) = 23/10 - 1/2 = 9/5$, so $s_{(5)}^2 = (9/5)/2 = 9/10$. The internally studentized residual is $r_5 = (-1/5)/\sqrt{(23/30)(2/25)} = -0.808$ and the externally studentized one is $t_5 = (-1/5)/\sqrt{(9/10)(2/25)} = -0.745$. Deleting this point raises the variance estimate, from $s^2 = 23/30$ to $s_{(5)}^2 = 9/10$, so $|t_5| < |r_5|$; part 4 of Theorem 1 shows that this happens exactly when $r_i^2 < 1$. Cook's distance is $D_5 = (r_5^2/2)\cdot(23/25)/(2/25)$; with $r_5^2 = (1/25)/[(23/30)(2/25)] = 15/23$, $D_5 = (15/46)\times(23/2) = 15/4 = 3.75$. A point with an internally studentized residual of only $-0.808$ has a Cook's distance far above the median of $F_{2,3}$, because it carries 92% of its own fit.
Problem 3 (when the bound of Proposition 3 is reached). Suppose eleven of the twelve adults lay exactly on a plane $\mathbf{x}^{\mathsf T}\boldsymbol\beta_0$ and the twelfth, case $i$, lay $\delta \ne 0$ above it. Find $r_i$ and $t_i$.
Solution. Then $\mathbf{y} = X\boldsymbol\beta_0 + \delta\mathbf{u}_i$, so $\mathbf{e} = (I - H)\mathbf{y} = \delta(I - H)\mathbf{u}_i$. Its $i$th entry is $e_i = \delta(1 - h_{ii})$ and its squared length is $\mathrm{RSS} = \delta^2\,\mathbf{u}_i^{\mathsf T}(I - H)\mathbf{u}_i = \delta^2(1 - h_{ii})$. Hence $$r_i^2 = \frac{e_i^2}{s^2(1 - h_{ii})} = \frac{\delta^2(1 - h_{ii})^2\,(n - p)}{\delta^2(1 - h_{ii})\,(1 - h_{ii})} = n - p,$$ the bound, with equality, whatever the size of $\delta$. For twelve adults and four coefficients, $|r_i| = \sqrt 8 = 2.83$ whether the case is off by 1 mm Hg or by 1,000. Equality in the Cauchy–Schwarz step of Proposition 3 requires $\mathbf{e}$ to be a multiple of $(I - H)\mathbf{u}_i$, so this is the only way the bound is reached. And $\mathrm{RSS}_{(i)} = \mathrm{RSS} - e_i^2/(1 - h_{ii}) = 0$: the other eleven fit perfectly, $s_{(i)} = 0$ and $t_i$ is infinite. The internally studentized residual cannot register the size of a single gross error; the externally studentized one registers it without limit.
Problem 4 (Cook's distance and $t$ from $r$ and $h$). For the 39-year-old woman, $r^2 = 4.9261$ to four places and $h = 0.5728$. Recover $D$ and $t$ without the deleted fit.
Solution. By Theorem 3, $D = (r^2/p)\,h/(1 - h) = (4.9261/4)\times(0.5728/0.4272) = 1.2315 \times 1.3408 = 1.65$. By Theorem 1, with $r = -\sqrt{4.9261} = -2.2195$, $t = r\sqrt{(n - p - 1)/(n - p - r^2)} = -2.2195\times\sqrt{7/3.0739} = -2.2195\times 1.5091 = -3.35$, matching the values from the deleted fit.
Summary
| Quantity | Formula | Proved in |
|---|---|---|
| Residual covariance | $\operatorname{Cov}(\mathbf{e}) = \sigma^2(I - H)$ | Proposition 1 |
| Leverage | $h_{ii} = 1/n + d_i^2/(n - 1)$ with an intercept; average $p/n$ | Proposition 2 |
| Internally studentized | $r_i = e_i/(s\sqrt{1 - h_{ii}})$, $r_i^2 \le n - p$ | Proposition 3 |
| Deleted coefficients | $\hat{\boldsymbol\beta} - \hat{\boldsymbol\beta}_{(i)} = M^{-1}\mathbf{x}_ie_i/(1 - h_{ii})$ | Theorem 1 |
| Deleted residual | $e_i/(1 - h_{ii})$ | Theorem 1 |
| Deleted RSS | $\mathrm{RSS} - e_i^2/(1 - h_{ii})$ | Theorem 1 |
| Externally studentized | $t_i = r_i\sqrt{(n - p - 1)/(n - p - r_i^2)} \sim t_{n-p-1}$ | Theorems 1, 2 |
| Cook's distance | $D_i = (r_i^2/p)\,h_{ii}/(1 - h_{ii})$ | Theorem 3 |
What we have learned
One least-squares fit contains its own leave-one-out analysis. The hat matrix gives each case's leverage, and the Sherman–Morrison formula turns $e_i$, $h_{ii}$ and $M^{-1}\mathbf{x}_i$ into the deleted coefficients, residual, variance estimate and studentized residual, with no refitting. The externally studentized residual is the $t$ statistic of a mean shift for one case, so it has an exact $t$ distribution under the normal model, and a Bonferroni bound turns it into a test over all cases. Cook's distance factors influence into lack of fit times potential, and reads as a confidence level. On the twelve adults one woman controls the sign of the age slope without being a significant outlier; on the 5,176 adults no one controls anything, and the ten flagged cases cannot be told apart, by these checks, from the right-skewed error distribution; they do not move the fit. Masking marks the limit of the method: single deletions cannot see groups.
Diagnostics ask whether the fit depends on a few cases. When it depends instead on many correlated inputs, the coefficients are unstable for a different reason, and the remedy is to shrink them: that is the subject of Ridge and Lasso Regression.
References
- Cook, R. D. (1977). Detection of influential observation in linear regression. Technometrics, 19(1), 15–18. https://doi.org/10.1080/00401706.1977.10489493 record
- Faraway, J. J. (2002). Practical Regression and Anova using R. Contributed documentation, The R Project for Statistical Computing. https://cran.r-project.org/doc/contrib/Faraway-PRA.pdf record
- R Core Team (n.d.). Regression deletion diagnostics (
influence.measures). R documentation, package stats, R-devel. Retrieved 18 November 2026 from https://stat.ethz.ch/R-manual/R-devel/library/stats/html/influence.measures.html record - Sherman, J., and Morrison, W. J. (1950). Adjustment of an inverse matrix corresponding to a change in one element of a given matrix. The Annals of Mathematical Statistics, 21(1), 124–127. https://doi.org/10.1214/aoms/1177729893 record