Among the unit's twelve adults, the youngest is 19 and the oldest 77. The difference of their pressures divided by the difference of their ages, $58$ years, estimates how fast pressure rises with age. It is unbiased if pressure follows a straight line in age with errors of mean zero. So is the least squares slope of Simple Linear Regression, which uses all twelve. Under equal error variances the two-point slope has 2.726 times the variance of the least squares slope. Is that an accident of these twelve ages, or can no linear unbiased estimator ever beat least squares?
The Gauss–Markov theorem says it cannot. Suppose the errors have mean zero, a common variance and no correlation. Then among all estimators that are linear in the responses and unbiased for every value of the coefficients, least squares has the smallest variance. That holds for every coefficient and every linear combination of coefficients. The conditions say nothing about normality or about the shape of the error distribution.
First we compute the mean and covariance of the least squares estimator and the expected residual sum of squares. Then we prove the theorem by one application of Pythagoras, in scalar and in matrix form. Next we show that under normal errors the word "linear" can be dropped. Then we drop the condition of uncorrelated, equal-variance errors and prove Aitken's theorem for generalized least squares, with the exact condition under which ordinary least squares remains best. Finally a seeded experiment shows what happens to the comparison when the conditions fail or nonlinear estimators are allowed. The article assumes The Geometry of Least Squares, whose notation it uses: $X$ is $n \times p$ with the intercept column counted in $p$, $H$ is the hat matrix, and $\mathbf{e} = (I - H)\mathbf{y}$.
The model and the conditions
The responses are random variables $Y_1, \ldots, Y_n$, collected in $\mathbf{Y}$, with
$$\mathbf{Y} = X\boldsymbol\beta + \boldsymbol\varepsilon,$$
where $X$ is a fixed $n \times p$ matrix, $\boldsymbol\beta$ an unknown vector of $p$ coefficients and $\boldsymbol\varepsilon$ a random vector of errors. The Gauss–Markov conditions are
$$E[\boldsymbol\varepsilon] = \mathbf{0}, \qquad \operatorname{Cov}(\boldsymbol\varepsilon) = \sigma^2 I,$$
that is, every error has mean zero and variance $\sigma^2$, and different errors are uncorrelated. Nothing else is assumed: the errors need not be normal, need not have the same distribution, and need not be independent. When the predictors are random, as ages drawn from a survey are, everything below holds conditionally on their observed values.
Two facts about random vectors are used throughout. For a fixed matrix $A$ and vector $\mathbf{c}$, $E[A\mathbf{Y} + \mathbf{c}] = AE[\mathbf{Y}] + \mathbf{c}$ and $\operatorname{Cov}(A\mathbf{Y}) = A\operatorname{Cov}(\mathbf{Y})A^{\mathsf T}$. Both are linearity of expectation and the bilinearity of covariance, applied entry by entry, from Theorems 3 and 6 of Expectation, Variance and Covariance. In particular a linear function $\mathbf{a}^{\mathsf T}\mathbf{Y}$ has variance $\sigma^2\lVert\mathbf{a}\rVert^2$ under the conditions.
An estimator of $\mathbf{c}^{\mathsf T}\boldsymbol\beta$, a linear combination of the coefficients such as a single slope or the mean response at given predictor values, is linear if it has the form $\mathbf{a}^{\mathsf T}\mathbf{Y}$ for a fixed vector $\mathbf{a}$, and unbiased if its expectation is $\mathbf{c}^{\mathsf T}\boldsymbol\beta$ for every $\boldsymbol\beta$. Since $E[\mathbf{a}^{\mathsf T}\mathbf{Y}] = \mathbf{a}^{\mathsf T}X\boldsymbol\beta$, a linear estimator is unbiased exactly when
$$X^{\mathsf T}\mathbf{a} = \mathbf{c}, \qquad (1)$$
because two linear functions of $\boldsymbol\beta$ agree for every $\boldsymbol\beta$ exactly when their coefficient vectors agree. So a linear unbiased estimator of $\mathbf{c}^{\mathsf T}\boldsymbol\beta$ exists exactly when $\mathbf{c}$ is a combination of the rows of $X$: when $\mathbf{c}^{\mathsf T}\boldsymbol\beta$ is estimable, in the sense of Theorem 6 of The Geometry of Least Squares.
The moments of least squares
Theorem 1 (moments). Under the Gauss–Markov conditions, with $X$ of rank $p$:
- $E[\hat{\boldsymbol\beta}] = \boldsymbol\beta$ and $\operatorname{Cov}(\hat{\boldsymbol\beta}) = \sigma^2(X^{\mathsf T}X)^{-1}$;
- $\operatorname{Cov}(\mathbf{e}) = \sigma^2(I - H)$, so $\operatorname{Var}(e_i) = \sigma^2(1 - h_{ii})$, and $\operatorname{Cov}(\hat{\boldsymbol\beta}, \mathbf{e}) = 0$;
- $E[\mathrm{RSS}] = (n - p)\sigma^2$, so $s^2 = \mathrm{RSS}/(n - p)$ is unbiased for $\sigma^2$.
Proof. (1) $\hat{\boldsymbol\beta} = (X^{\mathsf T}X)^{-1}X^{\mathsf T}\mathbf{Y}$ has mean $(X^{\mathsf T}X)^{-1}X^{\mathsf T}X\boldsymbol\beta = \boldsymbol\beta$ and covariance $(X^{\mathsf T}X)^{-1}X^{\mathsf T}(\sigma^2 I)X(X^{\mathsf T}X)^{-1} = \sigma^2(X^{\mathsf T}X)^{-1}$.
(2) $\mathbf{e} = (I - H)\mathbf{Y}$, and $I - H$ is symmetric and idempotent, so $\operatorname{Cov}(\mathbf{e}) = \sigma^2(I - H)(I - H)^{\mathsf T} = \sigma^2(I - H)$. The cross-covariance is $(X^{\mathsf T}X)^{-1}X^{\mathsf T}(\sigma^2 I)(I - H) = \sigma^2(X^{\mathsf T}X)^{-1}\big[(I - H)X\big]^{\mathsf T} = 0$, because $HX = X$.
(3) For any fixed symmetric $M$ and random vector $\mathbf{Y}$ with mean $\boldsymbol\mu$ and covariance $\Sigma$, the scalar $\mathbf{Y}^{\mathsf T}M\mathbf{Y}$ equals its own trace, $\operatorname{tr}(M\mathbf{Y}\mathbf{Y}^{\mathsf T})$, and the trace is linear, so
$$E[\mathbf{Y}^{\mathsf T}M\mathbf{Y}] = \operatorname{tr}\big(M\,E[\mathbf{Y}\mathbf{Y}^{\mathsf T}]\big) = \operatorname{tr}\big(M(\Sigma + \boldsymbol\mu\boldsymbol\mu^{\mathsf T})\big) = \operatorname{tr}(M\Sigma) + \boldsymbol\mu^{\mathsf T}M\boldsymbol\mu.$$
With $M = I - H$ (since $\mathrm{RSS} = \mathbf{Y}^{\mathsf T}(I - H)^{\mathsf T}(I - H)\mathbf{Y} = \mathbf{Y}^{\mathsf T}(I - H)\mathbf{Y}$), $\Sigma = \sigma^2 I$ and $\boldsymbol\mu = X\boldsymbol\beta$, the second term vanishes because $(I - H)X = 0$, and the first is $\sigma^2\operatorname{tr}(I - H) = \sigma^2(n - p)$ by Theorem 2 of The Geometry of Least Squares. $\square$
The divisor $n - p$ is the dimension of the space the residuals live in, $\mathcal{C}(X)^\perp$, and for $p = 1$ (the mean alone) it is the $n - 1$ of the sample variance. For the twelve adults, $n - p = 12 - 4 = 8$. Part 2 says that residuals have unequal variances even when errors do not: adult 2, with leverage 0.573, has a residual variance of only $0.427\sigma^2$. Leverage, Influence and Residual Diagnostics divides each residual by its own standard deviation for that reason. With normal errors, part 2 strengthens to independence of $\hat{\boldsymbol\beta}$ and $\mathbf{e}$, and $\mathrm{RSS}/\sigma^2$ becomes chi-squared with $n - p$ degrees of freedom; that is the distribution theory of The Normal Linear Model.
The theorem
Here is the idea in one sentence. Every unbiased coefficient vector $\mathbf{a}$ has the same projection onto the column space of $X$, that projection is the coefficient vector of least squares, and the variance $\sigma^2\lVert\mathbf{a}\rVert^2$ can only grow when a perpendicular part is added.
Theorem 2 (Gauss–Markov). Under the Gauss–Markov conditions, let $\mathbf{c}^{\mathsf T}\boldsymbol\beta$ be estimable, and let $\hat{\mathbf{b}}$ be any solution of the normal equations. Then $\mathbf{c}^{\mathsf T}\hat{\mathbf{b}}$ is a linear unbiased estimator of $\mathbf{c}^{\mathsf T}\boldsymbol\beta$, and every linear unbiased estimator $\mathbf{a}^{\mathsf T}\mathbf{Y}$ satisfies
$$\operatorname{Var}(\mathbf{a}^{\mathsf T}\mathbf{Y}) = \operatorname{Var}(\mathbf{c}^{\mathsf T}\hat{\mathbf{b}}) + \sigma^2\lVert(I - H)\mathbf{a}\rVert^2 \ \ge\ \operatorname{Var}(\mathbf{c}^{\mathsf T}\hat{\mathbf{b}}),$$
with equality only when $\mathbf{a}^{\mathsf T}\mathbf{Y} = \mathbf{c}^{\mathsf T}\hat{\mathbf{b}}$ for every value of $\mathbf{Y}$. In words, the least squares estimator is the best linear unbiased estimator (BLUE), and it is the only one.
Proof. Let $\mathbf{a}$ satisfy (1); one exists because $\mathbf{c}^{\mathsf T}\boldsymbol\beta$ is estimable. Split $\mathbf{a} = H\mathbf{a} + (I - H)\mathbf{a}$. The first part gives the least squares estimator: for every $\mathbf{y}$,
$$(H\mathbf{a})^{\mathsf T}\mathbf{y} = \mathbf{a}^{\mathsf T}H\mathbf{y} = \mathbf{a}^{\mathsf T}X\hat{\mathbf{b}} = \mathbf{c}^{\mathsf T}\hat{\mathbf{b}},$$
using the symmetry of $H$, the fact that $H\mathbf{y} = X\hat{\mathbf{b}}$ is the fitted vector for any solution $\hat{\mathbf{b}}$, and (1). (For rank-deficient $X$, $H$ denotes the projection onto $\mathcal{C}(X)$, which exists by the projection theorem.) In particular $\mathbf{c}^{\mathsf T}\hat{\mathbf{b}}$ is linear in $\mathbf{y}$, with coefficient vector $H\mathbf{a}$, and it is unbiased because $X^{\mathsf T}H\mathbf{a} = (HX)^{\mathsf T}\mathbf{a} = X^{\mathsf T}\mathbf{a} = \mathbf{c}$. The two parts of $\mathbf{a}$ are orthogonal, so
$$\operatorname{Var}(\mathbf{a}^{\mathsf T}\mathbf{Y}) = \sigma^2\lVert\mathbf{a}\rVert^2 = \sigma^2\lVert H\mathbf{a}\rVert^2 + \sigma^2\lVert(I - H)\mathbf{a}\rVert^2 = \operatorname{Var}(\mathbf{c}^{\mathsf T}\hat{\mathbf{b}}) + \sigma^2\lVert(I - H)\mathbf{a}\rVert^2.$$
Equality holds exactly when $(I - H)\mathbf{a} = \mathbf{0}$, that is, $\mathbf{a} = H\mathbf{a}$, and then $\mathbf{a}^{\mathsf T}\mathbf{y} = \mathbf{c}^{\mathsf T}\hat{\mathbf{b}}$ for every $\mathbf{y}$ by the display above. $\square$
The proof shows more than the statement. Any two unbiased vectors $\mathbf{a}$ and $\mathbf{a}'$ satisfy $X^{\mathsf T}(\mathbf{a} - \mathbf{a}') = \mathbf{0}$, so their difference is orthogonal to $\mathcal{C}(X)$ and $H\mathbf{a} = H\mathbf{a}'$. The least squares coefficient vector is the unique unbiased coefficient vector lying in the column space. Any other unbiased estimator costs exactly $\sigma^2$ times the squared length of its component outside the column space. In the rank-deficient case the theorem also explains why only estimable functions are worth reporting. They are the ones whose least squares value is the same for every solution, and for them that common value is best.
Corollary (matrix form). If $X$ has rank $p$ and $\tilde{\boldsymbol\beta} = A\mathbf{Y}$ is any linear estimator with $E[\tilde{\boldsymbol\beta}] = \boldsymbol\beta$ for every $\boldsymbol\beta$ (that is, $AX = I$), then $\operatorname{Cov}(\tilde{\boldsymbol\beta}) - \operatorname{Cov}(\hat{\boldsymbol\beta})$ is positive semidefinite. Consequently each $\operatorname{Var}(\tilde\beta_j) \ge \operatorname{Var}(\hat\beta_j)$, and the total $\sum_j\operatorname{Var}(\tilde\beta_j) \ge \sum_j\operatorname{Var}(\hat\beta_j)$.
Proof. For any $\mathbf{c}$, $\mathbf{c}^{\mathsf T}\tilde{\boldsymbol\beta}$ is a linear unbiased estimator of $\mathbf{c}^{\mathsf T}\boldsymbol\beta$, so by Theorem 2, $\mathbf{c}^{\mathsf T}\big[\operatorname{Cov}(\tilde{\boldsymbol\beta}) - \operatorname{Cov}(\hat{\boldsymbol\beta})\big]\mathbf{c} = \operatorname{Var}(\mathbf{c}^{\mathsf T}\tilde{\boldsymbol\beta}) - \operatorname{Var}(\mathbf{c}^{\mathsf T}\hat{\boldsymbol\beta}) \ge 0$. Taking $\mathbf{c}$ to be the coordinate vectors gives the diagonal, and the sum of the diagonal follows. $\square$
Two rivals on the twelve adults
Return to the opening, with the straight-line model $Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$ in age alone. For the twelve ages, $S_{xx} = 4{,}584.67$, and the least squares slope has variance $\sigma^2/S_{xx} = 0.000218\,\sigma^2$. Two other slopes are linear and unbiased, as (1) confirms by checking that their coefficients sum to 0 and that their weighted sum of the ages is 1:
- The endpoint slope, $(Y_{\text{oldest}} - Y_{\text{youngest}})/(77 - 19)$, has coefficient vector with $\pm1/58$ in two places and variance $2\sigma^2/58^2 = 0.000595\,\sigma^2$, 2.726 times that of least squares.
- The slope of group means, which splits the twelve into the six younger (mean age 29.50) and the six older (mean age 65.17) and divides the difference of the groups' mean pressures by $65.17 - 29.50$, has variance $0.000262\,\sigma^2$, 1.201 times that of least squares.
By Theorem 2 the excess of the group-means slope, $0.000044\,\sigma^2$, is $\sigma^2$ times the squared length of the part of its coefficient vector perpendicular to the column space; the script confirms that the projection of that vector onto the column space is exactly the least squares coefficient vector. Grouping is not a poor idea, and with equal groups it loses only about a fifth of the precision here, but it cannot win.
In the full model with sex and BMI, the variance of the age coefficient is $0.000366\,\sigma^2$, 1.676 times its variance in the model with age alone. That is not a contradiction of the theorem, which compares estimators of the same quantity in the same model. The age coefficient adjusted for sex and BMI is a different quantity from the unadjusted slope. By the Frisch–Waugh–Lovell theorem of The Geometry of Least Squares its variance is $\sigma^2$ divided by the squared length of age's residual after the other predictors, which is shorter than $S_{xx}$.
Under normal errors: best among all unbiased estimators
The restriction to linear estimators is what makes the proof a line of geometry. Can a nonlinear unbiased estimator do better? Under normal errors, no.
Theorem 3 (normal errors). Suppose the errors are independent and normal with mean 0 and known variance $\sigma^2$, and $X$ has rank $p$. Let $W = w(\mathbf{Y})$ be any estimator with finite variance and $E_{\boldsymbol\beta}[W] = \mathbf{c}^{\mathsf T}\boldsymbol\beta$ for every $\boldsymbol\beta$, and suppose that the integral defining $E_{\boldsymbol\beta}[W]$ may be differentiated with respect to each $\beta_j$ under the integral sign. Then $\operatorname{Var}(W) \ge \sigma^2\mathbf{c}^{\mathsf T}(X^{\mathsf T}X)^{-1}\mathbf{c} = \operatorname{Var}(\mathbf{c}^{\mathsf T}\hat{\boldsymbol\beta})$.
The idea is the Cramér–Rao argument: an unbiased estimator must covary with the score in a fixed way, and the Cauchy–Schwarz inequality turns that covariance into a floor on its variance.
Proof. The joint density of $\mathbf{Y}$ is $f(\mathbf{y}; \boldsymbol\beta) = (2\pi\sigma^2)^{-n/2}\exp\big(-\lVert\mathbf{y} - X\boldsymbol\beta\rVert^2/(2\sigma^2)\big)$, and its score is the gradient of $\ln f$ in $\boldsymbol\beta$,
$$\mathbf{U} = \frac{X^{\mathsf T}(\mathbf{Y} - X\boldsymbol\beta)}{\sigma^2},$$
which has mean $\mathbf{0}$ and covariance $X^{\mathsf T}(\sigma^2I)X/\sigma^4 = X^{\mathsf T}X/\sigma^2$. Differentiating $\int w(\mathbf{y})f(\mathbf{y}; \boldsymbol\beta)\,d\mathbf{y} = \mathbf{c}^{\mathsf T}\boldsymbol\beta$ with respect to each $\beta_j$ under the integral sign gives $E[W\,U_j] = c_j$, and since $E[U_j] = 0$, $\operatorname{Cov}(W, \mathbf{U}) = \mathbf{c}$. The differentiation is allowed by the theorem's last hypothesis, the same hypothesis that the Cramér–Rao bounds of Fisher Information and the Cramér–Rao Bound state; whether it holds for every unbiased estimator of finite variance in this model is not proved here. For any fixed vector $\mathbf{d} \ne \mathbf{0}$, the Cauchy–Schwarz inequality for the covariance of $W$ and $\mathbf{d}^{\mathsf T}\mathbf{U}$ (Probability Inequalities) gives
$$\operatorname{Var}(W) \ge \frac{\operatorname{Cov}(W, \mathbf{d}^{\mathsf T}\mathbf{U})^2}{\operatorname{Var}(\mathbf{d}^{\mathsf T}\mathbf{U})} = \frac{(\mathbf{d}^{\mathsf T}\mathbf{c})^2}{\mathbf{d}^{\mathsf T}X^{\mathsf T}X\mathbf{d}/\sigma^2}.$$
Choose $\mathbf{d} = (X^{\mathsf T}X)^{-1}\mathbf{c}$ (if $\mathbf{c} = \mathbf{0}$ there is nothing to prove). Then $\mathbf{d}^{\mathsf T}\mathbf{c} = \mathbf{c}^{\mathsf T}(X^{\mathsf T}X)^{-1}\mathbf{c} = \mathbf{d}^{\mathsf T}X^{\mathsf T}X\mathbf{d}$, and the bound becomes $\sigma^2\mathbf{c}^{\mathsf T}(X^{\mathsf T}X)^{-1}\mathbf{c}$, which is the variance of $\mathbf{c}^{\mathsf T}\hat{\boldsymbol\beta}$ by Theorem 1. $\square$
This is the several-parameter Cramér–Rao bound of the Fisher information article applied to independent observations that are not identically distributed; the proof above repeats its argument for that case. Knowing $\sigma^2$ is not essential. An estimator unbiased for every $\boldsymbol\beta$ and every $\sigma^2$ is unbiased for every $\boldsymbol\beta$ at each fixed $\sigma^2$, so the bound holds for it too.
Without normality, the question is open in a precise sense. Hansen (2022) argued that the classical theorem "can be restated omitting the unnatural restriction to linear estimators, without adding any extra conditions", so that least squares would be the best unbiased estimator, linear or not, under the Gauss–Markov conditions alone. Pötscher and Preinerstorfer (2024) replied that Theorem 4 of Hansen (2022) "applies to exactly the same class of estimators as does the classical Aitken theorem", and that his further theorems carry assumptions absent from the classical ones. Both statements are quoted from the papers' abstracts, which are all that this article's sources include; in the journal, the exchange continued with Hansen's (2024) reply, whose content this article does not report. The experiment below shows what is not in dispute: an estimator that is unbiased only for a narrower family of error distributions, such as least absolute deviations, which is unbiased for symmetric errors, can have smaller variance than least squares for some distributions in that family.
Correlated errors and unequal variances: Aitken's theorem
Suppose instead that $\operatorname{Cov}(\boldsymbol\varepsilon) = \sigma^2V$ for a known symmetric positive definite matrix $V$: errors with unequal variances (diagonal $V$), or correlated errors, such as repeated measurements on one person. Least squares is still unbiased, because Theorem 1's proof of unbiasedness used only $E[\boldsymbol\varepsilon] = \mathbf{0}$, but it is no longer best, and its covariance is no longer $\sigma^2(X^{\mathsf T}X)^{-1}$:
$$\operatorname{Cov}(\hat{\boldsymbol\beta}) = \sigma^2(X^{\mathsf T}X)^{-1}X^{\mathsf T}VX(X^{\mathsf T}X)^{-1}.$$
The remedy is to transform the problem back to the Gauss–Markov conditions. Every symmetric positive definite $V$ can be written $V = LL^{\mathsf T}$ with $L$ invertible, for example by the Cholesky factorization (a standard result of linear algebra, cited).
Theorem 4 (Aitken). If $\operatorname{Cov}(\boldsymbol\varepsilon) = \sigma^2V$ with $V$ known, symmetric and positive definite, and $X$ has rank $p$, the best linear unbiased estimator of $\boldsymbol\beta$ is the generalized least squares estimator
$$\hat{\boldsymbol\beta}_{G} = (X^{\mathsf T}V^{-1}X)^{-1}X^{\mathsf T}V^{-1}\mathbf{Y}, \qquad \operatorname{Cov}(\hat{\boldsymbol\beta}_{G}) = \sigma^2(X^{\mathsf T}V^{-1}X)^{-1},$$
in the sense of Theorem 2 and its corollary.
Proof. Let $\mathbf{Y}^* = L^{-1}\mathbf{Y}$ and $X^* = L^{-1}X$. Then $\mathbf{Y}^* = X^*\boldsymbol\beta + L^{-1}\boldsymbol\varepsilon$, and the new errors have mean $\mathbf{0}$ and covariance $\sigma^2L^{-1}LL^{\mathsf T}L^{-\mathsf T} = \sigma^2I$. So the transformed model satisfies the Gauss–Markov conditions, and $X^*$ has rank $p$ because $L^{-1}$ is invertible. Linear functions of $\mathbf{Y}^*$ and of $\mathbf{Y}$ are the same set, since $\mathbf{a}^{\mathsf T}\mathbf{Y} = (L^{\mathsf T}\mathbf{a})^{\mathsf T}\mathbf{Y}^*$, and unbiasedness does not depend on how an estimator is written. By Theorem 2 and its corollary, the best linear unbiased estimator is least squares in the transformed model, $(X^{*\mathsf T}X^*)^{-1}X^{*\mathsf T}\mathbf{Y}^*$. Since $L^{-\mathsf T}L^{-1} = (LL^{\mathsf T})^{-1} = V^{-1}$, this is the stated formula, with covariance $\sigma^2(X^{*\mathsf T}X^*)^{-1} = \sigma^2(X^{\mathsf T}V^{-1}X)^{-1}$ by Theorem 1. $\square$
Geometrically, $\hat{\boldsymbol\beta}_{G}$ is least squares in the inner product $\langle\mathbf{u}, \mathbf{v}\rangle = \mathbf{u}^{\mathsf T}V^{-1}\mathbf{v}$, as the last section of The Geometry of Least Squares describes. For diagonal $V$ it is weighted least squares with weights inversely proportional to the variances. Aitken's (1936) paper treated least squares and linear combinations of observations, and the result is known as Aitken's theorem. Pötscher and Preinerstorfer (2024) call it "the classical Aitken theorem" beside "the classical Gauss–Markov" theorem.
When ordinary least squares is still best
Generalized least squares needs $V$. When does ordinary least squares give the same estimates, so that $V$ does not matter? Kruskal (1968) posed the question in the title of a paper, "When are Gauss–Markov and least squares estimators identical?", and treated it by "a coordinate-free approach". The answer is a statement about the column space; the proof below is written out for this article.
Theorem 5 (equality of the two estimators). With $V$ symmetric positive definite and $X$ of rank $p$, $\hat{\boldsymbol\beta} = \hat{\boldsymbol\beta}_G$ for every $\mathbf{y}$ exactly when $V$ maps the column space into itself: $V\mathbf{v} \in \mathcal{C}(X)$ for every $\mathbf{v} \in \mathcal{C}(X)$.
Proof. $\hat{\boldsymbol\beta}_G$ is the unique solution of $X^{\mathsf T}V^{-1}(\mathbf{y} - X\mathbf{b}) = \mathbf{0}$. The ordinary estimate gives $\mathbf{y} - X\hat{\boldsymbol\beta} = (I - H)\mathbf{y}$, so the two agree for every $\mathbf{y}$ exactly when $X^{\mathsf T}V^{-1}(I - H)\mathbf{y} = \mathbf{0}$ for every $\mathbf{y}$, that is, when every column of $V^{-1}X$ is orthogonal to every vector of $\mathcal{C}(X)^\perp$. A vector orthogonal to $\mathcal{C}(X)^\perp$ lies in $\mathcal{C}(X)$ (by the projection theorem, as in the proof of estimability), so the condition is $\mathcal{C}(V^{-1}X) \subseteq \mathcal{C}(X)$. Both column spaces have dimension $p$, because $V^{-1}$ is invertible, so the inclusion is an equality. Applying $V$ to both sides turns $\mathcal{C}(V^{-1}X) = \mathcal{C}(X)$ into $\mathcal{C}(X) = \mathcal{C}(VX)$, which is the stated condition. Each step reverses. $\square$
Example (equicorrelated errors). Let every error have variance $\sigma^2$ and every pair correlation $\rho$, so $V = (1 - \rho)I + \rho J$ with $J$ the matrix of ones and $-1/(n-1) < \rho < 1$. For any $\mathbf{v}$, $V\mathbf{v} = (1 - \rho)\mathbf{v} + \rho(\mathbf{1}^{\mathsf T}\mathbf{v})\mathbf{1}$. If the model has an intercept, $\mathbf{1} \in \mathcal{C}(X)$ and $V$ maps $\mathcal{C}(X)$ into itself. So with an intercept, ordinary least squares is the best linear unbiased estimator under equicorrelated errors, whatever $\rho$. The script confirms the two estimates agree for the twelve adults with $\rho = 0.5$. But the usual standard errors are wrong. By the trace formula in Theorem 1, $E[s^2] = \sigma^2\operatorname{tr}\big((I - H)V\big)/(n - p) = (1 - \rho)\sigma^2$, since $(I - H)J = 0$ when $\mathbf{1} \in \mathcal{C}(X)$. For the twelve adults with $\rho = 0.5$ the true variance of the intercept is $1.977\,\sigma^2$, while the usual formula with $s^2$ averages $1.477\,\sigma^2$. For the age coefficient the two agree, $0.000183\,\sigma^2$, because the shared component of the errors moves only the intercept. A shared component is exactly the kind of dependence Linear Regression Assumptions warns about; here the estimate survives it and only the intercept's precision is overstated.
Example (autoregressive errors). If the errors of the twelve adults, ordered by age, are correlated with $\operatorname{Corr}(\varepsilon_i, \varepsilon_j) = 0.6^{\lvert k_i - k_j\rvert}$, where $k_i$ is adult $i$'s position in that order, then $V$ does not preserve the column space. The variance of the ordinary least squares age coefficient in the full model is $0.000685\,\sigma^2$ and that of generalized least squares $0.000578\,\sigma^2$: ordinary least squares is 1.184 times as variable, and by Theorem 4 no linear unbiased estimator does better than the generalized one.
What happens when the conditions fail: an experiment
The theorem compares linear unbiased estimators under its conditions. Two questions remain. How much do other estimators lose or gain when the errors are not normal? And how much does the comparison change when the variances are unequal? A seeded experiment measures both for one design.
Design. Fifty ages were drawn once from the 5,176 adults (seed 23,999,999). Responses follow the population line of pressure on age for these 5,176 adults, $100.21 + 0.5223\,\text{age}$ (Expectation, Variance and Covariance finds $100.09 + 0.5269\,\text{age}$ for its 5,247 adults, which include those without a body mass index), plus errors with standard deviation 17.4 mm Hg, the root mean square residual of that population line, in four scenarios:
- normal: normal errors;
- t3: errors from Student's $t$ distribution with 3 degrees of freedom, scaled to the same variance (heavy tails);
- Laplace: double-exponential errors with the same variance;
- unequal variances: normal errors whose standard deviation is proportional to age, with mean standard deviation 17.4 mm Hg over the fifty ages (variance grows with age).
Three estimators of the slope are compared with least squares: the slope of group means for the youngest and oldest thirds (16 adults each), which is linear and unbiased; the least absolute deviations slope, which minimizes $\sum_i\lvert y_i - a - bx_i\rvert$ and is not linear; and, in the last scenario, weighted least squares with the true weights, which is generalized least squares for that $V$. Each run generates 2,000 data sets and records each estimator's variance divided by that of least squares; there are 30 runs with seeds $23{,}000{,}000 + 1000k + s$ ($k$ the scenario, $s = 0, \ldots, 29$). Medians and interquartile ranges over the 30 runs:
| Errors | Group means / LS | Least absolute deviations / LS | Weighted LS / LS |
|---|---|---|---|
| normal | 1.184 (1.167 to 1.202) | 1.586 (1.564 to 1.603) | |
| t3 | 1.193 (1.171 to 1.211) | 0.667 (0.634 to 0.691) | |
| Laplace | 1.180 (1.172 to 1.191) | 0.732 (0.723 to 0.739) | |
| unequal variances, sd proportional to age | 1.353 (1.341 to 1.371) | 0.563 (0.557 to 0.572) |
Five observations, each about this experiment:
- The group-means slope is linear and unbiased, so under the first three scenarios Theorem 2 fixes its variance ratio exactly at $\sigma^2\lVert\mathbf{a}\rVert^2/\operatorname{Var}(\hat\beta_1) = 1.189$ for these fifty ages (the youngest and oldest thirds), whatever the error distribution. The simulated medians, 1.184, 1.193 and 1.180, scatter around it.
- With normal errors, least absolute deviations is 1.586 times as variable as least squares, as Theorem 3 requires of an unbiased competitor that meets its hypotheses. Least absolute deviations is unbiased for symmetric errors, provided its mean exists; that existence, and the finite variance and differentiability Theorem 3 asks for, are assumed here, not proved. The proof of unbiasedness is short. The minimizers of $\sum_i\lvert y_i - \mathbf{x}_i^{\mathsf T}\mathbf{b}\rvert$ form a bounded convex set, which need not be a single point: with an even number of observations it can be a segment with positive probability (for an intercept alone, every point between the two middle observations is a median). Write $\hat{\boldsymbol\beta}_L(\mathbf{y})$ for the centroid of that set. Replacing $\mathbf{y}$ by $X\mathbf{g} + \mathbf{y}$ moves the set, and so its centroid, by $\mathbf{g}$, because the objective for $X\mathbf{g} + \mathbf{y}$ at $\mathbf{b} + \mathbf{g}$ equals the objective for $\mathbf{y}$ at $\mathbf{b}$. Replacing $\mathbf{y}$ by $-\mathbf{y}$ negates them, because the objective for $-\mathbf{y}$ at $-\mathbf{b}$ equals that for $\mathbf{y}$ at $\mathbf{b}$. So $\hat{\boldsymbol\beta}_L(X\boldsymbol\beta - \boldsymbol\varepsilon) = 2\boldsymbol\beta - \hat{\boldsymbol\beta}_L(X\boldsymbol\beta + \boldsymbol\varepsilon)$. If $-\boldsymbol\varepsilon$ has the same distribution as $\boldsymbol\varepsilon$, the two sides have the same distribution, so the distribution of $\hat{\boldsymbol\beta}_L$ is symmetric about $\boldsymbol\beta$, and its mean, when it exists, is $\boldsymbol\beta$. The same argument applies to any rule for choosing a point of the set that commutes with the two maps. The script estimates only the slope, and takes the midpoint of the interval of minimizing slopes, which is such a rule. In this experiment that interval had positive length in 20 of the 180,000 data sets with symmetric errors that the least absolute deviations column uses, and the midpoint differed from the point the script's search first finds by less than $10^{-6}$.
- With heavy-tailed or double-exponential errors the nonlinear estimator wins, with about two thirds and three quarters of the variance of least squares. The Gauss–Markov conditions hold in both scenarios, so this does not contradict Theorem 2: the winner is not linear.
- With variance growing with age, weighted least squares has 0.563 of the variance of least squares, close to the exact ratio 0.560 that Theorem 4's covariance formula gives for this design.
- The group-means slope does worse under unequal variances, 1.353 against 1.189. Each estimator's variance is $\sum_i a_i^2\sigma_i^2$, an average of the error variances weighted by its squared weights. The group-means slope gives the oldest third 50% of its squared weight; least squares gives it 40% and the youngest third 57%, and the youngest have the smallest error variances. The exact ratio for this scenario, from that formula, is 1.356. Outside the Gauss–Markov conditions the theorem gives no ranking, and the ranking of two inferior estimators can change.
A biased estimator can also beat least squares in mean squared error, which adds the squared bias to the variance. Shrinking coefficients toward zero trades a little bias for a large reduction in variance when predictors are nearly collinear; that is the subject of Ridge and Lasso Regression. The Gauss–Markov theorem says nothing against it, because it compares only unbiased estimators.
What is proved here, what is cited, what the experiment shows
Proved here. The characterization (1) of linear unbiased estimators; the mean and covariance of the least squares estimator and of the residuals, and $E[s^2] = \sigma^2$ without normality (Theorem 1); the Gauss–Markov theorem for estimable functions with the exact excess variance, and its matrix form (Theorem 2 and corollary); the lower bound under normal errors for all unbiased estimators whose mean may be differentiated under the integral sign (Theorem 3, where that is a hypothesis); Aitken's theorem (Theorem 4) given a factorization $V = LL^{\mathsf T}$; the condition for ordinary and generalized least squares to coincide (Theorem 5) and the expected $s^2$ under equicorrelation.
Cited. Linearity and bilinearity of expectation and covariance (Expectation, Variance and Covariance); the hat matrix and estimability (The Geometry of Least Squares); the form of the Cramér–Rao argument (Fisher Information and the Cramér–Rao Bound); the Cauchy–Schwarz inequality (Probability Inequalities); the existence of the Cholesky factorization; the claims of Hansen (2022) and Pötscher and Preinerstorfer (2024), from their abstracts.
Computed. The exact variance ratios on the twelve adults' ages and the covariances under correlated errors are computed by the script tools/experiments/statistics/gauss-markov-theorem.py, the ratios in exact rational arithmetic. The variance ratios in the table are observations of a seeded experiment on one design with fifty ages and one error scale.
Worked problems
The problems use the unit's twelve adults and small exact cases, chosen to exercise Theorems 1, 2, 4 and 5. Every number is recomputed by the article's script.
Problem 1 (the endpoint slope). For the straight-line model in age on the twelve adults, show that the endpoint slope $(Y_{\text{oldest}} - Y_{\text{youngest}})/58$ is linear and unbiased, and find its variance relative to least squares from Theorem 2.
Solution. The ratio is 2.726. The estimator is $\mathbf{a}^{\mathsf T}\mathbf{Y}$ with $a = 1/58$ for the oldest adult (77), $-1/58$ for the youngest (19), and 0 elsewhere. With $\mathbf{c} = (0, 1)$ for the slope and rows $(1, x_i)$, condition (1) reads $\sum_i a_i = 0$ and $\sum_i a_ix_i = 1$; both hold, since $(77 - 19)/58 = 1$. Its variance is $\sigma^2\lVert\mathbf{a}\rVert^2 = 2\sigma^2/58^2 = 0.0005945\,\sigma^2$. The least squares slope has variance $\sigma^2/S_{xx}$ with $S_{xx} = 4{,}584.67$, which is $0.0002181\,\sigma^2$, so the ratio is $2 \times 4{,}584.67/3{,}364 = 2.726$. By Theorem 2 the excess, $0.0005945 - 0.0002181 = 0.0003764$ times $\sigma^2$, is $\sigma^2\lVert(I - H)\mathbf{a}\rVert^2$: the endpoint slope ignores ten observations, and its coefficient vector has a large component outside the column space.
Problem 2 (expected residual variance). For the twelve adults and the model with age, sex and BMI, find $E[\mathrm{RSS}]$ and the variance of the residual of adult 2, whose leverage is 0.573. Does either answer need normal errors?
Solution. $E[\mathrm{RSS}] = 8\sigma^2$ and $\operatorname{Var}(e_2) = 0.427\,\sigma^2$, and neither needs normality. By Theorem 1, $E[\mathrm{RSS}] = \sigma^2\operatorname{tr}(I - H) = \sigma^2(n - p) = 8\sigma^2$, and $\operatorname{Var}(e_i) = \sigma^2(1 - h_{ii})$, which for adult 2 is $(1 - 0.573)\sigma^2 = 0.427\,\sigma^2$. Both use only the mean and covariance of the errors through the trace formula and $\operatorname{Cov}(A\mathbf{Y}) = A\operatorname{Cov}(\mathbf{Y})A^{\mathsf T}$. Normality is needed only for the distributions, such as $\mathrm{RSS}/\sigma^2 \sim \chi^2_8$.
Problem 3 (when the correlation does not matter). Show that for the model with only an intercept, $Y_i = \mu + \varepsilon_i$, the sample mean is the best linear unbiased estimator of $\mu$ under equicorrelated errors, and find its variance and the expectation of the usual estimate $s^2/n$ of that variance.
Solution. Here $X = \mathbf{1}$, and $V\mathbf{1} = (1 - \rho)\mathbf{1} + \rho n\mathbf{1} = [1 + (n - 1)\rho]\mathbf{1}$ lies in $\mathcal{C}(X)$, so by Theorem 5 generalized least squares equals ordinary least squares, the sample mean, which is therefore best by Theorem 4. Its variance is $\mathbf{1}^{\mathsf T}V\mathbf{1}\,\sigma^2/n^2 = \sigma^2[1 + (n - 1)\rho]/n$, the formula of Expectation, Variance and Covariance for the variance of a mean of equicorrelated variables. The usual estimate has $E[s^2/n] = (1 - \rho)\sigma^2/n$, by the trace formula. For $\rho > 0$ it is too small by the factor $(1 - \rho)/[1 + (n - 1)\rho]$, which tends to 0 as $n$ grows: the estimate is best, and its reported precision is wrong.
Problem 4 (variance growing with the predictor). In the model through the origin $Y_i = \beta x_i + \varepsilon_i$ with $x = (1, 2, 3, 4)$ and $\operatorname{Var}(\varepsilon_i) = \sigma^2x_i$ (uncorrelated), find the generalized least squares estimator and compare its variance with that of ordinary least squares.
Solution. Generalized least squares gives $\sum_iY_i/\sum_ix_i$, with variance $0.1000\,\sigma^2$; ordinary least squares has $0.1111\,\sigma^2$, 1.111 times as much. Here $V = \operatorname{diag}(x_i)$ and $X = \mathbf{x}$, so Theorem 4 gives $\hat\beta_G = (\mathbf{x}^{\mathsf T}V^{-1}\mathbf{x})^{-1}\mathbf{x}^{\mathsf T}V^{-1}\mathbf{Y} = \sum_iY_i/\sum_ix_i$, because $V^{-1}\mathbf{x} = \mathbf{1}$, with variance $\sigma^2/\sum_ix_i = \sigma^2/10$. Ordinary least squares, $\hat\beta = \sum_ix_iY_i/\sum_ix_i^2$, has variance $\sigma^2\sum_ix_i^2\cdot x_i/(\sum_ix_i^2)^2 = \sigma^2\sum_ix_i^3/(\sum_ix_i^2)^2 = 100\sigma^2/900$. The column space is the line through $\mathbf{x}$, and $V\mathbf{x} = (1, 4, 9, 16)$ is not on it, so by Theorem 5 the two estimators had to differ.
Summary
| Result | Conditions | Statement |
|---|---|---|
| Moments (Theorem 1) | $E\boldsymbol\varepsilon = \mathbf{0}$, $\operatorname{Cov}\boldsymbol\varepsilon = \sigma^2I$ | $\operatorname{Cov}(\hat{\boldsymbol\beta}) = \sigma^2(X^{\mathsf T}X)^{-1}$; $E[s^2] = \sigma^2$ |
| Gauss–Markov (Theorem 2) | same | $\operatorname{Var}(\mathbf{a}^{\mathsf T}\mathbf{Y}) = \operatorname{Var}(\mathbf{c}^{\mathsf T}\hat{\mathbf{b}}) + \sigma^2\lVert(I - H)\mathbf{a}\rVert^2$ |
| Normal errors (Theorem 3) | independent normal errors | least squares is best among all unbiased estimators |
| Aitken (Theorem 4) | $\operatorname{Cov}\boldsymbol\varepsilon = \sigma^2V$, $V$ known | $\hat{\boldsymbol\beta}_G = (X^{\mathsf T}V^{-1}X)^{-1}X^{\mathsf T}V^{-1}\mathbf{Y}$ is best |
| Equality (Theorem 5) | same | $\hat{\boldsymbol\beta} = \hat{\boldsymbol\beta}_G$ for all $\mathbf{y}$ iff $V\mathcal{C}(X) \subseteq \mathcal{C}(X)$ |
What we have learned
Under uncorrelated errors with a common variance, least squares is the best linear unbiased estimator of every estimable combination of the coefficients. The proof is the geometry of the previous article: every unbiased coefficient vector has the same projection onto the column space, that projection is least squares, and any other estimator pays $\sigma^2$ times the squared length of its perpendicular part. Normality makes least squares best among all unbiased estimators. Correlated or unequal errors make generalized least squares best instead, and ordinary least squares keeps the title exactly when the error covariance maps the column space into itself. The experiment showed the limits: estimators that are not linear can win under heavy tails, and the ranking of inferior estimators can change when the variances are unequal.
The theorem concerns means and variances only. To test a coefficient or build an interval for it, the errors' distribution is needed. The Normal Linear Model: t and F Tests for Coefficients supplies it under normality, through the projection theorem of Sampling Distributions from the Normal.
References
- Aitken, A. C. (1936). On least squares and linear combination of observations. Proceedings of the Royal Society of Edinburgh, 55, 42–48. record
- Hansen, B. E. (2022). A modern Gauss–Markov theorem. Econometrica, 90(3), 1283–1294. record
- Hansen, B. E. (2024). Reply to: Comment on "A modern Gauss–Markov theorem". Econometrica, 92(3), 925–928. record
- Kruskal, W. (1968). When are Gauss–Markov and least squares estimators identical? A coordinate-free approach. The Annals of Mathematical Statistics, 39(1), 70–75. record
- Pötscher, B. M., and Preinerstorfer, D. (2024). A comment on: "A modern Gauss–Markov theorem". Econometrica, 92(3), 913–924. record