The Normal Linear Model: t and F Tests for Coefficients

The exact sampling theory of least-squares regression when the errors are independent and normal with a common variance. Every linear combination of the coefficients is normal and independent of the residual sum of squares, which is chi-squared on n - p degrees of freedom; from this follow, with proofs, the t test and interval for a coefficient, the interval for a mean response and the prediction interval for a new observation, the overall F test and its noncentral distribution, and the joint confidence region with its simultaneous intervals. Worked on 50 NHANES adults, with an experiment on how the intervals cover when the population is neither normal nor of constant variance, and worked problems on a hand-sized sample, the diabetes study of the data science pages, and a contrast with t and Scheffé intervals.

Fifty adults drawn at random from the 5,176 NHANES adults of this unit have systolic blood pressures that least squares relates to age, sex and body mass index (BMI) by

$$\widehat{\text{pressure}} = 86.04 + 0.4455\,\text{age} + 2.34\,\text{female} + 0.467\,\text{BMI},$$

in mm Hg, years and kg/m², with female equal to 1 for a woman and 0 for a man. For all 5,176 adults the same fit gives an age coefficient of 0.517 and a BMI coefficient of 0.385. A reader of the sample alone does not know those values. How far from the truth can the sample's 0.4455 be, is its 0.467 for BMI distinguishable from zero, and do the three inputs together explain more than chance would?

The normal linear model answers these questions exactly, for any sample size. It adds one assumption to those of the Gauss–Markov theorem: the errors are normal. Under it every estimated coefficient, less the true value and divided by its estimated standard error, has a $t$ distribution with $n - p$ degrees of freedom. The test that all slopes are zero has an $F$ distribution. These are the $t$ and $F$ tests that regression software prints beside every fit, and this article proves them.

First we state the model and collect what the earlier articles of the unit give. Then we prove the central theorem: a linear combination of the coefficients is normal, the residual sum of squares is chi-squared, and the two are independent. From it we derive the $t$ test and interval for one coefficient, and the intervals for a mean response and for a new observation. Then come the overall $F$ test and the joint confidence region, with the simultaneous intervals it implies. Finally an experiment asks how much of this survives when the population is not normal, as the NHANES pressures are not. The article assumes The Geometry of Least Squares and The Gauss–Markov Theorem, and it applies the projection theorem of Sampling Distributions from the Normal: Chi-Squared, t and F.

The model and what is already known

The data are $n$ responses $\mathbf{y} = (y_1, \ldots, y_n)^{\mathsf{T}}$ and an $n \times p$ design matrix $X$ whose first column is the vector of ones $\mathbf{1}$ and whose other columns hold the inputs. In the running example $n = 50$ and $p = 4$: the columns are $\mathbf{1}$, age, the indicator female, and BMI. The intercept is counted in $p$ here, as everywhere in this unit; the data science pages, such as Multiple Linear Regression, count $p$ inputs and $p + 1$ columns. We assume that $X$ has rank $p$, so that its columns are linearly independent.

The linear model says that

$$\mathbf{Y} = X\boldsymbol\beta + \boldsymbol\varepsilon,$$

where $\boldsymbol\beta$ is an unknown vector of $p$ coefficients, $X$ is treated as fixed, and $\boldsymbol\varepsilon$ is a random vector of errors. The Gauss–Markov conditions are that the errors have mean zero, a common variance $\sigma^2$, and are uncorrelated: $E[\boldsymbol\varepsilon] = \mathbf{0}$ and $\operatorname{Cov}(\boldsymbol\varepsilon) = \sigma^2 I$. The normal linear model asks more: the errors $\varepsilon_1, \ldots, \varepsilon_n$ are independent, and each is normal with mean 0 and variance $\sigma^2$. On this site the normal distribution with mean $\mu$ and standard deviation $\sigma$ is written $N(\mu, \sigma)$, with the standard deviation second, as the course page explains; this article describes it in words.

Four facts come from earlier articles of the unit.

  • The fit. The least-squares estimate is $\hat{\boldsymbol\beta} = (X^{\mathsf{T}}X)^{-1}X^{\mathsf{T}}\mathbf{y}$, the fitted values are $\hat{\mathbf{y}} = X\hat{\boldsymbol\beta} = H\mathbf{y}$ with the hat matrix $H = X(X^{\mathsf{T}}X)^{-1}X^{\mathsf{T}}$, and the residuals are $\mathbf{e} = (I - H)\mathbf{y}$. The residual sum of squares is $\mathrm{RSS} = \lVert\mathbf{e}\rVert^2$, and $s^2 = \mathrm{RSS}/(n - p)$.
  • The geometry (The Geometry of Least Squares, Theorems 2 and 4). $H$ is the projection onto the column space $\mathcal{C}(X)$: it is symmetric and idempotent with rank and trace $p$, $HX = X$, and $I - H$ is the projection onto the perpendicular complement, of rank $n - p$. The averaging matrix $P_{\mathbf 1} = \tfrac1n J$, which replaces every entry of a vector by the mean of its entries, is the projection onto the line through $\mathbf{1}$. Since $\mathbf{1}$ lies in $\mathcal{C}(X)$, the nested-projection rule of that article gives $HP_{\mathbf 1} = P_{\mathbf 1}H = P_{\mathbf 1}$, and $H - P_{\mathbf 1}$ is again a projection, of rank $p - 1$.
  • The moments (The Gauss–Markov Theorem, Theorem 1). Under the Gauss–Markov conditions alone, $E[\hat{\boldsymbol\beta}] = \boldsymbol\beta$, $\operatorname{Cov}(\hat{\boldsymbol\beta}) = \sigma^2(X^{\mathsf{T}}X)^{-1}$ and $E[s^2] = \sigma^2$. So the variance of the $j$th coefficient is $\sigma^2 v_{jj}$, where $v_{jj}$ is the $j$th diagonal entry of $(X^{\mathsf{T}}X)^{-1}$.
  • The projection theorem (Sampling Distributions from the Normal, Theorems 1 and 2). If a vector has independent normal coordinates with common variance $\sigma^2$ and mean vector $\mathbf{m}$, an orthogonal matrix turns it into a vector of the same kind with mean vector rotated. For projection matrices $P_1, \ldots, P_k$ with $P_iP_j = 0$ for $i \ne j$, the projections $P_i\mathbf{Y}$ are independent, and $\lVert P_i\mathbf{Y}\rVert^2/\sigma^2$ is chi-squared with the rank of $P_i$ as degrees of freedom and noncentrality $\lVert P_i\mathbf{m}\rVert^2/\sigma^2$, central when $P_i\mathbf{m} = \mathbf{0}$.

The moments say how large the error of each coefficient is on average. They do not give a probability: the chance that $\hat\beta_j$ lies within some distance of $\beta_j$ depends on the whole distribution, and $\sigma^2$ is unknown and must be replaced by $s^2$. The normal assumption supplies the distribution, and the projection theorem shows that replacing $\sigma^2$ by $s^2$ costs exactly a passage from the normal to the $t$ distribution.

The running sample. The 50 adults were drawn without replacement from the unit's population (numpy.random.default_rng(1)). Among them are 25 women, whose mean pressure is 125.64 mm Hg. The 50 have mean pressure 123.27 mm Hg, mean age 49.64 years and mean BMI 29.89. The fit above has $\mathrm{RSS} = 19{,}213.6$ on $n - p = 46$ degrees of freedom, so $s^2 = 417.69$ and $s = 20.44$ mm Hg.

The distribution of the estimates

Here is the idea. The estimate depends on the data only through its projection onto $\mathcal{C}(X)$, and the residuals are the projection onto the perpendicular space. For normal errors perpendicular projections are independent, and the second one is chi-squared.

Theorem 1 (the normal linear model). Under the normal linear model with $X$ of rank $p < n$:

  1. for every nonzero vector $\mathbf{a}$ of length $p$, the linear combination $\mathbf{a}^{\mathsf{T}}\hat{\boldsymbol\beta}$ is normal with mean $\mathbf{a}^{\mathsf{T}}\boldsymbol\beta$ and variance $\sigma^2\,\mathbf{a}^{\mathsf{T}}(X^{\mathsf{T}}X)^{-1}\mathbf{a}$;
  2. $\mathrm{RSS}/\sigma^2$ has the chi-squared distribution with $n - p$ degrees of freedom;
  3. the vector $\hat{\boldsymbol\beta}$ and the residual vector $\mathbf{e}$ are independent; in particular $\hat{\boldsymbol\beta}$ and $s^2$ are independent.

Proof. Part 1. Since $\mathbf{y} = X\boldsymbol\beta + \boldsymbol\varepsilon$,

$$\hat{\boldsymbol\beta} = (X^{\mathsf{T}}X)^{-1}X^{\mathsf{T}}(X\boldsymbol\beta + \boldsymbol\varepsilon) = \boldsymbol\beta + (X^{\mathsf{T}}X)^{-1}X^{\mathsf{T}}\boldsymbol\varepsilon,$$

so $\mathbf{a}^{\mathsf{T}}\hat{\boldsymbol\beta} - \mathbf{a}^{\mathsf{T}}\boldsymbol\beta = \mathbf{c}^{\mathsf{T}}\boldsymbol\varepsilon$ with $\mathbf{c} = X(X^{\mathsf{T}}X)^{-1}\mathbf{a}$. The vector $\mathbf{c}$ is not zero, since $X^{\mathsf{T}}\mathbf{c} = \mathbf{a} \ne \mathbf{0}$, and its squared length is

$$\lVert\mathbf{c}\rVert^2 = \mathbf{a}^{\mathsf{T}}(X^{\mathsf{T}}X)^{-1}X^{\mathsf{T}}X(X^{\mathsf{T}}X)^{-1}\mathbf{a} = \mathbf{a}^{\mathsf{T}}(X^{\mathsf{T}}X)^{-1}\mathbf{a}.$$

Let $\mathbf{u} = \mathbf{c}/\lVert\mathbf{c}\rVert$ and complete it by the Gram–Schmidt process to an orthogonal matrix $Q$ whose first row is $\mathbf{u}^{\mathsf{T}}$. The errors are independent normal variables with variance $\sigma^2$ and mean 0. By the rotation theorem (Theorem 1 of the projection article) the first coordinate of $Q\boldsymbol\varepsilon$, which is $\mathbf{u}^{\mathsf{T}}\boldsymbol\varepsilon$, is normal with mean 0 and variance $\sigma^2$. Hence $\mathbf{c}^{\mathsf{T}}\boldsymbol\varepsilon = \lVert\mathbf{c}\rVert\,\mathbf{u}^{\mathsf{T}}\boldsymbol\varepsilon$ is normal with mean 0 and variance $\sigma^2\lVert\mathbf{c}\rVert^2$, which is the claim.

Part 2. The residual vector is $\mathbf{e} = (I - H)\mathbf{y}$, so $\mathrm{RSS} = \lVert(I - H)\mathbf{y}\rVert^2$. The vector $\mathbf{y}$ has independent normal coordinates with variance $\sigma^2$ and mean vector $\mathbf{m} = X\boldsymbol\beta$. The matrix $I - H$ is a projection of rank $n - p$, and $(I - H)X\boldsymbol\beta = X\boldsymbol\beta - X\boldsymbol\beta = \mathbf{0}$ because $HX = X$. By the projection theorem, $\mathrm{RSS}/\sigma^2$ is central chi-squared with $n - p$ degrees of freedom.

Part 3. Because $H$ is symmetric and $HX = X$, also $X^{\mathsf{T}}H = X^{\mathsf{T}}$, so

$$\hat{\boldsymbol\beta} = (X^{\mathsf{T}}X)^{-1}X^{\mathsf{T}}H\mathbf{y}$$

is a function of $H\mathbf{y}$ alone. The residual vector is $(I - H)\mathbf{y}$. The two projections satisfy $H(I - H) = H - H^2 = 0$, so by part 1 of the projection theorem $H\mathbf{y}$ and $(I - H)\mathbf{y}$ are independent random vectors. Functions of independent random vectors are independent (Random Variables and Distribution Functions, Theorem 6 part 3 with the grouping fact stated there), so $\hat{\boldsymbol\beta}$ is independent of $\mathbf{e}$, and of $s^2 = \lVert\mathbf{e}\rVert^2/(n - p)$. $\square$

Two consequences are worth stating at once. The chi-squared variable with $r$ degrees of freedom has mean $r$ and variance $2r$ (the projection article), so $s^2 = \sigma^2 \cdot \chi^2_{n-p}/(n - p)$ has mean $\sigma^2$, as the Gauss–Markov article shows without normality. Its variance is $2\sigma^4/(n - p)$, which needs normality. With 46 degrees of freedom the standard deviation of $s^2$ is $\sigma^2\sqrt{2/46}$, about a fifth of $\sigma^2$: the running sample's $s^2 = 417.69$ estimates the error variance only to within roughly 20%.

Example (the arithmetic of a standard error). For the age coefficient, $\mathbf{a}$ is the second unit vector, and $\mathbf{a}^{\mathsf{T}}(X^{\mathsf{T}}X)^{-1}\mathbf{a}$ is the diagonal entry $v_{22} = 0.000062$ (0.0000619 before rounding). The estimated standard error is $s\sqrt{v_{22}} = 20.44 \times \sqrt{0.0000619} = 0.1608$ mm Hg per year. The entry $v_{22}$ is small because ages are widely spread. The formula $v_{22} = 1/S_{xx}$ of simple regression, with $S_{xx}$ the sum of squared deviations of age, is the case $p = 2$ (the first worked problem verifies it).

The t test and interval for one coefficient

Theorem 1 says that $\mathbf{a}^{\mathsf{T}}\hat{\boldsymbol\beta}$ is normal with a variance known up to $\sigma^2$, and that $s^2$ is an independent estimate of $\sigma^2$ with a known distribution. Dividing the first by the second is exactly the construction of Student's $t$, and $\sigma$ cancels.

Theorem 2 (the t statistic). Under the normal linear model, for every nonzero $\mathbf{a}$,

$$T = \frac{\mathbf{a}^{\mathsf{T}}\hat{\boldsymbol\beta} - \mathbf{a}^{\mathsf{T}}\boldsymbol\beta}{s\sqrt{\mathbf{a}^{\mathsf{T}}(X^{\mathsf{T}}X)^{-1}\mathbf{a}}}$$

has the $t$ distribution with $n - p$ degrees of freedom. In particular, for the $j$th coefficient, $(\hat\beta_j - \beta_j)/(s\sqrt{v_{jj}})$ has that distribution.

Proof. Write $v = \mathbf{a}^{\mathsf{T}}(X^{\mathsf{T}}X)^{-1}\mathbf{a}$. By part 1 of Theorem 1, $Z = (\mathbf{a}^{\mathsf{T}}\hat{\boldsymbol\beta} - \mathbf{a}^{\mathsf{T}}\boldsymbol\beta)/(\sigma\sqrt{v})$ is standard normal. By part 2, $V = \mathrm{RSS}/\sigma^2$ is chi-squared with $n - p$ degrees of freedom, and by part 3, $Z$ (a function of $\hat{\boldsymbol\beta}$) and $V$ (a function of $\mathbf{e}$) are independent. Since $s = \sigma\sqrt{V/(n - p)}$,

$$T = \frac{\sigma\sqrt{v}\,Z}{\sigma\sqrt{V/(n - p)}\,\sqrt{v}} = \frac{Z}{\sqrt{V/(n - p)}},$$

which is the definition of the $t$ distribution with $n - p$ degrees of freedom in The Chi-Squared and t Distributions. $\square$

The quantity $s\sqrt{v_{jj}}$ is the standard error of $\hat\beta_j$. Two procedures follow directly. With $t_{n-p,\,0.975}$ the upper 2.5% point of the $t$ distribution,

  • the interval $\hat\beta_j \pm t_{n-p,\,0.975}\,s\sqrt{v_{jj}}$ contains $\beta_j$ with probability exactly 0.95. The event that it does is the event $\lvert T\rvert \le t_{n-p,\,0.975}$.
  • The test of $H_0\!: \beta_j = 0$ rejects at level 0.05 when $\lvert\hat\beta_j\rvert/(s\sqrt{v_{jj}}) > t_{n-p,\,0.975}$, and its two-sided p-value is $P(\lvert T\rvert \ge \lvert t_{\text{obs}}\rvert)$ for $T$ with the $t$ distribution.

The test rejects exactly when the interval excludes zero, the duality of tests and intervals proved in Likelihood Ratio, Wald and Score Tests.

The running sample. With 46 degrees of freedom, $t_{46,\,0.975} = 2.0129$.

Coefficient Estimate Standard error $t$ p-value 95% interval
Intercept 86.04 15.49 5.55 $< 0.0001$
Age (per year) 0.4455 0.1608 2.77 0.0081 0.122 to 0.769
Female 2.34 5.89 0.40 0.69 $-9.52$ to 14.19
BMI (per kg/m²) 0.467 0.424 1.10 0.28 $-0.386$ to 1.320

Among adults of the same sex and BMI, each year of age goes with between 0.122 and 0.769 mm Hg of additional pressure, on the evidence of these 50. The population value 0.517 lies inside. Neither sex nor BMI is distinguishable from zero in this sample. The intercept is the predicted pressure of a man of age 0 with BMI 0, a point far outside the data; its test answers no useful question.

Example (not significant is not zero). The BMI interval runs from $-0.386$ to 1.320 and so contains zero, yet the coefficient for all 5,176 adults is 0.385, not zero. A test that fails to reject has not shown the coefficient to be zero; it has shown that this sample cannot tell. How often a sample of this design detects a coefficient of 0.385 is a question about power. Suppose the errors were normal with the population's residual variance, taken as the mean of the squared residuals of the population fit (their sum divided by 5,176), $\sigma^2 = 293.88$ ($\sigma = 17.14$). Then for the running sample's $X$ the statistic for BMI would be noncentral $t$ with noncentrality $0.385/(17.14\sqrt{v_{44}}) = 1.08$. The test would reject with probability 0.185 only. The experiment below draws fresh samples of 50 from the population. There the test for BMI rejected in a share 0.209 of samples (median over 30 seeds; interquartile range 0.202 to 0.213). The two rates need not agree, since each fresh sample brings its own inputs and its errors are not normal. In this experiment, about four samples in five missed the BMI association that the population has.

A second misreading concerns what the coefficient is. The test of $\beta_j = 0$ asks whether input $j$ adds to the prediction once the other columns are in the model, not whether input $j$ and the response are related. A coefficient can be insignificant because another column carries the same information, and its sign can change when a column is added. Multiple Linear Regression shows both on the diabetes data, and The Geometry of Least Squares proves the partialling-out identity behind them.

A note on history. Fisher (1925, §26) tested the slope of a fitted straight line with Student's distribution on $n' - 2$ degrees of freedom, $n'$ being the number of observations. He explained that the residuals "represent in reality only $n'-2$ degrees of freedom". In §29 he extended the test to partial regression coefficients with several independent variates, entering the $t$ table with $n' - p - 1$ degrees of freedom for $p$ variates. In this article's notation, where $p$ counts the intercept, that is $n - p$.

A mean response and a new observation

A fitted model is used to estimate the mean response at given inputs, and to predict one new response. The first is a linear combination of the coefficients, so Theorem 2 already covers it. The second adds the new observation's own error, which is independent of everything in the sample.

Theorem 3 (intervals at a point). Let $\mathbf{x}_0$ be a vector of inputs (with first entry 1), let $v_0 = \mathbf{x}_0^{\mathsf{T}}(X^{\mathsf{T}}X)^{-1}\mathbf{x}_0$, and let $Y_0 = \mathbf{x}_0^{\mathsf{T}}\boldsymbol\beta + \varepsilon_0$ be a new response whose error $\varepsilon_0$ is normal with mean 0 and variance $\sigma^2$ and independent of $\boldsymbol\varepsilon$. Under the normal linear model,

  1. $\mathbf{x}_0^{\mathsf{T}}\hat{\boldsymbol\beta} \pm t_{n-p,\,0.975}\,s\sqrt{v_0}$ contains the mean response $\mathbf{x}_0^{\mathsf{T}}\boldsymbol\beta$ with probability 0.95;
  2. $\mathbf{x}_0^{\mathsf{T}}\hat{\boldsymbol\beta} \pm t_{n-p,\,0.975}\,s\sqrt{1 + v_0}$ contains $Y_0$ with probability 0.95.

Proof. Part 1 is Theorem 2 with $\mathbf{a} = \mathbf{x}_0$. For part 2, $Y_0 - \mathbf{x}_0^{\mathsf{T}}\hat{\boldsymbol\beta} = \varepsilon_0 - (\mathbf{x}_0^{\mathsf{T}}\hat{\boldsymbol\beta} - \mathbf{x}_0^{\mathsf{T}}\boldsymbol\beta) = \varepsilon_0 - \mathbf{c}^{\mathsf{T}}\boldsymbol\varepsilon$ with $\mathbf{c} = X(X^{\mathsf{T}}X)^{-1}\mathbf{x}_0$, as in the proof of Theorem 1. The vector $(\varepsilon_1, \ldots, \varepsilon_n, \varepsilon_0)$ has $n + 1$ independent normal coordinates with variance $\sigma^2$ and mean 0. Applying the argument of part 1 of Theorem 1 to it, with the vector $(-\mathbf{c}, 1)$ of squared length $\lVert\mathbf{c}\rVert^2 + 1 = v_0 + 1$, shows that $Y_0 - \mathbf{x}_0^{\mathsf{T}}\hat{\boldsymbol\beta}$ is normal with mean 0 and variance $\sigma^2(1 + v_0)$. It is a function of $H\boldsymbol\varepsilon$ and $\varepsilon_0$ (since $\mathbf{c}^{\mathsf{T}}\boldsymbol\varepsilon = \mathbf{c}^{\mathsf{T}}H\boldsymbol\varepsilon$, $\mathbf{c}$ lying in $\mathcal{C}(X)$), while $s^2$ is a function of $(I - H)\boldsymbol\varepsilon$. These are independent: in $n + 1$ dimensions, the block matrices $P_1 = \begin{pmatrix} H & \mathbf{0} \\ \mathbf{0}^{\mathsf{T}} & 1\end{pmatrix}$ and $P_2 = \begin{pmatrix} I - H & \mathbf{0} \\ \mathbf{0}^{\mathsf{T}} & 0\end{pmatrix}$ are projections with $P_1P_2 = 0$, and the projection theorem applied to the vector $(\boldsymbol\varepsilon, \varepsilon_0)$ makes $P_1(\boldsymbol\varepsilon, \varepsilon_0) = (H\boldsymbol\varepsilon, \varepsilon_0)$ independent of $P_2(\boldsymbol\varepsilon, \varepsilon_0) = ((I - H)\boldsymbol\varepsilon, 0)$. Dividing as in the proof of Theorem 2 gives a $t$ variable with $n - p$ degrees of freedom, and the interval is the event that it lies within $\pm t_{n-p,\,0.975}$. $\square$

Example (the simple-regression formulas). With one input, $X$ has columns $\mathbf{1}$ and $\mathbf{x}$, and

$$X^{\mathsf{T}}X = \begin{pmatrix} n & \sum x_i \\ \sum x_i & \sum x_i^2\end{pmatrix}, \qquad (X^{\mathsf{T}}X)^{-1} = \frac{1}{nS_{xx}}\begin{pmatrix} \sum x_i^2 & -\sum x_i \\ -\sum x_i & n\end{pmatrix},$$

because the determinant is $n\sum x_i^2 - (\sum x_i)^2 = nS_{xx}$ with $S_{xx} = \sum(x_i - \bar{x})^2$. For $\mathbf{x}_0 = (1, x_0)$,

$$v_0 = \frac{\sum x_i^2 - 2x_0\sum x_i + nx_0^2}{nS_{xx}} = \frac{S_{xx} + n\bar{x}^2 - 2nx_0\bar{x} + nx_0^2}{nS_{xx}} = \frac1n + \frac{(x_0 - \bar{x})^2}{S_{xx}},$$

using $\sum x_i^2 = S_{xx} + n\bar{x}^2$. These are the standard errors $s\sqrt{1/n + (x_0 - \bar{x})^2/S_{xx}}$ and $s\sqrt{1 + 1/n + (x_0 - \bar{x})^2/S_{xx}}$ that Simple Linear Regression quotes for the two intervals; Theorem 3 proves them. The entry $n/(nS_{xx}) = 1/S_{xx}$ in the corner is the variance factor of the slope.

The running sample. For a 60-year-old woman with BMI 30, $\mathbf{x}_0 = (1, 60, 1, 30)$, the predicted pressure is 129.11 mm Hg and $v_0 = 0.0451$. The standard error of the mean response is $20.44\sqrt{0.0451} = 4.34$, and that of a prediction is $20.44\sqrt{1.0451} = 20.89$. The 95% interval for the mean pressure of all such women runs from 120.4 to 137.8 mm Hg. The 95% prediction interval for one such woman runs from 87.1 to 171.2. As the sample grows, the first shrinks toward zero width, since $v_0$ does. The half-width of the second approaches $1.96\,\sigma$ instead, since $t_{n-p,\,0.975}$ approaches 1.96, $s$ approaches $\sigma$, and the new woman's own error does not average away.

Testing all the slopes at once

The table tested each coefficient on its own. A different question is whether the inputs together explain anything: is every slope zero, $\beta_2 = \cdots = \beta_p = 0$? The answer compares the variation the fit explains with the variation it leaves.

Write $\mathrm{TSS} = \lVert\mathbf{y} - \bar{y}\mathbf{1}\rVert^2$ for the total sum of squares about the mean and $\mathrm{ESS} = \lVert\hat{\mathbf{y}} - \bar{y}\mathbf{1}\rVert^2$ for the explained sum of squares, so that $\mathrm{TSS} = \mathrm{ESS} + \mathrm{RSS}$ (The Geometry of Least Squares) and $R^2 = \mathrm{ESS}/\mathrm{TSS}$.

Theorem 4 (the overall F test). Under the normal linear model, with $\mathbf{1}$ a column of $X$ and $p \ge 2$, the statistic

$$F = \frac{\mathrm{ESS}/(p - 1)}{\mathrm{RSS}/(n - p)} = \frac{R^2/(p - 1)}{(1 - R^2)/(n - p)}$$

has the noncentral $F$ distribution with $p - 1$ and $n - p$ degrees of freedom and noncentrality

$$\lambda = \frac{\sum_i(\mu_i - \bar\mu)^2}{\sigma^2}, \qquad \boldsymbol\mu = X\boldsymbol\beta,\ \bar\mu = \tfrac1n\textstyle\sum_i\mu_i.$$

If all slopes are zero, $\lambda = 0$ and $F$ has the central $F$ distribution with $p - 1$ and $n - p$ degrees of freedom.

The idea: the explained part $\hat{\mathbf{y}} - \bar{y}\mathbf{1}$ is the projection of $\mathbf{y}$ onto the part of $\mathcal{C}(X)$ perpendicular to $\mathbf{1}$. That part is perpendicular to the residual space, so the two squared lengths are independent chi-squared variables.

Proof. Since $\hat{\mathbf{y}} = H\mathbf{y}$ and $\bar{y}\mathbf{1} = P_{\mathbf 1}\mathbf{y}$, $\mathrm{ESS} = \lVert(H - P_{\mathbf 1})\mathbf{y}\rVert^2$. As recalled above, $H - P_{\mathbf 1}$ is a projection of rank $p - 1$, and

$$(H - P_{\mathbf 1})(I - H) = H - H^2 - P_{\mathbf 1} + P_{\mathbf 1}H = H - H - P_{\mathbf 1} + P_{\mathbf 1} = 0.$$

By the projection theorem, $\mathrm{ESS}$ and $\mathrm{RSS}$ are independent. $\mathrm{RSS}/\sigma^2$ is central chi-squared with $n - p$ degrees of freedom (Theorem 1), and $\mathrm{ESS}/\sigma^2$ is chi-squared with $p - 1$ degrees of freedom and noncentrality $\lVert(H - P_{\mathbf 1})X\boldsymbol\beta\rVert^2/\sigma^2$. Now $HX\boldsymbol\beta = X\boldsymbol\beta = \boldsymbol\mu$ and $P_{\mathbf 1}\boldsymbol\mu = \bar\mu\mathbf{1}$, so the noncentrality is $\lVert\boldsymbol\mu - \bar\mu\mathbf{1}\rVert^2/\sigma^2 = \lambda$. A noncentral chi-squared variable over its degrees of freedom, divided by an independent central one over its own, is noncentral $F$ by the definition in the projection article. If $\beta_2 = \cdots = \beta_p = 0$, then $\boldsymbol\mu = \beta_1\mathbf{1}$ is constant and $\lambda = 0$. The second form of $F$ follows on dividing the numerator and the denominator by $\mathrm{TSS}$. $\square$

The converse also holds: if $\lambda = 0$ then $\boldsymbol\mu = X\boldsymbol\beta$ is a constant vector $c\mathbf{1} = X(c, 0, \ldots, 0)^{\mathsf{T}}$, and because the columns of $X$ are independent, $\boldsymbol\beta = (c, 0, \ldots, 0)^{\mathsf{T}}$. So the null hypothesis of the test is exactly "all slopes zero".

The running sample. $\mathrm{TSS} = 23{,}025.0$, $\mathrm{ESS} = 3{,}811.4$ and $R^2 = 0.1655$, so

$$F = \frac{3{,}811.4/3}{19{,}213.6/46} = 3.042,$$

on 3 and 46 degrees of freedom. The upper 5% point is 2.8068 and the p-value 0.0382: the three inputs together explain more than chance would, at the 5% level. That is no surprise after the age coefficient's $t$ of 2.77, but the two tests need not agree, and the next section says exactly when they can disagree.

Power. If the errors were normal with the population's variance and the coefficients were the population's, $\boldsymbol\mu = X\boldsymbol\beta$ would be computed from the population coefficients at the running sample's inputs, and $\lambda = 15.31$. The power of the test at the 5% level is then 0.897: the probability that a noncentral $F$ variable with 3 and 46 degrees of freedom and noncentrality 15.31 exceeds 2.8068. The noncentral $F$ distribution and its series are derived in the projection article.

Joint confidence regions and simultaneous intervals

The interval for the age coefficient covers it with probability 0.95, and so does each of the other three intervals. The probability that all four cover at once is lower, and depends on how the estimates are correlated. A statement about all the coefficients together needs a region in $p$ dimensions.

Theorem 5 (the joint confidence region). Under the normal linear model,

$$\frac{(\hat{\boldsymbol\beta} - \boldsymbol\beta)^{\mathsf{T}}X^{\mathsf{T}}X(\hat{\boldsymbol\beta} - \boldsymbol\beta)}{p\,s^2}$$

has the $F$ distribution with $p$ and $n - p$ degrees of freedom. Hence the ellipsoid of vectors $\mathbf{b}$ with $(\hat{\boldsymbol\beta} - \mathbf{b})^{\mathsf{T}}X^{\mathsf{T}}X(\hat{\boldsymbol\beta} - \mathbf{b}) \le p\,s^2F_{p,\,n-p,\,0.95}$ contains $\boldsymbol\beta$ with probability 0.95.

Proof. $X(\hat{\boldsymbol\beta} - \boldsymbol\beta) = H\mathbf{y} - X\boldsymbol\beta = H(\mathbf{y} - X\boldsymbol\beta) = H\boldsymbol\varepsilon$, so the numerator is $\lVert X(\hat{\boldsymbol\beta} - \boldsymbol\beta)\rVert^2 = \lVert H\boldsymbol\varepsilon\rVert^2$. The errors have mean vector $\mathbf{0}$, so by the projection theorem $\lVert H\boldsymbol\varepsilon\rVert^2/\sigma^2$ is central chi-squared with $p$ degrees of freedom and is independent of $\lVert(I - H)\boldsymbol\varepsilon\rVert^2/\sigma^2 = \mathrm{RSS}/\sigma^2$, which is chi-squared with $n - p$. The ratio, each over its degrees of freedom, is the statistic, with $\sigma^2$ canceled. $\square$

The region answers a question no list of intervals answers: which coefficient vectors are consistent with the data jointly. It also yields intervals for every linear combination at once.

Theorem 6 (simultaneous intervals). Let $c = \sqrt{p\,F_{p,\,n-p,\,0.95}}$. Under the normal linear model, the probability is 0.95 that

$$\big\lvert\mathbf{a}^{\mathsf{T}}\hat{\boldsymbol\beta} - \mathbf{a}^{\mathsf{T}}\boldsymbol\beta\big\rvert \le c\,s\sqrt{\mathbf{a}^{\mathsf{T}}(X^{\mathsf{T}}X)^{-1}\mathbf{a}} \quad\text{for every vector } \mathbf{a} \text{ at once}.$$

The idea: the largest standardized error over all directions $\mathbf{a}$ is the squared length that Theorem 5 controls. So bounding all of them is the same event as $\boldsymbol\beta$ lying in the ellipsoid.

Proof. Write $\boldsymbol\delta = \hat{\boldsymbol\beta} - \boldsymbol\beta$ and, for a nonzero $\mathbf{a}$, $\mathbf{u} = (X^{\mathsf{T}}X)^{-1}\mathbf{a}$. Then $\mathbf{a}^{\mathsf{T}}\boldsymbol\delta = (X\mathbf{u})\cdot(X\boldsymbol\delta)$, a dot product of two vectors in $n$ dimensions, and $\lVert X\mathbf{u}\rVert^2 = \mathbf{a}^{\mathsf{T}}(X^{\mathsf{T}}X)^{-1}\mathbf{a}$. By the Cauchy–Schwarz inequality for vectors (Probability Inequalities, corollary on sums of numbers),

$$(\mathbf{a}^{\mathsf{T}}\boldsymbol\delta)^2 \le \lVert X\mathbf{u}\rVert^2\,\lVert X\boldsymbol\delta\rVert^2 = \mathbf{a}^{\mathsf{T}}(X^{\mathsf{T}}X)^{-1}\mathbf{a}\;\cdot\;\boldsymbol\delta^{\mathsf{T}}X^{\mathsf{T}}X\boldsymbol\delta,$$

with equality when $\mathbf{a} = X^{\mathsf{T}}X\boldsymbol\delta$ (then $X\mathbf{u} = X\boldsymbol\delta$). So the largest value over nonzero $\mathbf{a}$ of $(\mathbf{a}^{\mathsf{T}}\boldsymbol\delta)^2/[\mathbf{a}^{\mathsf{T}}(X^{\mathsf{T}}X)^{-1}\mathbf{a}]$ is $\boldsymbol\delta^{\mathsf{T}}X^{\mathsf{T}}X\boldsymbol\delta$ if $\boldsymbol\delta \ne \mathbf{0}$, and 0 otherwise. All the inequalities of the theorem hold if and only if this largest value is at most $c^2s^2 = p\,s^2F_{p,\,n-p,\,0.95}$, which by Theorem 5 has probability 0.95. $\square$

The NIST/SEMATECH handbook (§7.4.7.2) calls the corresponding procedure for all contrasts among group means Scheffé's method, with multiplier $\sqrt{(r - 1)F}$ for $r$ groups. Theorem 6 is the same construction for every linear combination of regression coefficients. The price of simultaneity is a larger multiplier. For the running sample, $F_{4,\,46,\,0.95} = 2.5740$ and $c = \sqrt{4 \times 2.5740} = 3.2088$, against $t_{46,\,0.975} = 2.0129$ for one interval. The simultaneous interval for the age coefficient is $0.4455 \pm 3.2088 \times 0.1608$, from $-0.071$ to 0.961, and it contains zero. Age is significant on its own, but not after allowing for every question that might have been asked of these coefficients. If only the three slopes are of interest, the dimension is 3 and the multiplier $\sqrt{3F_{3,\,46,\,0.95}} = 2.9018$; The General Linear Hypothesis gives the region for any subset.

Theorem 6 also explains when the overall $F$ test and the individual $t$ tests can disagree. Applying the same argument to the slopes alone shows that the overall $F$ test rejects at level 0.05 exactly when some combination of the slopes $\mathbf{a}^{\mathsf{T}}\boldsymbol\beta$ (with $a_1 = 0$) has $\lvert t\rvert$ larger than $\sqrt{(p - 1)F_{p-1,\,n-p,\,0.95}}$; The General Linear Hypothesis proves that statement. The combination need not be a single coefficient. So a rejection by $F$ with no single significant $t$ means the evidence lies in a combination of slopes. A single significant $t$ with no rejection by $F$ means that $\lvert t\rvert$ lies between $t_{n-p,\,0.975}$ and $\sqrt{(p - 1)F_{p-1,\,n-p,\,0.95}}$, between 2.0129 and 2.9018 for the running sample.

How much depends on normality: an experiment

Everything above assumed that the errors are independent, normal and of constant variance. The NHANES population satisfies none of this exactly. Fit to all 5,176 adults, the model leaves residuals with skewness 0.88 and kurtosis 5.28 (a normal distribution has 0 and 3). Their root mean square is 11.7 mm Hg among adults aged 18 to 39 but 20.5 among those 60 and over. And no one believes pressure is exactly linear in age and BMI.

What can a sample of 50 estimate, then? The natural target is the population least-squares coefficient $\boldsymbol\beta^*$, the fit of the same model to all 5,176 adults: 90.05 for the intercept, 0.517 per year of age, $-1.92$ for women and 0.385 per kg/m² of BMI. It is the best linear description of the population in the mean-square sense, whether or not the population is linear. The experiment asks how often the intervals of this article contain it.

Design. For each of 30 seeds (numpy.random.default_rng(1000 + seed), seeds 0 to 29), 2,000 times: draw 51 adults without replacement and fit the model to the first 50. Compute the 95% $t$ intervals for the three slopes, the 95% joint region of Theorem 5 for all four coefficients, and the 95% prediction interval for the 51st adult. Record whether each contains its target. As a control, each replication also generates a response vector from the normal linear model itself, with the running sample's $X$, the coefficients $\boldsymbol\beta^*$, and normal errors with mean 0 and variance 293.88, the mean of the squared population residuals. The table gives the median coverage over the 30 seeds, with the interquartile range.

Interval (95%) Coverage, NHANES samples IQR Coverage, normal control IQR
Age slope 0.951 0.947 to 0.955 0.950 0.946 to 0.952
Female coefficient 0.950 0.947 to 0.952 0.951 0.948 to 0.952
BMI slope 0.947 0.944 to 0.951 0.951 0.947 to 0.953
Joint region, four coefficients 0.942 0.938 to 0.945 0.950 0.947 to 0.952
Prediction, one new adult 0.945 0.941 to 0.947

In the control the coverage is 0.95 to within the simulation's noise, as Theorems 2 and 5 require. In the NHANES samples the intervals for single coefficients covered the population coefficients at close to the nominal rate. The joint region covered somewhat less often, at 0.942, and the prediction interval at 0.945. These are observations of this experiment, for this population, this model and samples of 50; they are not a theorem about non-normal data. A plausible reading, not a proof: a coefficient is a weighted sum of many observations, the kind of quantity that central limit theorems make nearly normal. (Convergence in Distribution and the Proof of the Central Limit Theorem proves the case of equal weights and identical distributions.) A prediction interval rests on one new observation's error, to which no averaging applies. The prediction interval's overall rate also hides a split by age. Over the same replications, the new adult's pressure fell inside the interval in a share 0.987 of cases when the new adult was aged 18 to 39 (median over seeds; interquartile range 0.985 to 0.990). It did so in 0.905 when the new adult was aged 60 or over (0.900 to 0.911). One width for all ages is too wide for the young and too narrow for the old, as the residual spreads of 11.7 and 20.5 mm Hg suggest.

When the variance is not constant the usual standard error is not the right one. Linear Regression Assumptions describes the heteroscedasticity-consistent standard errors that replace it, with a simulation of what each failure of the assumptions costs. Leverage, Influence and Residual Diagnostics shows how to see such failures in a single sample's residuals.

What is proved, what is cited, what the experiment shows

Proved here. Theorem 1 (normality of every linear combination of the coefficients, the chi-squared distribution of the residual sum of squares, and their independence) from the rotation and projection theorems; the $t$ statistic for any linear combination (Theorem 2) with its tests and intervals; the intervals for a mean response and for a new observation (Theorem 3) and their simple-regression forms; the overall $F$ test with its noncentrality and the identification of its null hypothesis (Theorem 4); the joint confidence ellipsoid (Theorem 5); the simultaneous intervals for all linear combinations (Theorem 6).

Taken from other articles, where they are proved. The geometry of the hat matrix and the nested projections (The Geometry of Least Squares); the mean and covariance of $\hat{\boldsymbol\beta}$ and the unbiasedness of $s^2$ (The Gauss–Markov Theorem); the rotation and projection theorems, the moments of the chi-squared distribution and the noncentral $F$ distribution (Sampling Distributions from the Normal); the definition of $t$ (The Chi-Squared and t Distributions); independence of functions of independent vectors (Random Variables and Distribution Functions); the Cauchy–Schwarz inequality (Probability Inequalities).

Cited. Fisher's (1925) use of the $t$ distribution for regression coefficients; the name of Scheffé's method for contrasts (NIST/SEMATECH, §7.4.7.2). Percentage points and p-values of the $t$, $F$ and noncentral distributions are computed by scipy.

The experiment shows coverage rates for samples of 50 from one finite population, observed over 30 seeds; it shows nothing about other populations or sample sizes.

Worked problems

These problems were written for this article, on the unit's NHANES samples and on the diabetes study of the data science pages. The script tools/experiments/statistics/normal-linear-model.py recomputes every number.

Problem 1 (a hand-sized regression, two ways). The unit's 12-adult sample has ages 54, 39, 43, 61, 63, 69, 19, 21, 28, 27, 77 and 67 and mean systolic pressures (mm Hg) 129.33, 116.00, 119.33, 148.00, 144.00, 125.33, 106.67, 111.33, 119.33, 97.33, 112.67 and 123.33. Fit the simple regression of pressure on age, test the slope, give its 95% interval, and confirm that the general formula $s\sqrt{v_{22}}$ gives the simple-regression standard error.

Solution. The slope is 0.4145 mm Hg per year with standard error 0.1823, $t = 2.27$ on 10 degrees of freedom, p-value 0.0462, and the 95% interval runs from 0.008 to 0.821.

The mean age is 47.33 and the mean pressure 121.06. The sums of squares and products about the means are $S_{xx} = 4{,}584.67$ and $S_{xy} = 1{,}900.44$, so the slope is $b_1 = 1{,}900.44/4{,}584.67 = 0.4145$ and the intercept $b_0 = 121.06 - 0.4145 \times 47.33 = 101.43$. The residual sum of squares is 1,522.85 on $12 - 2 = 10$ degrees of freedom, so $s^2 = 152.29$, and the standard error is $\sqrt{152.29/4{,}584.67} = 0.1823$.

In matrix form, the ages sum to 568 and their squares to 31,470, so $X^{\mathsf{T}}X$ has entries 12, 568, 568 and 31,470, with determinant $12 \times 31{,}470 - 568^2 = 55{,}016 = 12 \times 4{,}584.67$. The lower right entry of the inverse is $12/55{,}016 = 1/4{,}584.67 = 1/S_{xx}$, as the example after Theorem 3 showed in general. So $s\sqrt{v_{22}}$ is the same 0.1823.

By Theorem 2 the statistic $0.4145/0.1823 = 2.27$ is to be referred to the $t$ distribution with 10 degrees of freedom, whose upper 2.5% point is 2.2281. Since $2.27 > 2.2281$ the slope differs from zero at the 5% level, barely: $p = 0.0462$. The interval is $0.4145 \pm 2.2281 \times 0.1823$, from 0.008 to 0.821. It is wide: the twelve adults say little more than that pressure rises with age.

Problem 2 (the standard errors of the data science pages). Multiple Linear Regression reports, for the 442 patients of the diabetes study (Efron, Hastie, Johnstone and Tibshirani, 2004), a BMI coefficient of 6.500 with standard error 0.697 in the model with BMI, blood pressure and S5, and a coefficient of $-13.04$ with standard error 5.46 and $p = 0.017$ for the sex indicator added to that model. Say what distribution justifies each p-value, and compute the 95% interval for the BMI coefficient.

Solution. Both come from Theorem 2: the BMI interval is 5.130 to 7.870 on 438 degrees of freedom, and the sex coefficient's $t = -2.39$ on 437 degrees of freedom gives $p = 0.0174$.

The first model has $p = 4$ columns, so $n - p = 442 - 4 = 438$ residual degrees of freedom. Its residual standard deviation is $s = 55.78$, the variance factor of the BMI coefficient is $v = 0.0001562$, and the standard error is $55.78\sqrt{0.0001562} = 0.697$. The statistic $6.500/0.697 = 9.32$ is referred to the $t$ distribution with 438 degrees of freedom, whose upper 2.5% point is 1.9654, so the interval is $6.500 \pm 1.9654 \times 0.697$, from 5.130 to 7.870. With the sex indicator the model has 5 columns and 437 degrees of freedom. Then $t = -13.04/5.46 = -2.39$, and $P(\lvert T\rvert \ge 2.39) = 0.0174$ for $T$ with the $t$ distribution on 437 degrees of freedom, whose 2.5% point is also 1.9654 to four places. The page's "$p = 0.017$" is this value rounded.

With hundreds of degrees of freedom the $t$ distribution is close to the standard normal, so normality of the errors matters little for these intervals. What the theorem still requires is independence and a common variance, which Linear Regression Assumptions examines for these data.

Problem 3 (one comparison, two kinds of interval). On the running sample, estimate how much higher the mean pressure of a 60-year-old woman with BMI 30 is than that of a 30-year-old man with BMI 25, with a 95% $t$ interval and with a 95% simultaneous interval of Theorem 6. Which interval is appropriate?

Solution. The estimate is 18.04 mm Hg; the $t$ interval runs from 3.19 to 32.88 and the simultaneous interval from $-5.63$ to 41.71.

The two adults differ by 30 years of age, by sex and by 5 units of BMI, so the difference of their mean pressures is $\mathbf{a}^{\mathsf{T}}\boldsymbol\beta$ with $\mathbf{a} = (0, 30, 1, 5)$; the intercept cancels. The estimate is $30 \times 0.4455 + 2.34 + 5 \times 0.467 = 18.04$. The variance factor is $\mathbf{a}^{\mathsf{T}}(X^{\mathsf{T}}X)^{-1}\mathbf{a} = 0.1303$, so the standard error is $20.44\sqrt{0.1303} = 7.38$ and $t = 18.04/7.38 = 2.44$. The $t$ interval is $18.04 \pm 2.0129 \times 7.38$, from 3.19 to 32.88, and the simultaneous interval is $18.04 \pm 3.2088 \times 7.38$, from $-5.63$ to 41.71. The population coefficients give 15.51, inside both.

The $t$ interval is correct if this comparison was chosen before the data were seen. Suppose instead it was picked, among all the comparisons that could have been made, because it looked large. Then only the simultaneous interval keeps its 95% guarantee, because Theorem 6 holds for every $\mathbf{a}$ at once, including one chosen after looking. Multiple Comparisons and the Bonferroni Correction discusses the same choice for group means.

Summary

Result Statement (normal errors, $X$ of rank $p$)
Theorem 1 $\mathbf{a}^{\mathsf{T}}\hat{\boldsymbol\beta}$ normal, mean $\mathbf{a}^{\mathsf{T}}\boldsymbol\beta$, variance $\sigma^2\mathbf{a}^{\mathsf{T}}(X^{\mathsf{T}}X)^{-1}\mathbf{a}$; $\mathrm{RSS}/\sigma^2 \sim \chi^2_{n-p}$; $\hat{\boldsymbol\beta}$ independent of $\mathbf{e}$
Theorem 2 $(\hat\beta_j - \beta_j)/(s\sqrt{v_{jj}})$ is $t$ with $n - p$ degrees of freedom
Theorem 3 mean response: $\mathbf{x}_0^{\mathsf{T}}\hat{\boldsymbol\beta} \pm t\,s\sqrt{v_0}$; new observation: $\pm t\,s\sqrt{1 + v_0}$
Theorem 4 $F = \frac{\mathrm{ESS}/(p-1)}{\mathrm{RSS}/(n-p)}$, noncentral $F$ with $\lambda = \lVert\boldsymbol\mu - \bar\mu\mathbf{1}\rVert^2/\sigma^2$
Theorem 5 $(\hat{\boldsymbol\beta} - \boldsymbol\beta)^{\mathsf{T}}X^{\mathsf{T}}X(\hat{\boldsymbol\beta} - \boldsymbol\beta)/(p s^2) \sim F_{p,\,n-p}$
Theorem 6 all $\mathbf{a}$ at once: multiplier $\sqrt{pF_{p,\,n-p,\,0.95}}$ in place of $t_{n-p,\,0.975}$

What we have learned

When the errors are independent and normal with a common variance, least-squares regression has an exact sampling theory. The coefficients depend on the data through one projection and the residuals through the perpendicular one, so they are independent. Every linear combination of the coefficients is normal, and the residual sum of squares is chi-squared on $n - p$ degrees of freedom. Dividing gives the $t$ statistic for any coefficient or combination, and with it the tests, the intervals for a mean response and the prediction intervals that regression software reports. The explained and residual sums of squares give the overall $F$ test, and the length of $X(\hat{\boldsymbol\beta} - \boldsymbol\beta)$ gives a joint region whose shadows are Scheffé's simultaneous intervals. For 50 NHANES adults, age is significant on its own ($t = 2.77$), sex and BMI are not, and the three together are ($F = 3.042$, $p = 0.0382$). In repeated samples from a population that is neither normal nor of constant variance the single-coefficient intervals covered at close to their nominal rate.

The overall test and the single-coefficient tests are two cases of one question: does a set of linear restrictions on $\boldsymbol\beta$ fit the data? Whether sex and BMI together add to age, whether two coefficients are equal, whether a set of group means coincide: The General Linear Hypothesis answers them all with one $F$ statistic.

References