In the 50 NHANES adults of The Normal Linear Model, sex and body mass index (BMI) were each indistinguishable from zero once age was in the model. Their $t$ statistics were 0.40 and 1.10. Two separate tests are not a test of the pair. Do sex and BMI together add anything to age?
The question is one hypothesis about two coefficients, $\beta_{\text{female}} = \beta_{\text{BMI}} = 0$. A general linear hypothesis is any set of $q$ linear restrictions on the coefficients, written $C\boldsymbol\beta = \mathbf{d}$ with a known $q \times p$ matrix $C$ and a known vector $\mathbf{d}$. That some coefficients are zero, that two coefficients are equal, that a coefficient has a stated value, that a set of group means coincide: all are of this form. This article proves that one $F$ statistic tests every such hypothesis exactly under the normal linear model. For sex and BMI it gives $F = 0.784$ on 2 and 46 degrees of freedom, $p = 0.462$: the pair adds nothing detectable.
First we derive the least-squares fit under the restrictions. The cost of imposing them, the increase in the residual sum of squares, turns out to equal a quadratic form in the estimated restrictions. Then we prove the $F$ distribution of the statistic, with its noncentrality under the alternative, and draw the joint confidence region it implies. Next we show that the $F$ test is the likelihood ratio test. The $t$ test, the overall $F$ test, one-way analysis of variance and the two-sample $t$-test are its cases. Finally we compute power, by the noncentral $F$ distribution and by an experiment. The article assumes The Normal Linear Model: t and F Tests for Coefficients and its notation, which is that of The Geometry of Least Squares.
Linear hypotheses
Throughout, $\mathbf{y} = X\boldsymbol\beta + \boldsymbol\varepsilon$ with $X$ of size $n \times p$ and rank $p$, $\hat{\boldsymbol\beta} = (X^{\mathsf{T}}X)^{-1}X^{\mathsf{T}}\mathbf{y}$, $H = X(X^{\mathsf{T}}X)^{-1}X^{\mathsf{T}}$, $\mathrm{RSS} = \lVert\mathbf{y} - X\hat{\boldsymbol\beta}\rVert^2$ and $s^2 = \mathrm{RSS}/(n - p)$. A hypothesis is
$$H_0\colon\ C\boldsymbol\beta = \mathbf{d},$$
where $C$ is $q \times p$ with linearly independent rows ($q \le p$) and $\mathbf{d}$ has length $q$. Independent rows mean that no restriction is implied by the others, so that $q$ counts restrictions. In the running example the coefficients are ordered intercept, age, female, BMI, and:
- sex and BMI both zero: $C = \begin{pmatrix} 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1\end{pmatrix}$, $\mathbf{d} = (0, 0)^{\mathsf{T}}$, $q = 2$;
- the age coefficient equals 0.5: $C = (0, 1, 0, 0)$, $\mathbf{d} = 0.5$, $q = 1$;
- all slopes zero: $C = \begin{pmatrix}\mathbf{0} & I_3\end{pmatrix}$, $\mathbf{d} = \mathbf{0}$, $q = 3$.
The statistic will be built from two quantities, and the first theorem proves them equal. One is the extra sum of squares: fit the model again with the restrictions imposed, and see how much the residual sum of squares grows. The other is the Wald form: estimate $C\boldsymbol\beta - \mathbf{d}$ by $C\hat{\boldsymbol\beta} - \mathbf{d}$ and measure its size in the metric its covariance supplies. By the Gauss–Markov article, $\operatorname{Cov}(C\hat{\boldsymbol\beta}) = \sigma^2 M$ with
$$M = C(X^{\mathsf{T}}X)^{-1}C^{\mathsf{T}},$$
so the natural size is $Q = (C\hat{\boldsymbol\beta} - \mathbf{d})^{\mathsf{T}}M^{-1}(C\hat{\boldsymbol\beta} - \mathbf{d})$. The matrix $M$ is invertible. For a nonzero vector $\mathbf{z}$ of length $q$, the vector $C^{\mathsf{T}}\mathbf{z}$ is not zero because the rows of $C$ are independent. Since $(X^{\mathsf{T}}X)^{-1}$ is positive definite, $\mathbf{z}^{\mathsf{T}}M\mathbf{z} = (C^{\mathsf{T}}\mathbf{z})^{\mathsf{T}}(X^{\mathsf{T}}X)^{-1}(C^{\mathsf{T}}\mathbf{z}) > 0$. Hence $M$ is positive definite, and so is $M^{-1}$.
Least squares under the restrictions
Here is the idea. Any coefficient vector $\mathbf{b}$ leaves a residual sum of squares equal to $\mathrm{RSS}$ plus a penalty for its distance from $\hat{\boldsymbol\beta}$. Minimizing the penalty over the vectors that satisfy the restrictions is a problem of projecting $\hat{\boldsymbol\beta}$ onto a flat set, and its solution can be written down and verified.
Theorem 1 (restricted least squares). Among all vectors $\mathbf{b}$ with $C\mathbf{b} = \mathbf{d}$, the residual sum of squares $\lVert\mathbf{y} - X\mathbf{b}\rVert^2$ is smallest at
$$\tilde{\boldsymbol\beta} = \hat{\boldsymbol\beta} - (X^{\mathsf{T}}X)^{-1}C^{\mathsf{T}}M^{-1}(C\hat{\boldsymbol\beta} - \mathbf{d}),$$
and only there. The minimum, $\mathrm{RSS}_0 = \lVert\mathbf{y} - X\tilde{\boldsymbol\beta}\rVert^2$, satisfies
$$\mathrm{RSS}_0 - \mathrm{RSS} = (C\hat{\boldsymbol\beta} - \mathbf{d})^{\mathsf{T}}M^{-1}(C\hat{\boldsymbol\beta} - \mathbf{d}) = Q.$$
Proof. A penalty for leaving $\hat{\boldsymbol\beta}$. For any $\mathbf{b}$, $\mathbf{y} - X\mathbf{b} = \mathbf{e} + X(\hat{\boldsymbol\beta} - \mathbf{b})$, where $\mathbf{e} = \mathbf{y} - X\hat{\boldsymbol\beta}$ is perpendicular to every column of $X$ by the normal equations. By Pythagoras,
$$\lVert\mathbf{y} - X\mathbf{b}\rVert^2 = \mathrm{RSS} + g(\mathbf{b}), \qquad g(\mathbf{b}) = (\hat{\boldsymbol\beta} - \mathbf{b})^{\mathsf{T}}X^{\mathsf{T}}X(\hat{\boldsymbol\beta} - \mathbf{b}).$$
So it is enough to minimize $g$ subject to $C\mathbf{b} = \mathbf{d}$.
The candidate satisfies the restrictions. Write $\mathbf{w} = M^{-1}(C\hat{\boldsymbol\beta} - \mathbf{d})$, so that $\hat{\boldsymbol\beta} - \tilde{\boldsymbol\beta} = (X^{\mathsf{T}}X)^{-1}C^{\mathsf{T}}\mathbf{w}$. Then $C\tilde{\boldsymbol\beta} = C\hat{\boldsymbol\beta} - M\mathbf{w} = C\hat{\boldsymbol\beta} - (C\hat{\boldsymbol\beta} - \mathbf{d}) = \mathbf{d}$.
It is the minimizer. Any $\mathbf{b}$ with $C\mathbf{b} = \mathbf{d}$ is $\tilde{\boldsymbol\beta} + \boldsymbol\delta$ with $C\boldsymbol\delta = \mathbf{0}$. Then $\hat{\boldsymbol\beta} - \mathbf{b} = (\hat{\boldsymbol\beta} - \tilde{\boldsymbol\beta}) - \boldsymbol\delta$, and the cross term in $g(\mathbf{b})$ vanishes:
$$\boldsymbol\delta^{\mathsf{T}}X^{\mathsf{T}}X(\hat{\boldsymbol\beta} - \tilde{\boldsymbol\beta}) = \boldsymbol\delta^{\mathsf{T}}C^{\mathsf{T}}\mathbf{w} = (C\boldsymbol\delta)^{\mathsf{T}}\mathbf{w} = 0.$$
So $g(\mathbf{b}) = g(\tilde{\boldsymbol\beta}) + \boldsymbol\delta^{\mathsf{T}}X^{\mathsf{T}}X\boldsymbol\delta$. The last term is $\lVert X\boldsymbol\delta\rVert^2$, which is positive unless $\boldsymbol\delta = \mathbf{0}$, because the columns of $X$ are independent. Hence $\tilde{\boldsymbol\beta}$ is the unique minimizer.
The minimum. $g(\tilde{\boldsymbol\beta}) = \mathbf{w}^{\mathsf{T}}C(X^{\mathsf{T}}X)^{-1}X^{\mathsf{T}}X(X^{\mathsf{T}}X)^{-1}C^{\mathsf{T}}\mathbf{w} = \mathbf{w}^{\mathsf{T}}M\mathbf{w} = (C\hat{\boldsymbol\beta} - \mathbf{d})^{\mathsf{T}}M^{-1}(C\hat{\boldsymbol\beta} - \mathbf{d})$, and $\mathrm{RSS}_0 - \mathrm{RSS} = g(\tilde{\boldsymbol\beta})$. $\square$
When the hypothesis sets some coefficients to zero, the restricted fit is simply the fit of the smaller model without those columns. The minimization over vectors with those entries zero is least squares on the remaining columns. Then $\mathrm{RSS}_0$ is that model's residual sum of squares, and the theorem says that the drop in residual sum of squares from adding the columns is the Wald form. The Geometry of Least Squares (Theorem 4) shows the same drop as the squared length of the difference of two nested projections.
The running sample. For sex and BMI, the restricted model is the regression on age alone: $\widehat{\text{pressure}} = 101.47 + 0.4392\,\text{age}$, with $\mathrm{RSS}_0 = 19{,}868.8$. The full model has $\mathrm{RSS} = 19{,}213.6$, so the extra sum of squares is 655.2. The Wald form gives the same number: here
$$M = \begin{pmatrix} 0.08301 & -0.001031 \\ -0.001031 & 0.0004301\end{pmatrix},$$
the lower right block of $(X^{\mathsf{T}}X)^{-1}$, and $C\hat{\boldsymbol\beta} = (2.34, 0.467)$; the script confirms that $Q$ equals $\mathrm{RSS}_0 - \mathrm{RSS}$ to rounding. The off-diagonal entry says that the two estimates are slightly negatively correlated, with correlation $-0.001031/\sqrt{0.08301 \times 0.0004301} = -0.173$.
The F statistic and its distribution
The extra sum of squares is a squared length, the length of a projection of the data, shifted by any vector of the form $X\mathbf{b}^*$ with $C\mathbf{b}^* = \mathbf{d}$, onto a $q$-dimensional subspace inside the column space of $X$. That subspace is perpendicular to the residuals, so the projection theorem gives its distribution and its independence of $s^2$ at once.
Theorem 2 (the F test of a general linear hypothesis). Under the normal linear model, the statistic
$$F = \frac{(\mathrm{RSS}_0 - \mathrm{RSS})/q}{\mathrm{RSS}/(n - p)} = \frac{(C\hat{\boldsymbol\beta} - \mathbf{d})^{\mathsf{T}}M^{-1}(C\hat{\boldsymbol\beta} - \mathbf{d})}{q\,s^2}$$
has the noncentral $F$ distribution with $q$ and $n - p$ degrees of freedom and noncentrality
$$\lambda = \frac{(C\boldsymbol\beta - \mathbf{d})^{\mathsf{T}}M^{-1}(C\boldsymbol\beta - \mathbf{d})}{\sigma^2}.$$
Under $H_0$, and only then, $\lambda = 0$, and $F$ has the central $F$ distribution with $q$ and $n - p$ degrees of freedom.
Proof. Let $A = X(X^{\mathsf{T}}X)^{-1}C^{\mathsf{T}}$, an $n \times q$ matrix whose columns lie in $\mathcal{C}(X)$, and note two identities: $A^{\mathsf{T}}A = C(X^{\mathsf{T}}X)^{-1}X^{\mathsf{T}}X(X^{\mathsf{T}}X)^{-1}C^{\mathsf{T}} = M$, and $A^{\mathsf{T}}X = C$. Define
$$P = AM^{-1}A^{\mathsf{T}}.$$
$P$ is a projection of rank $q$ inside $\mathcal{C}(X)$. It is symmetric, and $P^2 = AM^{-1}(A^{\mathsf{T}}A)M^{-1}A^{\mathsf{T}} = AM^{-1}A^{\mathsf{T}} = P$. Its trace is the trace of $M^{-1}A^{\mathsf{T}}A = I_q$, which is $q$; for a projection the rank is the trace (Sampling Distributions from the Normal, Lemma 2). Since the columns of $A$ lie in $\mathcal{C}(X)$, $HA = A$, so $HP = P$; taking transposes, $PH = P$, and therefore $P(I - H) = 0$.
The Wald form is a squared projection. Because the rows of $C$ are independent, the equation $C\mathbf{b} = \mathbf{d}$ has a solution $\mathbf{b}^*$. Put $\mathbf{z} = \mathbf{y} - X\mathbf{b}^*$. Then
$$A^{\mathsf{T}}\mathbf{z} = C(X^{\mathsf{T}}X)^{-1}X^{\mathsf{T}}\mathbf{y} - A^{\mathsf{T}}X\mathbf{b}^* = C\hat{\boldsymbol\beta} - C\mathbf{b}^* = C\hat{\boldsymbol\beta} - \mathbf{d},$$
so $Q = (A^{\mathsf{T}}\mathbf{z})^{\mathsf{T}}M^{-1}(A^{\mathsf{T}}\mathbf{z}) = \mathbf{z}^{\mathsf{T}}P\mathbf{z} = \lVert P\mathbf{z}\rVert^2$. Also $(I - H)\mathbf{z} = (I - H)\mathbf{y}$, because $(I - H)X = 0$, so $\mathrm{RSS} = \lVert(I - H)\mathbf{z}\rVert^2$.
The distributions. The vector $\mathbf{z}$ has independent normal coordinates with variance $\sigma^2$ and mean vector $\mathbf{m} = X(\boldsymbol\beta - \mathbf{b}^*)$. The projections $P$ and $I - H$ have product zero and ranks $q$ and $n - p$. By the projection theorem (Sampling Distributions from the Normal, Theorem 2), $Q/\sigma^2$ and $\mathrm{RSS}/\sigma^2$ are independent. $\mathrm{RSS}/\sigma^2$ is central chi-squared with $n - p$ degrees of freedom since $(I - H)\mathbf{m} = \mathbf{0}$; and $Q/\sigma^2$ is chi-squared with $q$ degrees of freedom and noncentrality
$$\frac{\lVert P\mathbf{m}\rVert^2}{\sigma^2} = \frac{\mathbf{m}^{\mathsf{T}}P\mathbf{m}}{\sigma^2} = \frac{(\boldsymbol\beta - \mathbf{b}^*)^{\mathsf{T}}C^{\mathsf{T}}M^{-1}C(\boldsymbol\beta - \mathbf{b}^*)}{\sigma^2} = \lambda,$$
using $A^{\mathsf{T}}X = C$ and $C\mathbf{b}^* = \mathbf{d}$. The ratio of the two, each over its degrees of freedom, is noncentral $F$ by definition. Since $M^{-1}$ is positive definite, $\lambda = 0$ exactly when $C\boldsymbol\beta = \mathbf{d}$. The two forms of $F$ agree by Theorem 1. $\square$
The test rejects $H_0$ at level $\alpha$ when $F$ exceeds the upper $\alpha$ point of the central $F$ distribution with $q$ and $n - p$ degrees of freedom. Large values are evidence against $H_0$: under the alternative the numerator has mean $\sigma^2(q + \lambda)/q$ (the noncentral chi-squared mean $q + \lambda$ of the projection article), while the denominator still estimates $\sigma^2$.
The running sample. For sex and BMI,
$$F = \frac{655.2/2}{19{,}213.6/46} = 0.784,$$
against an upper 5% point of 3.1996 for the $F$ distribution with 2 and 46 degrees of freedom; $p = 0.462$. The two coefficients together add nothing that this sample can detect to the model with age alone.
Example (two insignificant coefficients can be jointly significant, and the reverse). It is tempting to read the two separate $t$ tests as the joint test: both insignificant, so the pair is insignificant. The joint test measures the pair in the metric of $M$, which accounts for the correlation of the two estimates. Its region is an ellipse, not the rectangle of the two separate intervals. The figure shows both for the running sample. A point such as (12, 1.2) lies inside both 95% intervals, yet the $F$ statistic for the hypothesis $(\beta_{\text{female}}, \beta_{\text{BMI}}) = (12, 1.2)$ is 3.43, above 3.1996, so the joint test rejects it. Points near the left and right ends of the ellipse are the opposite case: the joint test accepts them while one of the separate intervals excludes them. Here the correlation is only $-0.173$ and the ellipse nearly fills the rectangle. The more strongly the two estimates are correlated, the longer and thinner the ellipse, and the larger the parts of the rectangle it leaves out.
The region of the test, and simultaneous intervals
Inverting the test gives a confidence region for $C\boldsymbol\beta$. The set of vectors $\boldsymbol\gamma$ with $(C\hat{\boldsymbol\beta} - \boldsymbol\gamma)^{\mathsf{T}}M^{-1}(C\hat{\boldsymbol\beta} - \boldsymbol\gamma) \le q\,s^2F_{q,\,n-p,\,0.95}$ contains $C\boldsymbol\beta$ with probability 0.95, since that event is the event that the test of $C\boldsymbol\beta$'s true value accepts. That is the ellipse of the figure. Its shadows are intervals for every combination of the restricted quantities, and they explain the relation between an $F$ test and the $t$ tests of the individual combinations.
Theorem 3 (the F test as the largest t). Let $\mathbf{r} = C\hat{\boldsymbol\beta} - \mathbf{d}$. For every nonzero vector $\mathbf{a}$ of length $q$, the $t$ statistic for the single hypothesis $\mathbf{a}^{\mathsf{T}}C\boldsymbol\beta = \mathbf{a}^{\mathsf{T}}\mathbf{d}$ is $t_{\mathbf a} = \mathbf{a}^{\mathsf{T}}\mathbf{r}/(s\sqrt{\mathbf{a}^{\mathsf{T}}M\mathbf{a}})$, and
$$\max_{\mathbf{a} \ne \mathbf{0}}\ t_{\mathbf a}^2 = qF,$$
attained at $\mathbf{a} = M^{-1}\mathbf{r}$ (when $\mathbf{r} \ne \mathbf{0}$). Consequently the $F$ test at level 0.05 rejects if and only if some combination has $\lvert t_{\mathbf a}\rvert > \sqrt{qF_{q,\,n-p,\,0.95}}$, and with probability 0.95 every combination satisfies $\lvert\mathbf{a}^{\mathsf{T}}C\hat{\boldsymbol\beta} - \mathbf{a}^{\mathsf{T}}C\boldsymbol\beta\rvert \le \sqrt{qF_{q,\,n-p,\,0.95}}\;s\sqrt{\mathbf{a}^{\mathsf{T}}M\mathbf{a}}$.
Proof. The $t$ statistic is that of The Normal Linear Model, Theorem 2, for the vector $C^{\mathsf{T}}\mathbf{a}$, whose variance factor is $\mathbf{a}^{\mathsf{T}}C(X^{\mathsf{T}}X)^{-1}C^{\mathsf{T}}\mathbf{a} = \mathbf{a}^{\mathsf{T}}M\mathbf{a}$. With $A$ as in the proof of Theorem 2 and $\mathbf{u} = M^{-1}\mathbf{r}$, the vectors $A\mathbf{a}$ and $A\mathbf{u}$ in $n$ dimensions have $(A\mathbf{a})\cdot(A\mathbf{u}) = \mathbf{a}^{\mathsf{T}}M\mathbf{u} = \mathbf{a}^{\mathsf{T}}\mathbf{r}$, $\lVert A\mathbf{a}\rVert^2 = \mathbf{a}^{\mathsf{T}}M\mathbf{a}$ and $\lVert A\mathbf{u}\rVert^2 = \mathbf{r}^{\mathsf{T}}M^{-1}\mathbf{r} = Q$. The Cauchy–Schwarz inequality for vectors (Probability Inequalities, corollary on sums of numbers) gives $(\mathbf{a}^{\mathsf{T}}\mathbf{r})^2 \le \mathbf{a}^{\mathsf{T}}M\mathbf{a}\cdot Q$, with equality at $\mathbf{a} = \mathbf{u}$. Dividing by $s^2\,\mathbf{a}^{\mathsf{T}}M\mathbf{a}$, $t_{\mathbf a}^2 \le Q/s^2 = qF$ with equality at $\mathbf{a} = \mathbf{u}$. The rejection statement follows. For the intervals, apply the same identity with $\mathbf{d}$ replaced by the true value $C\boldsymbol\beta$. All the inequalities hold exactly when the $F$ statistic for that true value is at most $F_{q,\,n-p,\,0.95}$, which by Theorem 2 has probability 0.95. $\square$
This is Scheffé's method, as the NIST/SEMATECH handbook (§7.4.7.2) describes it for contrasts among group means, in the generality of a linear model; The Normal Linear Model gave the case $C = I$.
The running sample. The overall test of The Normal Linear Model has $q = 3$ and $F = 3.042$, so the largest squared $t$ over combinations of the three slopes is $3 \times 3.042$, and the largest $\lvert t\rvert$ is 3.021. It exceeds $\sqrt{3F_{3,\,46,\,0.95}} = 2.9018$, as it must, since the $F$ test rejected. The combination that attains it has weights proportional to $M^{-1}\mathbf{r}$; scaled to give age the weight 1, it is age $+\ 0.008$ female $+\ 0.141$ BMI. The evidence against "all slopes zero" lies almost entirely in the age coefficient, helped a little by BMI.
The cases the course already knows
Four tests met earlier on the site are general linear hypotheses, and Theorem 2 gives each its exact distribution at once.
One coefficient, and $F = t^2$. With $q = 1$ and $C = \mathbf{c}^{\mathsf{T}}$, $M$ is the number $\mathbf{c}^{\mathsf{T}}(X^{\mathsf{T}}X)^{-1}\mathbf{c}$ and
$$F = \frac{(\mathbf{c}^{\mathsf{T}}\hat{\boldsymbol\beta} - d)^2}{s^2\,\mathbf{c}^{\mathsf{T}}(X^{\mathsf{T}}X)^{-1}\mathbf{c}} = t^2,$$
the square of the $t$ statistic of The Normal Linear Model, Theorem 2. The $F$ test with 1 and $n - p$ degrees of freedom is the two-sided $t$ test, because $t_{n-p}^2$ has the $F$ distribution with 1 and $n - p$ degrees of freedom (Sampling Distributions from the Normal). For the hypothesis that the age coefficient equals 0.5, $t = (0.4455 - 0.5)/0.1608 = -0.339$ and $F = 0.115$, $p = 0.736$. The restricted fit holds the age coefficient at 0.5 and adjusts the others: intercept 83.08, female 2.15, BMI 0.478.
All slopes. With $C = \begin{pmatrix}\mathbf{0} & I_{p-1}\end{pmatrix}$ and $\mathbf{d} = \mathbf{0}$, the restricted model has only the intercept, whose least-squares fit is $\bar{y}\mathbf{1}$, so $\mathrm{RSS}_0 = \mathrm{TSS}$ and $\mathrm{RSS}_0 - \mathrm{RSS} = \mathrm{ESS}$. Theorem 2 becomes the overall $F$ test of The Normal Linear Model, Theorem 4, and its noncentrality, $(C\boldsymbol\beta)^{\mathsf{T}}M^{-1}C\boldsymbol\beta/\sigma^2$, must equal the one found there, $\lVert\boldsymbol\mu - \bar\mu\mathbf{1}\rVert^2/\sigma^2$. Both are the squared length of the projection of the mean vector onto the same subspace: the columns of $A = X(X^{\mathsf{T}}X)^{-1}C^{\mathsf{T}}$ are perpendicular to $\mathbf{1}$, because $\mathbf{1} = X\mathbf{e}_1$ gives $\mathbf{1}^{\mathsf{T}}A = (C\mathbf{e}_1)^{\mathsf{T}} = \mathbf{0}^{\mathsf{T}}$, so the projection $P$ of Theorem 2 has rank $p - 1$ inside $\mathcal{C}(X)$ and perpendicular to $\mathbf{1}$, and is therefore $H - P_{\mathbf 1}$.
One-way analysis of variance. Let observation $j$ of group $i$ be $y_{ij}$, for $k$ groups of sizes $n_1, \ldots, n_k$. Code the model with one indicator column per group and no separate intercept: $\mathbf{y} = X\boldsymbol\mu + \boldsymbol\varepsilon$ with $\boldsymbol\mu = (\mu_1, \ldots, \mu_k)$, the group means. The indicators add to $\mathbf{1}$, so the column of ones lies in $\mathcal{C}(X)$ though it is not itself a column. Here $X^{\mathsf{T}}X$ is diagonal with entries $n_i$, so $\hat\mu_i = \bar{y}_i$, the group mean, and $\mathrm{RSS} = \sum_i\sum_j(y_{ij} - \bar{y}_i)^2 = SSW$. The hypothesis of equal means is $C\boldsymbol\mu = \mathbf{0}$ with the $k - 1$ rows $(1, -1, 0, \ldots, 0)$, $(1, 0, -1, \ldots, 0)$, …, $(1, 0, \ldots, 0, -1)$, which are independent. Under it every observation has the same mean, the restricted fit is the grand mean $\bar{y}$, and $\mathrm{RSS}_0 = \sum_i\sum_j(y_{ij} - \bar{y})^2 = SST$. So $\mathrm{RSS}_0 - \mathrm{RSS} = SST - SSW = SSB$, and
$$F = \frac{SSB/(k - 1)}{SSW/(n - k)},$$
the statistic of One-Way Analysis of Variance, with its $F$ distribution on $k - 1$ and $n - k$ degrees of freedom and its noncentrality $\sum_i n_i(\mu_i - \bar\mu)^2/\sigma^2$, with $\bar\mu = \sum_i n_i\mu_i/n$ the mean weighted by the group sizes, as Sampling Distributions from the Normal also derives. Take the course's worked sample of 10 adults from each of four age groups. The script fits the four indicators and finds group means from 108.267 to 130.333 mm Hg (grand mean 116.350), $SSW = 5{,}983.09$, $SSB = 2{,}850.01$, and $F = 5.716$ on 3 and 36 degrees of freedom, $p = 0.0026$: the course page's numbers.
The pooled two-sample $t$-test. With $k = 2$ the analysis above has $q = 1$, so $F = t^2$ for the $t$ statistic of the single restriction $\mu_1 - \mu_2 = 0$, whose variance factor is $1/n_1 + 1/n_2$; that $t$ is the pooled two-sample statistic. For the 20 men and 20 women of The Two-Sample t-Test, with means 126.950 and 122.817 mm Hg and pooled variance 376.06, $t = 0.674$ and $F = 0.454$, with $p = 0.5044$ either way. One-Way Analysis of Variance proves $F = t^2$ for two groups by direct algebra; here it is the case $q = 1$ of a general fact.
The F test is the likelihood ratio test
The $F$ statistic was built from least squares. It is also what the likelihood principle produces, and that connects this article to the general theory of tests in Likelihood Ratio, Wald and Score Tests.
Theorem 4 (likelihood ratio). Under the normal linear model with unknown $\boldsymbol\beta$ and $\sigma^2$, the likelihood ratio for $H_0\colon C\boldsymbol\beta = \mathbf{d}$ is
$$\Lambda = \frac{\sup_{C\mathbf{b} = \mathbf{d},\ \sigma^2 > 0} L(\mathbf{b}, \sigma^2)}{\sup_{\mathbf{b},\ \sigma^2 > 0} L(\mathbf{b}, \sigma^2)} = \Big(\frac{\mathrm{RSS}}{\mathrm{RSS}_0}\Big)^{n/2}, \qquad -2\ln\Lambda = n\ln\Big(1 + \frac{q}{n - p}\,F\Big).$$
So $-2\ln\Lambda$ is an increasing function of $F$, and the likelihood ratio test that rejects for large $-2\ln\Lambda$ is the $F$ test.
Proof. The observations are independent normal with means $(X\mathbf{b})_i$ and variance $\sigma^2$, so the log-likelihood is
$$\ell(\mathbf{b}, \sigma^2) = -\frac{n}{2}\ln(2\pi\sigma^2) - \frac{\lVert\mathbf{y} - X\mathbf{b}\rVert^2}{2\sigma^2}.$$
First, $\mathrm{RSS} > 0$ with probability 1: by Theorem 1 of The Normal Linear Model, $\mathrm{RSS}/\sigma^2$ is chi-squared with $n - p \ge 1$ degrees of freedom, a distribution with a density (The Chi-Squared and t Distributions), so it is 0 with probability 0; and $\mathrm{RSS}_0 \ge \mathrm{RSS}$. On that event both suprema below are finite. For fixed $\mathbf{b}$ with $R = \lVert\mathbf{y} - X\mathbf{b}\rVert^2 > 0$, the derivative with respect to $\sigma^2$ is $-n/(2\sigma^2) + R/(2\sigma^4)$, positive for $\sigma^2 < R/n$ and negative beyond, so the maximum over $\sigma^2$ is at $\sigma^2 = R/n$, with value $-\frac{n}{2}\ln(2\pi R/n) - \frac{n}{2}$. This decreases as $R$ increases, so the supremum over a set of $\mathbf{b}$ is attained where $R$ is smallest. That is at $\hat{\boldsymbol\beta}$ with $R = \mathrm{RSS}$ over all vectors, and at $\tilde{\boldsymbol\beta}$ with $R = \mathrm{RSS}_0$ under the restrictions (Theorem 1). The difference of the two maxima is $-\frac{n}{2}\ln(\mathrm{RSS}_0/\mathrm{RSS})$, which gives $\Lambda$. Finally $\mathrm{RSS}_0/\mathrm{RSS} = 1 + (\mathrm{RSS}_0 - \mathrm{RSS})/\mathrm{RSS} = 1 + qF/(n - p)$. $\square$
The proof also shows that the maximum likelihood estimate of $\sigma^2$ is $\mathrm{RSS}/n$, smaller than the unbiased $s^2 = \mathrm{RSS}/(n - p)$.
Likelihood Ratio, Wald and Score Tests states and cites the several-parameter form of Wilks's theorem. For large samples it says that $-2\ln\Lambda$ is approximately chi-squared with $q$ degrees of freedom under $H_0$. For the normal linear model no approximation is needed: Theorem 4 turns the exact $F$ distribution into the exact distribution of $-2\ln\Lambda$. The comparison shows what the approximation costs.
- For sex and BMI, $-2\ln\Lambda = 50\ln(19{,}868.8/19{,}213.6) = 1.677$. The chi-squared approximation with 2 degrees of freedom gives $p = 0.432$, against the exact 0.462.
- For the course's analysis of variance, $-2\ln\Lambda = 40\ln(1 + 3 \times 5.716/36) = 15.58$, with approximate $p = 0.0014$ against the exact 0.0026. The exact 5% critical value, $F_{3,\,36,\,0.95} = 2.8663$, corresponds to $-2\ln\Lambda = 40\ln(1 + 3 \times 2.8663/36) = 8.57$, while the chi-squared approximation would use 7.815. With 36 residual degrees of freedom the approximation rejects too easily: it treats $\sigma^2$ as known.
Power and sample size
Theorem 2 gives the power of every such test in closed form. It is the probability that a noncentral $F$ variable with $q$ and $n - p$ degrees of freedom and noncentrality $\lambda$ exceeds the critical value. The noncentrality depends on the design through $M$ and on the truth through $(C\boldsymbol\beta - \mathbf{d})/\sigma$.
The running design. Suppose the truth for sex and BMI were the coefficients of the whole population, $-1.92$ and 0.385, with the population's residual variance $\sigma^2 = 293.88$ (the mean of the squared residuals of the population fit, their sum divided by 5,176) (The Normal Linear Model). With the running sample's inputs,
$$\lambda = \frac{(-1.92, 0.385)\,M^{-1}\,(-1.92, 0.385)^{\mathsf{T}}}{293.88} = 1.21,$$
and the power of the 5% test is the probability that a noncentral $F$ variable with 2 and 46 degrees of freedom and noncentrality 1.21 exceeds 3.1996, which is 0.145. A sample of 50 has little chance of detecting effects of this size.
How many adults would be needed? Suppose the 50 adults' inputs were repeated $k$ times. Then $X^{\mathsf{T}}X$ would be multiplied by $k$, $M$ divided by $k$, and $\lambda$ multiplied by $k$, while the residual degrees of freedom would become $50k - 4$. The smallest such design with power at least 0.8 has 450 adults, with power 0.847. That is the general pattern: the noncentrality grows in proportion to the sample size, so the sample size for a target power can be read from the noncentral $F$ distribution.
An experiment. The calculation assumes normal errors and fixed inputs. For each of 30 seeds (numpy.random.default_rng(2000 + seed)), 2,000 times, the script drew 50 adults from the population and applied the test. As a control, it also applied the test to responses generated from the normal linear model with the running sample's inputs, the population coefficients and normal errors of variance 293.88. The control rejected in a share 0.147 of samples (median over seeds; interquartile range 0.144 to 0.150), as the calculation predicts. Samples drawn from the population itself rejected more often, in a share 0.177 (0.172 to 0.181). That is an observation about this population. Each such sample brings its own inputs and its own non-normal, unequally spread errors, and the calculation for one fixed design under normal errors does not describe it exactly. It is not a statement about the level of the test, since in the population the hypothesis is false.
What is proved, what is cited, what the experiment shows
Proved here. Restricted least squares, its uniqueness and the identity between the extra sum of squares and the Wald form (Theorem 1); the noncentral $F$ distribution of the statistic, with its noncentrality and the fact that the noncentrality vanishes only under the hypothesis (Theorem 2); the confidence region for $C\boldsymbol\beta$; the identity between $F$ and the largest squared $t$ over combinations, with the simultaneous intervals (Theorem 3); the likelihood ratio and its relation to $F$, with the maximum likelihood estimate of $\sigma^2$ (Theorem 4); the reductions to the $t$ test, the overall $F$ test, one-way analysis of variance and the pooled two-sample $t$-test.
Taken from other articles, where they are proved. The distribution theory of the normal linear model and the $t$ statistic for one combination (The Normal Linear Model); the covariance of $\hat{\boldsymbol\beta}$ (The Gauss–Markov Theorem); the projection theorem, the rank of a projection and the noncentral chi-squared and $F$ distributions (Sampling Distributions from the Normal); the Cauchy–Schwarz inequality (Probability Inequalities).
Cited. The several-parameter form of Wilks's theorem, through Likelihood Ratio, Wald and Score Tests; the name of Scheffé's method (NIST/SEMATECH, §7.4.7.2). Percentage points and noncentral probabilities are computed by scipy.
The experiment shows rejection rates for samples of 50 from one finite population and for a normal control, over 30 seeds.
Worked problems
These problems were written for this article, on the unit's NHANES samples and the course's analysis of variance. The script tools/experiments/statistics/general-linear-hypothesis.py recomputes every number.
Problem 1 (an extra sum of squares by hand). The unit's 12-adult sample (listed in the first worked problem of The Normal Linear Model) includes 4 women. The regression of pressure on age alone leaves a residual sum of squares of 1,522.85; adding the indicator for women and BMI reduces it to 1,047.54. Test whether sex and BMI together add to age.
Solution. $F = 1.815$ on 2 and 8 degrees of freedom, $p = 0.224$: no evidence that they do.
The hypothesis is $\beta_{\text{female}} = \beta_{\text{BMI}} = 0$, so $q = 2$, and the restricted model is the regression on age, whose residual sum of squares is $\mathrm{RSS}_0 = 1{,}522.85$ (the same value as in that article's problem). The full model has $p = 4$ coefficients and $n - p = 12 - 4 = 8$ residual degrees of freedom. The extra sum of squares is $1{,}522.85 - 1{,}047.54 = 475.32$ (from unrounded values), so
$$F = \frac{475.32/2}{1{,}047.54/8} = 1.815.$$
The upper 5% point of the $F$ distribution with 2 and 8 degrees of freedom is 4.4590, so the test does not reject, and $p = 0.224$. The full fit's coefficients, $-8.67$ for women and 1.379 for BMI, are large. With 8 residual degrees of freedom they are so uncertain that a reduction of 475.32 in the residual sum of squares is what chance would often produce.
Problem 2 (a hypothesis with stated values). Are the running sample's coefficients for sex and BMI compatible with the population's values, $-1.92$ and 0.385? Give the restricted fit and show how to compute it with ordinary least squares.
Solution. $F = 0.313$ on 2 and 46 degrees of freedom, $p = 0.732$: fully compatible. The restricted fit is $90.24 + 0.4531\,\text{age} - 1.92\,\text{female} + 0.385\,\text{BMI}$.
Here $\mathbf{d} = (-1.92, 0.385)$ and $\mathbf{r} = C\hat{\boldsymbol\beta} - \mathbf{d} = (2.34 + 1.92,\ 0.467 - 0.385) = (4.26, 0.082)$. With the same $M$ as in the text, $Q = \mathbf{r}^{\mathsf{T}}M^{-1}\mathbf{r} = 261.9$, so $F = (261.9/2)/417.69 = 0.313$.
To fit under the restrictions, note that with $\beta_{\text{female}}$ and $\beta_{\text{BMI}}$ fixed, the model says $y_i + 1.92\,\text{female}_i - 0.385\,\text{BMI}_i = \beta_1 + \beta_{\text{age}}\,\text{age}_i + \varepsilon_i$. The left side is the response minus a known offset, $-1.92\,\text{female}_i + 0.385\,\text{BMI}_i$, and the remaining coefficients are found by an ordinary regression of this adjusted response on age. That regression gives intercept 90.24 and slope 0.4531, and its residual sum of squares is $\mathrm{RSS}_0 = 19{,}475.5$; the script confirms that Theorem 1's formula gives the same vector. Then $\mathrm{RSS}_0 - \mathrm{RSS} = 19{,}475.5 - 19{,}213.6 = 261.9$, the Wald form again.
Problem 3 (the exact and approximate likelihood ratio for the analysis of variance). For the course's 40 blood pressures in four age groups ($F = 5.716$ on 3 and 36 degrees of freedom), compute the likelihood ratio statistic, its p-value by the chi-squared approximation, and the exact p-value. Which should be reported?
Solution. $-2\ln\Lambda = 15.58$; the approximation gives $p = 0.0014$; the exact p-value is 0.0026, and it is the one to report.
By Theorem 4 with $n = 40$, $p = 4$ (four group means) and $q = 3$, $-2\ln\Lambda = 40\ln(1 + 3 \times 5.716/36) = 40\ln(SST/SSW)$, with $SST = 5{,}983.09 + 2{,}850.01$, which is 15.58. Referred to the chi-squared distribution with 3 degrees of freedom it gives $p = 0.0014$. But the statistic is an increasing function of $F$, whose exact distribution is known, so $P(-2\ln\Lambda \ge 15.58) = P(F \ge 5.716) = 0.0026$ exactly under the normal model. The approximation halves the p-value because it ignores the uncertainty in $\sigma^2$, estimated here from only 36 degrees of freedom; the exact $F$ calibration is the one Theorem 2 justifies.
Summary
| Quantity | Formula |
|---|---|
| Hypothesis | $C\boldsymbol\beta = \mathbf{d}$, $C$ of size $q \times p$ with independent rows |
| $M$ | $C(X^{\mathsf{T}}X)^{-1}C^{\mathsf{T}}$, positive definite |
| Restricted fit | $\tilde{\boldsymbol\beta} = \hat{\boldsymbol\beta} - (X^{\mathsf{T}}X)^{-1}C^{\mathsf{T}}M^{-1}(C\hat{\boldsymbol\beta} - \mathbf{d})$ |
| Extra sum of squares | $\mathrm{RSS}_0 - \mathrm{RSS} = (C\hat{\boldsymbol\beta} - \mathbf{d})^{\mathsf{T}}M^{-1}(C\hat{\boldsymbol\beta} - \mathbf{d})$ |
| $F$ | $\frac{(\mathrm{RSS}_0 - \mathrm{RSS})/q}{\mathrm{RSS}/(n-p)}$, noncentral $F_{q,\,n-p}$ with $\lambda = (C\boldsymbol\beta - \mathbf{d})^{\mathsf{T}}M^{-1}(C\boldsymbol\beta - \mathbf{d})/\sigma^2$ |
| Largest $t$ | $\max_{\mathbf a} t_{\mathbf a}^2 = qF$ |
| Likelihood ratio | $-2\ln\Lambda = n\ln\big(1 + qF/(n - p)\big)$ |
| Cases | $q = 1$: $F = t^2$; all slopes: overall $F$; group means: one-way ANOVA |
What we have learned
A set of linear restrictions on the coefficients can be tested by fitting the model with and without them. The increase in the residual sum of squares equals a quadratic form in the estimated restrictions. It is the squared length of a projection perpendicular to the residuals, so under normal errors its ratio to the residual mean square has an exact $F$ distribution, noncentral when the restrictions are false. The same statistic is the likelihood ratio and the largest squared $t$ over combinations of the restrictions. In special cases it is the $t$ test for a coefficient, the overall regression $F$, the one-way analysis of variance and the pooled two-sample $t$-test. For the 50 adults, sex and BMI together add nothing detectable to age ($F = 0.784$, $p = 0.462$). A design would need about nine times as many adults to detect the population's coefficients with power 0.8.
Suppose the columns of $X$ come from two classifications with unequal numbers in the cells. Then "the effect of a factor" can be written as several different linear hypotheses, and statistical software reports several different sums of squares for it. Unbalanced Designs and the Types of Sums of Squares uses this article's theorems to say exactly which hypothesis each one tests.
References
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J), Blood Pressure (BPX_J) and Body Measures (BMX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 75 pages
- NIST/SEMATECH. e-Handbook of Statistical Methods, §7.4.7.2 (Scheffé's method). National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/ record · 36 pages