The worked example of One-Way Analysis of Variance draws 10 adults from each of four age groups and splits the variation of their 40 blood pressures into 2,850.01 between the groups and 5,983.09 within them. It divides the first by 3 and the second by 36, and refers the ratio, 5.716, to the $F$ distribution with 3 and 36 degrees of freedom. Why 3 and 36, why may the two pieces be treated as independent, and why does the same recipe reappear in the $t$-test, in regression and wherever normal data are analyzed? Because each sum of squares is the squared length of the projection of the data onto a subspace, the two subspaces are perpendicular and have dimensions 3 and 36, and a vector of independent normal variables with a common variance has independent projections onto perpendicular subspaces, with squared lengths that are chi-squared on as many degrees of freedom as the subspace has dimensions.
That sentence is a theorem, and this article proves it and draws out what follows from it. The chi-squared, $t$ and $F$ distributions are the distributions of a squared length, of a coordinate divided by a root mean squared length, and of a ratio of two mean squared lengths. Degrees of freedom are dimensions.
The course pages already prove several cases, and this article builds on them without repeating them. The Chi-Squared and t Distributions proves that a rotation of independent standard normal variables gives independent standard normal variables, uses Helmert's matrix to show that the mean and variance of a normal sample are independent, and derives the chi-squared and $t$ densities. The Two-Sample t-Test treats the pooled statistic. One-Way Analysis of Variance defines the $F$ distribution, proves by a rotation that its statistic has it, and shows that $F$ with one numerator degree of freedom is $t^2$. What is new here is the general theorem those arguments are instances of, Cochran's theorem, the $F$ density and its relatives, the noncentral chi-squared density, and two moment formulas that show exactly where normality is used.
First we extend the rotation to normal variables with any means and define the noncentral chi-squared distribution. Then we prove the projection theorem and read off the one-sample and one-way results. Next we prove Cochran's theorem. Then we derive the $F$ density and its properties, and the noncentral chi-squared density. Finally we ask what survives without normality. The article assumes the course page on the chi-squared and $t$ distributions and the linear algebra of orthogonal matrices used there.
Rotating a normal vector with any means
Throughout, $\mathbf{X} = (X_1, \ldots, X_n)$ is a vector of independent normal random variables with a common variance $\sigma^2 > 0$, and $X_i$ has mean $m_i$; write $\mathbf{m}$ for the vector of means. On this site the normal distribution with mean $\mu$ and standard deviation $\sigma$ is written $N(\mu, \sigma)$, with the standard deviation second. A matrix $Q$ is orthogonal if $QQ^{\mathsf{T}} = I$; its rows are then perpendicular unit vectors, and it preserves lengths, $\lVert Q\mathbf{x}\rVert = \lVert\mathbf{x}\rVert$.
Theorem 1 (rotation). If $Q$ is an orthogonal $n \times n$ matrix, then the coordinates of $\mathbf{W} = Q\mathbf{X}$ are independent normal variables with the common variance $\sigma^2$ and means $Q\mathbf{m}$.
The idea: the course page proves the case of mean 0 and variance 1, and the general case is that one shifted and scaled.
Proof. Let $Z_i = (X_i - m_i)/\sigma$. The $Z_i$ are independent standard normal variables. By Step 1 of the proof on the course page, the coordinates of $Q\mathbf{Z}$ are independent standard normal variables. Since $\mathbf{X} = \mathbf{m} + \sigma\mathbf{Z}$,
$$\mathbf{W} = Q\mathbf{X} = Q\mathbf{m} + \sigma\,Q\mathbf{Z}, \qquad W_i = (Q\mathbf{m})_i + \sigma\,(Q\mathbf{Z})_i.$$
Each $W_i$ is a function of the single coordinate $(Q\mathbf{Z})_i$, so the $W_i$ are independent, and $W_i$ is normal with mean $(Q\mathbf{m})_i$ and variance $\sigma^2$. $\square$
The random part of the vector is rotated and stays what it was; the vector of means is rotated as an ordinary vector. Normality is used in the cited step and nowhere else in this article's distribution theory, except that the proof of Theorem 5 writes down the density of one squared shifted normal variable. What the cited step uses is that the joint density of independent standard normal variables depends on a point only through its distance from the origin.
The first use of the theorem is to define a distribution. The chi-squared distribution with $r$ degrees of freedom is that of a sum of $r$ squared independent standard normal variables. What if the variables have means?
Lemma 1 (only the length of the mean vector matters). Let $Y_1, \ldots, Y_r$ be independent normal variables with variance 1 and means $b_1, \ldots, b_r$, and let $\lambda = b_1^2 + \cdots + b_r^2$. The distribution of $Y_1^2 + \cdots + Y_r^2$ depends on the means only through $\lambda$, and is that of
$$(Z_1 + \sqrt{\lambda})^2 + Z_2^2 + \cdots + Z_r^2$$
for independent standard normal variables $Z_1, \ldots, Z_r$.
Proof. If $\lambda = 0$ every mean is 0 and there is nothing to prove. Otherwise let $Q$ be an orthogonal matrix whose first row is the unit vector $\mathbf{b}/\sqrt{\lambda}$; the Gram–Schmidt process completes that vector to an orthonormal basis, as in One-Way Analysis of Variance. By Theorem 1 the coordinates of $\mathbf{W} = Q\mathbf{Y}$ are independent normal variables with variance 1 and means $Q\mathbf{b}$. The first of those means is $\mathbf{b}\cdot\mathbf{b}/\sqrt{\lambda} = \sqrt{\lambda}$, and the others are 0, because the other rows of $Q$ are perpendicular to $\mathbf{b}$. So $W_1 - \sqrt{\lambda}, W_2, \ldots, W_r$ are independent standard normal variables, and $\lVert\mathbf{Y}\rVert^2 = \lVert\mathbf{W}\rVert^2 = W_1^2 + \cdots + W_r^2$ has the stated form. $\square$
The distribution of the lemma is the noncentral chi-squared distribution with $r$ degrees of freedom and noncentrality $\lambda$, written $\chi^2_r(\lambda)$ here. With $\lambda = 0$ it is the ordinary, or central, chi-squared distribution. Conventions differ between books: some call $\lambda/2$ or $\sqrt{\lambda}$ the noncentrality. Here, as in the course pages and in scipy, it is the sum of the squared means.
Its mean and variance follow from the representation. For a standard normal $Z$ and a constant $\delta$, $(Z + \delta)^2 = Z^2 + 2\delta Z + \delta^2$ has mean $1 + \delta^2$. Its variance is $\operatorname{Var}(Z^2) + 4\delta^2\operatorname{Var}(Z) + 4\delta\operatorname{Cov}(Z^2, Z) = 2 + 4\delta^2$, since $\operatorname{Var}(Z^2) = 2$ (computed on the course page) and $\operatorname{Cov}(Z^2, Z) = E[Z^3] = 0$ by symmetry. Adding the $r - 1$ central terms, each with mean 1 and variance 2,
$$E\big[\chi^2_r(\lambda)\big] = r + \lambda, \qquad \operatorname{Var}\big(\chi^2_r(\lambda)\big) = 2r + 4\lambda.$$
Projections, and the theorem behind every sum of squares
A sum of squares in statistics is the squared length of a part of the data vector: the part along the vector of ones, the part that varies between groups, the part left over. "Part" means projection, and the matrices that carry out projections have a simple description.
A square matrix $P$ is a projection matrix if it is symmetric and idempotent: $P^{\mathsf{T}} = P$ and $P^2 = P$. The example to keep in mind is $P = \frac1n J$, where $J$ is the $n \times n$ matrix of ones. It replaces every coordinate of $\mathbf{x}$ by the mean $\bar{x}$, and doing so twice changes nothing more.
Lemma 2 (a projection matrix is $UU^{\mathsf{T}}$). Let $P$ be an $n \times n$ projection matrix of rank $r$. Then there is an $n \times r$ matrix $U$ whose columns are perpendicular unit vectors, with $P = UU^{\mathsf{T}}$. Consequently, for every vector $\mathbf{x}$,
$$\mathbf{x}^{\mathsf{T}}P\mathbf{x} = \lVert P\mathbf{x}\rVert^2 = \lVert U^{\mathsf{T}}\mathbf{x}\rVert^2,$$
and the rank of $P$ equals its trace, the sum of its diagonal entries.
Proof. Let $C$ be the column space of $P$, the set of all vectors $P\mathbf{b}$; its dimension is the rank $r$. Two facts. First, $P$ leaves the vectors of $C$ alone: if $\mathbf{c} = P\mathbf{b}$, then $P\mathbf{c} = P^2\mathbf{b} = P\mathbf{b} = \mathbf{c}$. Second, for any $\mathbf{x}$ the difference $\mathbf{x} - P\mathbf{x}$ is perpendicular to every $\mathbf{c}$ in $C$:
$$\mathbf{c}^{\mathsf{T}}(\mathbf{x} - P\mathbf{x}) = (P\mathbf{c})^{\mathsf{T}}(\mathbf{x} - P\mathbf{x}) = \mathbf{c}^{\mathsf{T}}P\,(\mathbf{x} - P\mathbf{x}) = \mathbf{c}^{\mathsf{T}}(P\mathbf{x} - P^2\mathbf{x}) = 0,$$
using the symmetry of $P$ in the second step. Now let $\mathbf{u}_1, \ldots, \mathbf{u}_r$ be an orthonormal basis of $C$, obtained by the Gram–Schmidt process, and let $U$ be the matrix with these columns. For any $\mathbf{x}$, the vector $P\mathbf{x}$ lies in $C$, so it equals $\sum_j (\mathbf{u}_j\cdot P\mathbf{x})\,\mathbf{u}_j$. By the second fact $\mathbf{u}_j\cdot(\mathbf{x} - P\mathbf{x}) = 0$, so $\mathbf{u}_j\cdot P\mathbf{x} = \mathbf{u}_j\cdot\mathbf{x}$. Hence
$$P\mathbf{x} = \sum_{j=1}^{r} (\mathbf{u}_j\cdot\mathbf{x})\,\mathbf{u}_j = UU^{\mathsf{T}}\mathbf{x} \quad\text{for every } \mathbf{x},$$
which says $P = UU^{\mathsf{T}}$. Since the columns of $U$ are orthonormal, $U^{\mathsf{T}}U = I_r$. Then $\mathbf{x}^{\mathsf{T}}P\mathbf{x} = \mathbf{x}^{\mathsf{T}}UU^{\mathsf{T}}\mathbf{x} = \lVert U^{\mathsf{T}}\mathbf{x}\rVert^2$, and also $\lVert P\mathbf{x}\rVert^2 = \mathbf{x}^{\mathsf{T}}P^{\mathsf{T}}P\mathbf{x} = \mathbf{x}^{\mathsf{T}}P\mathbf{x}$. For the trace, the trace of a product does not depend on the order of the factors, so the trace of $UU^{\mathsf{T}}$ equals the trace of $U^{\mathsf{T}}U = I_r$, which is $r$. $\square$
So $P\mathbf{x}$ is the component of $\mathbf{x}$ in the subspace $C$, the quadratic form $\mathbf{x}^{\mathsf{T}}P\mathbf{x}$ is its squared length, and $U^{\mathsf{T}}\mathbf{x}$ lists its $r$ coordinates in the basis $\mathbf{u}_1, \ldots, \mathbf{u}_r$.
Theorem 2 (projections of a normal vector). Let $\mathbf{X}$ have independent normal coordinates with common variance $\sigma^2$ and mean vector $\mathbf{m}$. Let $P_1, \ldots, P_k$ be projection matrices of ranks $r_1, \ldots, r_k$ with $P_iP_j = 0$ whenever $i \ne j$. Then:
- the random vectors $P_1\mathbf{X}, \ldots, P_k\mathbf{X}$ are independent of one another;
- each $\mathbf{X}^{\mathsf{T}}P_i\mathbf{X}/\sigma^2 = \lVert P_i\mathbf{X}\rVert^2/\sigma^2$ has the noncentral chi-squared distribution with $r_i$ degrees of freedom and noncentrality $\lambda_i = \lVert P_i\mathbf{m}\rVert^2/\sigma^2$;
- in particular it is central chi-squared with $r_i$ degrees of freedom when $P_i\mathbf{m} = \mathbf{0}$.
Here is the plan. Write each $P_i$ as $U_iU_i^{\mathsf{T}}$. The condition $P_iP_j = 0$ makes the columns of different $U_i$ perpendicular, so all the columns together are part of one orthonormal basis. Rotate to that basis. Then each projection uses its own block of coordinates of an independent normal vector.
Proof. By Lemma 2, $P_i = U_iU_i^{\mathsf{T}}$ with $U_i^{\mathsf{T}}U_i = I_{r_i}$. For $i \ne j$,
$$U_i^{\mathsf{T}}U_j = (U_i^{\mathsf{T}}U_i)\,U_i^{\mathsf{T}}U_j\,(U_j^{\mathsf{T}}U_j) = U_i^{\mathsf{T}}\,(U_iU_i^{\mathsf{T}})(U_jU_j^{\mathsf{T}})\,U_j = U_i^{\mathsf{T}}P_iP_jU_j = 0.$$
So the $r_1 + \cdots + r_k$ columns of $U_1, \ldots, U_k$ are perpendicular unit vectors. Complete them to an orthonormal basis of the whole space by the Gram–Schmidt process, and let $Q$ be the orthogonal matrix whose rows are the basis vectors, those of $U_1$ first, then those of $U_2$, and so on, with any added vectors last. By Theorem 1 the coordinates of $\mathbf{W} = Q\mathbf{X}$ are independent normal variables with variance $\sigma^2$ and means $Q\mathbf{m}$. The first $r_1$ coordinates of $\mathbf{W}$ are the vector $U_1^{\mathsf{T}}\mathbf{X}$, the next $r_2$ are $U_2^{\mathsf{T}}\mathbf{X}$, and so on.
Part 1. $P_i\mathbf{X} = U_i(U_i^{\mathsf{T}}\mathbf{X})$ is a function of the $i$th block of coordinates of $\mathbf{W}$. The blocks are disjoint sets of independent variables, and functions of disjoint sets of independent variables are independent (the third property cited on the course page, from Casella and Berger, 2002).
Part 2. By Lemma 2, $\lVert P_i\mathbf{X}\rVert^2 = \lVert U_i^{\mathsf{T}}\mathbf{X}\rVert^2$, the sum of the squares of the $r_i$ coordinates in block $i$. Divided by $\sigma$, those coordinates are independent normal variables with variance 1 and means $U_i^{\mathsf{T}}\mathbf{m}/\sigma$. By Lemma 1 the sum of their squares is $\chi^2_{r_i}(\lambda_i)$ with $\lambda_i = \lVert U_i^{\mathsf{T}}\mathbf{m}\rVert^2/\sigma^2 = \lVert P_i\mathbf{m}\rVert^2/\sigma^2$.
Part 3 is the case $\lambda_i = 0$. $\square$
The theorem contains the two rotations of the course pages.
One sample. Take $P_1 = \frac1n J$ and $P_2 = I - \frac1n J$. Both are symmetric. $J^2 = nJ$, so $P_1^2 = P_1$, and then $P_2^2 = I - 2P_1 + P_1 = P_2$ and $P_1P_2 = P_1 - P_1 = 0$. Their traces, hence ranks, are 1 and $n - 1$. $P_1\mathbf{X}$ has every coordinate equal to $\bar{X}$, and $P_2\mathbf{X}$ is the vector of deviations $X_i - \bar{X}$, so $\lVert P_2\mathbf{X}\rVert^2 = (n - 1)s^2$, where $s^2$ is the sample variance. For a sample from one normal population, $\mathbf{m}$ is a multiple of the vector of ones, so $P_2\mathbf{m} = \mathbf{0}$. The theorem gives at once that $\bar{X}$ and $s^2$ are independent and that $(n - 1)s^2/\sigma^2$ is $\chi^2_{n-1}$: the three facts of the course page. The matrix $Q$ built there is one explicit choice of the basis; matrices of that form are called Helmert matrices (Lancaster, 1965). The theorem shows that the choice does not matter, since only the subspaces enter the conclusion.
The one-way layout. Back to the 40 blood pressures. Let the data vector list the observations group by group, with group sizes $n_1, \ldots, n_k$ adding to $n$. Let $G$ be the matrix that replaces each observation by the mean of its group: it has a block $\frac{1}{n_i}J$ of size $n_i \times n_i$ for each group, and zeros elsewhere. Define
$$P_0 = \tfrac1n J, \qquad P_B = G - P_0, \qquad P_W = I - G.$$
$G$ is symmetric and idempotent, block by block, as $\frac1n J$ is. Replacing each entry of a constant vector by its group mean changes nothing, so $GJ = J$, and by symmetry $JG = J$; hence $GP_0 = P_0G = P_0$. Then
$$P_B^2 = G - GP_0 - P_0G + P_0 = G - P_0 = P_B, \qquad P_W^2 = I - 2G + G = P_W,$$
$$P_BP_W = (G - P_0)(I - G) = G - G - P_0 + P_0 = 0, \qquad P_0P_B = P_0 - P_0 = 0, \qquad P_0P_W = P_0 - P_0 = 0.$$
So the three are projection matrices with pairwise products 0, and $P_0 + P_B + P_W = I$. Their ranks are their traces. The trace of $G$ is $\sum_i n_i \cdot \frac{1}{n_i} = k$, so the ranks are 1, $k - 1$ and $n - k$. The vector $P_B\mathbf{x}$ has entries $\bar{x}_i - \bar{x}$, the group mean minus the grand mean, and $P_W\mathbf{x}$ has entries $x_{ij} - \bar{x}_i$. So the squared lengths are the sums of squares of the course page,
$$\mathbf{x}^{\mathsf{T}}P_B\mathbf{x} = \sum_i n_i(\bar{x}_i - \bar{x})^2 = SSB, \qquad \mathbf{x}^{\mathsf{T}}P_W\mathbf{x} = \sum_i\sum_j (x_{ij} - \bar{x}_i)^2 = SSW,$$
and $\mathbf{x}^{\mathsf{T}}\mathbf{x} = n\bar{x}^2 + SSB + SSW$ is the theorem of Pythagoras for three perpendicular components.
For the worked sample, the script builds the three $40 \times 40$ matrices and confirms that each is symmetric and idempotent, that the products vanish, and that the ranks are 1, 3 and 36. The squared lengths are $n\bar{x}^2 = 40 \times 116.350^2 = 541{,}492.90$, $SSB = 2{,}850.01$ and $SSW = 5{,}983.09$, adding to $\sum x^2 = 550{,}326.00$. The mean squares are $2{,}850.01/3 = 950.00$ and $5{,}983.09/36 = 166.20$, and their ratio is 5.716.
Now suppose the observations are independent and normal with a common variance $\sigma^2$, and those of group $i$ have mean $\mu_i$. The mean vector has the entry $\mu_i$ for each observation of group $i$. Then $P_W\mathbf{m} = \mathbf{0}$ whatever the $\mu_i$ are, and $P_B\mathbf{m}$ has entries $\mu_i - \bar\mu$ with $\bar\mu = \sum_i n_i\mu_i/n$. Theorem 2 gives, with no further work:
- $SSW/\sigma^2$ is $\chi^2_{n-k}$, whether or not the group means are equal;
- $SSB/\sigma^2$ is $\chi^2_{k-1}(\lambda)$ with $\lambda = \sum_i n_i(\mu_i - \bar\mu)^2/\sigma^2$, central exactly when the means are all equal;
- $SSB$, $SSW$ and the grand mean are independent.
These are the three facts that the course page proves one at a time, with a rotation whose first row is weighted by the group sizes, and its statement of the distribution under the alternative. The degrees of freedom 3 and 36 of the opening are the dimensions of two perpendicular subspaces of the 40-dimensional space of data vectors: the vectors that are constant within groups and sum to zero, and the vectors that sum to zero within every group.
The same theorem is the distribution theory of regression. There $P$ is the matrix that projects the data onto the space spanned by the predictors, $I - P$ gives the residuals, and, when the mean vector lies in that space and the predictors are linearly independent, the residual sum of squares over $\sigma^2$ is chi-squared with $n$ minus the number of fitted coefficients as degrees of freedom, independent of the fitted values. Both statements are Theorem 2 for the pair $P$ and $I - P$. That is the subject of Linear Models.
Cochran's theorem
Theorem 2 asks that the matrices be idempotent and that their products vanish. Checking those conditions took a paragraph of algebra for the one-way layout, and takes more in larger designs. Cochran's theorem says that when the quadratic forms add up to the total sum of squares, all of it can be replaced by a count: it is enough that the ranks add up to $n$.
Theorem 3 (Cochran). Let $A_1, \ldots, A_k$ be symmetric $n \times n$ matrices of ranks $r_1, \ldots, r_k$ with $A_1 + \cdots + A_k = I$. The following three conditions are equivalent:
- (a) $r_1 + \cdots + r_k = n$;
- (b) every $A_i$ is idempotent;
- (c) $A_iA_j = 0$ whenever $i \ne j$.
When they hold, and $\mathbf{X}$ has independent normal coordinates with common variance $\sigma^2$ and mean vector $\mathbf{m}$, the quadratic forms $\mathbf{X}^{\mathsf{T}}A_i\mathbf{X}/\sigma^2$ are independent, and the $i$th is $\chi^2_{r_i}(\lambda_i)$ with $\lambda_i = \mathbf{m}^{\mathsf{T}}A_i\mathbf{m}/\sigma^2$.
In words: if a sum of squares $\sum X_i^2$ is split into quadratic forms whose ranks add up to $n$, the pieces are independent chi-squared variables with their ranks as degrees of freedom, central if the means are zero and noncentral otherwise.
The idea of the proof: write each symmetric matrix through its eigenvectors. If the ranks add to $n$, the eigenvectors of all the matrices together are exactly $n$ vectors, and the equation $\sum A_i = I$ forces them to be an orthonormal basis and every nonzero eigenvalue to be 1.
Proof. (c) implies (b). Multiply $I = \sum_j A_j$ by $A_i$: $A_i = \sum_j A_iA_j = A_i^2$.
(b) implies (a). A symmetric idempotent matrix is a projection matrix, so by Lemma 2 its rank is its trace. The trace is additive, so $n$, the trace of $I$, equals $\sum_i r_i$.
(a) implies (b) and (c). This is the substantial step. By the spectral theorem for real symmetric matrices (Axler, 2024, §7B, result 7.29), each $A_i$ has an orthonormal basis of eigenvectors. Keeping the eigenvectors with nonzero eigenvalues, of which there are $r_i$, gives
$$A_i = U_i\,D_i\,U_i^{\mathsf{T}},$$
where $U_i$ is $n \times r_i$ with orthonormal columns and $D_i$ is an $r_i \times r_i$ diagonal matrix with no zero on its diagonal. Set the blocks side by side: $U = (U_1 \ \cdots\ U_k)$, and let $D$ be the diagonal matrix with the $D_i$ down its diagonal. By (a), $U$ and $D$ are $n \times n$. The hypothesis $\sum_i A_i = I$ reads
$$U\,D\,U^{\mathsf{T}} = I.$$
First, $U$ is invertible: if $U^{\mathsf{T}}\mathbf{x} = \mathbf{0}$ then $\mathbf{x} = UDU^{\mathsf{T}}\mathbf{x} = \mathbf{0}$, so the square matrix $U^{\mathsf{T}}$ has only the zero vector in its null space. Multiplying $UDU^{\mathsf{T}} = I$ by $U^{-1}$ on the left and by $(U^{\mathsf{T}})^{-1}$ on the right, $D = U^{-1}(U^{\mathsf{T}})^{-1} = (U^{\mathsf{T}}U)^{-1}$, so
$$U^{\mathsf{T}}U = D^{-1}.$$
The right side is a diagonal matrix. The diagonal entries of the left side are the squared lengths of the columns of $U$, and every column is a unit vector, so they are all 1. Hence $D^{-1} = I$: every nonzero eigenvalue is 1, and $U^{\mathsf{T}}U = I$, so $U$ is orthogonal. In blocks, $U_i^{\mathsf{T}}U_i = I$ and $U_i^{\mathsf{T}}U_j = 0$ for $i \ne j$. Therefore $A_i = U_iU_i^{\mathsf{T}}$, and
$$A_i^2 = U_i(U_i^{\mathsf{T}}U_i)U_i^{\mathsf{T}} = A_i, \qquad A_iA_j = U_i(U_i^{\mathsf{T}}U_j)U_j^{\mathsf{T}} = 0.$$
The distributions. Under (b) and (c) the $A_i$ are projection matrices with pairwise products 0, and Theorem 2 applies. By Lemma 2, $\lVert A_i\mathbf{m}\rVert^2 = \mathbf{m}^{\mathsf{T}}A_i\mathbf{m}$. $\square$
For the one-way layout the count is $1 + (k - 1) + (n - k) = n$. With Cochran's theorem, that count, the symmetry of the three matrices and the identity $P_0 + P_B + P_W = I$ are enough, and the algebra of the previous section need not have been done. As a check on the proof, the script computes the eigenvectors of the three $40 \times 40$ matrices of the worked sample, 1, 3 and 36 of them with nonzero eigenvalues, and confirms that together they form an orthogonal matrix and that every nonzero eigenvalue is 1.
Example (ranks that do not add up). The condition is not automatic. Take $n = 2$, $A_1 = \begin{pmatrix} 2 & 0 \\ 0 & 0\end{pmatrix}$ and $A_2 = \begin{pmatrix} -1 & 0 \\ 0 & 1\end{pmatrix}$. Both are symmetric and they add to $I$, but the ranks are 1 and 2, adding to 3. The forms are $2X_1^2$ and $X_2^2 - X_1^2$: the first is twice a chi-squared variable, the second can be negative, and they are not independent. A decomposition of the total sum of squares into quadratic forms is a decomposition into independent chi-squared pieces only when the dimensions fit.
The theorem is named for Cochran (1934), whose paper, by its abstract, set out to prove "the main-relevant results" on "the joint distribution of several quadratic forms in a univariate normal system", for use in the analysis of variance and covariance. The statement proved above is the form in which the theorem is used in this collection. The spectral theorem is cited from linear algebra; everything else in the proof is given.
The F distribution
With the chi-squared pieces in hand, the remaining distributions are ratios. The definitions are on the course pages: for independent $Z$ standard normal, $U$ chi-squared with $d_1$ degrees of freedom and $V$ chi-squared with $d_2$,
$$T = \frac{Z}{\sqrt{V/d_2}} \ \text{ is } t \text{ with } d_2 \text{ degrees of freedom}, \qquad F = \frac{U/d_1}{V/d_2} \ \text{ is } F \text{ with } d_1 \text{ and } d_2 \text{ degrees of freedom}.$$
Both are built to cancel an unknown $\sigma^2$: numerator and denominator are each a squared length divided by its dimension, so each estimates $\sigma^2$ when its mean vector projects to zero.
Moments of a chi-squared variable
The chi-squared density with $\nu$ degrees of freedom, derived on the course page, is $f_\nu(x) = x^{\nu/2 - 1}e^{-x/2}/[2^{\nu/2}\Gamma(\nu/2)]$ for $x > 0$. That page's formula (1) is $\int_0^\infty x^{a-1}e^{-x/2}\,dx = 2^a\,\Gamma(a)$ for $a > 0$. Applying it with $a = \nu/2 + s$ gives every power moment of a $\chi^2_\nu$ variable $V$:
$$E[V^s] = \frac{2^{\nu/2 + s}\,\Gamma(\nu/2 + s)}{2^{\nu/2}\,\Gamma(\nu/2)} = \frac{2^s\,\Gamma(\nu/2 + s)}{\Gamma(\nu/2)} \qquad (s > -\nu/2).$$
With $\Gamma(a + 1) = a\Gamma(a)$: $E[V] = \nu$, $E[V^2] = \nu(\nu + 2)$, $E[1/V] = 1/(\nu - 2)$ for $\nu > 2$, and $E[1/V^2] = 1/[(\nu - 2)(\nu - 4)]$ for $\nu > 4$.
The density
Theorem 4 (the F density). The $F$ distribution with $d_1$ and $d_2$ degrees of freedom has density
$$f(x) = \frac{\Gamma\!\big(\frac{d_1 + d_2}{2}\big)}{\Gamma\!\big(\frac{d_1}{2}\big)\,\Gamma\!\big(\frac{d_2}{2}\big)}\,\Big(\frac{d_1}{d_2}\Big)^{d_1/2}\,x^{d_1/2 - 1}\,\Big(1 + \frac{d_1x}{d_2}\Big)^{-(d_1 + d_2)/2}, \qquad x > 0.$$
The idea is the one the course page uses for the $t$ density: fix the value of the denominator, where the event $F \le x$ concerns the numerator alone, then average over the denominator.
Proof. Let $G_1$ be the distribution function of $U$. Since $U$ and $V$ are independent with joint density $f_{d_1}(u)f_{d_2}(v)$, and $F \le x$ is the event $U \le x\,d_1V/d_2$,
$$P(F \le x) = \int_0^\infty G_1\Big(\frac{x\,d_1v}{d_2}\Big)\,f_{d_2}(v)\,dv.$$
Differentiate with respect to $x$ under the integral sign. The derivative of the integrand is $\frac{d_1v}{d_2}\,f_{d_1}\big(\frac{x\,d_1v}{d_2}\big)\,f_{d_2}(v)$, which is $\frac1x\,u f_{d_1}(u)\,f_{d_2}(v)$ with $u = x\,d_1v/d_2$. The function $u f_{d_1}(u)$ is a constant times $u^{d_1/2}e^{-u/2}$, which is bounded, say by $K$. So for $x \ge x_0 > 0$ the derivative is at most $(K/x_0)\,f_{d_2}(v)$, which has a finite integral, and that justifies the step. Writing $c_j = 1/[2^{d_j/2}\Gamma(d_j/2)]$,
$$f(x) = \int_0^\infty \frac{d_1v}{d_2}\cdot c_1\Big(\frac{x\,d_1v}{d_2}\Big)^{d_1/2 - 1}e^{-x d_1 v/(2d_2)}\cdot c_2\,v^{d_2/2 - 1}e^{-v/2}\,dv = c_1c_2\Big(\frac{d_1}{d_2}\Big)^{d_1/2}x^{d_1/2 - 1}\int_0^\infty v^{a - 1}e^{-\frac{v}{2}\left(1 + \frac{d_1x}{d_2}\right)}\,dv,$$
with $a = (d_1 + d_2)/2$. Substituting $w = v\,(1 + d_1x/d_2)$ and using formula (1), the integral is $(1 + d_1x/d_2)^{-a}\,2^a\,\Gamma(a)$. The powers of 2 cancel, $2^a c_1c_2 = 1/[\Gamma(d_1/2)\Gamma(d_2/2)]$, which leaves the stated density. $\square$
This is the density given in the NIST handbook (§1.3.6.6.5). The script checks the formula against scipy's at several points for four pairs of degrees of freedom; for the worked example's $F_{3,36}$ the density at $x = 1$ is 0.4442.
The beta form
Analysis of variance reports the share of the variation between groups, $\eta^2 = SSB/(SSB + SSW)$, alongside $F$. The two are the same statistic on different scales. With $U = SSB/\sigma^2$ and $V = SSW/\sigma^2$,
$$B = \frac{U}{U + V} = \frac{d_1F}{d_2 + d_1F}, \qquad F = \frac{d_2}{d_1}\cdot\frac{B}{1 - B},$$
and $B$ is an increasing function of $F$.
Corollary 1 (the beta density). If $U$ and $V$ are independent chi-squared variables with $d_1$ and $d_2$ degrees of freedom, then $B = U/(U + V)$ has the density
$$f_B(y) = \frac{\Gamma(a + b)}{\Gamma(a)\,\Gamma(b)}\,y^{a - 1}(1 - y)^{b - 1}, \qquad 0 < y < 1, \qquad a = \frac{d_1}{2},\ b = \frac{d_2}{2}.$$
Proof. The map $x \mapsto y = d_1x/(d_2 + d_1x)$ is increasing from $(0, \infty)$ onto $(0, 1)$, with inverse $x = d_2y/[d_1(1 - y)]$ and $dx/dy = d_2/[d_1(1 - y)^2]$. So $f_B(y) = f(x)\,dx/dy$ with the $f$ of Theorem 4. At that $x$, $1 + d_1x/d_2 = 1/(1 - y)$ and $(d_1/d_2)^a x^{a-1} = (d_1/d_2)\,[y/(1 - y)]^{a - 1}$. Collecting the powers of $1 - y$, the exponent is $-(a - 1) + (a + b) - 2 = b - 1$, and the constants $d_1/d_2$ and $d_2/d_1$ cancel. $\square$
This is the beta distribution with parameters $a$ and $b$. Two things come with it. A density integrates to 1, so
$$\int_0^1 y^{a-1}(1 - y)^{b-1}\,dy = \frac{\Gamma(a)\,\Gamma(b)}{\Gamma(a + b)}$$
for all $a$ and $b$ that are positive multiples of $\tfrac12$, an identity between the beta integral and gamma functions obtained here from probability. Applying it with $a + 1$ in place of $a$ gives the mean, $E[B] = \frac{\Gamma(a + 1)\Gamma(a + b)}{\Gamma(a)\Gamma(a + b + 1)} = \frac{a}{a + b}$. And the distribution function of $F$ is the distribution function of a beta variable evaluated at $d_1x/(d_2 + d_1x)$, which is how the site's scripts and most software compute $F$ probabilities (the NIST handbook gives the same formula in its complementary form).
For the worked sample, $\eta^2 = 2{,}850.01/8{,}833.10 = 0.3227$. Under equal group means $\eta^2$ has the beta distribution with $a = 1.5$ and $b = 18$. Its expected value is $1.5/19.5 = 0.0769$, which is $(k - 1)/(n - 1)$: a share of that size is what chance alone produces on average. The test at the 0.05 level rejects when $\eta^2$ exceeds $3 \times 2.8663/(36 + 3 \times 2.8663) = 0.1928$, and the probability that this beta variable is at least 0.3227 is 0.0026, the $p$-value of the $F$ test, as it must be.
Relatives of F
The $t$ density again. If $T$ has the $t$ distribution with $\nu$ degrees of freedom, $T^2$ has the $F$ distribution with 1 and $\nu$ (One-Way Analysis of Variance). Since $T$ is symmetric about 0, for $t > 0$, $P(\lvert T\rvert \le t) = P(T^2 \le t^2)$, and differentiating, the density of $T$ at $\pm t$ is $t\,f(t^2)$ with the $F_{1,\nu}$ density $f$. From Theorem 4 with $d_1 = 1$, $d_2 = \nu$ and $\Gamma(\tfrac12) = \sqrt{\pi}$,
$$t\,f(t^2) = t\cdot\frac{\Gamma\!\big(\frac{\nu + 1}{2}\big)}{\sqrt{\pi}\,\Gamma\!\big(\frac{\nu}{2}\big)}\,\nu^{-1/2}\,(t^2)^{-1/2}\,\Big(1 + \frac{t^2}{\nu}\Big)^{-(\nu + 1)/2} = \frac{\Gamma\!\big(\frac{\nu + 1}{2}\big)}{\sqrt{\nu\pi}\,\Gamma\!\big(\frac{\nu}{2}\big)}\Big(1 + \frac{t^2}{\nu}\Big)^{-(\nu + 1)/2},$$
the $t$ density of the course page, reached by a second route. For 36 degrees of freedom the critical values agree: $t_{36,\,0.975} = 2.0281$ and $2.0281^2 = 4.1132 = F_{1,\,36,\,0.95}$.
The reciprocal. If $F$ has the $F$ distribution with $d_1$ and $d_2$ degrees of freedom, then $1/F = (V/d_2)/(U/d_1)$ has the $F$ distribution with $d_2$ and $d_1$, by the definition. Hence lower percentage points come from upper ones: $P(F \le x) = P(1/F \ge 1/x)$, so $F_{d_1,\,d_2,\,p} = 1/F_{d_2,\,d_1,\,1 - p}$.
Mean and variance. By independence and the moments of a chi-squared variable,
$$E[F] = \frac{d_2}{d_1}\,E[U]\,E\Big[\frac1V\Big] = \frac{d_2}{d_1}\cdot d_1\cdot\frac{1}{d_2 - 2} = \frac{d_2}{d_2 - 2} \qquad (d_2 > 2),$$
$$E[F^2] = \frac{d_2^2}{d_1^2}\,E[U^2]\,E\Big[\frac{1}{V^2}\Big] = \frac{d_2^2\,(d_1 + 2)}{d_1\,(d_2 - 2)(d_2 - 4)} \qquad (d_2 > 4).$$
Subtracting the square of the mean and putting the terms over a common denominator, the numerator is $(d_1 + 2)(d_2 - 2) - d_1(d_2 - 4) = 2(d_1 + d_2 - 2)$, so
$$\operatorname{Var}(F) = \frac{2\,d_2^2\,(d_1 + d_2 - 2)}{d_1\,(d_2 - 2)^2\,(d_2 - 4)} \qquad (d_2 > 4).$$
Both agree with the NIST handbook (§1.3.6.6.5). The mean does not depend on $d_1$ and is always above 1, although numerator and denominator both estimate $\sigma^2$ without bias: the reciprocal of an unbiased estimate is biased upward, an instance of Jensen's inequality (Probability Inequalities). For $F_{3,36}$ the mean is $36/34 = 1.0588$ and the variance 0.8642, a standard deviation of 0.9296.
Many denominator degrees of freedom. As $d_2 \to \infty$ with $d_1$ fixed, $d_1F$ converges in distribution to $\chi^2_{d_1}$. The denominator $V/d_2$ is the mean of $d_2$ independent squared standard normal variables, so it converges in probability to 1 by the weak law of large numbers, and the numerator $U$ does not change with $d_2$. Slutsky's theorem, proved in Convergence in Distribution and the Proof of the Central Limit Theorem, gives the limit. This is the sense in which a chi-squared test is an $F$ test with a known variance. The upper 5% point of $\chi^2_3$ is 7.815, and $3F_{3,\,d_2,\,0.95}$ is 8.599 for $d_2 = 36$, 7.889 for $d_2 = 360$ and 7.822 for $d_2 = 3{,}600$. With 36 degrees of freedom the estimation of $\sigma^2$ still raises the critical value by a tenth. The same argument with $d_1 = 1$ is the convergence of $t$ to the normal, the third worked problem of that article.
The noncentral chi-squared density
Under the alternative the between-groups sum of squares is noncentral chi-squared, and the power of a test is a probability under that distribution. Its density has a form that is both computable and easy to interpret: the number of extra degrees of freedom is random.
Theorem 5 (the noncentral chi-squared density is a Poisson mixture). The $\chi^2_r(\lambda)$ distribution has density
$$f_{r,\lambda}(x) = \sum_{j=0}^{\infty} e^{-\lambda/2}\,\frac{(\lambda/2)^j}{j!}\;f_{r + 2j}(x), \qquad x > 0,$$
where $f_\nu$ is the central chi-squared density with $\nu$ degrees of freedom.
The weights are the probabilities of a Poisson distribution with mean $\lambda/2$. So a $\chi^2_r(\lambda)$ variable can be produced in two stages: draw $J$ from that Poisson distribution, then draw a central chi-squared variable with $r + 2J$ degrees of freedom.
Proof. One degree of freedom. Let $Y = (Z + \delta)^2$ with $Z$ standard normal and $\delta = \sqrt{\lambda}$. For $x > 0$, $P(Y \le x) = P(-\sqrt{x} - \delta \le Z \le \sqrt{x} - \delta) = \Phi(\sqrt{x} - \delta) - \Phi(-\sqrt{x} - \delta)$. Differentiating, and writing $\phi$ for the standard normal density, the density of $Y$ is
$$\frac{\phi(\sqrt{x} - \delta) + \phi(\sqrt{x} + \delta)}{2\sqrt{x}} = \frac{e^{-(x + \delta^2)/2}}{2\sqrt{2\pi x}}\,\big(e^{\delta\sqrt{x}} + e^{-\delta\sqrt{x}}\big) = \frac{e^{-\lambda/2}\,e^{-x/2}}{\sqrt{2\pi x}}\,\sum_{j=0}^{\infty}\frac{(\lambda x)^j}{(2j)!},$$
because $e^{u} + e^{-u} = 2\sum_j u^{2j}/(2j)!$ and $(\delta\sqrt{x})^{2j} = (\lambda x)^j$. The $j$th term is $e^{-\lambda/2}\lambda^j\,x^{j - 1/2}e^{-x/2}/[\sqrt{2\pi}\,(2j)!]$. The claimed $j$th term is
$$e^{-\lambda/2}\,\frac{(\lambda/2)^j}{j!}\cdot\frac{x^{j - 1/2}\,e^{-x/2}}{2^{j + 1/2}\,\Gamma(j + \tfrac12)}.$$
The two agree if $\sqrt{2\pi}\,(2j)! = 2^j\,j!\,2^{j + 1/2}\,\Gamma(j + \tfrac12)$, that is, if
$$\sqrt{\pi}\,(2j)! = 2^{2j}\,j!\,\Gamma(j + \tfrac12).$$
This holds for $j = 0$, where it reads $\sqrt{\pi} = \Gamma(\tfrac12)$. If it holds for $j$, then multiplying the left side by $(2j + 2)(2j + 1)$ gives $\sqrt{\pi}\,(2j + 2)!$, and multiplying the right side by the same number, written as $4\,(j + 1)(j + \tfrac12)$, gives $2^{2j + 2}\,(j + 1)!\,\Gamma(j + \tfrac32)$, since $(j + \tfrac12)\Gamma(j + \tfrac12) = \Gamma(j + \tfrac32)$. So it holds for $j + 1$, and by induction for every $j$.
More degrees of freedom. For $r = 1$ the proof is complete. For $r \ge 2$, by Lemma 1, a $\chi^2_r(\lambda)$ variable is $Y + R$ with $Y = (Z_1 + \sqrt{\lambda})^2$ and $R$ an independent central $\chi^2_{r-1}$ variable. By the convolution formula cited on the course page, its density is
$$\int_0^x \Big[\sum_j w_j\,f_{1 + 2j}(u)\Big]\,f_{r - 1}(x - u)\,du = \sum_j w_j\int_0^x f_{1 + 2j}(u)\,f_{r-1}(x - u)\,du = \sum_j w_j\,f_{r + 2j}(x),$$
with $w_j = e^{-\lambda/2}(\lambda/2)^j/j!$. The last step holds because the inner integral is the density of the sum of independent $\chi^2_{1 + 2j}$ and $\chi^2_{r-1}$ variables, and independent chi-squared variables add. The exchange of the sum and the integral is allowed because every term is nonnegative; that is a theorem of the theory of integration (monotone convergence), used here without proof. $\square$
Integrating the density term by term gives the tail probability used for power,
$$P\big(\chi^2_r(\lambda) > x\big) = \sum_{j=0}^{\infty} e^{-\lambda/2}\,\frac{(\lambda/2)^j}{j!}\;P\big(\chi^2_{r + 2j} > x\big).$$
Each term is a central probability, and the tail probabilities do not decrease with $j$. A $\chi^2_{\nu + 2}$ variable is a $\chi^2_\nu$ variable plus an independent nonnegative $\chi^2_2$ variable, so $P(\chi^2_{\nu+2} > x) \ge P(\chi^2_\nu > x)$. A larger $\lambda$ moves the Poisson weights toward the higher terms, whose tail probabilities are the larger ones. One-Way Analysis of Variance proves, by a different argument, that the power of the test increases with $\lambda$.
The ratio distributions. The noncentral $F$ distribution with $d_1$ and $d_2$ degrees of freedom and noncentrality $\lambda$ is that of $(U'/d_1)/(V/d_2)$ with $U'$ distributed as $\chi^2_{d_1}(\lambda)$ and $V$ an independent central $\chi^2_{d_2}$. By the theorem above it is what $SSB/(k-1)$ over $SSW/(n-k)$ has when the group means differ. Conditioning on $V$ and applying the tail formula to $U'$, then exchanging the sum and the integral over $V$ as before,
$$P(F' > x) = \sum_{j=0}^{\infty} e^{-\lambda/2}\,\frac{(\lambda/2)^j}{j!}\;P\Big(F_{d_1 + 2j,\,d_2} > \frac{d_1x}{d_1 + 2j}\Big),$$
where $F_{d_1 + 2j,\,d_2}$ is a central $F$ variable: given $J = j$, the numerator $U'$ is $\chi^2_{d_1 + 2j}$, and $U'/d_1$ exceeds $xV/d_2$ exactly when $[U'/(d_1 + 2j)]/(V/d_2)$ exceeds $d_1x/(d_1 + 2j)$. The noncentral $t$ distribution is that of $(Z + \delta)/\sqrt{V/\nu}$. Its distribution function is derived as an integral in Power and Sample Size for a Test of a Mean, and Johnson, Kotz and Balakrishnan (1995) give its further properties. Its square is noncentral $F$ with 1 and $\nu$ degrees of freedom and noncentrality $\delta^2$, by the definitions.
The check on the running design. Take four normal populations with the age groups' means and a common variance $\sigma^2 = 290.94$, the setting of the course page's simulation, and 10 observations from each. There $\sum_i n_i(\mu_i - \bar\mu)^2/(k - 1) = 1{,}074.96$, so $\lambda = 3 \times 1{,}074.96/290.94 = 11.084$. The between-groups sum of squares over $\sigma^2$ is $\chi^2_3(11.084)$, with mean $3 + 11.084 = 14.084$ and variance $6 + 4 \times 11.084 = 50.337$. The script confirms that the Poisson mixture agrees with scipy's noncentral chi-squared density. The power of the $F$ test at the 0.05 level is $P(F' > 2.8663)$, which the mixture formula gives as 0.7583, in agreement with scipy's noncentral $F$. A simulation of the design, 10,000 sets of four normal samples for each of 30 seeds, rejected in a share 0.7594 of sets (median over the seeds; interquartile range 0.7561 to 0.7612). The fourth worked problem computes the first terms of the series by hand.
What depends on normality
Everything above rests on Theorem 1, which is special to the normal distribution. Two moment formulas, valid for every population with enough moments, show what is lost without it.
Theorem 6 (moments of the sample mean and variance). Let $X_1, \ldots, X_n$ be i.i.d. with mean $\mu$, variance $\sigma^2$, third central moment $\mu_3$ and fourth central moment $\mu_4$, all finite, and let $\kappa = \mu_4/\sigma^4$. Then
$$\operatorname{Cov}(\bar{X}, s^2) = \frac{\mu_3}{n}, \qquad \operatorname{Var}(s^2) = \frac{1}{n}\Big(\mu_4 - \frac{n - 3}{n - 1}\,\sigma^4\Big) = \frac{\sigma^4}{n}\Big(\kappa - \frac{n - 3}{n - 1}\Big).$$
Proof. Subtracting $\mu$ from every observation changes $\bar{X}$ by a constant and leaves $s^2$ unchanged, so assume $\mu = 0$. Then $E[X_i] = 0$, and the expected value of a product of observations vanishes whenever some observation appears in it exactly once, by independence. Write $A = \sum_i X_i^2$ and $B = \sum_{i < j} X_iX_j$. Since $(\sum_i X_i)^2 = A + 2B$,
$$(n - 1)\,s^2 = A - n\bar{X}^2 = A - \frac{A + 2B}{n} = \frac{n - 1}{n}\,A - \frac{2}{n}\,B.$$
The covariance. $E[\bar{X}A] = \frac1n\sum_j\sum_i E[X_jX_i^2]$. The terms with $j \ne i$ vanish and the $n$ terms with $j = i$ are $\mu_3$, so $E[\bar{X}A] = \mu_3$. Next, $E[\bar{X}B] = \frac1n\sum_l\sum_{i<j} E[X_lX_iX_j]$. In every term at least one of the three observations appears exactly once, because $i \ne j$, so $E[\bar{X}B] = 0$. Hence $E[\bar{X}(n - 1)s^2] = \frac{n-1}{n}\mu_3$, so $E[\bar{X}s^2] = \mu_3/n$, and since $E[\bar{X}] = 0$ this is the covariance.
The variance. The $X_i^2$ are independent with variance $\mu_4 - \sigma^4$, so $\operatorname{Var}(A) = n(\mu_4 - \sigma^4)$. The products $X_iX_j$ with $i < j$ have mean 0 and variance $E[X_i^2X_j^2] = \sigma^4$. Two different products are uncorrelated: $E[X_iX_j\,X_lX_m]$ contains an observation exactly once unless the pairs coincide. There are $n(n-1)/2$ products, so $\operatorname{Var}(B) = n(n-1)\sigma^4/2$. Finally $\operatorname{Cov}(A, B) = \sum_l\sum_{i<j} E[X_l^2X_iX_j] = 0$, for the same reason. Therefore
$$(n - 1)^2\operatorname{Var}(s^2) = \frac{(n-1)^2}{n^2}\cdot n(\mu_4 - \sigma^4) + \frac{4}{n^2}\cdot\frac{n(n - 1)}{2}\,\sigma^4 = \frac{(n-1)^2}{n}(\mu_4 - \sigma^4) + \frac{2(n - 1)}{n}\,\sigma^4.$$
Dividing by $(n - 1)^2$, $\operatorname{Var}(s^2) = \frac1n\big(\mu_4 - \sigma^4 + \frac{2}{n - 1}\sigma^4\big)$, and $-1 + \frac{2}{n-1} = -\frac{n - 3}{n - 1}$. $\square$
The variance formula is the one that One-Sided Confidence Bounds and Sample Size cites; it is now proved. The script checks both formulas in exact arithmetic on a small case, all 27 ordered samples of 3 from the population $\{0, 1, 3\}$: there $\sigma^2 = 1.5556$, $\mu_3 = 0.7407$ and $\mu_4 = 3.6296$, and enumeration gives $\operatorname{Cov}(\bar{X}, s^2) = 0.2469 = \mu_3/3$ and $\operatorname{Var}(s^2) = 1.2099$, as the formulas say.
The theorem measures the two failures.
- The scaled variance is not chi-squared. For a normal population $\kappa = 3$ and the formula gives $2\sigma^4/(n - 1)$, the variance that $\chi^2_{n-1}$ implies. For any other $\kappa$ the variance of $s^2$ is different at every sample size. The Delta Method finds the limiting distribution of $s^2$ and the limiting coverage of the chi-squared interval for a variance, 0.852 in place of 0.95 for the blood pressures.
- The mean and the variance are not independent. If the population is skewed, $\mu_3 \ne 0$ and they are correlated. Dividing the covariance by the standard deviations $\sigma/\sqrt{n}$ and $\sqrt{\operatorname{Var}(s^2)}$,
$$\operatorname{Corr}(\bar{X}, s^2) = \frac{\gamma}{\sqrt{\kappa - \frac{n - 3}{n - 1}}}, \qquad \gamma = \frac{\mu_3}{\sigma^3},$$
where $\gamma$ is the skewness. The correlation does not shrink as $n$ grows; it tends to $\gamma/\sqrt{\kappa - 1}$. For the blood pressures of the check below that limit is 0.541.
Example (zero covariance is not independence). A symmetric population has $\mu_3 = 0$, so $\bar{X}$ and $s^2$ are uncorrelated. They need not be independent. Let the population take the values $-1$ and $1$ with probability $\tfrac12$ each, and $n = 2$. The pair $(\bar{X}, s^2)$ is $(-1, 0)$, $(1, 0)$ or $(0, 2)$, with probabilities $\tfrac14$, $\tfrac14$ and $\tfrac12$. The covariance is 0, but $s^2 = 2$ forces $\bar{X} = 0$. Independence of the mean and the variance holds for the normal distribution alone. Lukacs (1942, p. 91) states it as a necessary and sufficient condition for the normality of the population, and proves that independence forces normality by characteristic functions, for a population that has a density and a variance. He credits the first proof to R. C. Geary; the course page cites both papers.
The check on two populations. The unit's two populations are the 5,247 NHANES blood pressures, with skewness 1.038 and $\kappa = 4.680$, and the 5,146 measurements of C-reactive protein of The Central Limit Theorem, with skewness 7.892 and $\kappa = 100.5$ (National Center for Health Statistics, 2020). For samples of 10 drawn with replacement, the script computed the correlation between $\bar{X}$ and $s^2$ across 10,000 samples, for each of 30 seeds.
| Population | Correlation by the formula | Simulated (median over seeds) | Interquartile range |
|---|---|---|---|
| Normal | 0.000 | 0.000 | $-0.008$ to 0.007 |
| Blood pressure | 0.525 | 0.524 | 0.519 to 0.531 |
| C-reactive protein | 0.790 | 0.790 | 0.786 to 0.796 |
For the blood pressures the mean and variance of a sample of 10 have correlation about one half, and for C-reactive protein about 0.8. This dependence is the explanation that the course page Moment Generating Functions and Slutsky's Theorem offers, as consistent with its simulations and not as a proved result, for the lopsided misses of the $t$ interval in skewed populations: a sample with a low mean tends to have a small $s$ as well.
What is proved, what is cited, what is computed
Proved here. Theorem 1 from the course page's rotation step; Lemmas 1 and 2; the projection theorem; Cochran's theorem, with the equivalence of its three conditions; the moments of a chi-squared variable; the $F$ density, its beta form, its mean and variance, the reciprocal rule and its chi-squared limit; the mean, variance and density of the noncentral chi-squared distribution and the series for noncentral $F$ probabilities; and the two moment formulas of Theorem 6.
Cited. Through the course page: the three properties of joint densities and independence, and the convolution formula (Casella and Berger, 2002). The spectral theorem for real symmetric matrices (Axler, 2024). The exchange of an infinite sum of nonnegative terms with an integral. The further theory of the noncentral $t$ distribution (Johnson, Kotz and Balakrishnan, 1995). The converse that independence of $\bar{X}$ and $s^2$ forces normality (Lukacs, 1942). This unit avoids measure theory: random variables here have densities, and the one place where integration theory is used beyond calculus is the exchange just named.
Computed. The projections, ranks and sums of squares of the worked sample, the percentage points and the power are computed by the script and checked against scipy where scipy has the function. The simulated power and the simulated correlations are observations from seeded experiments.
Worked problems
These problems were written for this article, on the sleep data of Student (1908) and on the worked sample and design of the course's analysis of variance. The script tools/experiments/statistics/normal-sampling-theory.py recomputes every number.
Problem 1 (Cochran's theorem by hand). The first four of Student's sleep differences are 1.2, 2.4, 1.3 and 1.3 hours. Split $\sum x_i^2$ into three quadratic forms: one for the mean, one for the contrast between the first pair and the second pair, and one for the differences within the pairs. Find the ranks, verify the decomposition, and say what distribution the ratio of the last two has for a normal sample.
Solution. The decomposition is $10.58 = 9.61 + 0.25 + 0.72$ with ranks 1, 1 and 2, and for a normal sample the contrast form over half the within-pairs form has the $F$ distribution with 1 and 2 degrees of freedom.
The three forms are
$$q_1 = \frac{(x_1 + x_2 + x_3 + x_4)^2}{4}, \qquad q_2 = \frac{(x_1 + x_2 - x_3 - x_4)^2}{4}, \qquad q_3 = \frac{(x_1 - x_2)^2}{2} + \frac{(x_3 - x_4)^2}{2}.$$
Each square is the squared coordinate of $\mathbf{x}$ along a unit vector: $(1, 1, 1, 1)/2$ for $q_1$, $(1, 1, -1, -1)/2$ for $q_2$, and $(1, -1, 0, 0)/\sqrt{2}$ and $(0, 0, 1, -1)/\sqrt{2}$ for $q_3$. So $q_1 = \mathbf{x}^{\mathsf{T}}A_1\mathbf{x}$ with $A_1 = \mathbf{u}\mathbf{u}^{\mathsf{T}}$ of rank 1, likewise $A_2$ of rank 1, and $A_3$ is a sum of two such matrices, of rank 2. The four unit vectors are perpendicular to one another, as a check of the six dot products shows, so they form an orthonormal basis of four-dimensional space. The sum of the squared coordinates of a vector in an orthonormal basis is its squared length, so $q_1 + q_2 + q_3 = \sum x_i^2$ for every $\mathbf{x}$, that is, $A_1 + A_2 + A_3 = I$. The ranks add to $1 + 1 + 2 = 4 = n$, the condition of Cochran's theorem.
With the numbers: the sum is 6.2 and the mean 1.55, so $q_1 = 6.2^2/4 = 9.61$. The contrast is $1.2 + 2.4 - 1.3 - 1.3 = 1.0$, so $q_2 = 0.25$. The within-pair differences are $-1.2$ and 0, so $q_3 = 1.44/2 + 0 = 0.72$. And $\sum x_i^2 = 1.44 + 5.76 + 1.69 + 1.69 = 10.58 = 9.61 + 0.25 + 0.72$. As a check, $q_2 + q_3 = 0.97$ is the sum of squared deviations from the mean, and $0.97/3 = 0.3233$ is the sample variance of the four values.
If the four differences are independent normal variables with a common mean $\mu$ and variance $\sigma^2$, the mean vector is $\mu(1, 1, 1, 1)$, which is perpendicular to the three unit vectors of $q_2$ and $q_3$. So $A_2\mathbf{m} = A_3\mathbf{m} = \mathbf{0}$, and by Cochran's theorem $q_2/\sigma^2$ and $q_3/\sigma^2$ are independent central chi-squared variables with 1 and 2 degrees of freedom. Their ratio, each over its degrees of freedom, is $F = q_2/(q_3/2) = 0.25/0.36 = 0.694$, to be referred to the $F$ distribution with 1 and 2 degrees of freedom; the probability of a value at least that large is 0.492. The split of the 3 degrees of freedom of $s^2$ into $1 + 2$ is one of many. Any orthonormal basis of the space perpendicular to the vector of ones gives another, and each gives independent pieces.
Problem 2 (the analysis of variance as a statement about a beta variable). For the worked sample, with $SSB = 2{,}850.01$ and $SSW = 5{,}983.09$ on 3 and 36 degrees of freedom, express the test in terms of $\eta^2$: its value, its distribution under equal means, its expected value under equal means, the rejection threshold at the 0.05 level, and the $p$-value.
Solution. $\eta^2 = 0.3227$; under equal means it is beta with parameters 1.5 and 18, with expected value 0.0769; the test rejects above 0.1928; and $p = 0.0026$.
By the projection theorem, under equal means $U = SSB/\sigma^2$ and $V = SSW/\sigma^2$ are independent central chi-squared variables with 3 and 36 degrees of freedom. Then $\eta^2 = SSB/(SSB + SSW) = U/(U + V)$, in which $\sigma^2$ has canceled, and by Corollary 1 it has the beta density with $a = 3/2$ and $b = 36/2 = 18$. Its value is $2{,}850.01/8{,}833.10 = 0.3227$, and its mean under equal means is $a/(a + b) = 1.5/19.5 = 0.0769$.
Since $\eta^2 = d_1F/(d_2 + d_1F)$ is an increasing function of $F$, the event $F > 2.8663$ is the event $\eta^2 > 3 \times 2.8663/(36 + 3 \times 2.8663) = 0.1928$. Likewise the event that $F$ is at least the observed 5.716 is the event that $\eta^2$ is at least 0.3227, so the two statistics give one $p$-value, 0.0026, computed here from the beta distribution function and on the course page from $F$.
The beta form answers a question that $F$ leaves obscure: how large a share of variation is unremarkable. With four groups of 10 and no real differences, the between-groups share averages 7.7%, and one sample in twenty shows more than 19.3%.
Problem 3 (the lower tail of F by the reciprocal rule). Many tables of $F$ give upper percentage points only. Find the point below which $F_{3,36}$ falls with probability 0.05, and the mean and standard deviation of $F_{3,36}$.
Solution. The lower 5% point is 0.1163, the mean 1.0588 and the standard deviation 0.9296.
By the reciprocal rule, $F_{3,\,36,\,0.05} = 1/F_{36,\,3,\,0.95}$. The upper 5% point of the $F$ distribution with 36 and 3 degrees of freedom is 8.6018, so the lower 5% point with 3 and 36 is $1/8.6018 = 0.1163$. Note that the degrees of freedom change places: the reciprocal of the upper point $F_{3,\,36,\,0.95} = 2.8663$ is not the answer.
The mean is $d_2/(d_2 - 2) = 36/34 = 1.0588$. The variance is
$$\frac{2 \times 36^2 \times (3 + 36 - 2)}{3 \times 34^2 \times 32} = \frac{95{,}904}{110{,}976} = 0.8642,$$
and its square root is 0.9296. So a between-groups mean square below 0.1163 times the within-groups mean square is as unusual, under equal means, as one above 2.8663 times it. A very small $F$ is no evidence of differing means, as the course page explains.
Problem 4 (power from the Poisson mixture). In the design of the check above, four normal populations with $\lambda = 11.084$ and 10 observations from each, suppose $\sigma^2$ were known. The test would then refer $SSB/\sigma^2$ to the $\chi^2_3$ distribution and reject above 7.815. Compute its power from the Poisson mixture, showing the terms, and compare with the power of the $F$ test.
Solution. The power with $\sigma^2$ known is 0.8072, against 0.7583 for the $F$ test.
Under equal means $SSB/\sigma^2$ is $\chi^2_3$, whose upper 5% point is 7.815. Under the alternative it is $\chi^2_3(\lambda)$ with $\lambda = 11.084$, and by the tail formula the power is $\sum_j w_j\,P(\chi^2_{3 + 2j} > 7.815)$ with Poisson weights $w_j = e^{-5.542}\,5.542^j/j!$, since $\lambda/2 = 5.542$.
| $j$ | Weight $w_j$ | $P(\chi^2_{3 + 2j} > 7.815)$ | Sum of the terms up to $j$ |
|---|---|---|---|
| 0 | 0.0039 | 0.0500 | 0.0002 |
| 1 | 0.0217 | 0.1667 | 0.0038 |
| 2 | 0.0602 | 0.3492 | 0.0248 |
| 3 | 0.1112 | 0.5529 | 0.0863 |
| 4 | 0.1540 | 0.7298 | 0.1987 |
| 5 | 0.1707 | 0.8555 | 0.3447 |
| 10 | 0.0295 | 0.9986 | 0.7807 |
| 20 | 0.0000 | 1.0000 | 0.8072 |
Rows 6 to 9 and 11 to 19 are not shown, and the running sum in the last column includes them. The first row is the level of the test: if $J = 0$ the statistic is central chi-squared with 3 degrees of freedom and exceeds 7.815 with probability 0.05. That row has weight 0.0039 only. Most of the weight sits near $j = 5$, the mean of the Poisson distribution being 5.542, where the statistic behaves like a central chi-squared variable with about 13 degrees of freedom and exceeds 7.815 with probability 0.8555. By $j = 20$ the sum has converged to four places: the power is 0.8072.
When $\sigma^2$ is not known it is estimated by $SSW/36$, and the test becomes the $F$ test, with power 0.7583 from the corresponding series for noncentral $F$. The difference, about 0.05, is the price of estimating the variance from 36 degrees of freedom. It is the same price seen in the critical values, where $3F_{3,\,36,\,0.95} = 8.599$ stands against 7.815.
Summary
| Result | Statement |
|---|---|
| Rotation | $Q$ orthogonal: $Q\mathbf{X}$ has independent normal coordinates, variance $\sigma^2$, means $Q\mathbf{m}$ |
| Projection | $P$ symmetric idempotent of rank $r$: $\mathbf{X}^{\mathsf{T}}P\mathbf{X}/\sigma^2$ is $\chi^2_r(\lVert P\mathbf{m}\rVert^2/\sigma^2)$ |
| Independence | $P_iP_j = 0$: $P_i\mathbf{X}$ and $P_j\mathbf{X}$ independent |
| Cochran | $\sum A_i = I$, symmetric: ranks add to $n$ exactly when the $A_i$ are projections with $A_iA_j = 0$ |
| Noncentral chi-squared | mean $r + \lambda$, variance $2r + 4\lambda$, density $\sum_j e^{-\lambda/2}\frac{(\lambda/2)^j}{j!}f_{r+2j}$ |
| $F$ | mean $d_2/(d_2 - 2)$; $1/F_{d_1,d_2}$ is $F_{d_2,d_1}$; $d_1F/(d_2 + d_1F)$ is beta$(d_1/2, d_2/2)$; $t_\nu^2$ is $F_{1,\nu}$ |
| Without normality | $\operatorname{Cov}(\bar{X}, s^2) = \mu_3/n$; $\operatorname{Var}(s^2) = \frac{\sigma^4}{n}\big(\kappa - \frac{n-3}{n-1}\big)$ |
What we have learned
A vector of independent normal variables with a common variance looks the same in every orthonormal basis, apart from its mean vector, which rotates with the basis. Projecting it onto perpendicular subspaces therefore gives independent pieces, and the squared length of each piece, over $\sigma^2$, is chi-squared with the dimension of the subspace as degrees of freedom, noncentral when the mean vector has a component there. Cochran's theorem turns the perpendicularity into a count of ranks. Ratios of such pieces give $t$ and $F$, whose densities and moments follow from one gamma integral, and the noncentral chi-squared distribution is a central one with a Poisson number of extra pairs of degrees of freedom. For the 40 blood pressures this accounts for the 3 and 36, for the independence of the two sums of squares, for the $p$-value 0.0026 and for the power 0.7583 of the design. None of it survives a change of population unchanged: in general the mean and variance of a sample have covariance $\mu_3/n$, and the variance of $s^2$ depends on the fourth moment.
This article closes the unit on probability theory. The next unit, Likelihood and Estimation, begins the theory of inference with the likelihood function, and Linear Models applies the projection theorem to regression.
References
- Axler, S. (2024). Linear Algebra Done Right (4th ed.). Springer. §7B, result 7.29 (the real spectral theorem). record
- Casella, G., and Berger, R. L. (2002). Statistical Inference (2nd ed.). Duxbury. Chapter 4, "Multiple Random Variables" (independence and joint densities), and Chapter 5, "Properties of a Random Sample" (the convolution formula). record · 11 pages
- Cochran, W. G. (1934). The distribution of quadratic forms in a normal system, with applications to the analysis of covariance. Mathematical Proceedings of the Cambridge Philosophical Society, 30(2), 178–191. record
- Johnson, N. L., Kotz, S., and Balakrishnan, N. (1995). Continuous Univariate Distributions, Volume 2 (2nd ed.). Wiley. The chapter "Noncentral t-Distributions". record · 2 pages
- Lancaster, H. O. (1965). The Helmert matrices. The American Mathematical Monthly, 72(1), 4–12. record
- Lukacs, E. (1942). A characterization of the normal distribution. The Annals of Mathematical Statistics, 13(1), 91–93. record · 2 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J), Blood Pressure (BPX_J) and High-Sensitivity C-Reactive Protein (HSCRP_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 67 pages
- NIST/SEMATECH. e-Handbook of Statistical Methods, §1.3.6.6.5 (F distribution). National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/ record · 34 pages
- Student (1908). The probable error of a mean. Biometrika, 6(1), 1–25. record · 14 pages