Covariance Matrices and Quadratic Forms

The algebra of covariance matrices and the distribution theory of quadratic forms in normal vectors, proved. Which matrices are covariance matrices; the spectral decomposition (cited) and what follows from it, including square roots, trace and determinant as sums and products of eigenvalues, and the largest variance of a linear combination; the Mahalanobis distance, its invariance to units and its chi-squared distribution; the exact condition for a quadratic form in a normal vector to be chi-squared, and for two forms, or a form and a linear function, to be independent; the unbiasedness of the sample covariance matrix, the independence of the sample mean and sample covariance of a normal sample, and the Wishart distribution with its moments. On four NHANES measurements, Mahalanobis outliers are checked against chi-squared quantiles and normal simulations, and samples of ten show what normality adds. Worked problems on correlations that cannot coexist, a Mahalanobis distance by hand, a quadratic form that is not chi-squared, and an interval for the variance of pulse pressure.

The National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 recorded systolic and diastolic blood pressure, height and weight for 5,161 adults. One of them has a systolic pressure of 162, a diastolic pressure of 54, a height of 181.4 cm and a weight of 123.4 kg. None of the four is two standard deviations from its mean, yet the combination is rare: high systolic pressure usually comes with high diastolic pressure, and this adult's diastolic pressure is low. How unusual is this adult, measured on one scale that respects the correlations, and how large must that measure be before we call an adult an outlier? The scale is a quadratic form in the adult's deviations from the mean, and the cutoff comes from its distribution. This article proves which quadratic forms in normal vectors are chi-squared, and applies the answer to the adults and to the sample covariance matrix itself.

A quadratic form in a vector $\mathbf{x}$ is $\mathbf{x}^{\mathsf T}A\mathbf{x} = \sum_j\sum_k a_{jk}x_jx_k$ for a symmetric matrix $A$. Variances of linear combinations, sums of squares in the analysis of variance, the Mahalanobis distance and the exponent of the normal density are all quadratic forms. This article is the home, for the Multivariate Statistics unit, of the algebra of covariance matrices and of the distribution theory of quadratic forms; The Multivariate Normal Distribution uses its square roots and its chi-squared result, and the later articles of the unit use the rest.

First we characterize covariance matrices and derive what the spectral theorem says about them. Then we define the Mahalanobis distance and prove that it is chi-squared for normal data. Next we find exactly when a quadratic form in a normal vector is chi-squared, and when two forms are independent, extending the projection theorem of Sampling Distributions from the Normal, and the conditions behind the Cochran's theorem proved there, from independent coordinates with a common variance to any covariance matrix. Then we turn to samples: the sample covariance matrix, its independence from the sample mean under normality, and the Wishart distribution. Finally we return to the adults. The article assumes The Multivariate Normal Distribution and the projection theorem and Cochran's theorem of Sampling Distributions from the Normal.

Which matrices are covariance matrices

The notation is that of The Multivariate Normal Distribution: a random vector $\mathbf{X}$ with $p$ coordinates, mean vector $\boldsymbol\mu$ and covariance matrix $\Sigma$, and $N_p(\boldsymbol\mu, \Sigma)$ for the multivariate normal distribution, with the covariance matrix (for $p = 1$ the variance) second. Two rules are proved in Expectation, Variance and Covariance (Theorem 6): $\operatorname{Cov}(B\mathbf{X} + \mathbf{c}) = B\Sigma B^{\mathsf T}$, and $\operatorname{Var}(\mathbf{a}^{\mathsf T}\mathbf{X}) = \mathbf{a}^{\mathsf T}\Sigma\mathbf{a}$. A symmetric matrix $M$ is positive semidefinite if $\mathbf{a}^{\mathsf T}M\mathbf{a} \ge 0$ for every $\mathbf{a}$, and positive definite if $\mathbf{a}^{\mathsf T}M\mathbf{a} > 0$ for every $\mathbf{a} \ne \mathbf{0}$.

Theorem 1 (covariance matrices).

  1. A matrix is the covariance matrix of some random vector if and only if it is symmetric and positive semidefinite.
  2. $\Sigma$ is singular if and only if some combination $\mathbf{a}^{\mathsf T}\mathbf{X}$ with $\mathbf{a} \ne \mathbf{0}$ equals the constant $\mathbf{a}^{\mathsf T}\boldsymbol\mu$ with probability 1. So $\Sigma$ is positive definite exactly when no nontrivial linear combination of the coordinates is constant (by Theorem 3, parts 1 and 2, a positive semidefinite matrix is positive definite exactly when it is nonsingular).

Proof. (1) A covariance matrix is symmetric because $\operatorname{Cov}(X_j, X_k) = \operatorname{Cov}(X_k, X_j)$, and positive semidefinite because $\mathbf{a}^{\mathsf T}\Sigma\mathbf{a}$ is a variance. Conversely, for a symmetric positive semidefinite $\Sigma$, Theorem 1 of The Multivariate Normal Distribution constructs a normal vector with covariance matrix $\Sigma$, using the square root of Theorem 3 below.

(2) If $\Sigma\mathbf{a} = \mathbf{0}$ for some $\mathbf{a} \ne \mathbf{0}$, then $\operatorname{Var}(\mathbf{a}^{\mathsf T}\mathbf{X}) = 0$, and the nonnegative variable $(\mathbf{a}^{\mathsf T}\mathbf{X} - \mathbf{a}^{\mathsf T}\boldsymbol\mu)^2$ has mean 0, so it is 0 with probability 1 (the lemma on a nonnegative variable with mean zero in Probability Inequalities). Conversely, if $\mathbf{a}^{\mathsf T}\mathbf{X}$ is constant with $\mathbf{a} \ne \mathbf{0}$, then $\mathbf{a}^{\mathsf T}\Sigma\mathbf{a} = 0$. By Theorem 3 below, $\mathbf{a}^{\mathsf T}\Sigma\mathbf{a} = \lVert\Sigma^{1/2}\mathbf{a}\rVert^2$, so $\Sigma^{1/2}\mathbf{a} = \mathbf{0}$ and $\Sigma\mathbf{a} = \Sigma^{1/2}\Sigma^{1/2}\mathbf{a} = \mathbf{0}$. $\square$

Example (pulse pressure). The vector (systolic, diastolic, systolic minus diastolic) has a singular covariance matrix, because the combination with $\mathbf{a} = (1, -1, -1)$ is identically 0. The Multivariate Normal Distribution (Theorem 8) works this example. A variable defined from others adds a row and a column to $\Sigma$ but no new direction of variation.

When every variance is positive, the correlation matrix is $R = D^{-1/2}\Sigma D^{-1/2}$, where $D$ is the diagonal matrix of the variances and $D^{-1/2}$ has the reciprocal standard deviations on its diagonal. It is the covariance matrix of the standardized vector $D^{-1/2}(\mathbf{X} - \boldsymbol\mu)$, so it too is positive semidefinite, with ones on its diagonal.

Example (correlations that cannot coexist). "Any numbers between $-1$ and 1 can serve as correlations." Not jointly. Three variables cannot have all three pairwise correlations equal to $-0.6$: with $\mathbf{a} = (1, 1, 1)$, $\mathbf{a}^{\mathsf T}R\mathbf{a} = 3 + 6 \times (-0.6) = -0.6$, the variance of the sum of the standardized variables, which cannot be negative. Problem 1 finds the smallest common correlation that $p$ variables can have.

The adults. Treat the 5,161 adults as a sample. Their mean vector of (systolic pressure, diastolic pressure, height, weight) is $(126.14, 72.45, 166.50, 82.57)$, in mm Hg, mm Hg, cm and kg. The sample covariance matrix, with divisor $n - 1$,

$$S = \frac{1}{n - 1}\sum_{i=1}^n(\mathbf{x}_i - \bar{\mathbf{x}})(\mathbf{x}_i - \bar{\mathbf{x}})^{\mathsf T},$$

is

$$S = \begin{pmatrix} 393.71 & 106.33 & -8.96 & 55.48 \\ 106.33 & 142.62 & 17.49 & 47.32 \\ -8.96 & 17.49 & 100.80 & 101.30 \\ 55.48 & 47.32 & 101.30 & 524.67\end{pmatrix},$$

and the correlations are 0.449 between the two pressures, 0.440 between height and weight, 0.173 between diastolic pressure and weight, 0.146 between diastolic pressure and height, 0.122 between systolic pressure and weight, and $-0.045$ between systolic pressure and height. $S$ is the covariance matrix of the distribution that puts probability $1/n$ on each adult, times $n/(n - 1)$, so it is positive semidefinite by Theorem 1. Its rank is at most $n - 1$, because the $n$ deviations $\mathbf{x}_i - \bar{\mathbf{x}}$ add up to zero and so span a space of dimension at most $n - 1$: a sample covariance matrix is singular whenever $n \le p$.

The spectral decomposition

A symmetric matrix can be rotated into a diagonal one. This is the one theorem of linear algebra on which the rest of the article stands, and it is cited.

Theorem 2 (the real spectral theorem; cited). For every real symmetric $p \times p$ matrix $\Sigma$ there are an orthogonal matrix $Q$, with columns $\mathbf{q}_1, \ldots, \mathbf{q}_p$ forming an orthonormal basis, and real numbers $\lambda_1, \ldots, \lambda_p$ such that $\Sigma\mathbf{q}_j = \lambda_j\mathbf{q}_j$ for each $j$. Equivalently,

$$\Sigma = Q\Lambda Q^{\mathsf T} = \sum_{j=1}^p\lambda_j\,\mathbf{q}_j\mathbf{q}_j^{\mathsf T}, \qquad \Lambda = \operatorname{diag}(\lambda_1, \ldots, \lambda_p).$$

This is result 7.29 of Axler (2024): an operator on a real inner product space is self-adjoint if and only if the space has an orthonormal basis of its eigenvectors. The two forms agree because $\Sigma Q = Q\Lambda$ column by column, and $Q^{-1} = Q^{\mathsf T}$. We order the eigenvalues $\lambda_1 \ge \cdots \ge \lambda_p$.

Theorem 3 (consequences for covariance matrices). Let $\Sigma = Q\Lambda Q^{\mathsf T}$ be symmetric.

  1. $\Sigma$ is positive semidefinite if and only if every $\lambda_j \ge 0$, and positive definite if and only if every $\lambda_j > 0$.
  2. $\operatorname{tr}\Sigma = \sum_j\lambda_j$ and $\det\Sigma = \prod_j\lambda_j$. The rank of $\Sigma$ is the number of nonzero eigenvalues, and its column space is spanned by the $\mathbf{q}_j$ with $\lambda_j \ne 0$.
  3. Square root. If $\Sigma$ is positive semidefinite, $\Sigma^{1/2} = Q\Lambda^{1/2}Q^{\mathsf T}$, with $\Lambda^{1/2} = \operatorname{diag}(\sqrt{\lambda_1}, \ldots, \sqrt{\lambda_p})$, is symmetric and positive semidefinite, and $\Sigma^{1/2}\Sigma^{1/2} = \Sigma$. Its determinant is $(\det\Sigma)^{1/2}$, and $\mathbf{a}^{\mathsf T}\Sigma\mathbf{a} = \lVert\Sigma^{1/2}\mathbf{a}\rVert^2$. If $\Sigma$ is positive definite, $\Sigma^{1/2}$ is invertible with inverse $\Sigma^{-1/2} = Q\Lambda^{-1/2}Q^{\mathsf T}$, and $\Sigma^{-1/2}\Sigma^{-1/2} = \Sigma^{-1} = Q\Lambda^{-1}Q^{\mathsf T}$.
  4. Extreme variances. Over unit vectors $\mathbf{a}$, $\lambda_p \le \mathbf{a}^{\mathsf T}\Sigma\mathbf{a} \le \lambda_1$, with the maximum at $\mathbf{a} = \mathbf{q}_1$ and the minimum at $\mathbf{q}_p$.

Proof. Write $\mathbf{b} = Q^{\mathsf T}\mathbf{a}$, the coordinates of $\mathbf{a}$ in the basis of eigenvectors. Since $Q$ is orthogonal, $\lVert\mathbf{b}\rVert = \lVert\mathbf{a}\rVert$, and

$$\mathbf{a}^{\mathsf T}\Sigma\mathbf{a} = \mathbf{b}^{\mathsf T}\Lambda\mathbf{b} = \sum_j\lambda_jb_j^2. \qquad (1)$$

(1) If every $\lambda_j \ge 0$, (1) is nonnegative; if every $\lambda_j > 0$, it is positive unless $\mathbf{b} = \mathbf{0}$. Conversely, $\mathbf{a} = \mathbf{q}_j$ gives $\mathbf{q}_j^{\mathsf T}\Sigma\mathbf{q}_j = \lambda_j$, so a negative (or zero) eigenvalue makes the form negative (or zero) at a nonzero vector.

(2) The trace of a product does not depend on the order of the factors, so $\operatorname{tr}(Q\Lambda Q^{\mathsf T}) = \operatorname{tr}(\Lambda Q^{\mathsf T}Q) = \operatorname{tr}\Lambda$. The determinant is multiplicative (Axler, 2024, 9.49), so $\det\Sigma = \det Q\,\det\Lambda\,\det Q^{\mathsf T} = \det\Lambda\,\det(QQ^{\mathsf T}) = \det\Lambda$, and the determinant of a diagonal matrix is the product of its diagonal (9.48). By the sum form, $\Sigma\mathbf{a} = \sum_j\lambda_jb_j\mathbf{q}_j$, which is $\mathbf{0}$ exactly when $b_j = 0$ for every $j$ with $\lambda_j \ne 0$. So the null space is spanned by the $\mathbf{q}_j$ with $\lambda_j = 0$, the column space by the others, and the rank is their number.

(3) $\Sigma^{1/2}$ is symmetric because $(Q\Lambda^{1/2}Q^{\mathsf T})^{\mathsf T} = Q\Lambda^{1/2}Q^{\mathsf T}$, and its eigenvalues $\sqrt{\lambda_j}$ are nonnegative, so it is positive semidefinite by part 1. Then $\Sigma^{1/2}\Sigma^{1/2} = Q\Lambda^{1/2}Q^{\mathsf T}Q\Lambda^{1/2}Q^{\mathsf T} = Q\Lambda Q^{\mathsf T} = \Sigma$. By part 2 its determinant is $\prod_j\sqrt{\lambda_j} = (\det\Sigma)^{1/2}$. Also $\mathbf{a}^{\mathsf T}\Sigma\mathbf{a} = \mathbf{a}^{\mathsf T}\Sigma^{1/2}\Sigma^{1/2}\mathbf{a} = \lVert\Sigma^{1/2}\mathbf{a}\rVert^2$, using symmetry. When every $\lambda_j > 0$, multiplying out shows $Q\Lambda^{-1/2}Q^{\mathsf T}$ is the inverse of $\Sigma^{1/2}$, that $Q\Lambda^{-1}Q^{\mathsf T}$ is the inverse of $\Sigma$, and that the square of the first is the second.

(4) By (1), $\sum_j\lambda_jb_j^2$ with $\sum_jb_j^2 = 1$ is a weighted average of the eigenvalues, so it lies between the smallest and the largest, and it equals $\lambda_1$ at $\mathbf{b} = (1, 0, \ldots, 0)$, that is, $\mathbf{a} = \mathbf{q}_1$. $\square$

The positive semidefinite square root is unique (Axler, 2024, 7.39), but nothing below needs uniqueness: any matrix $A$ with $AA^{\mathsf T} = \Sigma$ serves in the definition of the normal distribution, by Theorem 1 of The Multivariate Normal Distribution. Part 4 is the first step of Principal Component Analysis, which reaches it through a Lagrange multiplier: the linear combination of largest variance is along the first eigenvector, and its variance is the first eigenvalue.

The adults. The eigenvalues of $S$ are 579.35, 403.83, 107.37 and 71.25, adding up to the trace, the total variance 1,161.79; the first carries 49.9% of it. The first eigenvector is $(0.368, 0.193, 0.189, 0.890)$ and the second $(0.869, 0.274, -0.140, -0.389)$: the direction of largest variance is mostly weight, the next mostly systolic pressure. That says more about units than about the body. Weight in kilograms happens to have the largest variance, and the covariance matrix compares variances in mixed units. The correlation matrix puts all four on one scale; its eigenvalues are 1.653, 1.265, 0.607 and 0.475, adding up to 4. Principal Component Analysis discusses that choice.

The Mahalanobis distance

How far is a point from the center of a cloud whose spread differs by direction? Measure along each principal axis in units of the spread along that axis, then combine by Pythagoras.

For a positive definite $\Sigma$ the squared Mahalanobis distance of $\mathbf{x}$ from $\boldsymbol\mu$ is

$$\Delta^2(\mathbf{x}) = (\mathbf{x} - \boldsymbol\mu)^{\mathsf T}\Sigma^{-1}(\mathbf{x} - \boldsymbol\mu) = \lVert\Sigma^{-1/2}(\mathbf{x} - \boldsymbol\mu)\rVert^2 = \sum_{j=1}^p\frac{\big(\mathbf{q}_j^{\mathsf T}(\mathbf{x} - \boldsymbol\mu)\big)^2}{\lambda_j},$$

the second form by Theorem 3, part 3, and the third from $\Sigma^{-1} = \sum_j\lambda_j^{-1}\mathbf{q}_j\mathbf{q}_j^{\mathsf T}$. The third form says it plainly: the deviation is split into its components along the principal axes, each divided by the variance along that axis. A deviation along a direction of small variance counts for much. The name is from Mahalanobis (1936), "On the generalised distance in statistics". The contours $\Delta^2 = c$ are ellipsoids with semi-axes of length $\sqrt{c\lambda_j}$ along the $\mathbf{q}_j$, the contours of the normal density. For one variable $\Delta$ is the absolute $z$-score.

Theorem 4 (invariance). Let $C$ be a nonsingular $p \times p$ matrix, $\mathbf{c}$ a vector, and $\mathbf{Y} = C\mathbf{X} + \mathbf{c}$, with mean $C\boldsymbol\mu + \mathbf{c}$ and covariance $C\Sigma C^{\mathsf T}$. Then the Mahalanobis distance of $C\mathbf{x} + \mathbf{c}$ under the distribution of $\mathbf{Y}$ equals that of $\mathbf{x}$ under the distribution of $\mathbf{X}$.

Proof. The deviation becomes $C(\mathbf{x} - \boldsymbol\mu)$, and $(C\Sigma C^{\mathsf T})^{-1} = (C^{\mathsf T})^{-1}\Sigma^{-1}C^{-1}$, so the form is $(\mathbf{x} - \boldsymbol\mu)^{\mathsf T}C^{\mathsf T}(C^{\mathsf T})^{-1}\Sigma^{-1}C^{-1}C(\mathbf{x} - \boldsymbol\mu) = \Delta^2(\mathbf{x})$. $\square$

Changing units, standardizing the variables, or replacing them by any nonsingular set of linear combinations leaves the distance unchanged. Euclidean distance does not have this property: measuring height in meters instead of centimeters shrinks its contribution to a squared Euclidean distance ten-thousandfold. The script confirms on the adults that the Mahalanobis distances are the same with height in meters.

The same distance appears elsewhere on the site. In Leverage, Influence and Residual Diagnostics the leverage of an observation is $1/n$ plus its squared Mahalanobis distance among the inputs divided by $n - 1$. Detecting Outliers uses it to find unusual combinations, and robust versions of it. The Chi-Square Goodness-of-Fit Test shows that Pearson's statistic is a squared Mahalanobis distance of any $k - 1$ of the counts from their expected values (the full vector of $k$ counts has a singular covariance matrix).

Quadratic forms in normal vectors

We need three things: the mean of a quadratic form for any distribution, the distribution of the Mahalanobis distance for a normal one, and the general condition for a form to be chi-squared.

Theorem 5 (the mean of a quadratic form). If $\mathbf{X}$ has mean $\boldsymbol\mu$ and covariance matrix $\Sigma$, of any distribution, and $A$ is symmetric, then

$$E[\mathbf{X}^{\mathsf T}A\mathbf{X}] = \operatorname{tr}(A\Sigma) + \boldsymbol\mu^{\mathsf T}A\boldsymbol\mu.$$

Proof. A number is its own trace, and the trace of a product does not depend on the order, so $\mathbf{X}^{\mathsf T}A\mathbf{X} = \operatorname{tr}(A\mathbf{X}\mathbf{X}^{\mathsf T})$. The trace is a sum of entries and expectation is linear, so $E[\mathbf{X}^{\mathsf T}A\mathbf{X}] = \operatorname{tr}(A\,E[\mathbf{X}\mathbf{X}^{\mathsf T}])$, and $E[\mathbf{X}\mathbf{X}^{\mathsf T}] = \Sigma + \boldsymbol\mu\boldsymbol\mu^{\mathsf T}$ by expanding $E[(\mathbf{X} - \boldsymbol\mu)(\mathbf{X} - \boldsymbol\mu)^{\mathsf T}]$. Finally $\operatorname{tr}(A\boldsymbol\mu\boldsymbol\mu^{\mathsf T}) = \boldsymbol\mu^{\mathsf T}A\boldsymbol\mu$. $\square$

With $A = \Sigma^{-1}$ and the deviation $\mathbf{X} - \boldsymbol\mu$ in place of $\mathbf{X}$: $E[\Delta^2(\mathbf{X})] = \operatorname{tr}(I_p) = p$ for every distribution with a positive definite covariance matrix.

Theorem 6 (the Mahalanobis distance of a normal vector is chi-squared). If $\mathbf{X} \sim N_p(\boldsymbol\mu, \Sigma)$ with $\Sigma$ positive definite, then $\Delta^2(\mathbf{X}) = (\mathbf{X} - \boldsymbol\mu)^{\mathsf T}\Sigma^{-1}(\mathbf{X} - \boldsymbol\mu)$ has the chi-squared distribution with $p$ degrees of freedom. More generally, $\mathbf{X}^{\mathsf T}\Sigma^{-1}\mathbf{X}$ has the noncentral chi-squared distribution with $p$ degrees of freedom and noncentrality $\boldsymbol\mu^{\mathsf T}\Sigma^{-1}\boldsymbol\mu$.

Proof. $\mathbf{Z} = \Sigma^{-1/2}(\mathbf{X} - \boldsymbol\mu)$ is multivariate normal with mean $\mathbf{0}$ and covariance $\Sigma^{-1/2}\Sigma\Sigma^{-1/2} = I_p$ (The Multivariate Normal Distribution, Theorem 3), so its coordinates are independent standard normal variables (Theorem 5 there). By Theorem 3, part 3, $\Delta^2(\mathbf{X}) = \lVert\mathbf{Z}\rVert^2$, a sum of $p$ independent squared standard normal variables, which is the definition of $\chi^2_p$. For the general statement, $\Sigma^{-1/2}\mathbf{X}$ has independent normal coordinates with variance 1 and mean vector $\Sigma^{-1/2}\boldsymbol\mu$, so its squared length is noncentral chi-squared with noncentrality $\lVert\Sigma^{-1/2}\boldsymbol\mu\rVert^2 = \boldsymbol\mu^{\mathsf T}\Sigma^{-1}\boldsymbol\mu$ (Lemma 1 of Sampling Distributions from the Normal). $\square$

So the ellipsoid $\Delta^2 \le \chi^2_{p,1-\alpha}$, where $\chi^2_{p,1-\alpha}$ is the $1 - \alpha$ quantile of $\chi^2_p$, holds probability $1 - \alpha$, and its complement is the natural outlier rule for normal data. The moments of a chi-squared variable give $E[\Delta^2] = p$, $\operatorname{Var}(\Delta^2) = 2p$ and $E[\Delta^4] = p(p + 2)$ (Sampling Distributions from the Normal, the moments of a chi-squared variable).

Theorem 6 is one case of a general question. When is $\mathbf{X}^{\mathsf T}A\mathbf{X}$ chi-squared? For independent coordinates with a common variance $\sigma^2$, Theorem 2 and Cochran's theorem of Sampling Distributions from the Normal show that $\mathbf{X}^{\mathsf T}A\mathbf{X}$ is chi-squared when $\sigma^2A$ is a projection; Theorem 7 below shows that this is also necessary (for mean 0), and extends the condition to any covariance matrix. With a general $\Sigma$ the idea is to whiten first: $\mathbf{X} = \Sigma^{1/2}\mathbf{Z}$ turns $\mathbf{X}^{\mathsf T}A\mathbf{X}$ into a quadratic form in independent standard normal variables, whose matrix is $\Sigma^{1/2}A\Sigma^{1/2}$, and that matrix must be a projection.

Theorem 7 (when a quadratic form is chi-squared). Let $\mathbf{X} \sim N_p(\mathbf{0}, \Sigma)$, with $\Sigma$ positive semidefinite, possibly singular, and let $A$ be symmetric. Then $\mathbf{X}^{\mathsf T}A\mathbf{X}$ has the chi-squared distribution with $r$ degrees of freedom, for some whole number $r \ge 1$, if and only if

$$\Sigma A\Sigma A\Sigma = \Sigma A\Sigma \quad\text{and}\quad \Sigma A\Sigma \ne 0,$$

and then $r = \operatorname{tr}(A\Sigma)$. (If $\Sigma A\Sigma = 0$, the form is 0 with probability 1.) When $\Sigma$ is nonsingular the condition is $A\Sigma A = A$ with $A \ne 0$, that is, $A\Sigma$ is a nonzero idempotent matrix. If $\Sigma$ is nonsingular, $A\Sigma A = A \ne 0$ and the mean is $\boldsymbol\mu$ instead of $\mathbf{0}$, then $\mathbf{X}^{\mathsf T}A\mathbf{X}$ is noncentral chi-squared with $r$ degrees of freedom and noncentrality $\boldsymbol\mu^{\mathsf T}A\boldsymbol\mu$.

Proof. Reduction. By Theorem 1 of the companion article the distribution of $\mathbf{X}$ is that of $P\mathbf{Z}$, with $P = \Sigma^{1/2}$ and $\mathbf{Z}$ standard normal with $p$ coordinates, so $\mathbf{X}^{\mathsf T}A\mathbf{X}$ has the distribution of $\mathbf{Z}^{\mathsf T}M\mathbf{Z}$ with $M = PAP$, a symmetric matrix. By Theorem 2, $M = U\Gamma U^{\mathsf T}$ with $U$ orthogonal and eigenvalues $\gamma_1, \ldots, \gamma_p$. The vector $\mathbf{V} = U^{\mathsf T}\mathbf{Z}$ is normal with covariance $U^{\mathsf T}U = I$, so its coordinates are independent standard normal variables, and

$$\mathbf{Z}^{\mathsf T}M\mathbf{Z} = \mathbf{V}^{\mathsf T}\Gamma\mathbf{V} = \sum_{j=1}^p\gamma_jV_j^2.$$

Its moment generating function. $V_j^2$ is $\chi^2_1$, whose moment generating function is $(1 - 2s)^{-1/2}$ for $s < \tfrac12$ (it is the gamma distribution with shape $\tfrac12$ and rate $\tfrac12$, whose moment generating function is in Moment Generating and Characteristic Functions). So $E[e^{t\gamma_jV_j^2}] = (1 - 2t\gamma_j)^{-1/2}$ when $2t\gamma_j < 1$, and by independence, for $|t|$ small enough that every $2t\gamma_j < 1$,

$$E\big[e^{t\,\mathbf{Z}^{\mathsf T}M\mathbf{Z}}\big] = \prod_{j=1}^p(1 - 2t\gamma_j)^{-1/2}.$$

If. Suppose $M^2 = M$ and $M \ne 0$. Then each eigenvalue satisfies $\gamma_j^2 = \gamma_j$, so it is 0 or 1, and at least one is 1. The sum $\sum_j\gamma_jV_j^2$ is then the sum of $r \ge 1$ independent squared standard normal variables, $r$ being the number of ones, which is $\operatorname{tr}M$. That is $\chi^2_r$. (If $M = 0$ the form is identically 0.)

Only if. If $\mathbf{Z}^{\mathsf T}M\mathbf{Z}$ is $\chi^2_r$, the two moment generating functions agree near 0: $\prod_j(1 - 2t\gamma_j)^{-1/2} = (1 - 2t)^{-r/2}$. Squaring and inverting, the polynomials $\prod_j(1 - 2t\gamma_j)$ and $(1 - 2t)^r$ agree on an interval, so they are the same polynomial (their difference is a polynomial with infinitely many roots). The second has degree $r$ and the single root $\tfrac12$. The first has degree equal to the number of nonzero $\gamma_j$ and the roots $1/(2\gamma_j)$. So exactly $r$ of the $\gamma_j$ are nonzero and each equals 1: $M$ has eigenvalues 0 and 1 only, and $M^2 = U\Gamma^2U^{\mathsf T} = U\Gamma U^{\mathsf T} = M$.

The condition on $A$. $M^2 - M = P(A\Sigma A - A)P$, since $P^2 = \Sigma$. Write $N = A\Sigma A - A$ and $R = PNP$; we must show that $R = 0$ exactly when $PRP = \Sigma N\Sigma = \Sigma A\Sigma A\Sigma - \Sigma A\Sigma$ is 0. One direction is clear. For the other, suppose $PRP = 0$. For every $\mathbf{w}$, the vector $RP\mathbf{w}$ satisfies $P(RP\mathbf{w}) = \mathbf{0}$, and it equals $P(NP^2\mathbf{w})$, so it lies both in the null space of $P$ and in the column space of $P$. These are perpendicular for a symmetric $P$: if $P\mathbf{v} = \mathbf{0}$ then $(P\mathbf{u})^{\mathsf T}\mathbf{v} = \mathbf{u}^{\mathsf T}P\mathbf{v} = 0$. So $RP\mathbf{w} = \mathbf{0}$ for all $\mathbf{w}$, that is, $RP = 0$, and then $PR = (RP)^{\mathsf T} = 0$. Now every column of $R$ is in the null space of $P$, and also in its column space, since $R = P(NP)$. So $R = 0$. In the same way $M = P(AP)$ is 0 exactly when $PMP = \Sigma A\Sigma$ is 0 (the argument above with $M$ in place of $R$), so $M \ne 0$ exactly when $\Sigma A\Sigma \ne 0$. When $\Sigma$ is nonsingular, so is $P$, and $PNP = 0$ is the same as $N = 0$, that is, $A\Sigma A = A$. Finally, $r = \operatorname{tr}M = \operatorname{tr}(PAP) = \operatorname{tr}(AP^2) = \operatorname{tr}(A\Sigma)$.

The noncentral case. With $\Sigma$ nonsingular, $\mathbf{Y} = \Sigma^{-1/2}\mathbf{X}$ has independent coordinates with variance 1 and mean vector $\mathbf{m} = \Sigma^{-1/2}\boldsymbol\mu$, and $\mathbf{X}^{\mathsf T}A\mathbf{X} = \mathbf{Y}^{\mathsf T}M\mathbf{Y}$ with $M$ a projection of rank $r$. By Theorem 2 of Sampling Distributions from the Normal it is $\chi^2_r(\lVert M\mathbf{m}\rVert^2)$, and $\lVert M\mathbf{m}\rVert^2 = \mathbf{m}^{\mathsf T}M\mathbf{m} = \boldsymbol\mu^{\mathsf T}\Sigma^{-1/2}\Sigma^{1/2}A\Sigma^{1/2}\Sigma^{-1/2}\boldsymbol\mu = \boldsymbol\mu^{\mathsf T}A\boldsymbol\mu$. $\square$

Four cases show the range of the theorem.

  • The Mahalanobis distance. $A = \Sigma^{-1}$ gives $A\Sigma A = \Sigma^{-1} = A$ and $r = \operatorname{tr}I_p = p$: Theorem 6 again.
  • The sample variance. For $\mathbf{X} \sim N_n(\mu\mathbf{1}, \sigma^2I_n)$ and $A = (I - \tfrac1nJ)/\sigma^2$, with $J$ the matrix of ones, $A\Sigma A = (I - \tfrac1nJ)^2/\sigma^2 = A$ and $r = n - 1$. The mean vector $\mu\mathbf{1}$ gives noncentrality $\mu^2\mathbf{1}^{\mathsf T}(I - \tfrac1nJ)\mathbf{1}/\sigma^2 = 0$, so $(n - 1)s^2/\sigma^2$ is $\chi^2_{n-1}$, the fact proved on The Chi-Squared and t Distributions.
  • A singular covariance matrix. In The Chi-Square Goodness-of-Fit Test, the scaled deviations of multinomial counts tend to a normal vector with covariance $P = I - \mathbf{u}\mathbf{u}^{\mathsf T}$, a projection of rank $k - 1$, and Pearson's statistic is their squared length, $A = I$. The condition reads $PPP = P^3 = P = P^2$, true, with $r = \operatorname{tr}P = k - 1$.
  • A form that fails. For two correlated standard normal variables and $A = I$, the condition fails and the sum of squares is not chi-squared. Problem 3 works out how far it is from $\chi^2_2$, for two blood pressure readings of the same person.

The second question is when two forms, or a form and a linear function, are independent. The answer is a covariance condition.

Theorem 8 (independence of linear and quadratic forms). Let $\mathbf{X} \sim N_p(\boldsymbol\mu, \Sigma)$, let $A$ and $B$ be symmetric and $L$ any $q \times p$ matrix.

  1. If $L\Sigma A = 0$, then $L\mathbf{X}$ and $\mathbf{X}^{\mathsf T}A\mathbf{X}$ are independent.
  2. If $A\Sigma B = 0$, then $\mathbf{X}^{\mathsf T}A\mathbf{X}$ and $\mathbf{X}^{\mathsf T}B\mathbf{X}$ are independent.
  3. Conversely, if $\boldsymbol\mu = \mathbf{0}$, $\Sigma$ is nonsingular and $\mathbf{X}^{\mathsf T}A\mathbf{X}$ and $\mathbf{X}^{\mathsf T}B\mathbf{X}$ are independent, then $A\Sigma B = 0$.

The idea of parts 1 and 2: a quadratic form $\mathbf{X}^{\mathsf T}A\mathbf{X}$ is a function of the vector $A\mathbf{X}$, and two jointly normal vectors with zero cross-covariance are independent.

Proof. Write $A = \sum_j\alpha_j\mathbf{u}_j\mathbf{u}_j^{\mathsf T}$ by Theorem 2 and let $A^+ = \sum_{\alpha_j \ne 0}\alpha_j^{-1}\mathbf{u}_j\mathbf{u}_j^{\mathsf T}$. Then $AA^+ = \sum_{\alpha_j \ne 0}\mathbf{u}_j\mathbf{u}_j^{\mathsf T}$, and $AA^+A = \sum_{\alpha_j \ne 0}\alpha_j\mathbf{u}_j\mathbf{u}_j^{\mathsf T} = A$. Hence

$$\mathbf{X}^{\mathsf T}A\mathbf{X} = \mathbf{X}^{\mathsf T}AA^+A\mathbf{X} = (A\mathbf{X})^{\mathsf T}A^+(A\mathbf{X}),$$

a function of $A\mathbf{X}$.

(1) The stacked vector $(L\mathbf{X}, A\mathbf{X})$ is a linear image of $\mathbf{X}$, hence multivariate normal, with cross-covariance $L\Sigma A^{\mathsf T} = L\Sigma A = 0$. By Theorem 5 of The Multivariate Normal Distribution, $L\mathbf{X}$ and $A\mathbf{X}$ are independent, and so are $L\mathbf{X}$ and any function of $A\mathbf{X}$.

(2) The same argument with $(A\mathbf{X}, B\mathbf{X})$, whose cross-covariance is $A\Sigma B$, shows that $A\mathbf{X}$ and $B\mathbf{X}$ are independent, hence so are functions of them.

(3) Let $\tilde{A} = \Sigma^{1/2}A\Sigma^{1/2}$ and $\tilde{B} = \Sigma^{1/2}B\Sigma^{1/2}$, so that, as in Theorem 7, the pair of forms has the distribution of $(\mathbf{Z}^{\mathsf T}\tilde{A}\mathbf{Z}, \mathbf{Z}^{\mathsf T}\tilde{B}\mathbf{Z})$. For numbers $s$ and $t$ let $C = s\tilde{A} + t\tilde{B}$, with eigenvalues $c_j$. The proof of Theorem 7 gives $E[e^{\mathbf{Z}^{\mathsf T}C\mathbf{Z}}] = \prod_j(1 - 2c_j)^{-1/2} = \det(I - 2C)^{-1/2}$ when every $c_j < \tfrac12$; the determinant is the product of the eigenvalues $1 - 2c_j$ of $I - 2C$ by Theorem 3, part 2. Each $c_j$ equals $\mathbf{q}^{\mathsf T}C\mathbf{q}$ for a unit eigenvector $\mathbf{q}$, so by Theorem 3, part 4, each $|c_j|$ is at most $|s|$ times the largest $|\text{eigenvalue}|$ of $\tilde{A}$ plus $|t|$ times that of $\tilde{B}$, so the condition holds for $|s|, |t|$ below some $\delta > 0$. Independence makes the expected value of the product $e^{s\,\mathbf{Z}^{\mathsf T}\tilde{A}\mathbf{Z}}e^{t\,\mathbf{Z}^{\mathsf T}\tilde{B}\mathbf{Z}}$ the product of the expected values (Expectation, Variance and Covariance, Theorem 5), so for $|s|, |t| < \delta$

$$\det(I - 2s\tilde{A} - 2t\tilde{B}) = \det(I - 2s\tilde{A})\,\det(I - 2t\tilde{B}).$$

Both sides are polynomials in $s$ and $t$. For each fixed $t$ with $|t| < \delta$ they agree for all $|s| < \delta$, hence as polynomials in $s$; so each coefficient, a polynomial in $t$, agrees on $|t| < \delta$ and hence everywhere. So the identity holds for all $s$ and $t$. The Craig–Sakamoto theorem, as Olkin (1997) states it, says that for symmetric matrices this identity for all $s$ and $t$ holds if and only if $\tilde{A}\tilde{B} = 0$. That is $\Sigma^{1/2}A\Sigma B\Sigma^{1/2} = 0$, and multiplying by $\Sigma^{-1/2}$ on both sides gives $A\Sigma B = 0$. $\square$

Parts 1 and 2 are proved in full; part 3 rests on the cited matrix theorem. The subject goes back at least to Craig (1943), "Note on the independence of certain quadratic forms", whose name the matrix theorem carries; we have not read that note.

The sample mean and the sample covariance matrix

Back to samples. Let $\mathbf{x}_1, \ldots, \mathbf{x}_n$ be independent random vectors with a common distribution, mean $\boldsymbol\mu$ and covariance matrix $\Sigma$, with sample mean $\bar{\mathbf{x}}$ and sample covariance matrix $S$ as above. The first result needs no normality.

Theorem 9 (unbiasedness). $E[\bar{\mathbf{x}}] = \boldsymbol\mu$, $\operatorname{Cov}(\bar{\mathbf{x}}) = \Sigma/n$ and $E[S] = \Sigma$.

Proof. The first is linearity. For the second, $\operatorname{Cov}(\sum_i\mathbf{x}_i) = \sum_i\sum_l\operatorname{Cov}(\mathbf{x}_i, \mathbf{x}_l) = n\Sigma$, since the cross-covariances of independent vectors are 0; dividing the sum by $n$ divides the covariance by $n^2$. For the third, expand around $\boldsymbol\mu$:

$$\sum_{i=1}^n(\mathbf{x}_i - \bar{\mathbf{x}})(\mathbf{x}_i - \bar{\mathbf{x}})^{\mathsf T} = \sum_{i=1}^n(\mathbf{x}_i - \boldsymbol\mu)(\mathbf{x}_i - \boldsymbol\mu)^{\mathsf T} - n(\bar{\mathbf{x}} - \boldsymbol\mu)(\bar{\mathbf{x}} - \boldsymbol\mu)^{\mathsf T},$$

because the cross terms are $-\sum_i(\mathbf{x}_i - \boldsymbol\mu)(\bar{\mathbf{x}} - \boldsymbol\mu)^{\mathsf T} = -n(\bar{\mathbf{x}} - \boldsymbol\mu)(\bar{\mathbf{x}} - \boldsymbol\mu)^{\mathsf T}$ and its transpose. The expected value is $n\Sigma - n(\Sigma/n) = (n - 1)\Sigma$. $\square$

For normal samples much more is true, and one rotation proves it, the multivariate form of the rotation on The Chi-Squared and t Distributions. Define the Wishart distribution $W_p(\Sigma, m)$ as the distribution of $\sum_{k=1}^m\mathbf{y}_k\mathbf{y}_k^{\mathsf T}$, where $\mathbf{y}_1, \ldots, \mathbf{y}_m$ are independent $N_p(\mathbf{0}, \Sigma)$ vectors. For $p = 1$ it is $\sigma^2$ times $\chi^2_m$.

Theorem 10 (normal samples). If the $\mathbf{x}_i$ are independent $N_p(\boldsymbol\mu, \Sigma)$ vectors, then

  1. $\bar{\mathbf{x}} \sim N_p(\boldsymbol\mu, \Sigma/n)$;
  2. $\bar{\mathbf{x}}$ and $S$ are independent;
  3. $(n - 1)S$ has the Wishart distribution $W_p(\Sigma, n - 1)$: it equals $\sum_{k=2}^n\mathbf{y}_k\mathbf{y}_k^{\mathsf T}$ for independent $N_p(\mathbf{0}, \Sigma)$ vectors $\mathbf{y}_2, \ldots, \mathbf{y}_n$ that are also independent of $\bar{\mathbf{x}}$.

Here is the plan. Take an orthogonal $n \times n$ matrix $H$ whose first row is $(1, \ldots, 1)/\sqrt{n}$, and mix the observations with it. The first mixture is $\sqrt{n}\,\bar{\mathbf{x}}$, the others have mean zero, the mixtures are independent because $H$ is orthogonal, and the total sum of squares and products is unchanged.

Proof. Such an $H$ exists: complete the unit vector $(1, \ldots, 1)/\sqrt{n}$ to an orthonormal basis by the Gram–Schmidt process, as in Sampling Distributions from the Normal (proof of Lemma 1). Let $\mathbf{y}_k = \sum_i h_{ki}\mathbf{x}_i$ for $k = 1, \ldots, n$.

Joint normality. The $np$ numbers in $\mathbf{x}_1, \ldots, \mathbf{x}_n$ form one multivariate normal vector, since stacks of independent normal vectors are normal (Theorem 3, part 2, of The Multivariate Normal Distribution, applied $n - 1$ times). The $\mathbf{y}_k$ together are a linear image of that vector, so they are jointly multivariate normal.

Means and covariances. $E[\mathbf{y}_k] = (\sum_ih_{ki})\boldsymbol\mu$. For $k = 1$ this is $\sqrt{n}\,\boldsymbol\mu$. For $k \ge 2$, $\sum_ih_{ki} = \sqrt{n}\,\sum_ih_{ki}h_{1i} = 0$ because rows 1 and $k$ are perpendicular, so the mean is $\mathbf{0}$. Since different observations are independent,

$$\operatorname{Cov}(\mathbf{y}_k, \mathbf{y}_l) = \sum_i\sum_{i'} h_{ki}h_{li'}\operatorname{Cov}(\mathbf{x}_i, \mathbf{x}_{i'}) = \Big(\sum_ih_{ki}h_{li}\Big)\Sigma,$$

which is $\Sigma$ for $k = l$ and 0 for $k \ne l$, by the orthogonality of the rows. By Theorem 5 of The Multivariate Normal Distribution, applied to the $n$ blocks (its proof with Lemma 1 extended to $n$ blocks, as noted after that lemma), $\mathbf{y}_1, \ldots, \mathbf{y}_n$ are independent, $\mathbf{y}_1 \sim N_p(\sqrt{n}\boldsymbol\mu, \Sigma)$ and $\mathbf{y}_k \sim N_p(\mathbf{0}, \Sigma)$ for $k \ge 2$.

The sums of squares and products. Since the columns of $H$ are also orthonormal ($H^{\mathsf T}H = I$),

$$\sum_{k=1}^n\mathbf{y}_k\mathbf{y}_k^{\mathsf T} = \sum_i\sum_{i'}\Big(\sum_kh_{ki}h_{ki'}\Big)\mathbf{x}_i\mathbf{x}_{i'}^{\mathsf T} = \sum_{i=1}^n\mathbf{x}_i\mathbf{x}_i^{\mathsf T}.$$

Also $\mathbf{y}_1 = \sqrt{n}\,\bar{\mathbf{x}}$, and the identity of Theorem 9 with $\boldsymbol\mu$ replaced by $\mathbf{0}$ gives $(n - 1)S = \sum_i\mathbf{x}_i\mathbf{x}_i^{\mathsf T} - n\bar{\mathbf{x}}\bar{\mathbf{x}}^{\mathsf T}$. Hence $(n - 1)S = \sum_{k=2}^n\mathbf{y}_k\mathbf{y}_k^{\mathsf T}$, a function of $\mathbf{y}_2, \ldots, \mathbf{y}_n$ alone, while $\bar{\mathbf{x}} = \mathbf{y}_1/\sqrt{n}$. All three statements follow. $\square$

The Wishart distribution inherits simple properties from its definition.

Lemma (fourth moments of a normal vector). If $\mathbf{Y} \sim N_p(\mathbf{0}, \Sigma)$, then for any indices $j, k, l, h$

$$E[Y_jY_kY_lY_h] = \sigma_{jk}\sigma_{lh} + \sigma_{jl}\sigma_{kh} + \sigma_{jh}\sigma_{kl}.$$

Proof. Write $\mathbf{Y} = A\mathbf{Z}$ with $AA^{\mathsf T} = \Sigma$ and $\mathbf{Z}$ standard normal (Theorem 1 of the companion article). For standard normal coordinates, $E[Z_aZ_bZ_cZ_d]$ is $E[Z^4] = 3$ if all four indices are equal, $E[Z_a^2]E[Z_c^2] = 1$ if they form two different pairs, and 0 otherwise, because an index appearing once or three times contributes an odd moment, which is 0. In every case this is $\delta_{ab}\delta_{cd} + \delta_{ac}\delta_{bd} + \delta_{ad}\delta_{bc}$, where $\delta_{ab}$ is 1 if $a = b$ and 0 otherwise. Expanding $Y_j = \sum_aA_{ja}Z_a$ and summing,

$$E[Y_jY_kY_lY_h] = \sum_{a,b,c,d}A_{ja}A_{kb}A_{lc}A_{hd}\,(\delta_{ab}\delta_{cd} + \delta_{ac}\delta_{bd} + \delta_{ad}\delta_{bc}) = (AA^{\mathsf T})_{jk}(AA^{\mathsf T})_{lh} + (AA^{\mathsf T})_{jl}(AA^{\mathsf T})_{kh} + (AA^{\mathsf T})_{jh}(AA^{\mathsf T})_{kl}. \ \square$$

$E[Z^4] = 3$ is derived on The Chi-Squared and t Distributions, in the variance of a chi-squared variable. The formula is a case of the product-moment formula of Isserlis (1918), whose title announces a formula for product moments of any order of a normal distribution in any number of variables.

Theorem 11 (properties of the Wishart distribution). Let $W \sim W_p(\Sigma, m)$.

  1. $E[W] = m\Sigma$.
  2. For any fixed $\mathbf{a}$ with $\mathbf{a}^{\mathsf T}\Sigma\mathbf{a} > 0$, $\mathbf{a}^{\mathsf T}W\mathbf{a}/\mathbf{a}^{\mathsf T}\Sigma\mathbf{a}$ is $\chi^2_m$.
  3. For a $q \times p$ matrix $C$, $CWC^{\mathsf T} \sim W_q(C\Sigma C^{\mathsf T}, m)$; the sum of independent $W_p(\Sigma, m_1)$ and $W_p(\Sigma, m_2)$ matrices is $W_p(\Sigma, m_1 + m_2)$.
  4. $W$ has rank at most $m$, so it is singular when $m < p$.
  5. $\operatorname{Cov}(W_{jk}, W_{lh}) = m(\sigma_{jl}\sigma_{kh} + \sigma_{jh}\sigma_{kl})$. In particular, for a normal sample, $\operatorname{Var}(S_{jk}) = (\sigma_{jk}^2 + \sigma_{jj}\sigma_{kk})/(n - 1)$ and $\operatorname{Var}(S_{jj}) = 2\sigma_{jj}^2/(n - 1)$.

Proof. Let $W = \sum_k\mathbf{y}_k\mathbf{y}_k^{\mathsf T}$ as in the definition. (1) $E[\mathbf{y}_k\mathbf{y}_k^{\mathsf T}] = \Sigma$. (2) $\mathbf{a}^{\mathsf T}W\mathbf{a} = \sum_k(\mathbf{a}^{\mathsf T}\mathbf{y}_k)^2$, and the $\mathbf{a}^{\mathsf T}\mathbf{y}_k$ are independent normal with mean 0 and variance $\mathbf{a}^{\mathsf T}\Sigma\mathbf{a}$. (3) $CWC^{\mathsf T} = \sum_k(C\mathbf{y}_k)(C\mathbf{y}_k)^{\mathsf T}$ with $C\mathbf{y}_k \sim N_q(\mathbf{0}, C\Sigma C^{\mathsf T})$ independent; two independent sums of independent outer products of $N_p(\mathbf{0}, \Sigma)$ vectors together form one such sum with $m_1 + m_2$ terms. (4) $W$ is a sum of $m$ matrices of rank at most 1, and its column space lies in the span of $\mathbf{y}_1, \ldots, \mathbf{y}_m$. (5) The terms are independent, so the covariance is $m$ times $\operatorname{Cov}(Y_jY_k, Y_lY_h) = E[Y_jY_kY_lY_h] - \sigma_{jk}\sigma_{lh}$, which by the lemma is $\sigma_{jl}\sigma_{kh} + \sigma_{jh}\sigma_{kl}$. For the sample, $S = W/(n - 1)$ with $m = n - 1$ by Theorem 10, so $\operatorname{Var}(S_{jk}) = (n - 1)(\sigma_{jk}^2 + \sigma_{jj}\sigma_{kk})/(n - 1)^2$. $\square$

Part 4 is the rank bound of the first section, now for the population. When $m \ge p$ and $\Sigma$ is nonsingular, $W$ has a density on the positive definite matrices, which we do not derive. Wishart (1928) is the paper on this distribution, as its title, "The generalised product moment distribution in samples from a normal multivariate population", shows; we have read only the title. Bishop (2006, Appendix B, eq. B.78) lists a density under the name Wishart, as a prior distribution for the inverse of a covariance matrix, with mean $\nu W$ for $\nu$ degrees of freedom and scale matrix $W$ (eq. B.80). Its mean and its one-dimensional case, the gamma distribution with shape $\nu/2$ and rate $1/(2W)$, which is $W$ times $\chi^2_\nu$, agree with parts 1 and 2 when $W = \Sigma$ and $\nu = m$. That Bishop's density is the density of the sum of outer products defined here rests on Wishart's title and on these two checks; neither source that we read proves it. Hotelling's T-Squared Test uses Theorem 10 to test a mean vector.

What normality adds

Theorem 9 holds for any population; Theorems 10 and 11 need normality. How much do they change for real measurements? Draw samples of $n = 10$ adults, with replacement, from the 5,161 adults, so that the observations are independent draws from the population of adults (covariances with divisor $N$ in this section). Repeat 10,000 times for each of 30 seeds, and do the same with samples from the normal distribution with the same mean vector and covariance matrix. The tables give medians over the 30 seeds, with interquartile ranges.

Weight has population variance 524.57, skewness 1.08 and kurtosis 4.94 (a normal variable has 0 and 3). The covariance of systolic and diastolic pressure is 106.31. For a sample of 10:

Quantity Normal theory Any population (exact) Samples of adults Normal samples
Mean of $S_{ww}$ (weight variance) 524.57 524.57 524.4 (522.8 to 526.6) 524.7 (524.1 to 525.9)
Mean of $S_{12}$ (pressure covariance) 106.31 106.31 106.3 (105.6 to 107.0) 106.5 (105.9 to 107.0)
Variance of $S_{ww}$ 61,149 114,548 115,627 (113,168 to 117,107) 60,582 (60,173 to 61,585)
Variance of $S_{12}$ 7,492 10,806 (10,629 to 10,985) 7,509 (7,419 to 7,586)
Correlation of $\bar{x}_w$ and $S_{ww}$ 0 0.528 0.530 (0.527 to 0.535) -0.002 (-0.009 to 0.005)

The normal-theory column comes from Theorems 10 and 11: $2\sigma^4/9$ for the variance of a sample variance, $(\sigma_{12}^2 + \sigma_{11}\sigma_{22})/9$ for a sample covariance, and independence. The exact column comes from Theorem 9 and from Theorem 6 of Sampling Distributions from the Normal, which gives $\operatorname{Var}(s^2) = (\mu_4 - \tfrac{n-3}{n-1}\sigma^4)/n$ and $\operatorname{Cov}(\bar{X}, s^2) = \mu_3/n$ for any population with four moments; it has no entry for the covariance, whose general formula this article does not derive. Three observations, all of this experiment:

  • Unbiasedness survives, as Theorem 9 says it must: the averages of $S$ match the population values for the adults as well as for normal samples.
  • The spread of $S$ does not. The variance of the weight variance is 1.9 times the normal-theory value, as the exact formula predicts from the kurtosis, and the variance of the sample covariance of the two pressures is larger by about 44%.
  • Independence fails. In samples of adults the sample mean weight and the sample variance of weight have correlation 0.53, because heavy outliers raise both at once; in normal samples they are uncorrelated, as Theorem 10 says.

Intervals and tests built on Theorems 10 and 11 inherit these errors; Problem 4 measures the effect on an interval for a variance.

Back to the adults

Return to the 5,161 adults and their four measurements. With the sample mean and $S$ in place of $\boldsymbol\mu$ and $\Sigma$, each adult has a squared distance $d_i^2 = (\mathbf{x}_i - \bar{\mathbf{x}})^{\mathsf T}S^{-1}(\mathbf{x}_i - \bar{\mathbf{x}})$.

The average is fixed. Whatever the data, the average of the $d_i^2$ is exactly $p(n - 1)/n$. By the trace argument of Theorem 5, $\sum_id_i^2 = \operatorname{tr}\big(S^{-1}\sum_i(\mathbf{x}_i - \bar{\mathbf{x}})(\mathbf{x}_i - \bar{\mathbf{x}})^{\mathsf T}\big) = \operatorname{tr}\big(S^{-1}(n - 1)S\big) = (n - 1)p$. Here it is $4 \times 5{,}160/5{,}161 = 3.9992$. A check of fit must look at the shape of the distances, not their mean.

The reference distribution. For a normal population and known parameters, Theorem 6 makes the distances $\chi^2_4$. With estimated parameters this holds approximately in large samples: by the weak law of large numbers (The Laws of Large Numbers) $\bar{\mathbf{x}}$ and every entry of $S$ converge in probability to their population values, and $d_i^2$ is a continuous function of $\mathbf{x}_i$, $\bar{\mathbf{x}}$ and $S$. This is an argument, not a proof here: the continuous mapping and Slutsky theorems of Convergence in Distribution are proved there for real variables, and their extension to vectors is not. The simulation below measures the approximation directly, with 30 samples of 5,161 from the normal distribution with the adults' mean vector and $S$ (seeds 1 to 30), each with its own estimated mean and covariance.

The quantiles of $\chi^2_4$ at 0.5, 0.9, 0.975 and 0.999 are 3.357, 7.779, 11.143 and 18.467. The shares of adults beyond them:

Beyond the quantile 0.5 0.9 0.975 0.999
Expected share, $\chi^2_4$ 50% 10% 2.5% 0.1%
Normal samples, median (IQR) 50.0 (49.9 to 50.3) 10.0 (9.8 to 10.1) 2.5 (2.4 to 2.6) 0.10 (0.08 to 0.14)
Adults 44.2 10.2 4.5 1.07
Adults, logarithms of systolic pressure and weight 48.0 10.2 3.2 0.48
Sorted squared Mahalanobis distances of the 5,161 adults against the quantiles of the chi-squared distribution with 4 degrees of freedom, for the raw measurements and with logarithms of systolic pressure and weight, with the band that holds the middle 95% of 30 normal samples of the same size, at every fifth rank. Notice that both curves sit slightly below the band through the bulk (too many small distances) and cross above it between squared distances of 8 and 9, the raw measurements rising much farther above it than the transformed ones.
Sorted squared Mahalanobis distances of the 5,161 adults against the quantiles of the chi-squared distribution with 4 degrees of freedom, for the raw measurements and with logarithms of systolic pressure and weight, with the band that holds the middle 95% of 30 normal samples of the same size, at every fifth rank. Notice that both curves sit slightly below the band through the bulk (too many small distances) and cross above it between squared distances of 8 and 9, the raw measurements rising much farther above it than the transformed ones.

The fourth moment tells the same story in one number. Under normality $E[\Delta^4] = p(p + 2) = 24$, and the normal samples give an average $d_i^4$ of 24.00 (23.88 to 24.06). The adults give 30.10, and 26.28 after the logarithms. The adults have too many small distances and too many large ones: 55 of them, 1.07%, lie beyond the 0.999 quantile, nearly eleven times the normal share. The logarithms help because systolic pressure and weight are right-skewed (skewness 0.99 and 1.08, becoming 0.50 and 0.22), but the tail stays heavier than normal. Mardia (1970) developed measures of multivariate skewness and kurtosis and a test of multivariate normality built on them (by the paper's abstract); the average of $d_i^4$ computed here is in the same spirit.

Who is far out. The largest distance, 49.6, belongs to an adult weighing 219.6 kg, $z = 5.98$ on weight alone. The third and fourth largest belong to adults with recorded diastolic pressures of 12 and 8 mm Hg, far below every other reading; distances flag them whatever their cause, and only a look at the records can say whether they are errors. In these four measurements every adult beyond the 0.999 contour is at least 2 standard deviations from the mean on at least one measurement by itself: 0 adults within 2 standard deviations on all four are beyond it. Combinations show at the 0.975 contour, where 12 adults within 2 standard deviations on every measurement lie beyond. The adult of the opening is the farthest of them, with $d^2 = 15.91$: systolic 162.0 ($z = 1.81$), diastolic 54.0 ($z = -1.55$), height 181.4 ($z = 1.48$) and weight 123.4 ($z = 1.78$). High systolic with low diastolic pressure runs against their positive correlation, and that is what the distance counts. Detecting Outliers takes the practical side further, with robust estimates of the mean and covariance that outliers cannot distort.

What is proved here, what is cited, and what the experiments show

Proved here. Theorem 1 (covariance matrices are exactly the symmetric positive semidefinite matrices; singularity and degenerate combinations); Theorem 3 (eigenvalues and definiteness, trace and determinant, rank, square roots, extreme variances), given the spectral theorem; Theorem 4 (invariance of the Mahalanobis distance); Theorem 5 (the mean of a quadratic form, for any distribution); Theorem 6 (the Mahalanobis distance is chi-squared); Theorem 7 (the necessary and sufficient condition for a quadratic form in a normal vector to be chi-squared, for any covariance matrix, and the noncentral case); parts 1 and 2 of Theorem 8 (independence of a linear and a quadratic form, and of two quadratic forms), and part 3 given the Craig–Sakamoto theorem; Theorem 9 (unbiasedness of the sample mean and covariance matrix); Theorem 10 (normal samples: the sample mean and the sample covariance matrix are independent, and the scaled sample covariance matrix is Wishart); the lemma on fourth moments; Theorem 11 (mean, quadratic forms, transformations, rank and covariances of a Wishart matrix); the average squared distance $p(n - 1)/n$.

Cited. The real spectral theorem (Axler, 2024, 7.29) and the determinant facts 9.48 and 9.49 there; the Craig–Sakamoto theorem, as stated in the abstract of Olkin (1997); the Wishart density (Wishart, 1928; Bishop, 2006, eq. B.78). From the site: the projection theorem and Cochran's theorem for independent coordinates (Sampling Distributions from the Normal), and the results on the multivariate normal distribution in The Multivariate Normal Distribution, with the uniqueness theorem for characteristic functions that they cite. The large-sample chi-squared approximation for distances with estimated parameters is argued, not proved.

What the experiments show. On four NHANES measurements, Mahalanobis distances are close to chi-squared in magnitude in the bulk but systematically smaller (the raw curve lies below the band of normal samples at 76% of the plotted ranks), and far too frequent in the upper tail, nearly eleven times the normal share beyond the 0.999 quantile; logarithms of the two skewed measurements cut that excess by more than half without removing it. In samples of ten from the adults, the sample covariance matrix is unbiased as Theorem 9 requires, but its entries vary up to twice as much as normal theory says, and the sample mean and sample variance of weight are strongly correlated. These are observations about this population and these sample sizes.

Worked problems

These problems were written for this article; none comes from a textbook. They use the NHANES measurements and small exact cases, and every number is computed and asserted by the script named at the end.

Problem 1 (the smallest common correlation). Which numbers $\rho$ can be the common correlation of $p$ variables, all pairs having correlation $\rho$? Check the opening example of three correlations of $-0.6$.

Solution. The correlation matrix is $R = (1 - \rho)I + \rho J$, with $J = \mathbf{1}\mathbf{1}^{\mathsf T}$ the matrix of ones. $J\mathbf{1} = p\mathbf{1}$, so $R\mathbf{1} = (1 - \rho + p\rho)\mathbf{1}$: the vector of ones is an eigenvector with eigenvalue $1 + (p - 1)\rho$. Any $\mathbf{v}$ perpendicular to $\mathbf{1}$ has $J\mathbf{v} = \mathbf{1}(\mathbf{1}^{\mathsf T}\mathbf{v}) = \mathbf{0}$, so $R\mathbf{v} = (1 - \rho)\mathbf{v}$: the $p - 1$ dimensions perpendicular to $\mathbf{1}$ are eigenvectors with eigenvalue $1 - \rho$. By Theorem 3, part 1, $R$ is a correlation matrix exactly when both eigenvalues are nonnegative, that is,

$$-\frac{1}{p - 1} \le \rho \le 1.$$

Every $\rho$ in that range is realized, by a normal vector (Theorem 1). For $p = 3$ the bound is $-\tfrac12$. With $\rho = -0.6$ the eigenvalue along $\mathbf{1}$ is $1 + 2(-0.6) = -0.2$, negative, and the vector $\mathbf{1}$ gives the negative variance $-0.6 = 3 \times (-0.2)$ found in the opening example. Many variables can all be positively correlated, but they cannot all be strongly negatively correlated: as $p$ grows, the lower bound $-1/(p - 1)$ rises toward 0.

Problem 2 (a Mahalanobis distance by hand). For the 5,232 adults of The Multivariate Normal Distribution, with mean $(126.31, 72.49)$ and covariance entries $\sigma_{11} = 402.65$, $\sigma_{12} = 109.05$, $\sigma_{22} = 144.14$, find the squared Mahalanobis distance of an adult with systolic pressure 150 and diastolic pressure 60. Compare it with the sum of the squared $z$-scores, and with the share of adults at least as far out.

Solution. The deviations are $\delta_1 = 23.69$ and $\delta_2 = -12.49$, so the $z$-scores are 1.18 and $-1.04$, and the sum of their squares is 2.48. The determinant of the covariance matrix is $402.65 \times 144.14 - 109.05^2 = 46{,}144.2$, and with the $2 \times 2$ inverse of the companion article,

$$d^2 = \frac{\sigma_{22}\delta_1^2 - 2\sigma_{12}\delta_1\delta_2 + \sigma_{11}\delta_2^2}{\sigma_{11}\sigma_{22} - \sigma_{12}^2} = \frac{144.14(23.69)^2 + 2(109.05)(23.69)(12.49) + 402.65(12.49)^2}{46{,}144.2} = 4.51,$$

computed from unrounded values. The middle term is added, not subtracted, because the deviations have opposite signs while the correlation is positive: the combination runs against the correlation, and the distance exceeds the sum of squared $z$-scores. Under a bivariate normal model $d^2$ is $\chi^2_2$, whose tail is $e^{-d^2/2}$, so the share of adults at least this far out would be $e^{-4.51/2} = 10.48\%$. In the data it is 9.86%.

Problem 3 (a sum of squares that is not chi-squared). A person's first two systolic readings, standardized, have correlation $\rho = 0.966$ among the 4,730 adults with three readings. Treat the standardized pair $\mathbf{Z}$ as bivariate normal with this correlation. Is $Z_1^2 + Z_2^2$ chi-squared with 2 degrees of freedom? Find its mean, variance and the probability that it exceeds the $\chi^2_2$ 0.95 quantile, 5.991.

Solution. Here $\Sigma = R = \begin{pmatrix} 1 & \rho \\ \rho & 1\end{pmatrix}$ and $A = I$. Theorem 7 asks whether $RIRIR = RIR$, that is, $R^3 = R^2$; since $R$ is nonsingular this means $R = I$, false for $\rho \ne 0$. So the form is not chi-squared. To find its distribution, follow the proof: $R$ has eigenvectors $(1, 1)/\sqrt{2}$ and $(1, -1)/\sqrt{2}$ with eigenvalues $1 + \rho = 1.966$ and $1 - \rho = 0.034$, so

$$Z_1^2 + Z_2^2 \ \text{has the distribution of} \ 1.966\,V_1^2 + 0.034\,V_2^2$$

with $V_1, V_2$ independent standard normal. The mean is $1.966 + 0.034 = 2$, the same as for $\chi^2_2$ (Theorem 5: $\operatorname{tr}R = 2$). The variance is $2(1.966^2 + 0.034^2) = 4(1 + \rho^2) = 7.734$, against 4 for $\chi^2_2$. Nearly all of the sum is one squared normal variable doubled, so it is far more spread out. The probability that it exceeds 5.991, by numerical integration over $V_2^2$, is 8.2% instead of 5%. The correct form is $\mathbf{Z}^{\mathsf T}R^{-1}\mathbf{Z}$, which is $\chi^2_2$ by Theorem 6. In the data the shares above 5.991 are 6.3% for the sum of squares and 6.1% for the correct form; the readings are not exactly normal, and the gap between the two forms is smaller than the normal model predicts.

Problem 4 (an interval for the variance of pulse pressure). Ten adults drawn at random from the 5,161 (numpy default_rng(2026), without replacement) have pulse pressures $89\tfrac13$, $90\tfrac23$, $66\tfrac23$, 52, 36, $41\tfrac13$, $57\tfrac13$, $28\tfrac23$, 36 and $33\tfrac13$ mm Hg (each pressure is a mean of up to three readings; all the adults' pressures are whole numbers of thirds). Assuming normality, give a 95% confidence interval for the variance of pulse pressure, and find how often such intervals cover the population variance when samples of 10 are drawn from the adults.

Solution. Pulse pressure is $\mathbf{a}^{\mathsf T}\mathbf{x}$ with $\mathbf{a} = (1, -1, 0, 0)$. If the measurements are normal, Theorem 10 and Theorem 11, part 2, make $9\,\mathbf{a}^{\mathsf T}S\mathbf{a}/\mathbf{a}^{\mathsf T}\Sigma\mathbf{a}$ a $\chi^2_9$ variable, where $\mathbf{a}^{\mathsf T}S\mathbf{a}$ is the sample variance of the ten pulse pressures, 515.36. With the $\chi^2_9$ quantiles 2.7004 and 19.0228,

$$P\Big(2.7004 \le \frac{9 \times \text{sample variance}}{\sigma^2_{\text{pp}}} \le 19.0228\Big) = 0.95 \quad\Longrightarrow\quad \Big(\frac{9 \times 515.36}{19.0228},\ \frac{9 \times 515.36}{2.7004}\Big) = (243.8,\ 1{,}717.6).$$

It contains the variance of pulse pressure among the 5,161 adults, 323.61 (divisor $N$; it differs slightly from the 328.68 of the companion article, which uses the 5,232 adults with both pressures). The interval is wide, because nine degrees of freedom pin a variance down poorly. How reliable is the procedure? Over 30 seeds of 10,000 samples of 10 drawn with replacement from the adults, the interval covered the population variance in a median of 86.3% (interquartile range 86.1% to 86.5%) of samples; for normal samples with the same variance the median was 95.0% (94.9% to 95.1%). Pulse pressure has skewness 1.24 and kurtosis 5.12, and as in the section on what normality adds, the chi-squared distribution of Theorem 11 understates how much a sample variance varies. The Delta Method finds the same effect in the limit for systolic pressure.

Summary

  • Covariance matrices are exactly the symmetric positive semidefinite matrices; $\Sigma$ is singular exactly when some nontrivial linear combination is constant (Theorem 1). Not every matrix of correlations between $-1$ and 1 is a correlation matrix.
  • $\Sigma = Q\Lambda Q^{\mathsf T}$ (cited); definiteness is read from the eigenvalues, $\operatorname{tr}\Sigma = \sum\lambda_j$, $\det\Sigma = \prod\lambda_j$, $\Sigma^{1/2} = Q\Lambda^{1/2}Q^{\mathsf T}$, and the largest variance of a unit combination is $\lambda_1$ (Theorem 3).
  • The Mahalanobis distance $\Delta^2 = (\mathbf{x} - \boldsymbol\mu)^{\mathsf T}\Sigma^{-1}(\mathbf{x} - \boldsymbol\mu)$ is unchanged by any nonsingular affine recoding (Theorem 4), has mean $p$ for any distribution (Theorem 5), and is $\chi^2_p$ for a normal vector (Theorem 6).
  • $\mathbf{X}^{\mathsf T}A\mathbf{X}$ with $\mathbf{X} \sim N_p(\mathbf{0}, \Sigma)$ is chi-squared exactly when $\Sigma A\Sigma A\Sigma = \Sigma A\Sigma \ne 0$ ($A\Sigma A = A \ne 0$ for nonsingular $\Sigma$), with $\operatorname{tr}(A\Sigma)$ degrees of freedom (Theorem 7).
  • $L\mathbf{X}$ and $\mathbf{X}^{\mathsf T}A\mathbf{X}$ are independent when $L\Sigma A = 0$; two forms are independent when $A\Sigma B = 0$, and for nonsingular $\Sigma$ and zero mean only then (Theorem 8).
  • $E[S] = \Sigma$ for any population (Theorem 9); for normal samples $\bar{\mathbf{x}}$ and $S$ are independent and $(n - 1)S \sim W_p(\Sigma, n - 1)$ (Theorem 10), with $\operatorname{Var}(S_{jk}) = (\sigma_{jk}^2 + \sigma_{jj}\sigma_{kk})/(n - 1)$ (Theorem 11).
  • On NHANES measurements the distances are slightly too small in the bulk and far too large in the tail, and normal-theory variances of $S$ are too small by up to half.

What we have learned

Covariance matrices are positive semidefinite, and the spectral theorem turns every question about them into a question about their eigenvalues. Quadratic forms in normal vectors reduce, after whitening, to sums of weighted squared standard normal variables, and they are chi-squared exactly when the weights are all 0 or 1. Their independence reduces to a covariance condition. Applied to a sample, the same tools prove that the sample mean and sample covariance matrix of normal data are independent and that the covariance matrix is Wishart, which are the facts that multivariate tests rest on. On real measurements the theory needs care: the sample covariance matrix stays unbiased, but its variability and the outlier cutoffs depend on tails that normal data do not have.

Hotelling's T-Squared Test uses Theorem 10, and the representation of $(n - 1)S$ as a sum of outer products, to test a mean vector, the multivariate form of the $t$-test.

References