Hotelling's T-Squared Test

The t test for a mean vector. Hotelling's statistic is defined as a squared Mahalanobis distance of the sample mean from the hypothesized mean; its exact F distribution under normality, noncentral under the alternative, is proved without Wishart theory by a rotation and a residual sum of squares; the statistic is proved invariant under every nonsingular linear change of variables, equal to the largest squared t statistic over all linear combinations (which gives simultaneous confidence intervals for every combination), and a monotone function of the likelihood ratio. The two-sample version follows from the same lemma. Simulations and NHANES blood pressures show when the test is more powerful than separate t tests with a Bonferroni correction (effects that run against the correlation, or several moderate effects) and when it is less (one effect among many variables, or an effect along the correlation). Worked problems on a four-point sample in exact fractions, the sample size for comparing two age groups, and an interval for the difference between systolic and diastolic pressure.

Hypothesis Tests for a Mean drew 25 adults aged 60 and over from the 5,247 NHANES adults (National Center for Health Statistics, 2020) and asked whether their mean systolic pressure differs from the all-adult mean of 126.36 mm Hg. It found $t = 1.820$ and $p = 0.081$ (0.0813 to four places, recomputed here): not significant at the 5% level, although the true mean of the 1,923 older adults is 136.85. The same 25 adults also have diastolic pressures. Their mean diastolic pressure is 68.51 against an all-adult mean of 72.49, and its $t$ test gives $t = -2.329$, $p = 0.0286$. Two tests on one sample need a correction, and the Bonferroni correction tests each at 0.025, so neither rejects. Is there a single test of the hypothesis that both means equal the all-adult values, and what does it conclude?

There is. Hotelling's $T^2$ measures the distance of the sample mean vector from the hypothesized one in the metric of the sample covariance matrix. For the 25 adults it gives $T^2 = 9.240$, which corresponds to $F = 4.427$ on 2 and 23 degrees of freedom and $p = 0.0236$: the joint hypothesis is rejected at the 5% level. Hotelling (1931) published it under the title "The generalization of Student's ratio" (we have read only the title), and with one variable it is indeed the square of Student's $t$.

The route is as follows. First we define the statistic and compute it for the 25 adults. Then we prove its exact distribution under normality, with a lemma that later serves the two-sample test as well. Next come three properties, each proved: invariance under linear changes of variables, the identity with the largest $t$ statistic over all linear combinations (which gives simultaneous intervals), and the identity with the likelihood ratio test. Then the two-sample version. Finally we compare the test with separate $t$ tests and a Bonferroni correction, by exact power calculations, by simulation and on NHANES samples, to see when each is the better choice. The article assumes The Multivariate Normal Distribution and Covariance Matrices and Quadratic Forms, whose notation it uses: $N_p(\boldsymbol\mu, \Sigma)$ is the $p$-variate normal with mean vector $\boldsymbol\mu$ and covariance matrix $\Sigma$. With $p = 1$ the second argument is then the variance, while the site's one-dimensional $N(\mu, \sigma)$ has the standard deviation second.

The statistic

Let $\mathbf{x}_1, \ldots, \mathbf{x}_n$ be independent observations of a $p$-dimensional random vector with mean vector $\boldsymbol\mu$ and covariance matrix $\Sigma$. Write the sample mean and sample covariance matrix as

$$\bar{\mathbf{x}} = \frac1n\sum_{i=1}^n \mathbf{x}_i, \qquad S = \frac{1}{n-1}\sum_{i=1}^n (\mathbf{x}_i - \bar{\mathbf{x}})(\mathbf{x}_i - \bar{\mathbf{x}})^{\mathsf T},$$

and let $A = (n-1)S$, the matrix of sums of squares and products. The hypothesis is $H_0\colon \boldsymbol\mu = \boldsymbol\mu_0$ for a given vector $\boldsymbol\mu_0$. Hotelling's statistic is

$$T^2 = n\,(\bar{\mathbf{x}} - \boldsymbol\mu_0)^{\mathsf T} S^{-1} (\bar{\mathbf{x}} - \boldsymbol\mu_0),$$

defined when $S$ is invertible, which (as shown below) requires $n > p$.

Here is why this is the right size of $\bar{\mathbf{x}} - \boldsymbol\mu_0$. Its covariance matrix is $\Sigma/n$, so with $\Sigma$ known the natural size would be the squared Mahalanobis distance $n(\bar{\mathbf{x}} - \boldsymbol\mu_0)^{\mathsf T}\Sigma^{-1}(\bar{\mathbf{x}} - \boldsymbol\mu_0)$. That distance is chi-squared with $p$ degrees of freedom under $H_0$ and normality (Covariance Matrices and Quadratic Forms, Theorem 6). It weighs each direction by the inverse of the variance in that direction. A deviation along a direction in which the data scatter little counts for more than the same deviation along a direction in which they scatter widely. $T^2$ replaces $\Sigma$ by its estimate $S$, as Student's $t$ replaces $\sigma$ by $s$. With $p = 1$, $S = s^2$ and $T^2 = n(\bar{x} - \mu_0)^2/s^2 = t^2$.

The running sample

The 25 adults were drawn with replacement from the 1,923 adults aged 60 and over, with the course script's seed, so they are the sample of Hypothesis Tests for a Mean. For each adult the two variables are the mean systolic and the mean diastolic pressure over the readings taken (mm Hg; a diastolic reading of 0 counts as missing, as on the course pages). The hypothesized vector is the pair of all-adult means: 126.36 for systolic pressure over the 5,247 adults, and 72.49 for diastolic pressure over the 5,232 of them with a diastolic value. In the population of older adults the true means are 136.85 and 70.91, so $H_0$ is false in both coordinates, in opposite directions.

The sample gives

$$\bar{\mathbf{x}} = \begin{pmatrix} 133.19 \\ 68.51\end{pmatrix}, \qquad S = \begin{pmatrix} 351.78 & 9.00 \\ 9.00 & 73.05\end{pmatrix},$$

with a sample correlation of only 0.056 between the two pressures in these 25 adults. The deviation is $\bar{\mathbf{x}} - \boldsymbol\mu_0 = (6.83, -3.98)$. Because the correlation is small, $S^{-1}$ is nearly diagonal and $T^2$ is close to the sum of the two squared $t$ statistics, $1.820^2 + 2.329^2 = 8.735$. The exact value, from the full matrix, is $T^2 = 9.240$.

The exact distribution

How large must $T^2$ be to reject? Under normality its distribution is a multiple of an $F$ distribution, and the proof needs only the univariate theory of the site plus one rotation.

Here is the plan. The sample mean and the matrix $A$ are independent, and $A$ is a sum of $n - 1$ outer products of independent normal vectors with mean zero (Lemma 1). For any fixed vector, the quadratic form in $A^{-1}$ can be rotated so that the vector points along the first axis. Its value is then the vector's squared length divided by a single number, the residual sum of squares of a regression of one column of normal data on the others (Lemma 3). That number is chi-squared, and the ratio is $F$.

Lemma 1 (the sample mean and the sums of squares). Let $\mathbf{x}_1, \ldots, \mathbf{x}_n$ be independent with distribution $N_p(\boldsymbol\mu, \Sigma)$. Then $\sqrt{n}\,\bar{\mathbf{x}}$ has the distribution $N_p(\sqrt{n}\,\boldsymbol\mu, \Sigma)$, it is independent of $A$, and $A$ has the distribution of $\sum_{k=2}^n\mathbf{y}_k\mathbf{y}_k^{\mathsf T}$ for independent $\mathbf{y}_2, \ldots, \mathbf{y}_n$ distributed as $N_p(\mathbf{0}, \Sigma)$.

This is Theorem 10 of Covariance Matrices and Quadratic Forms, proved there by rotating the sample with Helmert's orthogonal matrix, whose first row is $(1, \ldots, 1)/\sqrt{n}$, as The Chi-Squared and t Distributions does for one variable: the rotated vectors $\mathbf{y}_k = \sum_iH_{ki}\mathbf{x}_i$ are independent, $\mathbf{y}_1 = \sqrt{n}\,\bar{\mathbf{x}}$, and the others have mean zero and add up, as outer products, to $A$. That article calls the distribution of such a sum of $m$ outer products the Wishart distribution $W_p(\Sigma, m)$, so that $A \sim W_p(\Sigma, n - 1)$, and cites Wishart (1928) for its density. The proof below never uses the density; it uses only the representation as a sum.

Lemma 2 (when the sum is invertible). Let $\mathbf{z}_1, \ldots, \mathbf{z}_m$ be independent $N_p(\mathbf{0}, \Sigma)$ with $\Sigma$ positive definite, and $B = \sum_k \mathbf{z}_k\mathbf{z}_k^{\mathsf T}$. If $m \ge p$, then $B$ is positive definite with probability 1. If $m < p$, it is singular.

Proof. Let $Z$ be the $m \times p$ matrix with rows $\mathbf{z}_k^{\mathsf T}$, so that $B = Z^{\mathsf T}Z$ and $\mathbf{v}^{\mathsf T}B\mathbf{v} = \lVert Z\mathbf{v}\rVert^2$ for every vector $\mathbf{v}$. Hence $B$ is positive definite exactly when $Z\mathbf{v} \ne \mathbf{0}$ for all $\mathbf{v} \ne \mathbf{0}$, that is, when the $p$ columns of $Z$ are linearly independent. That is impossible when $m < p$. For $m \ge p$, write $\mathbf{z}_k = \Sigma^{1/2}\mathbf{u}_k$ with independent standard normal $\mathbf{u}_k$, so $Z = U\Sigma^{1/2}$, which has independent columns exactly when $U$ does. The columns $\mathbf{c}_1, \ldots, \mathbf{c}_p$ of $U$ are independent $N_m(\mathbf{0}, I_m)$ vectors. Given $\mathbf{c}_1, \ldots, \mathbf{c}_{j-1}$, their span is a subspace of dimension at most $j - 1 < m$. A subspace of lower dimension has volume zero in $\mathbb{R}^m$, so the vector $\mathbf{c}_j$, which has a density, lies in it with probability 0. So with probability 1 no column lies in the span of the earlier ones. $\square$

So $S$ is invertible with probability 1 exactly when $n - 1 \ge p$, which is the condition $n > p$ stated with the definition.

Lemma 3 (one diagonal entry of an inverse). Let $B$ be a positive definite $p \times p$ matrix, partitioned with a scalar $b_{11}$ in the corner, a row $\mathbf{b}_{12}^{\mathsf T}$ beside it, and the lower right block $B_{22}$ of size $p - 1$. Then the first diagonal entry of $B^{-1}$ is

$$(B^{-1})_{11} = \frac{1}{b_{11} - \mathbf{b}_{12}^{\mathsf T}B_{22}^{-1}\mathbf{b}_{12}},$$

and the denominator is positive. (For $p = 1$ it is $1/b_{11}$.)

This is the case of a one-dimensional first block in Theorem 7 of The Multivariate Normal Distribution, which proves for any positive definite matrix that the upper-left block of the inverse is the inverse of the Schur complement. The denominator is positive because it is $1/(B^{-1})_{11}$ and $(B^{-1})_{11} = \mathbf{e}_1^{\mathsf T}B^{-1}\mathbf{e}_1 > 0$, $B^{-1}$ being positive definite.

The denominator is the conditional variance of one coordinate given the others in that article. Here it has a second reading. If $B = Z^{\mathsf T}Z$ and $Z = [\mathbf{y}\ X]$ with first column $\mathbf{y}$ and the remaining columns $X$, then $b_{11} = \mathbf{y}^{\mathsf T}\mathbf{y}$, $\mathbf{b}_{12} = X^{\mathsf T}\mathbf{y}$ and $B_{22} = X^{\mathsf T}X$. The denominator is $\mathbf{y}^{\mathsf T}\mathbf{y} - \mathbf{y}^{\mathsf T}X(X^{\mathsf T}X)^{-1}X^{\mathsf T}\mathbf{y} = \lVert(I - H_X)\mathbf{y}\rVert^2$, with $H_X$ the hat matrix of $X$. That is the residual sum of squares of the least-squares regression of $\mathbf{y}$ on the columns of $X$, without an intercept (The Geometry of Least Squares).

Theorem 1 (the distribution of a quadratic form in an inverse sum of squares). Let $\mathbf{d}$ have the distribution $N_p(\boldsymbol\nu, \Sigma)$ with $\Sigma$ positive definite, and let it be independent of $B = \sum_{k=1}^m \mathbf{z}_k\mathbf{z}_k^{\mathsf T}$, where $\mathbf{z}_1, \ldots, \mathbf{z}_m$ are independent $N_p(\mathbf{0}, \Sigma)$ and $m \ge p$. Then

$$\frac{m - p + 1}{p}\;\mathbf{d}^{\mathsf T}B^{-1}\mathbf{d}$$

has the noncentral $F$ distribution with $p$ and $m - p + 1$ degrees of freedom and noncentrality $\lambda = \boldsymbol\nu^{\mathsf T}\Sigma^{-1}\boldsymbol\nu$. When $\boldsymbol\nu = \mathbf{0}$ it is the central $F$ distribution.

Proof. Reduction to $\Sigma = I$. Put $\mathbf{u} = \Sigma^{-1/2}\mathbf{d}$ and $\mathbf{w}_k = \Sigma^{-1/2}\mathbf{z}_k$, with the symmetric square root of Covariance Matrices and Quadratic Forms. Then $\mathbf{u}$ has the distribution $N_p(\Sigma^{-1/2}\boldsymbol\nu, I_p)$, the $\mathbf{w}_k$ are independent $N_p(\mathbf{0}, I_p)$ and independent of $\mathbf{u}$, and $C = \sum_k\mathbf{w}_k\mathbf{w}_k^{\mathsf T} = \Sigma^{-1/2}B\Sigma^{-1/2}$. Since $B^{-1} = \Sigma^{-1/2}C^{-1}\Sigma^{-1/2}$, we have $\mathbf{d}^{\mathsf T}B^{-1}\mathbf{d} = \mathbf{u}^{\mathsf T}C^{-1}\mathbf{u}$. $C$ is invertible with probability 1 by Lemma 2.

Rotating $\mathbf{u}$ onto the first axis. With probability 1, $\mathbf{u} \ne \mathbf{0}$. For each nonzero vector $\mathbf{u}$ choose an orthogonal matrix $Q_{\mathbf u}$ with $Q_{\mathbf u}\mathbf{u} = \lVert\mathbf{u}\rVert\mathbf{e}_1$; the reflection $I - 2\mathbf{h}\mathbf{h}^{\mathsf T}/\lVert\mathbf{h}\rVert^2$ with $\mathbf{h} = \mathbf{u} - \lVert\mathbf{u}\rVert\mathbf{e}_1$ is one (and $Q_{\mathbf u} = I$ when $\mathbf{u}$ already points along $\mathbf{e}_1$). Then

$$\mathbf{u}^{\mathsf T}C^{-1}\mathbf{u} = (Q_{\mathbf u}\mathbf{u})^{\mathsf T}(Q_{\mathbf u}CQ_{\mathbf u}^{\mathsf T})^{-1}(Q_{\mathbf u}\mathbf{u}) = \lVert\mathbf{u}\rVert^2\,\big[(Q_{\mathbf u}CQ_{\mathbf u}^{\mathsf T})^{-1}\big]_{11}.$$

Now condition on $\mathbf{u}$, and let $\mathbf{w}_k' = Q_{\mathbf u}\mathbf{w}_k$. The $\mathbf{w}_k$ are independent of $\mathbf{u}$, so given $\mathbf{u}$ they are still independent $N_p(\mathbf{0}, I_p)$, and $Q_{\mathbf u}$ is a fixed orthogonal matrix. A fixed orthogonal map sends $N_p(\mathbf{0}, I_p)$ to $N_p(\mathbf{0}, Q_{\mathbf u}Q_{\mathbf u}^{\mathsf T}) = N_p(\mathbf{0}, I_p)$, so given $\mathbf{u}$ the $\mathbf{w}_k'$ are independent $N_p(\mathbf{0}, I_p)$, whatever the value of $\mathbf{u}$. Also $Q_{\mathbf u}CQ_{\mathbf u}^{\mathsf T} = \sum_k\mathbf{w}_k'\mathbf{w}_k'^{\mathsf T}$.

The diagonal entry is the reciprocal of a chi-squared variable. Let $W$ be the $m \times p$ matrix with rows $\mathbf{w}_k'^{\mathsf T}$, first column $\mathbf{y}$ and remaining columns $X$. By Lemma 3 and the remark after it, $1/[(Q_{\mathbf u}CQ_{\mathbf u}^{\mathsf T})^{-1}]_{11} = R$, the residual sum of squares of $\mathbf{y}$ on $X$ (and $R = \mathbf{y}^{\mathsf T}\mathbf{y}$ when $p = 1$). Given $\mathbf{u}$, the entries of $W$ are independent standard normal variables. So given $\mathbf{u}$ and $X$, the vector $\mathbf{y}$ is $N_m(\mathbf{0}, I_m)$ and $X$ has rank $p - 1 < m$ with probability 1 (Lemma 2 applied to the $m$ rows of $X$, which are independent $N_{p-1}(\mathbf{0}, I_{p-1})$ vectors). This is the normal linear model with design $X$, coefficients $\mathbf{0}$ and error variance 1, and by The Normal Linear Model (Theorem 1, part 2) $R$ is chi-squared with $m - (p - 1) = m - p + 1$ degrees of freedom. For $p = 1$, $R = \mathbf{y}^{\mathsf T}\mathbf{y}$ is chi-squared with $m$ degrees of freedom directly. The conditional distribution of $R$ is the same whatever $\mathbf{u}$ and $X$ are, so $R$ is chi-squared with $m - p + 1$ degrees of freedom and independent of $\mathbf{u}$.

The ratio. We have $\mathbf{d}^{\mathsf T}B^{-1}\mathbf{d} = \lVert\mathbf{u}\rVert^2/R$. The numerator is a sum of squares of $p$ independent normal variables with variance 1 and mean vector $\Sigma^{-1/2}\boldsymbol\nu$, so it has the noncentral chi-squared distribution with $p$ degrees of freedom and noncentrality $\lVert\Sigma^{-1/2}\boldsymbol\nu\rVert^2 = \boldsymbol\nu^{\mathsf T}\Sigma^{-1}\boldsymbol\nu$ (Sampling Distributions from the Normal, Lemma 1). It is independent of the denominator. Hence

$$\frac{m - p + 1}{p}\,\mathbf{d}^{\mathsf T}B^{-1}\mathbf{d} = \frac{\lVert\mathbf{u}\rVert^2/p}{R/(m - p + 1)}$$

is a noncentral $F$ variable as stated, by the definition in Sampling Distributions from the Normal. $\square$

The proof shows where the degrees of freedom go. Of the $m$ independent normal vectors that make up $B$, the quadratic form "uses up" $p - 1$ dimensions on the directions perpendicular to $\mathbf{d}$, the columns of $X$ in the proof, and the remaining $m - p + 1$ enter the denominator. In Wishart language the theorem rests on the fact that $\mathbf{d}^{\mathsf T}\Sigma^{-1}\mathbf{d}/\mathbf{d}^{\mathsf T}B^{-1}\mathbf{d}$ is chi-squared with $m - p + 1$ degrees of freedom independently of $\mathbf{d}$, which is exactly $R$ above.

Corollary 1 (the one-sample test). If $\mathbf{x}_1, \ldots, \mathbf{x}_n$ are independent $N_p(\boldsymbol\mu, \Sigma)$ with $\Sigma$ positive definite and $n > p$, then

$$F = \frac{n - p}{p(n - 1)}\,T^2$$

has the noncentral $F$ distribution with $p$ and $n - p$ degrees of freedom and noncentrality $\lambda = n(\boldsymbol\mu - \boldsymbol\mu_0)^{\mathsf T}\Sigma^{-1}(\boldsymbol\mu - \boldsymbol\mu_0)$. Under $H_0$, and only then, $\lambda = 0$.

Proof. By Lemma 1, $\mathbf{d} = \sqrt{n}(\bar{\mathbf{x}} - \boldsymbol\mu_0)$ has the distribution $N_p(\sqrt{n}(\boldsymbol\mu - \boldsymbol\mu_0), \Sigma)$ and is independent of $A$, which is distributed as a sum of $m = n - 1$ outer products of independent $N_p(\mathbf{0}, \Sigma)$ vectors. Theorem 1 concerns only the joint distribution of $\mathbf{d}$ and the sum, so it applies. Since $S^{-1} = (n-1)A^{-1}$, $T^2 = (n - 1)\,\mathbf{d}^{\mathsf T}A^{-1}\mathbf{d}$. Theorem 1 with $m = n - 1$ gives the distribution of $\frac{n - p}{p}\mathbf{d}^{\mathsf T}A^{-1}\mathbf{d} = \frac{n - p}{p(n-1)}T^2$, with $\boldsymbol\nu = \sqrt{n}(\boldsymbol\mu - \boldsymbol\mu_0)$. As $\Sigma^{-1}$ is positive definite, $\lambda = 0$ exactly when $\boldsymbol\mu = \boldsymbol\mu_0$. $\square$

The test rejects $H_0$ at level $\alpha$ when $T^2 > \frac{p(n-1)}{n-p}F_{p,\,n-p,\,1-\alpha}$, where $F_{p,\,n-p,\,1-\alpha}$ is the upper $\alpha$ point of the central $F$ distribution. The NIST/SEMATECH handbook (§6.5.4.3) gives the same scaling, $T^2 \sim \frac{p(n-1)}{n-p}F_{p,\,n-p}$. With $p = 1$ it reads $T^2 = t^2 \sim F_{1,\,n-1}$, the square of Student's theorem.

The running sample. With $n = 25$ and $p = 2$, $F = \frac{23}{2 \times 24}\times 9.240 = 4.427$. The upper 5% point of $F_{2,\,23}$ is 3.4221, so the critical value of $T^2$ is $\frac{2 \times 24}{23}\times 3.4221 = 7.142$, and $T^2 = 9.240$ exceeds it; $p = 0.0236$. Each coordinate alone missed: the Bonferroni critical value for two $t$ tests at total level 0.05 is $t_{24,\,0.9875} = 2.3909$, above both $|t| = 1.820$ and $|t| = 2.329$. Holm's procedure (Multiple Comparisons and the Bonferroni Correction) compares the smaller p-value, 0.0286, with 0.025 first and stops there.

The 25 older adults: the 95% confidence ellipse for their mean systolic and diastolic pressures (solid), the box of the two Bonferroni intervals (dashed) and the box of the two T-squared simultaneous intervals (dotted). Notice that the all-adult means (square) lie inside both boxes but outside the ellipse: the joint test rejects while neither coordinate does. The ellipse's axes are nearly parallel to the coordinate axes because the two pressures are almost uncorrelated in this sample.
The 25 older adults: the 95% confidence ellipse for their mean systolic and diastolic pressures (solid), the box of the two Bonferroni intervals (dashed) and the box of the two T-squared simultaneous intervals (dotted). Notice that the all-adult means (square) lie inside both boxes but outside the ellipse: the joint test rejects while neither coordinate does. The ellipse's axes are nearly parallel to the coordinate axes because the two pressures are almost uncorrelated in this sample.

The figure shows the reason. The test accepts $\boldsymbol\mu_0$ when it lies inside the ellipse $\{\mathbf{m} : n(\bar{\mathbf{x}} - \mathbf{m})^{\mathsf T}S^{-1}(\bar{\mathbf{x}} - \mathbf{m}) \le 7.142\}$, the 95% confidence region for $\boldsymbol\mu$. The Bonferroni procedure accepts when $\boldsymbol\mu_0$ lies inside the box of the two intervals at level 0.975 each. The box has corners that the ellipse cuts off, and $\boldsymbol\mu_0$ sits in one of them: both coordinates deviate moderately, neither enough alone, and the deviations add.

Invariance under linear maps

Blood pressure could be recorded in kilopascals instead of mm Hg, or as the difference and a weighted average of the two readings instead of the readings themselves. A test of "the mean vector equals $\boldsymbol\mu_0$" should not depend on such a choice, because the hypothesis does not.

Theorem 2 (invariance). Let $C$ be a nonsingular $p \times p$ matrix and $\mathbf{b}$ a vector, and replace each observation by $\mathbf{x}_i' = C\mathbf{x}_i + \mathbf{b}$ and the hypothesized mean by $\boldsymbol\mu_0' = C\boldsymbol\mu_0 + \mathbf{b}$. Then $T^2$ is unchanged.

Proof. The new sample mean is $\bar{\mathbf{x}}' = C\bar{\mathbf{x}} + \mathbf{b}$, so $\bar{\mathbf{x}}' - \boldsymbol\mu_0' = C(\bar{\mathbf{x}} - \boldsymbol\mu_0)$, and the new deviations are $\mathbf{x}_i' - \bar{\mathbf{x}}' = C(\mathbf{x}_i - \bar{\mathbf{x}})$, so $S' = CSC^{\mathsf T}$. Since $C$ is invertible, $(S')^{-1} = (C^{\mathsf T})^{-1}S^{-1}C^{-1}$, and

$$T'^2 = n(\bar{\mathbf{x}} - \boldsymbol\mu_0)^{\mathsf T}C^{\mathsf T}(C^{\mathsf T})^{-1}S^{-1}C^{-1}C(\bar{\mathbf{x}} - \boldsymbol\mu_0) = T^2. \qquad\square$$

The hypothesis is also unchanged: $\boldsymbol\mu = \boldsymbol\mu_0$ exactly when $C\boldsymbol\mu + \mathbf{b} = \boldsymbol\mu_0'$. So any two analysts who record the same information in different linear coordinates reach the same $T^2$ and the same p-value.

Separate $t$ tests do not have this property. Each $t$ statistic is unchanged by rescaling its own variable, but a mixing map changes the variables being tested. Take the 25 adults and record, instead of the two pressures, their difference (systolic minus diastolic) and the weighted average (systolic plus twice diastolic)/3. This is the map $C = \begin{pmatrix} 1 & -1 \\ 1/3 & 2/3\end{pmatrix}$, which is nonsingular. The script confirms that $T^2 = 9.240$ again. But the two new $t$ statistics are 2.679 for the difference (sample mean 64.68 against the hypothesized 53.87) and $-0.217$ for the weighted average (90.07 against 90.44). Now the first exceeds the Bonferroni value 2.3909, and the Bonferroni procedure rejects. The same data and the same hypothesis give a Bonferroni rejection in one set of coordinates and none in the other. $T^2$ gives one answer.

The largest t statistic

Why should a test of a vector reject when no coordinate does? Because the coordinates are only $p$ of the infinitely many linear combinations of the variables, and the hypothesis $\boldsymbol\mu = \boldsymbol\mu_0$ says that every combination has its hypothesized mean. $T^2$ is the most extreme of all their $t$ statistics.

Theorem 3 (the largest t). For a nonzero vector $\mathbf{a}$, the $t$ statistic for the hypothesis $\mathbf{a}^{\mathsf T}\boldsymbol\mu = \mathbf{a}^{\mathsf T}\boldsymbol\mu_0$, computed from the values $\mathbf{a}^{\mathsf T}\mathbf{x}_i$, is

$$t_{\mathbf a} = \frac{\sqrt{n}\,\mathbf{a}^{\mathsf T}(\bar{\mathbf{x}} - \boldsymbol\mu_0)}{\sqrt{\mathbf{a}^{\mathsf T}S\mathbf{a}}}, \qquad\text{and}\qquad \max_{\mathbf{a} \ne \mathbf{0}}\, t_{\mathbf a}^2 = T^2,$$

attained at $\mathbf{a} = S^{-1}(\bar{\mathbf{x}} - \boldsymbol\mu_0)$ and its nonzero multiples (when $\bar{\mathbf{x}} \ne \boldsymbol\mu_0$).

Proof. The values $\mathbf{a}^{\mathsf T}\mathbf{x}_i$ have sample mean $\mathbf{a}^{\mathsf T}\bar{\mathbf{x}}$ and sample variance $\frac{1}{n-1}\sum_i(\mathbf{a}^{\mathsf T}\mathbf{x}_i - \mathbf{a}^{\mathsf T}\bar{\mathbf{x}})^2 = \mathbf{a}^{\mathsf T}S\mathbf{a}$, which gives the formula for $t_{\mathbf a}$. Write $\mathbf{v} = \bar{\mathbf{x}} - \boldsymbol\mu_0$ and use the symmetric square roots of the positive definite $S$: $\mathbf{a}^{\mathsf T}\mathbf{v} = (S^{1/2}\mathbf{a})^{\mathsf T}(S^{-1/2}\mathbf{v})$. The Cauchy–Schwarz inequality for vectors (Probability Inequalities) gives

$$(\mathbf{a}^{\mathsf T}\mathbf{v})^2 \le \lVert S^{1/2}\mathbf{a}\rVert^2\,\lVert S^{-1/2}\mathbf{v}\rVert^2 = (\mathbf{a}^{\mathsf T}S\mathbf{a})(\mathbf{v}^{\mathsf T}S^{-1}\mathbf{v}),$$

with equality exactly when $S^{1/2}\mathbf{a}$ is a multiple of $S^{-1/2}\mathbf{v}$, that is, when $\mathbf{a}$ is a multiple of $S^{-1}\mathbf{v}$. Multiplying by $n/(\mathbf{a}^{\mathsf T}S\mathbf{a})$ gives $t_{\mathbf a}^2 \le n\mathbf{v}^{\mathsf T}S^{-1}\mathbf{v} = T^2$, with equality at those $\mathbf{a}$. $\square$

So the $T^2$ test rejects exactly when some linear combination has $|t_{\mathbf a}|$ above $\sqrt{p(n-1)F_{p,\,n-p,\,1-\alpha}/(n - p)}$, a critical value larger than the $t$ quantile because the combination is chosen after seeing the data. The General Linear Hypothesis (Theorem 3) proves the same identity for the $F$ test of linear restrictions in regression, where the variance in the denominator is known up to the scalar $\sigma^2$. Here the whole matrix is estimated, and the critical value comes from Corollary 1.

For the 25 adults the maximizing combination is proportional to $S^{-1}(\bar{\mathbf{x}} - \boldsymbol\mu_0)$. Scaled to give systolic pressure the weight 1, it is systolic minus 2.735 times diastolic, and its $|t|$ is $\sqrt{9.240} = 3.040$. It exceeds the critical value $\sqrt{7.142} = 2.672$, as it must, since the test rejects. The script checks the maximum by computing $t_{\mathbf a}^2$ over 200,001 directions.

Simultaneous intervals

Inverting the test gives intervals for every combination at once. The event that the test of the true $\boldsymbol\mu$ accepts has probability $1 - \alpha$, and by Theorem 3 it is the event that every combination satisfies

$$\big|\mathbf{a}^{\mathsf T}\bar{\mathbf{x}} - \mathbf{a}^{\mathsf T}\boldsymbol\mu\big| \le c_T\sqrt{\frac{\mathbf{a}^{\mathsf T}S\mathbf{a}}{n}}, \qquad c_T = \sqrt{\frac{p(n-1)}{n-p}F_{p,\,n-p,\,1-\alpha}}.$$

So with probability $1 - \alpha$ the intervals $\mathbf{a}^{\mathsf T}\bar{\mathbf{x}} \pm c_T\sqrt{\mathbf{a}^{\mathsf T}S\mathbf{a}/n}$ cover their targets simultaneously for all $\mathbf{a}$, including combinations chosen after looking at the data. This is the multivariate form of Scheffé's method, which The General Linear Hypothesis derives for regression.

The Bonferroni intervals for the $p$ coordinates alone replace $c_T$ by $c_B = t_{n-1,\,1-\alpha/(2p)}$; they cover all $p$ targets with probability at least $1 - \alpha$ by the union bound (Multiple Comparisons and the Bonferroni Correction). For the 25 adults at 95%, $c_T = 2.672$ and $c_B = 2.3909$:

  • systolic: $T^2$ interval (123.16, 143.21), Bonferroni interval (124.22, 142.16);
  • diastolic: $T^2$ interval (63.94, 73.07), Bonferroni interval (64.42, 72.59).

The $T^2$ intervals are wider by the factor 1.118, and all four contain the all-adult values. That is no contradiction with the rejection: the $T^2$ test rejects because some combination's interval misses its hypothesized value, not necessarily a coordinate's. The difference systolic minus diastolic has the $T^2$ interval (53.90, 75.46), which just excludes the hypothesized 53.87.

Example (which intervals to report). It is tempting to report the $T^2$ intervals for the coordinates because they come from the "right" multivariate test. If only the $p$ coordinates are of interest, chosen in advance, the Bonferroni intervals are the shorter valid choice. The $T^2$ intervals pay for protection over every combination, and the price grows quickly with $p$. For $n = 25$ at 95%:

$p$ $c_T$ $c_B$ ratio
1 2.064 2.064 1.00
2 2.672 2.391 1.12
3 3.159 2.574 1.23
5 4.033 2.797 1.44
10 6.380 3.091 2.06
20 20.918 3.376 6.20

With $p = 1$ the two agree, since $F_{1,\,n-1,\,0.95} = t_{n-1,\,0.975}^2$. With 20 variables and 25 observations the $T^2$ intervals are six times as wide: only 5 degrees of freedom remain for the denominator. The $T^2$ intervals are the right choice when the combinations to be examined are not known in advance.

The likelihood ratio test

The statistic was introduced as a distance. It is also what the likelihood principle produces, which connects it to the general theory of Likelihood Ratio, Wald and Score Tests.

Lemma 4 (the normal estimate of a covariance matrix). Let $M$ be a positive definite $p \times p$ matrix and $n > 0$. Over positive definite $\Sigma$, the function $g(\Sigma) = -\frac{n}{2}\ln\det\Sigma - \frac12\operatorname{tr}(\Sigma^{-1}M)$ has its unique maximum at $\Sigma = M/n$, with value $-\frac{n}{2}\ln\det(M/n) - \frac{np}{2}$.

Proof. Let $K = M^{1/2}\Sigma^{-1}M^{1/2}$, which is positive definite, with eigenvalues $\kappa_1, \ldots, \kappa_p > 0$. Then $\operatorname{tr}(\Sigma^{-1}M) = \operatorname{tr}(M^{1/2}\Sigma^{-1}M^{1/2}) = \sum_j\kappa_j$ (the trace is unchanged by cycling a product), and $\det K = \det M/\det\Sigma$, so $\ln\det\Sigma = \ln\det M - \sum_j\ln\kappa_j$. Hence

$$g(\Sigma) = -\frac{n}{2}\ln\det M + \sum_{j=1}^p\Big(\frac{n}{2}\ln\kappa_j - \frac{\kappa_j}{2}\Big).$$

Each term of the sum is a function of $\kappa_j > 0$ with derivative $\frac{n}{2\kappa_j} - \frac12$, positive below $n$ and negative above, so it is largest at $\kappa_j = n$, uniquely. So $g$ is largest exactly when every eigenvalue of the symmetric matrix $K$ is $n$, that is, $K = nI$, which means $\Sigma = M/n$. The value there is $-\frac{n}{2}\ln\det M + \frac{np}{2}\ln n - \frac{np}{2} = -\frac{n}{2}\ln\det(M/n) - \frac{np}{2}$. $\square$

Theorem 4 (likelihood ratio). For independent $N_p(\boldsymbol\mu, \Sigma)$ observations with both parameters unknown and $n > p$, the likelihood ratio for $H_0\colon\boldsymbol\mu = \boldsymbol\mu_0$ satisfies

$$\Lambda^{2/n} = \frac{\det\hat\Sigma}{\det\hat\Sigma_0} = \frac{1}{1 + T^2/(n-1)}, \qquad -2\ln\Lambda = n\ln\Big(1 + \frac{T^2}{n - 1}\Big),$$

where $\hat\Sigma = A/n$ and $\hat\Sigma_0 = \frac1n\sum_i(\mathbf{x}_i - \boldsymbol\mu_0)(\mathbf{x}_i - \boldsymbol\mu_0)^{\mathsf T}$. So the likelihood ratio test, which rejects for large $-2\ln\Lambda$, is the $T^2$ test.

Proof. The log-likelihood, from the density of The Multivariate Normal Distribution, is

$$\ell(\boldsymbol\mu, \Sigma) = -\frac{np}{2}\ln(2\pi) - \frac{n}{2}\ln\det\Sigma - \frac12\sum_{i=1}^n(\mathbf{x}_i - \boldsymbol\mu)^{\mathsf T}\Sigma^{-1}(\mathbf{x}_i - \boldsymbol\mu).$$

Since a scalar equals its trace and the trace of a product is unchanged by cycling it, the sum equals $\operatorname{tr}\big(\Sigma^{-1}\sum_i(\mathbf{x}_i - \boldsymbol\mu)(\mathbf{x}_i - \boldsymbol\mu)^{\mathsf T}\big)$. Expanding about $\bar{\mathbf{x}}$, the cross terms sum to zero and

$$\sum_i(\mathbf{x}_i - \boldsymbol\mu)(\mathbf{x}_i - \boldsymbol\mu)^{\mathsf T} = A + n(\bar{\mathbf{x}} - \boldsymbol\mu)(\bar{\mathbf{x}} - \boldsymbol\mu)^{\mathsf T}.$$

Unrestricted maximum. For fixed $\Sigma$, the sum is $\operatorname{tr}(\Sigma^{-1}A) + n(\bar{\mathbf{x}} - \boldsymbol\mu)^{\mathsf T}\Sigma^{-1}(\bar{\mathbf{x}} - \boldsymbol\mu)$, smallest at $\boldsymbol\mu = \bar{\mathbf{x}}$. Then Lemma 4 with $M = A$, which is positive definite with probability 1 by Lemmas 1 and 2, gives the maximum at $\hat\Sigma = A/n$ with value $-\frac{np}{2}\ln(2\pi) - \frac{n}{2}\ln\det\hat\Sigma - \frac{np}{2}$.

Restricted maximum. With $\boldsymbol\mu = \boldsymbol\mu_0$, Lemma 4 with $M = A + n\mathbf{v}\mathbf{v}^{\mathsf T}$, $\mathbf{v} = \bar{\mathbf{x}} - \boldsymbol\mu_0$, gives the maximum at $\hat\Sigma_0 = M/n = \hat\Sigma + \mathbf{v}\mathbf{v}^{\mathsf T}$, with the same expression in $\det\hat\Sigma_0$.

The ratio. Subtracting, $\ln\Lambda = -\frac{n}{2}\ln(\det\hat\Sigma_0/\det\hat\Sigma)$. For the determinant, write $\hat\Sigma_0 = \hat\Sigma^{1/2}(I + \mathbf{w}\mathbf{w}^{\mathsf T})\hat\Sigma^{1/2}$ with $\mathbf{w} = \hat\Sigma^{-1/2}\mathbf{v}$. The matrix $I + \mathbf{w}\mathbf{w}^{\mathsf T}$ has $\mathbf{w}$ as an eigenvector with eigenvalue $1 + \lVert\mathbf{w}\rVert^2$, and every vector perpendicular to $\mathbf{w}$ as an eigenvector with eigenvalue 1, so its determinant is $1 + \lVert\mathbf{w}\rVert^2 = 1 + \mathbf{v}^{\mathsf T}\hat\Sigma^{-1}\mathbf{v}$. Finally $\hat\Sigma^{-1} = nA^{-1} = \frac{n}{n-1}S^{-1}$, so $\mathbf{v}^{\mathsf T}\hat\Sigma^{-1}\mathbf{v} = T^2/(n-1)$. $\square$

For the 25 adults, $\det\hat\Sigma = 23{,}607$ and $\det\hat\Sigma_0 = 32{,}696$ (in mm Hg to the fourth power), whose ratio is $\Lambda^{2/n} = 0.7220 = 1/(1 + 9.240/24)$, and $-2\ln\Lambda = 8.142$. Wilks's approximation, as stated in Likelihood Ratio, Wald and Score Tests, refers this to the chi-squared distribution with 2 degrees of freedom and gives $p = 0.0171$. The exact p-value is 0.0236, from Corollary 1. As with the $F$ test of The General Linear Hypothesis, the approximation treats the estimated covariance matrix as known and makes the evidence look stronger than it is.

Two samples

The more common question compares two groups: do men and women have the same mean systolic and diastolic pressures? Let $\mathbf{x}_{11}, \ldots, \mathbf{x}_{1n_1}$ be a sample from $N_p(\boldsymbol\mu_1, \Sigma)$ and $\mathbf{x}_{21}, \ldots, \mathbf{x}_{2n_2}$ an independent sample from $N_p(\boldsymbol\mu_2, \Sigma)$, with the same covariance matrix. Let $\bar{\mathbf{x}}_1, \bar{\mathbf{x}}_2$ be the sample means, $A_1, A_2$ the matrices of sums of squares and products about them, and

$$S_p = \frac{A_1 + A_2}{n_1 + n_2 - 2}$$

the pooled covariance matrix, the matrix version of the pooled variance of The Two-Sample t-Test. The two-sample statistic is

$$T^2 = \frac{n_1n_2}{n_1 + n_2}(\bar{\mathbf{x}}_1 - \bar{\mathbf{x}}_2)^{\mathsf T}S_p^{-1}(\bar{\mathbf{x}}_1 - \bar{\mathbf{x}}_2).$$

Theorem 5 (the two-sample test). Under these assumptions, with $n_1 + n_2 - 2 \ge p$,

$$F = \frac{n_1 + n_2 - p - 1}{p(n_1 + n_2 - 2)}\,T^2$$

has the noncentral $F$ distribution with $p$ and $n_1 + n_2 - p - 1$ degrees of freedom and noncentrality $\lambda = \frac{n_1n_2}{n_1 + n_2}(\boldsymbol\mu_1 - \boldsymbol\mu_2)^{\mathsf T}\Sigma^{-1}(\boldsymbol\mu_1 - \boldsymbol\mu_2)$, central exactly when $\boldsymbol\mu_1 = \boldsymbol\mu_2$.

Proof. Apply Lemma 1 to each sample: $\bar{\mathbf{x}}_1$ and $A_1$ are independent, $A_1$ is distributed as a sum of $n_1 - 1$ outer products of independent $N_p(\mathbf{0}, \Sigma)$ vectors, and the same holds for the second sample with $n_2 - 1$. The samples are independent, so $\bar{\mathbf{x}}_1$, $\bar{\mathbf{x}}_2$, $A_1$ and $A_2$ are independent, and $B = A_1 + A_2$ is distributed as a sum of $m = n_1 + n_2 - 2$ outer products of independent $N_p(\mathbf{0}, \Sigma)$ vectors, as in Theorem 1. The vector $\mathbf{d} = \sqrt{n_1n_2/(n_1 + n_2)}\,(\bar{\mathbf{x}}_1 - \bar{\mathbf{x}}_2)$ is normal with covariance $\frac{n_1n_2}{n_1 + n_2}\big(\frac{\Sigma}{n_1} + \frac{\Sigma}{n_2}\big) = \Sigma$ and mean $\boldsymbol\nu = \sqrt{n_1n_2/(n_1 + n_2)}\,(\boldsymbol\mu_1 - \boldsymbol\mu_2)$, and it is independent of $B$. Since $S_p^{-1} = mB^{-1}$, $T^2 = m\,\mathbf{d}^{\mathsf T}B^{-1}\mathbf{d}$, and Theorem 1 gives the distribution of $\frac{m - p + 1}{p}\mathbf{d}^{\mathsf T}B^{-1}\mathbf{d} = \frac{m - p + 1}{pm}T^2$, with $m - p + 1 = n_1 + n_2 - p - 1$. $\square$

Theorems 2 and 3 carry over with the same proofs. $T^2$ is unchanged by a nonsingular linear change of variables applied to both samples. It is the largest squared two-sample $t$ statistic over all combinations $\mathbf{a}^{\mathsf T}\mathbf{x}$, attained at $\mathbf{a} = S_p^{-1}(\bar{\mathbf{x}}_1 - \bar{\mathbf{x}}_2)$. That direction is Fisher's linear discriminant, the subject of Linear Discriminant Analysis. Written with the sample Mahalanobis distance between the two means, $D^2 = (\bar{\mathbf{x}}_1 - \bar{\mathbf{x}}_2)^{\mathsf T}S_p^{-1}(\bar{\mathbf{x}}_1 - \bar{\mathbf{x}}_2)$, the statistic is $T^2 = \frac{n_1n_2}{n_1 + n_2}D^2$. Theorem 4 needs a new step, because under the hypothesis the common mean is unknown rather than given.

Theorem 6 (the two-sample likelihood ratio). Under the assumptions of Theorem 5, with $N = n_1 + n_2 > p + 1$, the likelihood ratio for $H_0\colon\boldsymbol\mu_1 = \boldsymbol\mu_2$ (with $\Sigma$ unknown) satisfies

$$\Lambda^{2/N} = \frac{1}{1 + T^2/(N - 2)}, \qquad -2\ln\Lambda = N\ln\Big(1 + \frac{T^2}{n_1 + n_2 - 2}\Big),$$

an increasing function of $T^2$, so the likelihood ratio test is the two-sample $T^2$ test.

Proof. Write $B = A_1 + A_2$, $\mathbf{d} = \bar{\mathbf{x}}_1 - \bar{\mathbf{x}}_2$ and $c = n_1n_2/N$. As in the proof of Theorem 4, the sum of squares and products of sample $k$ about a vector $\mathbf{m}$ is $A_k + n_k(\bar{\mathbf{x}}_k - \mathbf{m})(\bar{\mathbf{x}}_k - \mathbf{m})^{\mathsf T}$, and the log-likelihood is $-\frac{Np}{2}\ln(2\pi) - \frac N2\ln\det\Sigma - \frac12\operatorname{tr}(\Sigma^{-1}M)$, where $M$ is the total of these matrices.

Unrestricted maximum. For fixed $\Sigma$, the terms $n_k(\bar{\mathbf{x}}_k - \boldsymbol\mu_k)^{\mathsf T}\Sigma^{-1}(\bar{\mathbf{x}}_k - \boldsymbol\mu_k)$ vanish at $\boldsymbol\mu_k = \bar{\mathbf{x}}_k$, leaving $M = B$, which is positive definite with probability 1 (Lemmas 1 and 2, $N - 2 \ge p$). Lemma 4 gives the maximum at $\hat\Sigma = B/N$.

Restricted maximum. With a common mean $\boldsymbol\mu$, for fixed $\Sigma$ we must minimize $n_1q(\bar{\mathbf{x}}_1 - \boldsymbol\mu) + n_2q(\bar{\mathbf{x}}_2 - \boldsymbol\mu)$, with $q(\mathbf{v}) = \mathbf{v}^{\mathsf T}\Sigma^{-1}\mathbf{v}$. Its gradient in $\boldsymbol\mu$ is $-2\Sigma^{-1}\big(n_1(\bar{\mathbf{x}}_1 - \boldsymbol\mu) + n_2(\bar{\mathbf{x}}_2 - \boldsymbol\mu)\big)$, which vanishes only at the pooled mean $\bar{\mathbf{x}} = (n_1\bar{\mathbf{x}}_1 + n_2\bar{\mathbf{x}}_2)/N$, and the function is convex (a sum of positive definite quadratic forms), so that is the minimum, whatever $\Sigma$ is. There $\bar{\mathbf{x}}_1 - \bar{\mathbf{x}} = \frac{n_2}{N}\mathbf{d}$ and $\bar{\mathbf{x}}_2 - \bar{\mathbf{x}} = -\frac{n_1}{N}\mathbf{d}$, so the two outer products add to $\frac{n_1n_2^2 + n_2n_1^2}{N^2}\mathbf{d}\mathbf{d}^{\mathsf T} = c\,\mathbf{d}\mathbf{d}^{\mathsf T}$, and $M = B + c\,\mathbf{d}\mathbf{d}^{\mathsf T}$. Lemma 4 gives the maximum at $\hat\Sigma_0 = (B + c\,\mathbf{d}\mathbf{d}^{\mathsf T})/N$.

The ratio. By Lemma 4 the two maxima differ by $-\frac N2\ln(\det\hat\Sigma_0/\det\hat\Sigma)$. The determinant step of Theorem 4, with $\hat\Sigma = B/N$ and $\mathbf{v} = \sqrt{c/N}\,\mathbf{d}$, gives $\det\hat\Sigma_0/\det\hat\Sigma = 1 + \mathbf{v}^{\mathsf T}\hat\Sigma^{-1}\mathbf{v} = 1 + c\,\mathbf{d}^{\mathsf T}B^{-1}\mathbf{d}$. Since $S_p^{-1} = (N - 2)B^{-1}$, $c\,\mathbf{d}^{\mathsf T}B^{-1}\mathbf{d} = T^2/(N - 2)$. $\square$

For the 20 men and 20 women below, $T^2 = 2.417$ gives $-2\ln\Lambda = 40\ln(1 + 2.417/38) = 2.467$.

Men and women

The course's two-sample example is 20 men and 20 women drawn from the NHANES adults (The Two-Sample t-Test), whose mean systolic pressures, 126.950 and 122.817, gave the pooled $t = 0.674$, $p = 0.504$ (0.5044 to four places, recomputed here). The same 40 adults have mean diastolic pressures of 75.63 for the men and 69.90 for the women, a difference of 5.73 mm Hg. Its two-sample $t$ is 1.520, with $p = 0.137$. The pooled covariance matrix is

$$S_p = \begin{pmatrix} 376.06 & 141.56 \\ 141.56 & 142.19\end{pmatrix},$$

a correlation of 0.612. Theorem 5 gives $T^2 = 10 \times 0.2417 = 2.417$, so $D^2 = 0.2417$, and $F = \frac{37}{2 \times 38}\times 2.417 = 1.177$ on 2 and 37 degrees of freedom (upper 5% point 3.2519), $p = 0.320$. The joint test finds nothing either. The direction of the largest $t$ puts weight $-7.02$ on diastolic pressure per unit weight on systolic pressure. Reversing the sign, it is the diastolic pressure minus a seventh of the systolic. Here the men's systolic and diastolic means are both higher, in the direction the two pressures already move together, and the test gains nothing from combining them. The next section measures how general that is.

Against separate t tests

The two examples point in opposite directions. For the older adults, $T^2$ rejected while Bonferroni-corrected $t$ tests did not; for men and women nothing was found either way. Which test has more power depends on how the true difference sits relative to the covariance matrix, and Theorem 5 says exactly how for $T^2$: its power is a function of $\lambda$ alone. For Bonferroni-corrected $t$ tests, the power depends on the $p$ individual standardized differences.

Here is the comparison in one sentence. $\lambda$ is a squared Mahalanobis distance. It is large when the difference points along a direction of small variance, such as against the correlation, and $T^2$ then gains. It is small relative to the individual effects when the difference points along the correlation, or when one coordinate carries it alone among many, and $T^2$ then pays for the degrees of freedom it spends on the other directions.

Take two variables with unit variances and correlation $\rho$, and a difference of $\delta$ standard deviations in each. Then $\Sigma^{-1}$ gives $(\delta, \delta)\Sigma^{-1}(\delta, \delta)^{\mathsf T} = 2\delta^2/(1 + \rho)$ for a shift in the same direction, and $2\delta^2/(1 - \rho)$ for a shift in opposite directions. With $\rho = 0.6$ the second is four times the first. The individual $t$ statistics are the same size in both cases, so the Bonferroni procedure cannot tell them apart.

Size and power

The script simulated two samples of 20 from normal distributions with unit variances, in seven scenarios, 2,000 samples for each of 30 seeds, and applied both tests at level 0.05. The table gives the median rejection rate over the seeds and, in parentheses, the interquartile range. The last column is the exact power of $T^2$ from the noncentral $F$ distribution of Theorem 5.

Scenario ($p$, correlation, difference in SDs) $\lambda$ $T^2$ Bonferroni $T^2$ exact
null: $p = 2$, $\rho = 0.6$, $(0, 0)$ 0 0.052 (0.049 to 0.055) 0.048 (0.045 to 0.051) 0.05
same direction: $\rho = 0.6$, $(0.5, 0.5)$ 3.13 0.304 (0.300 to 0.318) 0.350 (0.344 to 0.358) 0.310
opposite directions: $\rho = 0.6$, $(0.5, -0.5)$ 12.50 0.866 (0.861 to 0.875) 0.461 (0.455 to 0.472) 0.869
one variable: $\rho = 0.6$, $(0.7, 0)$ 7.66 0.659 (0.651 to 0.664) 0.474 (0.465 to 0.481) 0.659
same direction, $\rho = 0$, $(0.5, 0.5)$ 5.00 0.472 (0.465 to 0.481) 0.421 (0.415 to 0.430) 0.470
null: $p = 10$, $\rho = 0.3$, all 0 0 0.049 (0.047 to 0.054) 0.046 (0.043 to 0.049) 0.05
one of ten: $p = 10$, $\rho = 0.3$, $(0.9, 0, \ldots, 0)$ 10.63 0.434 (0.428 to 0.445) 0.480 (0.469 to 0.488) 0.435

Four observations, each about this experiment:

  • Under the null hypothesis $T^2$ rejected at close to 5%, as Theorem 5 requires, and in every scenario the exact power lies inside the interquartile range of the simulated rates. The Bonferroni procedure rejected less than 5% of the time, as a union bound allows.
  • With correlated variables and a shift against the correlation, $T^2$ rejected almost twice as often (0.866 against 0.461).
  • A shift in one of two correlated variables also favored $T^2$ (0.659 against 0.474). The second variable, unshifted but correlated with the first, lets $T^2$ subtract part of the first variable's noise. In Mahalanobis terms $\lambda = 10 \times 0.7^2/(1 - 0.6^2) = 7.66$, larger than the $10 \times 0.7^2 = 4.9$ that the first variable alone would give.
  • Bonferroni won in two scenarios: the shift along the correlation (0.350 against 0.304) and a single shifted variable among ten (0.480 against 0.434), where $T^2$ spreads its attention over ten dimensions and keeps only $40 - 10 - 1 = 29$ denominator degrees of freedom.

Samples from NHANES

The normal simulation fixes the covariance and the shape. NHANES pressures are skewed, and the covariance matrices of different groups differ. The 1,578 adults aged 40 to 59 have mean pressures of 126.31 and 77.41 with a correlation of 0.637 between them; the 1,912 aged 60 and over (with a diastolic value) have 136.76 and 70.91 with a correlation of 0.377. Systolic pressure is 10.44 mm Hg higher in the older group and diastolic pressure 6.50 lower: a difference against the correlation. The 2,541 men and 2,691 women differ by 2.12 and 3.04 mm Hg (means 127.40 and 74.05 for men, 125.28 and 71.01 for women): a difference along it. With the two groups' covariance matrices averaged, the squared Mahalanobis distance between the group means is 1.155 for the age groups and 0.065 for the sexes.

For each of 30 seeds the script drew 2,000 pairs of samples, of equal size, without replacement within each group, and applied both tests at level 0.05. A third comparison shifted every 40-to-59 adult's pair of pressures by the difference in means, so that the null hypothesis holds exactly in the population while the shapes and covariances stay different. Medians over the seeds, with interquartile ranges:

Comparison Adults per group $T^2$ Bonferroni
60+ against 40 to 59 10 0.510 (0.503 to 0.516) 0.268 (0.261 to 0.274)
60+ against 40 to 59 20 0.855 (0.848 to 0.860) 0.538 (0.531 to 0.545)
60+ against 40 to 59 30 0.968 (0.965 to 0.969) 0.748 (0.744 to 0.754)
men against women 20 0.098 (0.093 to 0.102) 0.099 (0.095 to 0.102)
men against women 50 0.182 (0.179 to 0.189) 0.187 (0.181 to 0.195)
men against women 100 0.332 (0.324 to 0.338) 0.341 (0.335 to 0.345)
null (shifted) 10 0.049 (0.046 to 0.051) 0.044 (0.042 to 0.047)
null (shifted) 20 0.049 (0.046 to 0.052) 0.044 (0.041 to 0.047)

The real data follow the normal pattern. For the age groups, whose pressures move in opposite directions, $T^2$ detected the difference far more often at every sample size. For the sexes, whose pressures move together, the two procedures were within a percentage point of each other, with Bonferroni slightly ahead. With equal group sizes, the shifted null kept $T^2$ close to its nominal level despite skewed data and unequal covariance matrices. That is an observation for this population and equal groups. Theorem 5 assumes a common covariance matrix, and these samples do not tell what happens with unequal group sizes, where the pooled $t$ test of Welch's t-Test already goes wrong in one dimension.

What is proved here, what is cited, what the experiments show

Proved here. Invertibility of a sum of outer products exactly when there are at least $p$ of them (Lemma 2); the reading of a Schur complement as a residual sum of squares (after Lemma 3); the noncentral $F$ distribution of $\mathbf{d}^{\mathsf T}B^{-1}\mathbf{d}$ (Theorem 1) and of the one-sample and two-sample statistics (Corollary 1, Theorem 5); invariance under nonsingular linear maps (Theorem 2); $T^2$ as the largest squared $t$ over combinations and the simultaneous intervals (Theorem 3); the normal maximum likelihood estimate of a covariance matrix (Lemma 4) and the likelihood ratios of the one-sample and two-sample tests (Theorems 4 and 6); the noncentralities that explain the power comparison.

Taken from other articles, where they are proved. Closure of the multivariate normal under linear maps and the inverse of a partitioned matrix (Lemma 3; The Multivariate Normal Distribution, Theorems 3 and 7); symmetric square roots, the chi-squared law of the Mahalanobis distance, and the independence of $\bar{\mathbf{x}}$ and $S$ with $A$ distributed as a sum of $n - 1$ outer products (Lemma 1; Covariance Matrices and Quadratic Forms, Theorems 3, 6 and 10); Helmert's matrix (The Chi-Squared and t Distributions); the chi-squared distribution of a residual sum of squares (The Normal Linear Model, Theorem 1); noncentral chi-squared and $F$ distributions (Sampling Distributions from the Normal); the Cauchy–Schwarz inequality (Probability Inequalities); the union bound behind Bonferroni (Multiple Comparisons and the Bonferroni Correction).

Cited. The origin of the statistic (Hotelling, 1931) and of the distribution of $A$ (Wishart, 1928), neither used in a proof; the scaling of the $F$ distribution as the NIST/SEMATECH handbook states it (§6.5.4.3), a check on Corollary 1; Wilks's approximation through Likelihood Ratio, Wald and Score Tests. Percentage points and noncentral $F$ probabilities are computed by SciPy.

The experiments show rejection rates of $T^2$ and of Bonferroni-corrected $t$ tests in seven normal scenarios and in samples drawn from the NHANES population, each over 30 seeds; they agree with the exact theory where it applies and describe this population where it does not.

Worked problems

These problems were written for this article, on a small exact sample and on the unit's NHANES data. The script tools/experiments/statistics/hotellings-t-squared.py checks every number.

Problem 1 ($T^2$ by hand). Four observations of two variables are $(2, 1)$, $(4, 3)$, $(3, 5)$ and $(7, 3)$. Test $H_0\colon\boldsymbol\mu = (2, 2)$ with Hotelling's $T^2$, assuming normality, and give the exact p-value.

Solution. $T^2 = 38/9 = 4.222$, $F = 38/27 = 1.407$ on 2 and 2 degrees of freedom, and $p = 27/65 = 0.415$: no evidence against $H_0$.

The sample mean is $\bar{\mathbf{x}} = (16/4, 12/4) = (4, 3)$, so $\bar{\mathbf{x}} - \boldsymbol\mu_0 = (2, 1)$. The deviations from the mean are $(-2, -2)$, $(0, 0)$, $(-1, 2)$ and $(3, 0)$. Their sums of squares and products are $4 + 0 + 1 + 9 = 14$, $4 + 0 - 2 + 0 = 2$ and $4 + 0 + 4 + 0 = 8$, and dividing by $n - 1 = 3$,

$$S = \begin{pmatrix} 14/3 & 2/3 \\ 2/3 & 8/3\end{pmatrix}, \qquad \det S = \frac{112}{9} - \frac{4}{9} = 12, \qquad S^{-1} = \frac{1}{12}\begin{pmatrix} 8/3 & -2/3 \\ -2/3 & 14/3\end{pmatrix}.$$

Then $(2, 1)S^{-1}(2, 1)^{\mathsf T} = \frac{1}{12}\big(\frac{8}{3}\cdot 4 - 2\cdot\frac{2}{3}\cdot 2 + \frac{14}{3}\cdot 1\big) = \frac{1}{12}\cdot\frac{32 - 8 + 14}{3} = \frac{38}{36}$, and $T^2 = 4 \times \frac{38}{36} = \frac{38}{9}$. By Corollary 1, $F = \frac{n - p}{p(n - 1)}T^2 = \frac{2}{6}\cdot\frac{38}{9} = \frac{38}{27}$. With 2 and 2 degrees of freedom the $F$ distribution has the tail $P(F > x) = 1/(1 + x)$: the density of Theorem 4 of Sampling Distributions from the Normal with $d_1 = d_2 = 2$ is $(1 + x)^{-2}$, whose integral from $x$ to infinity is $1/(1 + x)$. So $p = 1/(1 + 38/27) = 27/65$, and the 5% critical value is the $x$ with $1/(1 + x) = 0.05$, which is 19. With only four observations, the denominator has 2 degrees of freedom, and only a very large $T^2$ would reject.

Problem 2 (how many adults per group). Plan a two-sample $T^2$ test at level 0.05 of the systolic and diastolic means of adults aged 60 and over against those aged 40 to 59, with power at least 0.9, taking the population difference and the average of the two covariance matrices as the truth. How many adults per group are needed, and how many would a $t$ test of systolic pressure alone need for the same power?

Solution. 24 adults per group for $T^2$ (power 0.907), and 77 per group for systolic pressure alone.

By Theorem 5 with $n$ adults per group, $\lambda = \frac{n \cdot n}{2n}\Delta^2 = \frac{n}{2}\Delta^2$, with the population's $\Delta^2 = 1.155$, and $F$ has 2 and $2n - 3$ degrees of freedom. The power is the probability that a noncentral $F$ variable with these parameters exceeds the upper 5% point of the central $F$. It increases with $n$. At $n = 23$ it is 0.893; at $n = 24$, $\lambda = 13.86$ and the power is 0.907, so 24 per group are needed. For systolic pressure alone the standardized difference is 0.528 standard deviations, and the two-sample $t$ test (the case $p = 1$ of Theorem 5) with 24 per group has power only 0.433. The smallest group size giving power 0.9 for systolic alone is 77 per group. The diastolic means move the other way, against a positive correlation, and that is what makes the joint test so much cheaper here.

Problem 3 (one combination chosen in advance, and one chosen afterwards). For the 25 older adults, compute the 95% $t$ interval for the mean of systolic minus diastolic pressure, as if that difference had been the only question planned. Compare it with the $T^2$ simultaneous interval (53.90, 75.46) of the text, and say when each may be reported.

Solution. The $t$ interval is (56.35, 73.01). It may be reported only if the difference was chosen before the data were seen; the wider $T^2$ interval is valid however it was chosen.

The values $x_{i1} - x_{i2}$ have mean $64.68$ and variance $\mathbf{a}^{\mathsf T}S\mathbf{a}$ with $\mathbf{a} = (1, -1)$, that is $351.78 - 2 \times 9.00 + 73.05$, which from the unrounded entries of $S$ is 406.82, a standard deviation of 20.17. The $t$ interval is $64.68 \pm 2.0639 \times 20.17/\sqrt{25}$, which is (56.35, 73.01), with $t_{24,\,0.975} = 2.0639$. Both intervals exclude the hypothesized 53.87, the planned one by more. The $T^2$ interval replaces 2.0639 by $c_T = 2.672$. If the difference was picked because the data made it look interesting, as the direction of the largest $t$ suggests it might be, only the $T^2$ interval keeps its 95% guarantee (Theorem 3).

Summary

Quantity Formula
One-sample statistic $T^2 = n(\bar{\mathbf{x}} - \boldsymbol\mu_0)^{\mathsf T}S^{-1}(\bar{\mathbf{x}} - \boldsymbol\mu_0)$
Its distribution $\frac{n - p}{p(n - 1)}T^2 \sim F_{p,\,n-p}(\lambda)$, $\lambda = n(\boldsymbol\mu - \boldsymbol\mu_0)^{\mathsf T}\Sigma^{-1}(\boldsymbol\mu - \boldsymbol\mu_0)$
Two-sample statistic $T^2 = \frac{n_1n_2}{n_1 + n_2}(\bar{\mathbf{x}}_1 - \bar{\mathbf{x}}_2)^{\mathsf T}S_p^{-1}(\bar{\mathbf{x}}_1 - \bar{\mathbf{x}}_2)$
Its distribution $\frac{n_1 + n_2 - p - 1}{p(n_1 + n_2 - 2)}T^2 \sim F_{p,\,n_1 + n_2 - p - 1}(\lambda)$
Largest $t$ $\max_{\mathbf a}t_{\mathbf a}^2 = T^2$ at $\mathbf{a} \propto S^{-1}(\bar{\mathbf{x}} - \boldsymbol\mu_0)$
Simultaneous intervals $\mathbf{a}^{\mathsf T}\bar{\mathbf{x}} \pm c_T\sqrt{\mathbf{a}^{\mathsf T}S\mathbf{a}/n}$, $c_T^2 = \frac{p(n-1)}{n-p}F_{p,\,n-p,\,1-\alpha}$
Likelihood ratio $-2\ln\Lambda = n\ln\big(1 + T^2/(n-1)\big)$
Two-sample likelihood ratio $-2\ln\Lambda = N\ln\big(1 + T^2/(N-2)\big)$, $N = n_1 + n_2$
Invariance unchanged by $\mathbf{x} \mapsto C\mathbf{x} + \mathbf{b}$, $C$ nonsingular

What we have learned

Hotelling's $T^2$ is the squared Mahalanobis distance of the sample mean from the hypothesized mean, measured with the estimated covariance matrix. Under normality a fixed multiple of it has an exact $F$ distribution, noncentral when the hypothesis is false. The proof needs only a rotation and the chi-squared law of a residual sum of squares. The statistic does not depend on the linear coordinates in which the data are recorded. It is the largest squared $t$ over all linear combinations, which yields intervals valid for every combination at once, and it is the likelihood ratio test. Against separate $t$ tests with a Bonferroni correction it wins when the difference runs against the correlation of the variables or when several moderate differences combine, as for older against middle-aged adults' blood pressures. It loses, modestly, when the difference runs along the correlation or sits in one variable among many. For the 25 older adults it rejected the all-adult means ($p = 0.0236$) where neither coordinate could.

The direction that makes the two-sample $t$ statistic largest, $S_p^{-1}(\bar{\mathbf{x}}_1 - \bar{\mathbf{x}}_2)$, separates the two groups best in a precise sense. Linear Discriminant Analysis turns it into a rule for classifying a new observation and proves that it is the Bayes rule when the groups are normal with a common covariance matrix.

References