A study measures the systolic blood pressure of 10 adults aged 60 and over and 30 adults aged 18 to 39, drawn from the NHANES survey of 2017 to 2018. The older group averages 26.8 mm Hg higher. The pooled test of The Two-Sample t-Test reports $p = 0.00002$ and a 95% interval of 15.67 to 37.97 mm Hg for the difference. That test assumes the two populations share one variance, and these two do not. Can its answer be trusted? No: it claims more precision than the data contain. The test that drops the assumption gives $p = 0.0068$ and an interval of 9.19 to 44.45 mm Hg.
Welch's $t$-test is that test. It estimates the variance of each group separately and refers the statistic to a $t$ distribution whose degrees of freedom are computed from the data. Where the pooled test is exact under its assumptions, Welch's test is approximate, and this article says exactly where.
Here is the route. First we derive Welch's statistic and the Satterthwaite degrees of freedom, and say exactly which of the steps are approximations. Then we work the test on the older and younger adults. Simulations then show when the pooled test fails and by how much. Finally we show how to choose between the two tests, and why the traditional answer, a preliminary $F$ test of the variances, is a poor one. Worked problems close the article. The article assumes The Two-Sample t-Test: its notation, the standard normal variable $Z$ of its first section, the pooled variance $s_p^2$ and the proof that the pooled statistic has a $t$ distribution.
Unequal variances: Welch's t-test
The pooled test needs the two populations to share one variance, and often they do not. Blood pressure varies much more among older adults than among younger ones. The NHANES adults aged 60 and over have a standard deviation of 21.02 mm Hg, and those aged 18 to 39 one of 12.85. The ratio is 1.64, and the variance ratio is 2.68.
Without a common variance there is nothing to pool. The natural statistic estimates each variance separately:
$$t = \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)}{\sqrt{s_1^2/n_1 + s_2^2/n_2}}.$$
The square of its denominator is an unbiased estimate of the true variance of the difference whatever the two variances are, since $E[S_1^2/n_1 + S_2^2/n_2] = \sigma_1^2/n_1 + \sigma_2^2/n_2$. The difficulty is its distribution. For normal populations, the square of the denominator is
$$\frac{S_1^2}{n_1} + \frac{S_2^2}{n_2} = \frac{\sigma_1^2}{n_1(n_1 - 1)}\,V_1 + \frac{\sigma_2^2}{n_2(n_2 - 1)}\,V_2,$$
where $V_i = (n_i - 1)S_i^2/\sigma_i^2$ are independent chi-squared variables. It is a weighted sum of two chi-squared variables with unknown weights. Only when the two weights are equal is it a multiple of $V_1 + V_2$, a single chi-squared variable, and the proof of the pooled test in The Two-Sample t-Test then applies. Otherwise the proof has nothing to offer. The weighted sum is then not a multiple of a single chi-squared variable (proved below, under the shape of $V$), and the distribution of the ratio depends on the unknown ratio $\sigma_1^2/\sigma_2^2$, so no single $t$ distribution with a fixed number of degrees of freedom fits every case. Welch (1947) and Satterthwaite (1946) give approximations to it, derived in the next section.
Testing equality of two normal means without assuming equal variances is known as the Behrens–Fisher problem. The name comes from a solution proposed by Behrens in 1929 and defended by Fisher, which Welch (1947) discusses and rejects. The problem has no solution as simple as the one Student found for one mean.
The Satterthwaite approximation
Here is the idea in one sentence. We cannot make the statistic exactly $t$, so we find the $t$ distribution it most resembles, by choosing the degrees of freedom that make its denominator behave most like a chi-squared variable.
The formal version. Write $V = S_1^2/n_1 + S_2^2/n_2$ for the estimated variance of the difference and $\theta = \sigma_1^2/n_1 + \sigma_2^2/n_2$ for the true one. The statistic is
$$t = \frac{(\bar{X}_1 - \bar{X}_2) - (\mu_1 - \mu_2)}{\sqrt{\theta}} \Big/ \sqrt{\frac{V}{\theta}}.$$
Its first factor is the standard normal $Z$ of the first section of The Two-Sample t-Test. $V$ is independent of it, because $V$ depends only on the two sample variances and $Z$ only on the two sample means (Step 3 of the pooled proof there, which did not use the equality of the variances). So if $\nu V/\theta$ were chi-squared with $\nu$ degrees of freedom, the statistic would be exactly $t$ with $\nu$ degrees of freedom, by the definition of the $t$ distribution. It is not, but we can choose $\nu$ to make $V$ match $\theta\chi^2_\nu/\nu$ in its first two moments.
The mean. $E[V] = \sigma_1^2/n_1 + \sigma_2^2/n_2 = \theta$. A chi-squared variable with $\nu$ degrees of freedom has mean $\nu$, so $E[\theta\chi^2_\nu/\nu] = \theta$ for every $\nu$. The means agree automatically.
The variance of a chi-squared variable is $2\nu$. A $\chi^2_\nu$ variable is a sum of $\nu$ independent squares $Z_j^2$ of standard normal variables, so its variance is $\nu\operatorname{Var}(Z^2)$, and $\operatorname{Var}(Z^2) = E[Z^4] - (E[Z^2])^2 = E[Z^4] - 1$. The fourth moment comes from one integration by parts. The standard normal density $\varphi$ satisfies $\varphi'(z) = -z\varphi(z)$, so
$$E[Z^4] = \int_{-\infty}^{\infty} z^3 \cdot z\varphi(z)\,dz = \Big[-z^3\varphi(z)\Big]_{-\infty}^{\infty} + 3\int_{-\infty}^{\infty} z^2\varphi(z)\,dz = 0 + 3E[Z^2] = 3.$$
Hence $\operatorname{Var}(Z^2) = 2$ and $\operatorname{Var}(\chi^2_\nu) = 2\nu$. It follows that $\operatorname{Var}(\theta\chi^2_\nu/\nu) = (\theta^2/\nu^2)(2\nu) = 2\theta^2/\nu$.
The variance of $V$. Since $(n_i - 1)S_i^2/\sigma_i^2$ is chi-squared with $n_i - 1$ degrees of freedom, its variance is $2(n_i - 1)$, so
$$\operatorname{Var}(S_i^2) = \frac{\sigma_i^4}{(n_i - 1)^2}\cdot 2(n_i - 1) = \frac{2\sigma_i^4}{n_i - 1}.$$
The two samples are independent, so the variances of the two terms of $V$ add:
$$\operatorname{Var}(V) = \frac{\operatorname{Var}(S_1^2)}{n_1^2} + \frac{\operatorname{Var}(S_2^2)}{n_2^2} = \frac{2(\sigma_1^2/n_1)^2}{n_1 - 1} + \frac{2(\sigma_2^2/n_2)^2}{n_2 - 1}.$$
Matching $2\theta^2/\nu = \operatorname{Var}(V)$ and solving for $\nu$:
$$\nu = \frac{\left(\dfrac{\sigma_1^2}{n_1} + \dfrac{\sigma_2^2}{n_2}\right)^2}{\dfrac{(\sigma_1^2/n_1)^2}{n_1 - 1} + \dfrac{(\sigma_2^2/n_2)^2}{n_2 - 1}}.$$
The population variances are unknown, so the sample variances are substituted. That gives the degrees of freedom in the course text:
$$\nu = \frac{\left(\dfrac{s_1^2}{n_1} + \dfrac{s_2^2}{n_2}\right)^2}{\dfrac{(s_1^2/n_1)^2}{n_1 - 1} + \dfrac{(s_2^2/n_2)^2}{n_2 - 1}}.$$
The statistic is referred to a $t$ distribution with $\nu$ degrees of freedom. This is Welch's $t$-test, also called the unequal-variance or separate-variance $t$-test.
What is exact and what is approximate. Three statements in the derivation are exact for normal populations: the numerator over $\sqrt{\theta}$ is standard normal, $V$ is independent of it, and $V$ has mean $\theta$ and the variance computed above. Two steps are approximations, and neither comes with a proof that the test's level is exactly $\alpha$.
- The shape of $V$. The distribution of $V$ is replaced by that of $\theta\chi^2_\nu/\nu$, which shares its mean and variance. The replacement is exact if and only if the two weights of the weighted sum above are equal; the proof follows this list.
- The plug-in. The $\nu$ that matches the moments depends on the unknown $\sigma_1^2$ and $\sigma_2^2$. The test uses the value computed from $s_1^2$ and $s_2^2$, which is itself random and changes from sample to sample.
Proof that $V$ is a multiple of a chi-squared variable only when the weights are equal. Write $V = c_1V_1 + c_2V_2$ with $c_i = \sigma_i^2/[n_i(n_i - 1)]$, $p = n_1 - 1$ and $q = n_2 - 1$. If $c_1 = c_2 = c$, then $V = c(V_1 + V_2)$ is $c$ times a $\chi^2_{p+q}$ variable, and the moment-matched $\nu$ is $p + q = n_1 + n_2 - 2$. For the converse we need the third central moment of a chi-squared variable. For a standard normal $Z$, the integration by parts above gives $E[Z^6] = 5E[Z^4] = 15$, so
$$E\big[(Z^2 - 1)^3\big] = E[Z^6] - 3E[Z^4] + 3E[Z^2] - 1 = 15 - 9 + 3 - 1 = 8.$$
For independent $A$ and $B$ with means zero, $E[(A + B)^3] = E[A^3] + 3E[A^2]E[B] + 3E[A]E[B^2] + E[B^3] = E[A^3] + E[B^3]$, so third central moments of independent variables add, as variances do. A $\chi^2_k$ variable therefore has mean $k$, variance $2k$ and third central moment $8k$. Now suppose $V$ had the distribution of $c\chi^2_\nu$ for some $c > 0$ and $\nu > 0$. Equating the mean, the variance and the third central moment of the two,
$$c_1p + c_2q = c\nu, \qquad 2(c_1^2p + c_2^2q) = 2c^2\nu, \qquad 8(c_1^3p + c_2^3q) = 8c^3\nu.$$
Then $(c_1^2p + c_2^2q)^2 = c^4\nu^2 = (c_1p + c_2q)(c_1^3p + c_2^3q)$. That is equality in the Cauchy–Schwarz inequality for the pairs $(\sqrt{c_1p}, \sqrt{c_2q})$ and $(c_1^{3/2}\sqrt{p}, c_2^{3/2}\sqrt{q})$, which holds only when the pairs are proportional. The ratios of their entries are $c_1$ and $c_2$, so $c_1 = c_2$. $\blacksquare$
The test is therefore an approximate one, due to Welch (1947) and Satterthwaite (1946). Its accuracy is a matter of evidence, which this article's simulations supply for the designs they cover.
Who derived it. Welch (1947) obtained the approximation by matching the first two moments, as above. He treated $\nu$ as an effective number of degrees of freedom and noted that he had used the method in papers of 1936 and 1938. With the population variances, the formula is his exactly. For the version with sample variances he suggested a slightly different estimate: $n_i + 1$ in place of $n_i - 1$ in the denominator, and 2 subtracted from the result, so that its numerator and denominator have the right averages over repeated samples. Satterthwaite (1946) proposed the same moment-matching rule for any linear combination of independent mean squares. He substituted the observed mean squares, as here, and applied the rule to the difference of two means. Software and the course text use the plain substitution, and $\nu$ is often called the Welch–Satterthwaite degrees of freedom. The NIST handbook (§1.3.5.3 and §7.3.1) gives the test and the formula in this form.
Three properties of $\nu$ are worth knowing.
- It is usually not a whole number. Software uses it as it is, while tables need it rounded. The course text rounds it down to the nearest integer, which is slightly conservative: fewer degrees of freedom give a larger critical value.
- It always lies between $\min(n_1, n_2) - 1$ and $n_1 + n_2 - 2$. The proof follows this list. It reaches the upper end exactly when the two terms are in proportion to their degrees of freedom, $(s_1^2/n_1)/(n_1 - 1) = (s_2^2/n_2)/(n_2 - 1)$, as with equal sample sizes and equal sample variances. It falls toward the lower end when one group's term dominates the standard error. The uncertainty about the standard error is then essentially the uncertainty in that one group's variance.
- With equal sample sizes, Welch's statistic is the pooled one. With $n_1 = n_2 = n$ the pooled variance is the plain average $(s_1^2 + s_2^2)/2$, so $s_p^2(1/n + 1/n) = (s_1^2 + s_2^2)/n = s_1^2/n + s_2^2/n$. The two standard errors coincide, and only the degrees of freedom differ.
Proof of the bounds. Write $a = s_1^2/n_1$, $b = s_2^2/n_2$, $p = n_1 - 1$ and $q = n_2 - 1$, so that $\nu = (a + b)^2/(a^2/p + b^2/q)$ with $a, b > 0$.
Upper bound. By the Cauchy–Schwarz inequality applied to the pairs $(a/\sqrt{p},\, b/\sqrt{q})$ and $(\sqrt{p},\, \sqrt{q})$,
$$(a + b)^2 = \left(\frac{a}{\sqrt{p}}\sqrt{p} + \frac{b}{\sqrt{q}}\sqrt{q}\right)^2 \le \left(\frac{a^2}{p} + \frac{b^2}{q}\right)(p + q),$$
so $\nu \le p + q = n_1 + n_2 - 2$. Equality holds exactly when the two pairs are proportional, that is, when $a/p = b/q$.
Lower bound. Let $m = \min(p, q)$. Then $a^2/p + b^2/q \le (a^2 + b^2)/m \le (a + b)^2/m$, the last step because $2ab > 0$. So $\nu \ge m = \min(n_1, n_2) - 1$, and the inequality is strict. $\blacksquare$
The same proof applies to the version with population variances, so the moment-matched $\nu$ has the same bounds.
For the 20 men and 20 women of The Two-Sample t-Test, whose pooled test gave $t = 0.674$ and $p = 0.504$, Welch's test gives the same $t = 0.674$ on $\nu = 37.47$ degrees of freedom, $p = 0.504$, and the interval $[-8.29, 16.55]$.
How good is the approximation? The script checked it for normal populations with the NHANES standard deviations, 21.02 and 12.85 mm Hg, and samples of 10 and 30. With the true standard deviations, $\nu = 11.33$ and the variance of $V$ is 436.25. Across 300,000 simulated pairs of samples the variance of $V$ was 435.98. The upper 2.5% point of Welch's statistic was 2.186, against 2.193 for the $t$ distribution with 11.33 degrees of freedom. The Type I error rates in the section on when pooling fails show the approximation holding its level in every design simulated there. These are results of this article's experiments, not a general theorem.
Older and younger adults
Back to the age groups. The script drew 10 adults aged 60 and over and 30 aged 18 to 39 (seed 1). In this design the smaller group comes from the more variable population:
| Sample | $n$ | $\bar{x}$ | $s$ | $s^2$ | $s^2/n$ |
|---|---|---|---|---|---|
| Aged 60 and over | 10 | 142.333 | 24.311 | 591.04 | 59.104 |
| Aged 18 to 39 | 30 | 115.511 | 10.707 | 114.63 | 3.821 |
The difference in means is $142.333 - 115.511 = 26.822$ mm Hg. Welch's test proceeds in three steps.
Step 1, the standard error. $\sqrt{59.104 + 3.821} = \sqrt{62.925} = 7.933$ mm Hg. Almost all of it comes from the small, variable group.
Step 2, the degrees of freedom.
$$\nu = \frac{(59.104 + 3.821)^2}{59.104^2/9 + 3.821^2/29} = \frac{3959.53}{388.139 + 0.503} = 10.19.$$
That is close to the lower limit of 9, as expected when one group dominates. Rounded down, as in the course text, it is 10.
Step 3, the test and interval. $t = 26.822/7.933 = 3.381$, and $p = 0.0068$ on 10.19 degrees of freedom (0.0070 on 10). The 95% interval is $26.822 \pm 2.2226 \times 7.933 = [9.19, 44.45]$ mm Hg. With 10 degrees of freedom, $t_{10,\,0.975} = 2.2281$ and the interval is $[9.15, 44.50]$.
The pooled test on the same samples gives a different picture. The pooled variance is
$$s_p^2 = \frac{9(591.04) + 29(114.63)}{38} = 227.47, \qquad s_p = 15.082,$$
dominated by the 30 young adults' small variance. The standard error is $15.082\sqrt{1/10 + 1/30} = 5.507$ mm Hg, only 0.69 times Welch's. The pooled test reports $t = 4.870$ on 38 degrees of freedom, $p = 0.00002$, and the interval $[15.67, 37.97]$.
| Test | Standard error | $t$ | df | $p$ | 95% interval |
|---|---|---|---|---|---|
| Pooled | 5.507 | 4.870 | 38 | 0.00002 | $[15.67, 37.97]$ |
| Welch | 7.933 | 3.381 | 10.19 | 0.0068 | $[9.19, 44.45]$ |
Both tests reject, as they should: the true difference is 22.01 mm Hg. But the pooled test claims far more precision than the data contain. The true standard error of the difference in means for this design, computed from the population standard deviations, is $\sqrt{21.02^2/10 + 12.85^2/30} = 7.05$ mm Hg. The pooled formula, even with the population variances plugged in, would give 5.55. So the pooled standard error is too small by construction, not by bad luck in this sample. The next section measures what that does to the error rate.
When pooling fails
The Two-Sample t-Test derives the bias of the pooled estimate of the variance of the difference in means, $(n_1 - n_2)(\sigma_1^2 - \sigma_2^2)(n_1 + n_2 - 1)/[n_1 n_2(n_1 + n_2 - 2)]$. The pooled estimate is unbiased exactly when the variances are equal or the sample sizes are. It is too small on average when the smaller sample comes from the more variable population, and too large when the larger sample does. This section measures what that does to the test.
Example (the direction of the failure). When the smaller sample comes from the more variable population, as with the 10 older and 30 younger adults, the pooled test rejects a true null hypothesis more often than 5%. The large sample's small variance dominates $s_p^2$, so the standard error is too small and $t$ too large. When the larger sample comes from the more variable population, the error runs the other way, and the test is too cautious.
The script measured the Type I error rate, the share of samples in which a test at $\alpha = 0.05$ rejects a true null hypothesis. It used two normal populations with equal means, group 1 having the larger standard deviation, and drew 10,000 samples for each of 30 seeds (medians reported). The last column gives the expected pooled variance of the difference divided by the true variance.
| SD ratio | $n_1$ (larger SD) | $n_2$ | Pooled | Welch | Pooled variance ÷ true |
|---|---|---|---|---|---|
| 1 | 10 | 40 | 0.050 | 0.051 | 1.00 |
| 1.5 | 15 | 15 | 0.051 | 0.049 | 1.00 |
| 1.5 | 10 | 40 | 0.121 | 0.052 | 0.62 |
| 1.5 | 40 | 10 | 0.014 | 0.051 | 1.61 |
| 2 | 15 | 15 | 0.053 | 0.050 | 1.00 |
| 2 | 10 | 40 | 0.181 | 0.051 | 0.46 |
| 2 | 40 | 10 | 0.005 | 0.050 | 2.15 |
| 3 | 15 | 15 | 0.055 | 0.050 | 1.00 |
| 3 | 10 | 40 | 0.257 | 0.051 | 0.34 |
| 3 | 40 | 10 | 0.001 | 0.049 | 2.88 |
Three patterns stand out.
- With equal sample sizes the pooled test is nearly unaffected. It rises only to 0.055 at a threefold difference in standard deviations. Its estimate of the variance of the difference is unbiased then, by the bias formula, and only its degrees of freedom are too generous.
- With unequal sizes it fails badly, in the direction set by which group is more variable. When the small group has three times the standard deviation, a nominal 5% test rejects true null hypotheses a quarter of the time.
- Welch's test holds its level within a point in every case.
These are observations on the ten designs of the table, all with normal populations. The next table repeats four of them on real data, with the same result. The script shifted the NHANES older adults so that their mean equals the younger adults', keeping their spread and skewness, and sampled from the two groups:
| $n$, aged 60+ | $n$, aged 18–39 | Pooled | Welch |
|---|---|---|---|
| 10 | 40 | 0.142 | 0.058 |
| 25 | 25 | 0.052 | 0.051 |
| 40 | 10 | 0.012 | 0.050 |
| 10 | 10 | 0.053 | 0.049 |
With 10 older and 40 younger adults the pooled test's error rate is almost three times its nominal level. Welch's is close to 5%, a little above it because the populations are skewed as well as unequally spread.
So the pooled test kept its level in these simulations only when the variances were equal or the sizes were. The next section shows that the data are a poor guide to which case holds.
Choosing between the tests
The F test and why it is discouraged
The traditional procedure is to test the equality of the variances first, and to pool if that test does not reject. The test usually used is the $F$ test. The NIST handbook (§1.3.5.9) defines it by the statistic $F = s_1^2/s_2^2$. For normal populations with equal variances, $F$ has an $F$ distribution with $n_1 - 1$ and $n_2 - 1$ degrees of freedom. The $F$ distribution is defined in One-Way Analysis of Variance as that of the ratio of two independent chi-squared variables, each over its degrees of freedom. Here the two chi-squared variables are $V_i = (n_i - 1)S_i^2/\sigma^2$ from the pooled proof of The Two-Sample t-Test, and the common $\sigma^2$ cancels:
$$\frac{V_1/(n_1 - 1)}{V_2/(n_2 - 1)} = \frac{S_1^2/\sigma^2}{S_2^2/\sigma^2} = \frac{S_1^2}{S_2^2} = F.$$
The test rejects equality when $F$ falls below the lower $\alpha/2$ point or above the upper one.
- Men and women: $F = 331.27/420.85 = 0.787$ on 19 and 19 degrees of freedom, $p = 0.61$.
- Age groups: $F = 591.04/114.63 = 5.156$ on 9 and 29 degrees of freedom, $p = 0.0007$.
Three problems make this a poor guide to choosing a $t$-test.
It is far more sensitive to non-normality than the test it guards. The $F$ test's null distribution relies on normality much more heavily than the $t$-test's does. The variance of a sample variance depends on the fourth moment of the population, and so on its tails, while the variance of a sample mean depends only on the population variance (Box, 1953). The script drew two samples from the same population, so that the variances were exactly equal, and counted how often the $F$ test at 0.05 rejected.
Example (larger samples do not cure the $F$ test). For the two skewed populations tried here, larger samples made the $F$ test's false alarms more frequent, not less:
| Population | $n = 10$ each | $n = 25$ each | $n = 100$ each |
|---|---|---|---|
| Normal | 0.050 | 0.050 | 0.049 |
| NHANES adults' blood pressure (skewness 1.04) | 0.111 | 0.132 | 0.143 |
| Lognormal | 0.408 | 0.506 | 0.604 |
For the moderately skewed blood pressure population, the $F$ test reports unequal variances in 11 to 14% of comparisons in which they are equal, rising with the sample size. For a strongly skewed population it is wrong most of the time.
Box (1953) contrasted exactly these two kinds of test, tests on means and tests on variances, under non-normal errors. He found the tests on means far more robust. Box later recalled that the paper was the first to use the word "robust" to mean "insensitive to violation of assumptions" (Box, 1982). Of using a test of variances as a preliminary check before an analysis of means, he wrote: "To make the preliminary test on variances is rather like putting to sea in a rowing boat to find out whether conditions are sufficiently calm for an ocean liner to leave port" (Box, 1953, as recalled in Box, 1982). His target was Bartlett's test before an analysis of variance, of which the $F$ test is the two-group counterpart.
It has little power when it matters most. Unequal variances do the most damage with small, unequal samples, which is where the $F$ test is weakest. For normal populations whose standard deviations differ by a factor of 1.5, the $F$ test detected the difference in 20% of samples of 10 and 10, 39% of samples of 10 and 40, and 49% of samples of 25 and 25. In the design where pooling fails worst, it usually gives the go-ahead to pool. Satterthwaite (1946) had already warned that a non-significant $F$ is not evidence that the variances are equal, especially when one of the variances rests on few degrees of freedom.
The two-stage procedure inherits both problems. The script used the $F$ test at 0.05 to choose between the pooled and Welch tests, for samples of 10 and 40 with the larger SD in the small group. The combination had Type I error rates of 0.086 (SD ratio 1.5) and 0.073 (SD ratio 2), against Welch's 0.052 and 0.051. The choice of test depends on the same data as the test itself, and the combination is worse than simply using Welch's test.
Levene's test compares the groups' average absolute deviations from their centers. It is much less sensitive to non-normality, and the NIST handbook (§1.3.5.10) recommends its version based on the median as a good default. It still has low power with small samples, and a two-stage procedure built on it still makes the error rate depend on the outcome of a preliminary test. Delacre, Lakens and Leys (2017) review these objections, reach the same conclusions by simulation, and recommend that Welch's test be used by default. Not every reference agrees: the NIST handbook's comparison of two process means (§7.3.1) advises testing the equality of the standard deviations before pooling. The simulations here side with the recommendation to skip the preliminary test.
What Welch's test costs
If the variances really are equal, the pooled test is exact, and Welch's test approximates it with fewer degrees of freedom, so some power is lost. For normal populations with equal variances and a true difference of one standard deviation:
| Sample sizes | Power, pooled | Power, Welch |
|---|---|---|
| 5 and 5 | 0.285 | 0.264 |
| 10 and 10 | 0.563 | 0.557 |
| 10 and 40 | 0.792 | 0.746 |
With equal samples of 10 or more the difference is under a point. With very small or very unequal samples it is a few points. Delacre and colleagues (2017) reached the same conclusion in their simulations: Welch's test "has practically the same power as Student's t-test, even when SDR = 1", their SDR being the ratio of the standard deviations. Against that, the pooled test's error rate can be several times its nominal level when the variances differ and the sizes are unequal.
A sensible rule follows. Use Welch's test unless there is a reason from outside the data, such as the design of the measurement, to believe the variances are equal. And when the groups can be made equal in size, make them so: that protects the pooled test and costs Welch almost nothing.
Intervals for the difference
Each test has a matching confidence interval, built from the same standard error and degrees of freedom. An analysis should report the interval of the test it uses:
$$\text{pooled: } (\bar{x}_1 - \bar{x}_2) \pm t_{n_1+n_2-2,\,1-\alpha/2}\; s_p\sqrt{\frac{1}{n_1} + \frac{1}{n_2}}, \qquad \text{Welch: } (\bar{x}_1 - \bar{x}_2) \pm t_{\nu,\,1-\alpha/2}\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}.$$
Each interval contains zero exactly when its test fails to reject at level $\alpha$. The interval is for $\mu_1 - \mu_2$ in the order the groups were numbered, and the sign of every number in the report should follow that order. As in the one-sample case, the interval says more than the test. For the age groups, Welch's test says the difference is not zero. Welch's interval says it is plausibly anywhere from 9 to 44 mm Hg, which is what a reader needs to judge whether it matters.
Summary
| Pooled two-sample $t$ | Welch's $t$ | |
|---|---|---|
| Assumes | Independent samples, normal populations, $\sigma_1 = \sigma_2$ | Independent samples, normal populations |
| Standard error | $s_p\sqrt{1/n_1 + 1/n_2}$ | $\sqrt{s_1^2/n_1 + s_2^2/n_2}$ |
| Degrees of freedom | $n_1 + n_2 - 2$ | $\nu$, the Welch–Satterthwaite formula |
| Exact? | Yes, when its assumptions hold | Approximate, accurate in practice |
| When $\sigma_1 \ne \sigma_2$ | Fine with equal $n$; wrong error rates with unequal $n$ | Holds its level |
Both tests rely on normality for exact small-sample error rates. In this article's simulations from the skewed NHANES age groups, both stayed within 0.003 of the 0.05 level when the two samples were the same size (25 and 25, or 10 and 10). Part of the reason can be stated exactly. For independent $A$ and $B$ with means zero, $E[(A - B)^3] = E[A^3] - 3E[A^2]E[B] + 3E[A]E[B^2] - E[B^3] = E[A^3] - E[B^3]$. So the third central moment of $\bar{X}_1 - \bar{X}_2$ is $\mu_{3,1}/n_1^2 - \mu_{3,2}/n_2^2$, where $\mu_{3,i}$ is the third central moment of population $i$ and $\mu_{3,i}/n_i^2$ that of its sample mean. When both populations are skewed in the same direction the two terms partly cancel, and with equal sample sizes they cancel completely if the populations have the same third central moment. The NHANES age groups do not: their third central moments are 2,090 mm Hg$^3$ (aged 18 to 39, skewness 0.99) and 6,976 mm Hg$^3$ (aged 60 and over, skewness 0.75). So the cancellation there is partial. With $n$ in each group, the skewness of the difference in means is $0.33/\sqrt{n}$, against $0.75/\sqrt{n}$ for the older group's mean alone. That is consistent with the small errors in the simulations, but does not by itself prove them. For strongly skewed data or outliers, the Wilcoxon rank-sum test compares two independent groups by ranks.
Worked problems
These three problems are posed here. The first two use the data of an exercise in the course text's chapter on the comparison of two means (Pagano, Gauvreau and Mattie, 2022), which is solved with the pooled test as the worked problem of The Two-Sample t-Test. The third uses the 20 men and 20 women of that article. The script recomputes every number.
Problem 1 (bone mineral content without the equal-variance assumption). A sample of 77 newborns whose mothers smoked during pregnancy has mean bone mineral content $\bar{x}_1 = 0.098$ g/cm and standard deviation $s_1 = 0.026$ g/cm. A sample of 161 newborns whose mothers did not smoke has $\bar{x}_2 = 0.095$ g/cm and $s_2 = 0.025$ g/cm. The pooled test of these data, in The Two-Sample t-Test, gave $t = 0.855$ on 236 degrees of freedom, $p = 0.39$ and the interval $[-0.0039, 0.0099]$ g/cm. Repeat the test and the interval with Welch's test.
Solution. Welch's test gives $t = 0.843$ on $\nu = 144.6$ degrees of freedom, $p = 0.40$, and the interval $[-0.0040, 0.0100]$ g/cm: the same conclusion.
The two terms of the standard error are $s_1^2/n_1 = 0.000676/77 = 8.779 \times 10^{-6}$ and $s_2^2/n_2 = 0.000625/161 = 3.882 \times 10^{-6}$. Their sum is $1.2661 \times 10^{-5}$, so the standard error is $\sqrt{1.2661 \times 10^{-5}} = 0.003558$ and $t = 0.003/0.003558 = 0.843$. The degrees of freedom are
$$\nu = \frac{(1.2661 \times 10^{-5})^2}{\dfrac{(8.779 \times 10^{-6})^2}{76} + \dfrac{(3.882 \times 10^{-6})^2}{160}} = \frac{1.6031 \times 10^{-10}}{1.0141 \times 10^{-12} + 9.419 \times 10^{-14}} = 144.6,$$
between the proved bounds $\min(77, 161) - 1 = 76$ and $77 + 161 - 2 = 236$. Rounded down it is 144. With $t_{144.6,\,0.975} = 1.9765$, the interval is $0.003 \pm 1.9765 \times 0.003558 = [-0.0040, 0.0100]$, and $p = 0.40$ with either 144.6 or 144 degrees of freedom.
The pooled standard error, 0.003509, is slightly smaller than Welch's 0.003558. That is the direction the bias formula predicts: the smaller sample has the larger variance. With sample variances this close, the two tests barely differ.
Problem 2 (testing the variances). Apply the $F$ test for equal variances to the data of Problem 1, two-sided at 0.05.
Solution. $F = 1.082$ on 76 and 160 degrees of freedom, with $p = 0.67$: the test does not reject equal variances.
The statistic is $F = s_1^2/s_2^2 = 0.000676/0.000625 = 1.0816$. Under equal variances and normal populations it has the $F$ distribution with $77 - 1 = 76$ and $161 - 1 = 160$ degrees of freedom. The two-sided test at 0.05 rejects below $F_{76,\,160,\,0.025} = 0.669$ or above $F_{76,\,160,\,0.975} = 1.455$, and 1.0816 lies between them. The $p$-value is twice the smaller tail, 0.67.
By this article's argument, that result is not what justifies pooling. The $F$ test is sensitive to non-normality, and a non-significant $F$ is not evidence of equality. Here the question hardly matters, since Problem 1 shows the two tests agreeing.
Problem 3 (Welch's degrees of freedom with equal sample sizes). Show that when $n_1 = n_2 = n$, Welch's degrees of freedom depend on the data only through the ratio $R = s_1^2/s_2^2$:
$$\nu = (n - 1)\,\frac{(1 + R)^2}{1 + R^2}.$$
Evaluate it for the 20 men and 20 women, and find its largest and smallest possible values.
Solution. For the men and women $\nu = 37.47$. The largest value is $2(n - 1)$, at $R = 1$, and $\nu$ decreases toward $n - 1$ as $R$ goes to zero or to infinity.
With $n_1 = n_2 = n$, every term of the formula carries the same power of $n$:
$$\nu = \frac{(s_1^2/n + s_2^2/n)^2}{\dfrac{(s_1^2/n)^2}{n - 1} + \dfrac{(s_2^2/n)^2}{n - 1}} = (n - 1)\,\frac{(s_1^2 + s_2^2)^2}{s_1^4 + s_2^4}.$$
Dividing the numerator and the denominator by $s_2^4$ gives the stated form. For the men and women, $s_1^2 = 331.27$ and $s_2^2 = 420.85$, so $R = 0.7871$ and
$$\nu = 19 \times \frac{(1 + R)^2}{1 + R^2} = 19 \times \frac{3.1939}{1.6196} = 37.47,$$
the value found in the article (the squares are computed with the unrounded ratio). For the extremes, write $(1 + R)^2/(1 + R^2) = 1 + 2R/(1 + R^2)$. Since $1 + R^2 - 2R = (1 - R)^2 \ge 0$, the fraction $2R/(1 + R^2)$ is at most 1, with equality only at $R = 1$, and it tends to 0 as $R \to 0$ or $R \to \infty$. So $n - 1 < \nu \le 2(n - 1)$, which are the general bounds for equal sizes. With $n = 20$: 38 at $R = 1$, 34.20 at $R = 2$, 31.45 at $R = 2.68$ (the variance ratio of this article's two age groups) and 22.76 at $R = 10$. Equal sample sizes keep the standard error honest whatever the variances are, but unequal variances still cost degrees of freedom.
What we have learned
Without a common variance, Welch's statistic and the Satterthwaite degrees of freedom give an approximate test that held its level in every design simulated here. Two of its steps are approximations: the chi-squared shape given to the estimated variance, which is exact only when the two weights are equal, and the use of the sample variances in the degrees of freedom. Pooling was nearly harmless with equal group sizes and failed badly with unequal ones, rejecting a true null hypothesis a quarter of the time in the worst design simulated here. A preliminary $F$ test does not rescue it, so Welch's test is the sensible default.
Sample Size for Comparing Two Means derives the number of subjects a study needs in each group to detect a given difference, and the best allocation when the two standard deviations differ. Comparing more than two groups at once is the subject of one-way analysis of variance, where Welch's analysis of variance extends this test to $k$ groups.
In Python
import numpy as np
from scipy import stats
def welch_df(s1, n1, s2, n2):
v1, v2 = s1 ** 2 / n1, s2 ** 2 / n2
return (v1 + v2) ** 2 / (v1 ** 2 / (n1 - 1) + v2 ** 2 / (n2 - 1))
print(welch_df(24.311, 10, 10.707, 30)) # about 10.19
# from summary statistics: the older and younger adults above
print(stats.ttest_ind_from_stats(142.333, 24.311, 10, 115.511, 10.707, 30, equal_var=True))
print(stats.ttest_ind_from_stats(142.333, 24.311, 10, 115.511, 10.707, 30, equal_var=False))
# pooled: t about 4.87, p about 2e-05; Welch: t about 3.38, p about 0.0068
# from raw data: res = stats.ttest_ind(x1, x2, equal_var=False)
# res.df is Welch's nu; res.confidence_interval(0.95) is the matching interval
scipy's ttest_ind pools by default (equal_var=True); Welch's test has to be requested. R's t.test does the opposite. Its argument var.equal defaults to FALSE, and the Welch approximation to the degrees of freedom is used unless it is set to TRUE (R Core Team, t.test documentation). Every number in this article is computed by tools/experiments/applied-statistics/two-sample-t-test.py in the site's repository. The script implements the $t$ and $F$ distributions directly and was checked against scipy 1.17.1.
References
- Box, G. E. P. (1953). Non-normality and tests on variances. Biometrika, 40(3–4), 318–335. record · 2 pages
- Box, G. E. P. (1982). This week's citation classic: Non-normality and tests on variances. Current Contents, (4), 18. record
- Delacre, M., Lakens, D., and Leys, C. (2017). Why psychologists should by default use Welch's t-test instead of Student's t-test. International Review of Social Psychology, 30(1), 92–101. record · 2 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 52 pages
- NIST/SEMATECH. e-Handbook of Statistical Methods, §1.3.5.3 (Two-sample t-test for equal means), §1.3.5.9 (F-test for equality of two variances), §1.3.5.10 (Levene test for equality of variances) and §7.3.1 (Do two processes have the same mean?). National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/ record · 32 pages
- Pagano, M., Gauvreau, K., and Mattie, H. (2022). Principles of Biostatistics (3rd ed.). CRC Press. The course text. record · 48 pages
- R Core Team.
t.test: Student's t-Test. Documentation for package stats, R. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/t.test.html record - Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. Biometrics Bulletin, 2(6), 110–114. record
- Welch, B. L. (1947). The generalization of "Student's" problem when several different population variances are involved. Biometrika, 34(1–2), 28–35. record · 2 pages