The Two-Sample t-Test: Pooled and Welch

Comparing the means of two independent groups. The sampling distribution of a difference in means, the pooled two-sample t-test derived from a common variance, Welch's test for unequal variances with the Satterthwaite degrees of freedom derived and checked, intervals for the difference, worked on NHANES blood pressures of men and women and of younger and older adults, simulations of when pooling fails, and why a preliminary F test of the variances is discouraged.

In the NHANES survey of 2017 to 2018, the 2,547 men examined had a mean systolic blood pressure of 127.41 mm Hg and the 2,700 women 125.37 mm Hg. A study that measures 20 men and 20 women will not see those numbers. In one such sample, drawn by this article's script, the men averaged 4.13 mm Hg higher than the women. Each sample mean is off from its population mean by an amount that depends on the spread of blood pressures and the size of the sample. Does a gap of 4.13 mm Hg between two sample means show a real difference between the populations?

Unlike the paired designs of the previous article, the men and women here have nothing to do with each other. No man is matched to a particular woman, and the groups could as well have different sizes. The two samples are independent, and the comparison has to account for the variation in both. The two-sample $t$-test makes that comparison.

Here is the route. First we find the sampling distribution of a difference between two means. Then we derive the classical test, which assumes the two populations share one variance. Next we drop that assumption and derive Welch's test, whose degrees of freedom come from an approximation due to Satterthwaite. Simulations then show when the classical test fails and by how much. Finally we ask how to choose between the two tests, and why the traditional answer, a preliminary $F$ test of the variances, is a poor one.

Two independent samples

The notation follows the course's formula sheet. From a population with mean $\mu_1$ and standard deviation $\sigma_1$, a random sample of size $n_1$ is drawn, with mean $\bar{x}_1$ and standard deviation $s_1$. From a second population, independently, a sample of size $n_2$ is drawn, with mean $\bar{x}_2$ and standard deviation $s_2$. The hypotheses are

$$H_0: \mu_1 = \mu_2 \ \ (\text{equivalently } \mu_1 - \mu_2 = 0), \qquad H_A: \mu_1 \ne \mu_2,$$

or a one-sided alternative such as $H_A: \mu_1 > \mu_2$.

The natural estimate of $\mu_1 - \mu_2$ is $\bar{X}_1 - \bar{X}_2$, and its mean is $\mu_1 - \mu_2$.

Check your understanding. The two means are subtracted. Is the variance of $\bar{X}_1 - \bar{X}_2$ the difference of their variances?

No, it is the sum. Each mean wobbles on its own, and the wobbles do not cancel; they compound. Because the samples are independent,

$$\operatorname{Var}(\bar{X}_1 - \bar{X}_2) = \operatorname{Var}(\bar{X}_1) + \operatorname{Var}(\bar{X}_2) = \frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}.$$

The covariance term that made paired differences precise is zero here, because the samples are independent.

Each sample mean is normal for a normal population. It is approximately normal for large samples from any population, by the central limit theorem. A difference of independent normal variables is normal, so

$$Z = \frac{(\bar{X}_1 - \bar{X}_2) - (\mu_1 - \mu_2)}{\sqrt{\sigma_1^2/n_1 + \sigma_2^2/n_2}}$$

is standard normal. If $\sigma_1$ and $\sigma_2$ were known, this would be the test statistic: the two-sample $z$-test, which appears on the formula sheet for completeness. In practice the standard deviations are estimated, and how to estimate them is where the two versions of the test part ways.

Equal variances: the pooled t-test

Suppose first that the two populations have the same standard deviation, $\sigma_1 = \sigma_2 = \sigma$. The variance of the difference in means becomes

$$\sigma^2\left(\frac{1}{n_1} + \frac{1}{n_2}\right),$$

and only one unknown, $\sigma^2$, has to be estimated. Both samples carry information about it: $s_1^2$ estimates $\sigma^2$ from $n_1 - 1$ degrees of freedom, and $s_2^2$ from $n_2 - 1$. The pooled variance combines them, weighting each by its degrees of freedom:

$$s_p^2 = \frac{(n_1 - 1)s_1^2 + (n_2 - 1)s_2^2}{n_1 + n_2 - 2}.$$

The numerator is the sum of the squared deviations of every observation from its own group's mean. The denominator is the total number of observations less the two means estimated. The weights make sense: a variance computed from 40 observations deserves more say than one computed from 10. Since $E[S_1^2] = E[S_2^2] = \sigma^2$, the weighted average is unbiased, $E[S_p^2] = \sigma^2$.

The t distribution with n1 + n2 - 2 degrees of freedom

The test statistic replaces $\sigma$ by $s_p$:

$$t = \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)}{\sqrt{s_p^2\left(\dfrac{1}{n_1} + \dfrac{1}{n_2}\right)}}.$$

Here is the plan of the derivation. The argument that gave the one-sample $t$ distribution goes through with two samples, once the two variance estimates are combined into one chi-squared variable. The three facts about normal samples, and the definitions of the chi-squared and $t$ distributions, are in Confidence Intervals for a Mean.

The formal version. Applied to two normal populations with a common $\sigma$, the three facts give:

  1. $Z$ above, with $\sigma_1 = \sigma_2 = \sigma$, is standard normal.
  2. $(n_1 - 1)S_1^2/\sigma^2$ and $(n_2 - 1)S_2^2/\sigma^2$ are independent chi-squared variables on $n_1 - 1$ and $n_2 - 1$ degrees of freedom. Independent chi-squared variables add, so $$V = \frac{(n_1 + n_2 - 2)S_p^2}{\sigma^2}$$ is chi-squared with $n_1 + n_2 - 2$ degrees of freedom.
  3. Within each normal sample the mean is independent of the variance, and the samples are independent of each other, so $Z$ and $V$ are independent.

Dividing,

$$\frac{Z}{\sqrt{V/(n_1 + n_2 - 2)}} = \frac{(\bar{X}_1 - \bar{X}_2) - (\mu_1 - \mu_2)}{\sigma\sqrt{1/n_1 + 1/n_2}} \cdot \frac{\sigma}{S_p} = t.$$

The unknown $\sigma$ cancels, and $t$ has the form that defines the $t$ distribution, here with $n_1 + n_2 - 2$ degrees of freedom. This is the pooled two-sample $t$-test. It is exact for normal populations with equal variances at any sample sizes. The NIST/SEMATECH e-Handbook of Statistical Methods (§1.3.5.3) gives it in the same form.

The $100(1-\alpha)\%$ confidence interval for $\mu_1 - \mu_2$ comes from the same pivot:

$$(\bar{x}_1 - \bar{x}_2) \pm t_{n_1+n_2-2,\,1-\alpha/2}\; s_p\sqrt{\frac{1}{n_1} + \frac{1}{n_2}}.$$

As for one mean, the two-sided test at level $\alpha$ rejects $H_0: \mu_1 = \mu_2$ exactly when this interval excludes zero.

Men and women

Back to the 20 men and 20 women. The NHANES population used in this course splits into groups with these spreads:

Population (NHANES 2017–2018 adults) Size Mean (mm Hg) Standard deviation (mm Hg)
Men 2,547 127.41 18.37
Women 2,700 125.37 21.60
Aged 18 to 39 1,745 114.83 12.85
Aged 60 and over 1,923 136.85 21.02

Men's mean is 2.04 mm Hg higher than women's, and women's standard deviation is 1.18 times men's. With equal group sizes, a difference in spread that small does the pooled test little harm, as the simulations in the section on when pooling fails show. The script drew 20 men and 20 women at random (seed 1):

Sample $n$ $\bar{x}$ $s$ $s^2$
Men 20 126.950 18.201 331.27
Women 20 122.817 20.515 420.85

Step 1, the pooled variance. With equal sample sizes it is the plain average of the two variances:

$$s_p^2 = \frac{19(331.27) + 19(420.85)}{38} = 376.06, \qquad s_p = 19.392.$$

Step 2, the standard error. $s_p\sqrt{1/20 + 1/20} = 19.392 \times 0.3162 = 6.132$ mm Hg.

Step 3, the test statistic. The difference in means is $126.950 - 122.817 = 4.133$ mm Hg, so

$$t = \frac{4.133}{6.132} = 0.674$$

on $20 + 20 - 2 = 38$ degrees of freedom, and $p = 2\,P(T_{38} \ge 0.674) = 0.504$.

Step 4, the interval. With $t_{38,\,0.975} = 2.0244$, the 95% interval is $4.133 \pm 2.0244 \times 6.132 = [-8.28, 16.55]$ mm Hg.

The test does not reject, and the interval shows why the result says little. It contains the true difference of 2.04 mm Hg. It also contains every difference from 8 mm Hg in favor of women to 16.5 mm Hg in favor of men. Forty people cannot resolve a difference of two millimeters of mercury between populations whose members vary by twenty. How many people could is the subject of Sample Size for Comparing Two Means.

Men and women have similar spreads. What if two groups clearly do not?

Unequal variances: Welch's t-test

The pooled test needs the two populations to share one variance, and often they do not. Blood pressure varies much more among older adults than among younger ones. The NHANES adults aged 60 and over have a standard deviation of 21.02 mm Hg, and those aged 18 to 39 one of 12.85. The ratio is 1.64, and the variance ratio is 2.68.

Systolic blood pressure of NHANES adults aged 18 to 39 (filled) and aged 60 and over (outline). Notice that the older group is not only centered 22 mm Hg higher but also much more spread out, with a standard deviation 1.64 times as large: a comparison of these two groups should not assume a common variance.
Systolic blood pressure of NHANES adults aged 18 to 39 (filled) and aged 60 and over (outline). Notice that the older group is not only centered 22 mm Hg higher but also much more spread out, with a standard deviation 1.64 times as large: a comparison of these two groups should not assume a common variance.

Without a common variance there is nothing to pool. The natural statistic estimates each variance separately:

$$t = \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)}{\sqrt{s_1^2/n_1 + s_2^2/n_2}}.$$

Its denominator estimates the true standard error correctly whatever the two variances are. The difficulty is its distribution. The square of the denominator is now a weighted sum of two independent chi-squared variables with different, unknown weights, and that sum is not a multiple of a single chi-squared variable. So the ratio is not exactly $t$ with any number of degrees of freedom, and its exact distribution depends on the unknown ratio $\sigma_1^2/\sigma_2^2$.

Testing equality of two normal means without assuming equal variances is known as the Behrens–Fisher problem. The name comes from a solution proposed by Behrens in 1929 and defended by Fisher, which Welch (1947, §5) discusses and rejects. The problem has no solution as simple as the one Student found for one mean.

The Satterthwaite approximation

Here is the idea in one sentence. We cannot make the statistic exactly $t$, so we find the $t$ distribution it most resembles, by choosing the degrees of freedom that make its denominator behave most like a chi-squared variable.

The formal version. Write $V = S_1^2/n_1 + S_2^2/n_2$ for the estimated variance of the difference and $\theta = \sigma_1^2/n_1 + \sigma_2^2/n_2$ for the true one. Suppose $V$ behaved like $\theta$ times a chi-squared variable with $\nu$ degrees of freedom, divided by $\nu$. Then the statistic would be exactly $t$ with $\nu$ degrees of freedom, by the argument of the pooled test. So choose $\nu$ to make $V$ match such a variable in its first two moments.

The mean. $E[V] = \sigma_1^2/n_1 + \sigma_2^2/n_2 = \theta$, and $E[\theta\chi^2_\nu/\nu] = \theta$ for every $\nu$. The means agree automatically.

The variance. A chi-squared variable with $\nu$ degrees of freedom has variance $2\nu$, so $\operatorname{Var}(\theta\chi^2_\nu/\nu) = 2\theta^2/\nu$. For $V$: since $(n_i - 1)S_i^2/\sigma_i^2$ is chi-squared with $n_i - 1$ degrees of freedom, $\operatorname{Var}(S_i^2) = 2\sigma_i^4/(n_i - 1)$. The two samples are independent, so

$$\operatorname{Var}(V) = \frac{2(\sigma_1^2/n_1)^2}{n_1 - 1} + \frac{2(\sigma_2^2/n_2)^2}{n_2 - 1}.$$

Matching $2\theta^2/\nu = \operatorname{Var}(V)$ and solving for $\nu$:

$$\nu = \frac{\left(\dfrac{\sigma_1^2}{n_1} + \dfrac{\sigma_2^2}{n_2}\right)^2}{\dfrac{(\sigma_1^2/n_1)^2}{n_1 - 1} + \dfrac{(\sigma_2^2/n_2)^2}{n_2 - 1}}.$$

The population variances are unknown, so the sample variances are substituted. That gives the degrees of freedom on the formula sheet:

$$\nu = \frac{\left(\dfrac{s_1^2}{n_1} + \dfrac{s_2^2}{n_2}\right)^2}{\dfrac{(s_1^2/n_1)^2}{n_1 - 1} + \dfrac{(s_2^2/n_2)^2}{n_2 - 1}}.$$

The statistic is referred to a $t$ distribution with $\nu$ degrees of freedom. This is Welch's $t$-test, also called the unequal-variance or separate-variance $t$-test.

Who derived it. Welch (1947, §4) obtained the approximation by matching the first two moments, as above. He called $\nu$ the "effective degrees of freedom" and noted that he had used the method in papers of 1936 and 1938. With the population variances, the formula is his exactly. For the version with sample variances he suggested a slightly different estimate: $n_i + 1$ in place of $n_i - 1$ in the denominator, and 2 subtracted from the result, so that its numerator and denominator have the right averages over repeated samples. Satterthwaite (1946) proposed the same moment-matching rule for any linear combination of independent mean squares. He substituted the observed mean squares, as here, and applied the rule to the difference of two means. Software and the formula sheet use the plain substitution, and $\nu$ is often called the Welch–Satterthwaite degrees of freedom. The NIST handbook (§1.3.5.3 and §7.3.1) gives the test and the formula in this form.

Three properties of $\nu$ are worth knowing.

  • It is usually not a whole number. Software uses it as it is, while tables need it rounded. The course's convention rounds down, which is slightly conservative.
  • It always lies between $\min(n_1, n_2) - 1$ and $n_1 + n_2 - 2$. It reaches the upper end when the two terms are in proportion to their degrees of freedom, $(s_1^2/n_1)/(n_1 - 1) = (s_2^2/n_2)/(n_2 - 1)$, as with equal sample sizes and equal sample variances. It falls toward the lower end when one group's term dominates the standard error. The uncertainty about the standard error is then essentially the uncertainty in that one group's variance.
  • With equal sample sizes, Welch's statistic is the pooled one. Since $s_p^2(2/n) = (s_1^2 + s_2^2)/n$, the two standard errors coincide, and only the degrees of freedom differ.

For the men and women above, Welch's test gives the same $t = 0.674$ on $\nu = 37.47$ degrees of freedom, $p = 0.504$, and the interval $[-8.29, 16.55]$.

How good is the approximation? The script checked it for normal populations with the NHANES standard deviations, 21.02 and 12.85 mm Hg, and samples of 10 and 30. With the true standard deviations, $\nu = 11.33$ and the variance of $V$ is 436.25. Across 300,000 simulated pairs of samples the variance of $V$ was 435.98. The upper 2.5% point of Welch's statistic was 2.186, against 2.193 for the $t$ distribution with 11.33 degrees of freedom. The Type I error rates below confirm that the approximation holds its level across a wide range of designs.

Older and younger adults

Back to the age groups. The script drew 10 adults aged 60 and over and 30 aged 18 to 39 (seed 1). In this design the smaller group comes from the more variable population:

Sample $n$ $\bar{x}$ $s$ $s^2$ $s^2/n$
Aged 60 and over 10 142.333 24.311 591.04 59.104
Aged 18 to 39 30 115.511 10.707 114.63 3.821

The difference in means is $142.333 - 115.511 = 26.822$ mm Hg. Welch's test proceeds in three steps.

Step 1, the standard error. $\sqrt{59.104 + 3.821} = \sqrt{62.925} = 7.933$ mm Hg. Almost all of it comes from the small, variable group.

Step 2, the degrees of freedom.

$$\nu = \frac{(59.104 + 3.821)^2}{59.104^2/9 + 3.821^2/29} = \frac{3959.53}{388.139 + 0.503} = 10.19.$$

That is close to the lower limit of 9, as expected when one group dominates. The course's convention rounds down to 10.

Step 3, the test and interval. $t = 26.822/7.933 = 3.381$, and $p = 0.0068$ on 10.19 degrees of freedom (0.0070 on 10). The 95% interval is $26.822 \pm 2.2226 \times 7.933 = [9.19, 44.45]$ mm Hg. With 10 degrees of freedom, $t_{10,\,0.975} = 2.2281$ and the interval is $[9.15, 44.50]$.

The pooled test on the same samples gives a different picture. The pooled variance is

$$s_p^2 = \frac{9(591.04) + 29(114.63)}{38} = 227.47, \qquad s_p = 15.082,$$

dominated by the 30 young adults' small variance. The standard error is $15.082\sqrt{1/10 + 1/30} = 5.507$ mm Hg, only 0.69 times Welch's. The pooled test reports $t = 4.870$ on 38 degrees of freedom, $p = 0.00002$, and the interval $[15.67, 37.97]$.

Test Standard error $t$ df $p$ 95% interval
Pooled 5.507 4.870 38 0.00002 $[15.67, 37.97]$
Welch 7.933 3.381 10.19 0.0068 $[9.19, 44.45]$

Both tests reject, as they should: the true difference is 22.01 mm Hg. But the pooled test claims far more precision than the data contain. The true standard error of the difference in means for this design, computed from the population standard deviations, is $\sqrt{21.02^2/10 + 12.85^2/30} = 7.05$ mm Hg. The pooled formula, even with the population variances plugged in, would give 5.55. So the pooled standard error is too small by construction, not by bad luck in this sample. What does that do to the error rate?

When pooling fails

The pooled standard error weights each sample variance by its degrees of freedom, $n_i - 1$. The true variance of the difference weights each population variance by $1/n_i$. The two weightings agree only when the variances are equal or the sample sizes are.

Check your understanding. Suppose the smaller sample comes from the more variable population, as with the 10 older and 30 younger adults. Will the pooled test reject a true null hypothesis more often than 5%, or less often?

More often. The large sample's small variance dominates $s_p^2$, so the standard error is too small and $t$ too large. When the larger sample comes from the more variable population, the error runs the other way, and the test is too cautious.

The script measured the Type I error rate, the share of samples in which a test at $\alpha = 0.05$ rejects a true null hypothesis. It used two normal populations with equal means, group 1 having the larger standard deviation, and drew 10,000 samples for each of 30 seeds (medians reported). The last column gives the expected pooled variance of the difference divided by the true variance.

SD ratio $n_1$ (larger SD) $n_2$ Pooled Welch Pooled variance ÷ true
1 10 40 0.050 0.051 1.00
1.5 15 15 0.051 0.049 1.00
1.5 10 40 0.121 0.052 0.62
1.5 40 10 0.014 0.051 1.61
2 15 15 0.053 0.050 1.00
2 10 40 0.181 0.051 0.46
2 40 10 0.005 0.050 2.15
3 15 15 0.055 0.050 1.00
3 10 40 0.257 0.051 0.34
3 40 10 0.001 0.049 2.88
Type I error rate at alpha = 0.05 for two normal populations with equal means, samples of 10 and 40, against the ratio of the standard deviation of the group of 10 to that of the group of 40 (log scale). Notice that Welch's test (squares) stays at 0.05 throughout, while the pooled test (circles) is correct only at a ratio of 1: it rejects a true null hypothesis 18% of the time when the small group's SD is twice the large group's, and almost never when it is half.
Type I error rate at alpha = 0.05 for two normal populations with equal means, samples of 10 and 40, against the ratio of the standard deviation of the group of 10 to that of the group of 40 (log scale). Notice that Welch's test (squares) stays at 0.05 throughout, while the pooled test (circles) is correct only at a ratio of 1: it rejects a true null hypothesis 18% of the time when the small group's SD is twice the large group's, and almost never when it is half.

Three patterns stand out.

  • With equal sample sizes the pooled test is nearly unaffected. It rises only to 0.055 at a threefold difference in standard deviations. Its standard error is unbiased then, and only its degrees of freedom are too generous.
  • With unequal sizes it fails badly, in the direction set by which group is more variable. When the small group has three times the standard deviation, a nominal 5% test rejects true null hypotheses a quarter of the time.
  • Welch's test holds its level within a point in every case.

Real data behave the same way. The script shifted the NHANES older adults so that their mean equals the younger adults', keeping their spread and skewness, and sampled from the two groups:

$n$, aged 60+ $n$, aged 18–39 Pooled Welch
10 40 0.142 0.058
25 25 0.052 0.051
40 10 0.012 0.050
10 10 0.053 0.049

With 10 older and 40 younger adults the pooled test's error rate is almost three times its nominal level. Welch's is close to 5%, a little above it because the populations are skewed as well as unequally spread.

So the pooled test is safe only when the variances are equal or the sizes are. Can the data tell us which case we are in?

Choosing between the tests

The F test and why it is discouraged

The traditional procedure is to test the equality of the variances first, and to pool if that test does not reject. The test usually used is the $F$ test. The NIST handbook (§1.3.5.9) defines it by the statistic $F = s_1^2/s_2^2$. For normal populations with equal variances, $F$ has an $F$ distribution with $n_1 - 1$ and $n_2 - 1$ degrees of freedom: the ratio of two independent chi-squared variables, each over its degrees of freedom, as defined in One-Way Analysis of Variance. The test rejects equality when $F$ falls below the lower $\alpha/2$ point or above the upper one.

  • Men and women: $F = 331.27/420.85 = 0.787$ on 19 and 19 degrees of freedom, $p = 0.61$.
  • Age groups: $F = 591.04/114.63 = 5.156$ on 9 and 29 degrees of freedom, $p = 0.0007$.

Three problems make this a poor guide to choosing a $t$-test.

It is far more sensitive to non-normality than the test it guards. The $F$ test's null distribution relies on normality much more heavily than the $t$-test's does, because the variance of a sample variance depends on the tails of the population. The script drew two samples from the same population, so that the variances were exactly equal, and counted how often the $F$ test at 0.05 rejected.

Check your understanding. For a skewed population, do larger samples cure the $F$ test's false alarms?

They make them worse:

Population $n = 10$ each $n = 25$ each $n = 100$ each
Normal 0.050 0.050 0.049
NHANES adults' blood pressure (skewness 1.04) 0.111 0.132 0.143
Lognormal 0.408 0.506 0.604

For the moderately skewed blood pressure population, the $F$ test reports unequal variances in 11 to 14% of comparisons in which they are equal, rising with the sample size. For a strongly skewed population it is wrong most of the time.

Box (1953) contrasted exactly these two kinds of test, tests on means and tests on variances, under non-normal errors. He found the tests on means far more robust. Box later recalled that the paper was the first to use the word "robust" to mean "insensitive to violation of assumptions" (Box, 1982). Of using a test of variances as a preliminary check before an analysis of means, he wrote: "To make the preliminary test on variances is rather like putting to sea in a rowing boat to find out whether conditions are sufficiently calm for an ocean liner to leave port" (Box, 1953, as recalled in Box, 1982). His target was Bartlett's test before an analysis of variance, of which the $F$ test is the two-group counterpart.

It has little power when it matters most. Unequal variances do the most damage with small, unequal samples, which is where the $F$ test is weakest. For normal populations whose standard deviations differ by a factor of 1.5, the $F$ test detected the difference in 20% of samples of 10 and 10, 39% of samples of 10 and 40, and 49% of samples of 25 and 25. In the design where pooling fails worst, it usually gives the go-ahead to pool. Satterthwaite (1946, §4) had already warned that "a non-significant F is not evidence that the variances are equal", especially when one of the variances rests on few degrees of freedom.

The two-stage procedure inherits both problems. The script used the $F$ test at 0.05 to choose between the pooled and Welch tests, for samples of 10 and 40 with the larger SD in the small group. The combination had Type I error rates of 0.086 (SD ratio 1.5) and 0.073 (SD ratio 2), against Welch's 0.052 and 0.051. The choice of test depends on the same data as the test itself, and the combination is worse than simply using Welch's test.

Levene's test compares the groups' average absolute deviations from their centers. It is much less sensitive to non-normality, and the NIST handbook (§1.3.5.10) recommends its version based on the median as a good default. It still has low power with small samples, and a two-stage procedure built on it still makes the error rate depend on the outcome of a preliminary test. Delacre, Lakens and Leys (2017) review these objections, reach the same conclusions by simulation, and recommend that Welch's test be used by default. Not every reference agrees: the NIST handbook's comparison of two process means (§7.3.1) advises testing the equality of the standard deviations before pooling. The simulations here side with the recommendation to skip the preliminary test.

What Welch's test costs

If the variances really are equal, the pooled test is exact, and Welch's test approximates it with fewer degrees of freedom, so some power is lost. For normal populations with equal variances and a true difference of one standard deviation:

Sample sizes Power, pooled Power, Welch
5 and 5 0.285 0.264
10 and 10 0.563 0.557
10 and 40 0.792 0.746

With equal samples of 10 or more the difference is under a point. With very small or very unequal samples it is a few points. Delacre and colleagues (2017) reached the same conclusion in their simulations: Welch's test "has practically the same power as Student's t-test, even when SDR = 1", their SDR being the ratio of the standard deviations. Against that, the pooled test's error rate can be several times its nominal level when the variances differ and the sizes are unequal.

A sensible rule follows. Use Welch's test unless there is a reason from outside the data, such as the design of the measurement, to believe the variances are equal. And when the groups can be made equal in size, make them so: that protects the pooled test and costs Welch almost nothing.

Intervals for the difference

Each test has a matching confidence interval, built from the same standard error and degrees of freedom. An analysis should report the interval of the test it uses:

$$\text{pooled: } (\bar{x}_1 - \bar{x}_2) \pm t_{n_1+n_2-2,\,1-\alpha/2}\; s_p\sqrt{\frac{1}{n_1} + \frac{1}{n_2}}, \qquad \text{Welch: } (\bar{x}_1 - \bar{x}_2) \pm t_{\nu,\,1-\alpha/2}\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}.$$

Each interval contains zero exactly when its test fails to reject at level $\alpha$. The interval is for $\mu_1 - \mu_2$ in the order the groups were numbered, and the sign of every number in the report should follow that order. As in the one-sample case, the interval says more than the test. For the age groups, Welch's test says the difference is not zero. Welch's interval says it is plausibly anywhere from 9 to 44 mm Hg, which is what a reader needs to judge whether it matters.

Summary

Pooled two-sample $t$ Welch's $t$
Assumes Independent samples, normal populations, $\sigma_1 = \sigma_2$ Independent samples, normal populations
Standard error $s_p\sqrt{1/n_1 + 1/n_2}$ $\sqrt{s_1^2/n_1 + s_2^2/n_2}$
Degrees of freedom $n_1 + n_2 - 2$ $\nu$, the Welch–Satterthwaite formula
Exact? Yes, when its assumptions hold Approximate, accurate in practice
When $\sigma_1 \ne \sigma_2$ Fine with equal $n$; wrong error rates with unequal $n$ Holds its level

Both tests rely on normality for exact small-sample error rates. Both are as tolerant of moderate skewness as the one-sample $t$-test, and more so with equal and larger samples. For strongly skewed data or outliers, the Wilcoxon rank-sum test compares two independent groups by ranks.

What we have learned

The variance of a difference between independent means is the sum of the two variances. With a common variance, pooling the two estimates gives an exact $t$ test on $n_1 + n_2 - 2$ degrees of freedom. Without one, Welch's statistic and the Satterthwaite degrees of freedom give an approximate test that held its level in every design simulated here. Pooling is safe with equal group sizes and can fail badly with unequal ones, rejecting a true null hypothesis a quarter of the time in the worst design simulated here. A preliminary $F$ test does not rescue it, so Welch's test is the sensible default. For 20 men and 20 women, both tests gave $p = 0.504$ and an interval too wide to say much.

That last result leaves a planning question: how many subjects would a study need in each group to detect a difference of a few mm Hg? Sample Size for Comparing Two Means answers it. Comparing more than two groups at once is the subject of one-way analysis of variance. Its within-groups variance $s_W^2$ is the pooled variance extended to $k$ groups, and it assumes equal variances in the same way the pooled test does.

In Python

import numpy as np
from scipy import stats

def welch_df(s1, n1, s2, n2):
    v1, v2 = s1 ** 2 / n1, s2 ** 2 / n2
    return (v1 + v2) ** 2 / (v1 ** 2 / (n1 - 1) + v2 ** 2 / (n2 - 1))

print(welch_df(24.311, 10, 10.707, 30))          # about 10.19
# from summary statistics: the older and younger adults above
print(stats.ttest_ind_from_stats(142.333, 24.311, 10, 115.511, 10.707, 30, equal_var=True))
print(stats.ttest_ind_from_stats(142.333, 24.311, 10, 115.511, 10.707, 30, equal_var=False))
# pooled: t about 4.87, p about 2e-05; Welch: t about 3.38, p about 0.0068

# from raw data: res = stats.ttest_ind(x1, x2, equal_var=False)
# res.df is Welch's nu; res.confidence_interval(0.95) is the matching interval

scipy's ttest_ind pools by default (equal_var=True); Welch's test has to be requested. R's t.test does the opposite. Its argument var.equal defaults to FALSE, and the Welch approximation to the degrees of freedom is used unless it is set to TRUE (R Core Team, t.test documentation). Every number in this article is computed by tools/experiments/applied-statistics/two-sample-t-test.py in the site's repository. The script implements the $t$ and $F$ distributions directly and was checked against scipy 1.17.1.

References

  • Box, G. E. P. (1953). Non-normality and tests on variances. Biometrika, 40(3–4), 318–335. record · 2 pages
  • Box, G. E. P. (1982). This week's citation classic: Non-normality and tests on variances. Current Contents, (4), 18. record
  • Delacre, M., Lakens, D., and Leys, C. (2017). Why psychologists should by default use Welch's t-test instead of Student's t-test. International Review of Social Psychology, 30(1), 92–101. record · 2 pages
  • National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
  • NIST/SEMATECH. e-Handbook of Statistical Methods, §1.3.5.3 (Two-sample t-test for equal means), §1.3.5.9 (F-test for equality of two variances), §1.3.5.10 (Levene test for equality of variances) and §7.3.1 (Do two processes have the same mean?). National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/ record · 30 pages
  • R Core Team. t.test: Student's t-Test. Documentation for package stats, R. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/t.test.html record
  • Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. Biometrics Bulletin, 2(6), 110–114. record
  • Welch, B. L. (1947). The generalization of "Student's" problem when several different population variances are involved. Biometrika, 34(1–2), 28–35. record · 2 pages