Comparison of Two Means

Comparing the means of two groups, the most common question in medical research. Paired and independent designs and why the design decides the analysis; the paired t-test on Student's sleep data and repeated NHANES blood pressure readings; the pooled two-sample t-test and Welch's unequal-variance test with Satterthwaite's degrees of freedom, with simulations of when pooling fails and why a preliminary F test is discouraged; and the sample size for comparing two means, including the factor of 2 and the savings from pairing.

Every patient in the sleep trial whose data Student analyzed in 1908 took both drugs, and the previous unit tested the ten differences as one sample. Most studies are not built that way. They compare a treated group with a control group, exposed with unexposed people, or men with women, and neither group's mean is known. Is the gap between two sample means a real difference between the populations, or the luck of who was sampled?

A comparison of two means answers that question, and the answer depends first on how the data were collected. When each subject is measured twice, or subjects come in matched pairs, the comparison reduces to one sample of differences, and the variation between subjects drops out. When the two groups are independent, the variation in both groups enters the comparison. The test then depends on whether the two populations can be assumed to share a variance.

This unit takes the two designs in turn, then plans a study, in three steps.

  1. What does pairing buy, and what does ignoring it cost? The variance of a difference shows where the gain comes from. On Student's data, the same numbers give $p = 0.0028$ analyzed as designed and $p = 0.079$ with the pairing ignored.
  2. How do we compare two independent groups? The pooled $t$-test assumes a common variance; Welch's test does not. Simulations show when pooling fails, and why testing the variances first does not fix it.
  3. How many subjects does a two-group study need? About four times as many as a one-sample study for independent groups, and often a small fraction of that for a paired design.

The examples use systolic blood pressures from the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018: men and women, younger and older adults, and the repeated readings taken from each participant, which form a natural paired design with a known truth. Student's sleep data are the classic paired example. The notation follows the course's formula sheets, including $s_p^2$ for the pooled variance and $\delta = \mu_1 - \mu_2$ for the mean difference of paired data. Every number comes from a script in the site's repository.

Articles

  • The Paired Samples t-Test: why does measuring each subject twice help so much? Paired and independent designs, and the variance of a difference, which shows where the gain from pairing comes from. The paired $t$-test as a one-sample test on the differences, with its interval and one-sided bound. Student's sleep data analyzed as designed and with the pairing ignored ($p = 0.0028$ against $p = 0.079$), repeated NHANES readings, and the correlation below which pairing loses.
  • The Two-Sample t-Test: Pooled and Welch: how do we compare two groups that have nothing to do with each other? The sampling distribution of a difference in means, the pooled variance and the $t$ distribution with $n_1 + n_2 - 2$ degrees of freedom. Welch's statistic, with the Satterthwaite degrees of freedom derived by matching moments (Welch, 1947). Worked examples on men and women and on younger and older adults, Type I error rates of both tests by simulation, the $F$ test for equal variances and the reasons it is discouraged, and intervals for the difference.
  • Sample Size for Comparing Two Means: how many subjects does a two-group study need? The per-group formula derived from the power of the two-sample test, the factor of 2, the exact noncentral $t$ answer and a simulation check. Then the optimal allocation when standard deviations differ, the cost of unequal group sizes, and how much a crossover design saves over parallel groups.

The unit builds on Hypothesis Testing. Hypothesis Tests for a Mean gives the one-sample $t$-test that a paired test applies to differences. Power and Sample Size for a Test of a Mean gives the power function and the noncentral $t$ distribution. The $t$ distribution itself is derived in Confidence Intervals for a Mean.

Comparing more than two groups is the next unit, Analysis of Variance, whose within-groups variance generalizes the pooled variance. For skewed data or outliers, the sign and signed-rank tests replace the paired $t$-test, and the Wilcoxon rank-sum test replaces the two-sample test. Two proportions are compared in Comparing Two Proportions. In information retrieval, Significance Testing applies paired tests to comparing two search engines over the same queries.

References

  • Box, G. E. P. (1953). Non-normality and tests on variances. Biometrika, 40(3–4), 318–335. record · 2 pages
  • Delacre, M., Lakens, D., and Leys, C. (2017). Why psychologists should by default use Welch's t-test instead of Student's t-test. International Review of Social Psychology, 30(1), 92–101. record · 2 pages
  • NIST/SEMATECH. e-Handbook of Statistical Methods, §1.3.5.3 (Two-sample t-test for equal means) and §7.3.1 (Do two processes have the same mean?). National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/ record · 30 pages
  • Pagano, M., Gauvreau, K., and Mattie, H. (2022). Principles of Biostatistics (3rd ed.). CRC Press. The course text. record · 35 pages
  • Student (1908). The probable error of a mean. Biometrika, 6(1), 1–25. record · 11 pages
  • Welch, B. L. (1947). The generalization of "Student's" problem when several different population variances are involved. Biometrika, 34(1–2), 28–35. record · 2 pages