Sample Size for Comparing Two Means

How many subjects a study comparing two means needs. The per-group sample size formula derived from the power of the two-sample test, the factor of 2 that makes a two-arm trial four times the size of a one-sample study, the exact noncentral t answer and a simulation check on NHANES blood pressures, unequal standard deviations and the optimal allocation, the cost of unequal group sizes, how much a paired or crossover design saves, and what moves the answer.

Power and Sample Size for a Test of a Mean planned a study of adults aged 60 and over that would compare their mean systolic blood pressure with a known value, the mean of all adults. It found that 141 people give an 80% chance of detecting a difference of 5 mm Hg. Most clinical questions are not like that. A trial of a blood pressure drug compares a treated group with a control group, and neither group's mean is known in advance. Both have to be estimated, and both estimates carry error. How many older adults would such a trial need for the same 5 mm Hg?

About four times as many. For two independent groups with equal standard deviations, each group needs about twice as many subjects as the one-sample study needs in total: 279 per group, 558 in all, against 141. This article derives that answer and the choices that change it.

Here is the route. First we derive the per-group formula from the power of the two-sample test, and see where the factor of 2 comes from. Then we plan a two-arm trial, correct the answer for the $t$ distribution and check it by simulation from NHANES. Next we turn to the choices that move the answer: unequal standard deviations, unequal group sizes, and pairing. The article assumes the power function, the noncentral $t$ distribution and the one-sample formula of Power and Sample Size for a Test of a Mean, and uses them without re-deriving them. The formulas are those on the course's formula sheet, written in its notation.

The power of a two-sample test

Here is the idea in one sentence. The one-sample power calculation carries over unchanged, except that the standard error is now that of a difference between two means.

Consider two independent groups of $n$ subjects each, from populations with means $\mu_1$ and $\mu_2$ and standard deviations $\sigma_1$ and $\sigma_2$, and the two-sided test of $H_0: \mu_1 = \mu_2$ at level $\alpha$. As in the one-sample case, we start with the $z$ version, as if the standard deviations were known. From The Two-Sample t-Test, the difference in sample means has standard error

$$SE = \sqrt{\frac{\sigma_1^2}{n} + \frac{\sigma_2^2}{n}} = \frac{\sqrt{\sigma_1^2 + \sigma_2^2}}{\sqrt{n}},$$

and the test rejects when $|z| \ge z_{1-\alpha/2}$, where $z = (\bar{x}_1 - \bar{x}_2)/SE$.

The one-sample article showed that the power of a $z$ test depends on the design only through $\lambda$, the true difference measured in standard errors. Its power function, $\Phi(\lambda - z_{1-\alpha/2}) + \Phi(-\lambda - z_{1-\alpha/2})$, applies here as it stands. Only $\lambda$ is new:

$$\lambda = \frac{\mu_1 - \mu_2}{SE} = \frac{(\mu_1 - \mu_2)\sqrt{n}}{\sqrt{\sigma_1^2 + \sigma_2^2}}.$$

Solving for a target power exactly as there, by dropping the negligible chance of rejecting in the wrong direction and setting $\lambda - z_{1-\alpha/2} = z_{1-\beta}$, gives

$$\frac{(\mu_1 - \mu_2)\sqrt{n}}{\sqrt{\sigma_1^2 + \sigma_2^2}} = z_{1-\alpha/2} + z_{1-\beta} \quad\Longrightarrow\quad n = \left(\frac{z_{1-\alpha/2} + z_{1-\beta}}{\mu_1 - \mu_2}\right)^2 (\sigma_1^2 + \sigma_2^2),$$

rounded up. This is the formula on the course's formula sheet. Note that $n$ is the number of subjects in each group, so the study enrolls $2n$. The standard deviations need not be equal. For a one-sided test, $z_{1-\alpha}$ replaces $z_{1-\alpha/2}$.

The factor of 2

When the two populations have the same standard deviation $\sigma$, $\sigma_1^2 + \sigma_2^2 = 2\sigma^2$ and

$$n = 2\left[\frac{(z_{1-\alpha/2} + z_{1-\beta})\,\sigma}{\mu_1 - \mu_2}\right]^2 \text{ per group}.$$

The bracket is the one-sample formula. So each group needs twice the one-sample study's total, and the whole study four times as many subjects. The two factors of 2 have separate causes.

  • Both means are estimated. The variance of $\bar{x}_1 - \bar{x}_2$ is $2\sigma^2/n$, twice the variance of a single mean compared with a known value.
  • There are two groups to fill.

A one-sample design against a known value is cheaper because it borrows the comparison value from outside the study. That is legitimate only when the value really is known, as the all-adult NHANES mean was in the earlier article. When the comparison has to be measured, so does its uncertainty.

For 80% power at $\alpha = 0.05$, $2(z_{0.975} + z_{0.80})^2 = 2(1.9600 + 0.8416)^2 = 15.70$, so roughly

$$n \approx 16\left(\frac{\sigma}{\mu_1 - \mu_2}\right)^2 \text{ per group},$$

a rule of thumb worth remembering. For 90% power the multiplier is $2(1.9600 + 1.2816)^2 = 21.01$.

Planning a two-arm trial

Back to the older adults. Suppose a trial in adults aged 60 and over will randomize participants to a new treatment or to usual care. It wants an 80% chance of detecting a 5 mm Hg difference in mean systolic pressure with a two-sided test at 0.05. For planning, both arms are given the standard deviation of the NHANES adults aged 60 and over, $\sigma = 21.024$ mm Hg.

Check your understanding. The one-sample study, for the same difference and standard deviation, had a $z$-formula answer of 138.77. What will the two-group formula give per group?

Exactly twice as much, since the bracket is the same:

$$n = \left(\frac{1.9600 + 0.8416}{5}\right)^2 (21.024^2 + 21.024^2) = 277.54, \quad\text{so } 278 \text{ per group},$$

and 556 in all, a ratio to 138.77 of exactly 2.000 per group. For 90% power, $z_{0.90} = 1.2816$ gives 371.55, so 372 per group.

Correcting for t

The trial will be analyzed with the pooled two-sample $t$-test, so the $z$ answer needs the same check as in the one-sample case. Under the alternative, the pooled statistic has a noncentral $t$ distribution, as the one-sample statistic did, with two changes. It has $2n - 2$ degrees of freedom, and its noncentrality is $\lambda = (\mu_1 - \mu_2)/(\sigma\sqrt{2/n})$. The power is the noncentral-$t$ tail sum of Power and Sample Size for a Test of a Mean, with $t_{2n-2,\,1-\alpha/2}$ as the critical value:

$$1 - \beta = P(T' \ge t_{2n-2,\,1-\alpha/2}) + P(T' \le -t_{2n-2,\,1-\alpha/2}),$$

with $T'$ noncentral $t$ as scipy defines it (scipy.stats.nct).

Check your understanding. For one sample, the $t$ correction added two subjects (139 to 141). Will it add more or fewer per group here?

Fewer. With $2n - 2 = 554$ degrees of freedom instead of 138, the $t$ distribution is much closer to the normal. At 278 per group the power is 0.7993, a hair short of 0.80, and the smallest $n$ that reaches 0.80 is 279 (power 0.8007). The NIST/SEMATECH e-Handbook of Statistical Methods (§7.2.2.2) suggests iterating with $t$ quantiles for the one-sample case. Doing so here, $t_{554,\,0.975} = 1.9643$ and $t_{554,\,0.80} = 0.8423$ give 278.52, so 279, and the iteration stays there. For 90% power the exact answer is 373 per group. Across the table below, the exact answer is the $z$ formula's plus one.

A simulation checks the plan against real data. The script drew two samples of 279 from the NHANES older adults, added 5 mm Hg to every value in one of them, and applied the pooled $t$-test, 10,000 pairs of samples for each of 30 seeds. The power was 0.801 (median; interquartile range 0.799 to 0.803). The skewness of real blood pressures makes no practical difference at this size.

Power to detect a 5 mm Hg difference with a two-sided t-test at alpha = 0.05, sigma = 21.0 mm Hg, against sample size. Solid: one sample compared with a known mean, which reaches 80% power at n = 141. Dashed: two independent groups of n each, which need 279 per group. Notice the gap at 139 per group, the size that sufficed for the one-sample design: there the two-group trial has power of only about 0.51.
Power to detect a 5 mm Hg difference with a two-sided t-test at alpha = 0.05, sigma = 21.0 mm Hg, against sample size. Solid: one sample compared with a known mean, which reaches 80% power at n = 141. Dashed: two independent groups of n each, which need 279 per group. Notice the gap at 139 per group, the size that sufficed for the one-sample design: there the two-group trial has power of only about 0.51.

A study that sized each arm by the one-sample formula, 139 per group, would have power 0.506 (0.508 in the simulation). That is a coin toss for a difference it was designed to detect.

A table

Sample size per group for the two-sided pooled $t$-test at $\alpha = 0.05$, by the standardized difference $(\mu_1 - \mu_2)/\sigma$:

$(\mu_1 - \mu_2)/\sigma$ 80% power, $z$ formula exact $t$ 90% power, $z$ formula exact $t$
0.20 393 394 526 527
0.25 252 253 337 338
0.50 63 64 85 86
0.75 28 29 38 39
1.00 16 17 22 23

The total study is twice each entry. Compared with the one-sample table of the earlier article (8 at a standardized difference of 1, 197 at 0.2, for 80% power), each entry is about double.

So far both groups were alike. What changes when they are not?

Unequal standard deviations

The sheet's formula allows $\sigma_1 \ne \sigma_2$. Suppose a study will compare an outcome between a younger and an older group whose spreads are like those of systolic pressure in NHANES. Then $\sigma_1 = 21.024$ mm Hg for adults aged 60 and over and $\sigma_2 = 12.849$ for those aged 18 to 39. The study wants 80% power to detect a 5 mm Hg difference. With equal groups,

$$n = \left(\frac{1.9600 + 0.8416}{5}\right)^2 (21.024^2 + 12.849^2) = 190.60, \quad\text{so } 191 \text{ per group, 382 in all}.$$

The analysis would use Welch's test, since the variances are not assumed equal. Simulating it from the two NHANES groups, with the older group's values shifted so that the means differ by exactly 5 mm Hg, gave power 0.805.

The optimal allocation

Equal groups are not the most efficient design when the standard deviations differ.

Check your understanding. With 382 subjects to place, should the more variable group get more than half of them, or fewer?

More. The derivation below shows how many more.

Here is the plan of the derivation. The power depends on the sample sizes only through the standard error of the difference, $\sqrt{\sigma_1^2/n_1 + \sigma_2^2/n_2}$. So the question is how to make that standard error a given size $\sqrt{V}$ with the fewest subjects.

The formal version. Minimize $n_1 + n_2$ subject to

$$\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2} = V.$$

With a Lagrange multiplier $\kappa$, set the derivatives of $n_1 + n_2 + \kappa(\sigma_1^2/n_1 + \sigma_2^2/n_2 - V)$ with respect to $n_1$ and $n_2$ to zero. That gives $1 = \kappa\sigma_i^2/n_i^2$, so $n_i = \sqrt{\kappa}\,\sigma_i$: each group's size should be proportional to its standard deviation,

$$\frac{n_1}{n_2} = \frac{\sigma_1}{\sigma_2}.$$

Substituting back, the constraint gives $\sqrt{\kappa} = (\sigma_1 + \sigma_2)/V$. So the minimum total is $(\sigma_1 + \sigma_2)^2/V$, against $2(\sigma_1^2 + \sigma_2^2)/V$ for equal groups. The ratio of the two,

$$\frac{2(\sigma_1^2 + \sigma_2^2)}{(\sigma_1 + \sigma_2)^2},$$

is at least 1, with equality only when $\sigma_1 = \sigma_2$, since $2(a^2 + b^2) - (a + b)^2 = (a - b)^2$.

For the age groups, $\sigma_1/\sigma_2 = 1.636$. With $V$ set by the target power, $V = [5/(z_{0.975} + z_{0.80})]^2$, the optimal sizes are $n_1 = 223.58$ older and $n_2 = 136.64$ younger adults. They round up to 224 and 137, a total of 361. Equal groups need $382/361 = 1.058$ times as many, about 6% more. The simulated Welch power at the optimal allocation was 0.800.

The saving is modest unless the standard deviations differ a great deal. It is also the reason to give a more variable group more subjects rather than fewer.

Unequal group sizes

Even with equal standard deviations, studies sometimes enroll more subjects in one group than the other. A trial may want more experience with a new treatment; one group may be cheaper to recruit; or the groups may be found rather than assigned, as in the comparison of men and women. What does the imbalance cost?

Suppose the larger group is $k$ times the smaller, with $n$ in the smaller. The variance of the difference is $\sigma^2(1/n + 1/(kn))$. Setting it equal to the variance $2\sigma^2/m$ of a balanced design with $m$ per group gives $n = m(1 + k)/(2k)$. The total, $n(1 + k)$, is

$$\frac{(1 + k)^2}{4k} \times 2m,$$

the balanced total inflated by the factor $(1 + k)^2/(4k)$.

Check your understanding. A 2:1 design is common in trials. How much larger must its total be than a balanced design's?

Only 12.5% larger, since $(1 + 2)^2/(4 \times 2) = 1.125$. The table shows how the cost grows:

Allocation $k:1$ 1:1 1.5:1 2:1 3:1 4:1 9:1
Total sample ÷ balanced total 1.000 1.042 1.125 1.333 1.563 2.778
The total number of subjects needed for the same power, relative to equal groups, when one group is k times the size of the other and the standard deviations are equal. Notice how flat the curve starts: mild imbalance costs little (2:1 needs 12.5% more subjects), but the cost grows steadily, so that 4:1 needs 56% more and 9:1 (beyond the plotted range) nearly three times as many.
The total number of subjects needed for the same power, relative to equal groups, when one group is k times the size of the other and the standard deviations are equal. Notice how flat the curve starts: mild imbalance costs little (2:1 needs 12.5% more subjects), but the cost grows steadily, so that 4:1 needs 56% more and 9:1 (beyond the plotted range) nearly three times as many.

For the trial in older adults, a 2:1 allocation by the $z$ formula needs a smaller arm of 208.16, so 209, with 418 in the larger arm, and the exact $t$ calculation confirms 209 (power 0.8004). The total, 627, is 1.124 times the balanced 558.

Mild imbalance is cheap, which is why 2:1 trials are common. But the factor grows without limit, because the smaller group's variance always remains in the standard error, and adding subjects to the larger group cannot remove it.

Unequal groups cost subjects. Can a design save them instead?

Paired designs need far fewer subjects

For a paired design, such as a crossover trial in which every patient receives both treatments, the comparison is a one-sample test on the differences (The Paired Samples t-Test). So the number of pairs comes from the one-sample formula with the standard deviation of the differences, $\sigma_d$, in place of $\sigma$:

$$n = \left[\frac{(z_{1-\alpha/2} + z_{1-\beta})\,\sigma_d}{\delta}\right]^2 \text{ pairs},$$

where $\delta$ is the mean difference to be detected. What is new here is the comparison with a parallel design. Suppose the two measurements on a subject each have standard deviation $\sigma$ and correlation $\rho$. The variance identity of the paired article gives $\sigma_d^2 = 2\sigma^2(1 - \rho)$, so

$$n_{\text{pairs}} = 2\left[\frac{(z_{1-\alpha/2} + z_{1-\beta})\,\sigma}{\delta}\right]^2(1 - \rho) = (1 - \rho) \times n_{\text{per group}}.$$

A paired study needs $(1 - \rho)$ times as many subjects as one arm of a parallel-group study, and so $(1 - \rho)/2$ times the parallel study's total.

  • When $\rho = 0$, it needs half as many subjects, because each subject provides two measurements.
  • When $\rho = 0.5$, it needs a quarter as many.
  • When $\rho = 0.8$, it needs a tenth as many.

Student's sleep data show what this means in practice. The crossover trial planned in the one-sample article, for 0.5 hours at 80% power with $\sigma_d = 1.23$ hours from Student's ten differences (Student, 1908), needed 50 patients (48 by the $z$ formula).

A parallel-group trial of the same question, with each drug given to a different group, faces the variation between patients in full. The standard deviations of the two drugs' values in Student's table are 1.789 and 2.002 hours, with pooled value $s_p = 1.899$, and

$$n = \left(\frac{1.9600 + 0.8416}{0.5}\right)^2 \times 2 \times 1.899^2 = 226.35, \quad\text{so } 227 \text{ per group}.$$

The exact $t$ calculation gives 228 per group: 456 patients against 50, more than nine times as many. The correlation between the drugs in Student's data, 0.80, accounts for the saving: $(1 - 0.80)/2 = 0.10$, close to the actual ratio of 50 to 456. The identity holds only approximately, because the two drugs' standard deviations are not quite equal.

The saving rests on assumptions that the parallel design does not need. The crossover has to be possible at all, which requires a condition that is stable over the course of the trial, and treatments whose effects wear off between periods. And the planning value of $\rho$, or of $\sigma_d$, comes from earlier data that may be as small as Student's ten patients. When those conditions hold, pairing is the single most effective way to reduce the size of a study.

What moves the sample size

As in the one-sample case, the standard deviation and the difference to be detected enter squared and dominate. The baseline is the trial in older adults with 279 per group (80% power for 5 mm Hg, two-sided at 0.05):

Change Effect
One-sided test at 0.05 instead of two-sided Power at 279 rises to 0.877; 220 per group suffice for 80%
$\alpha = 0.01$ instead of 0.05 Power at 279 falls to 0.589; 80% needs 415 per group
True $\sigma$ 10% larger than planned Power falls to 0.722
True $\sigma$ 25% larger than planned Power falls to 0.612
True difference 2.5 mm Hg instead of 5 Power falls to 0.289; 80% needs 1,112 per group
15% of participants expected to drop out Recruit $279/(1 - 0.15) = 328.2$, so 329 per group

The power figures match the one-sample article's almost exactly. Both baselines were sized for 80% power, so they start from nearly the same $\lambda$. A change in $\sigma$ or in the difference rescales $\lambda$ by the same factor in both, and a change in $\alpha$ or in the number of sides moves the critical value by almost the same amount. The cautions carry over unchanged:

  • the difference $\mu_1 - \mu_2$ should be the smallest one that would matter, not the one the investigators hope for;
  • the planning standard deviation is itself an estimate, and a cautious, larger value protects the power;
  • a one-sided test is a choice for the protocol, not for the analysis, as One-Sided Confidence Bounds and Sample Size explains;
  • power is a property of the design, and computing it afterwards from the observed difference adds nothing to the $p$-value, as Power and Sample Size for a Test of a Mean showed.

Summary

Design Sample size for power $1 - \beta$, two-sided at $\alpha$
One sample against a known mean $n = [(z_{1-\alpha/2} + z_{1-\beta})\,\sigma/(\mu_1 - \mu_0)]^2$
Paired (number of pairs) $n = [(z_{1-\alpha/2} + z_{1-\beta})\,\sigma_d/\delta]^2$
Two independent groups (each group) $n = [(z_{1-\alpha/2} + z_{1-\beta})/(\mu_1 - \mu_2)]^2(\sigma_1^2 + \sigma_2^2)$
Two groups, equal $\sigma$ $n = 2[(z_{1-\alpha/2} + z_{1-\beta})\,\sigma/(\mu_1 - \mu_2)]^2$; about $16(\sigma/(\mu_1-\mu_2))^2$ for 80% power
Optimal allocation, unequal $\sigma$ $n_1/n_2 = \sigma_1/\sigma_2$
Allocation $k:1$, equal $\sigma$ total inflated by $(1 + k)^2/(4k)$

Always round up, and check the $z$ answer against the exact $t$ power, which for two groups usually adds one subject per group. The same reasoning extends to proportions, where the variance of each group depends on its own proportion, in Comparing Two Proportions.

What we have learned

Comparing two independent groups doubles the variance of the estimate, so each group needs twice the one-sample total: 279 per group for the older-adult trial, against 141 for one sample. The $t$ correction adds about one per group. When the standard deviations differ, groups sized in proportion to them save a little, about 6% for the age groups. Unequal allocation costs little at 2:1 and grows without limit beyond. Pairing saves the most: Student's crossover trial needs 50 patients where a parallel trial needs 456.

This closes the unit on two means. Many studies compare three or more groups: several doses, several diets, several age bands. Running a two-sample test on every pair inflates the Type I error rate, and the pooled variance needs extending to $k$ groups. The next topic, Analysis of Variance, does both, starting with One-Way Analysis of Variance.

In Python

import math
from statistics import NormalDist
from scipy import stats

def two_sample_power(d_over_sigma, n1, n2, alpha=0.05):
    """Power of the two-sided pooled two-sample t test (normal populations, common sigma)."""
    df = n1 + n2 - 2
    nc = d_over_sigma / math.sqrt(1 / n1 + 1 / n2)
    c = stats.t.ppf(1 - alpha / 2, df)
    return stats.nct.sf(c, df, nc) + stats.nct.cdf(-c, df, nc)

z = NormalDist().inv_cdf
d = 5 / 21.024
n = math.ceil(2 * ((z(0.975) + z(0.80)) / d) ** 2)
print(n, two_sample_power(d, n, n))     # 278, about 0.7993
while two_sample_power(d, n, n) < 0.80:
    n += 1
print(n, two_sample_power(d, n, n))     # 279, about 0.8007

Every number in this article is computed by tools/experiments/applied-statistics/sample-size-for-comparing-two-means.py in the site's repository. The script implements the noncentral $t$ distribution directly and was checked against scipy 1.17.1.

References