Power and Sample Size for a Test of a Mean

How likely a test is to detect a difference that is really there, and how many observations it needs to do so reliably. The power function of the z test derived and drawn as power curves, the noncentral t distribution for the t test, the sample size formula derived from the power function and checked against the exact t calculation and simulation on NHANES blood pressures and Student's sleep data, what moves the answer, and why power computed from the observed data adds nothing to the p-value.

A study of 25 adults aged 60 and over, in Hypothesis Tests for a Mean, tested whether their mean systolic blood pressure differed from the all-adult mean. It found $p = 0.081$, although the true difference in the NHANES population is 10.5 mm Hg. Type I and Type II Errors measured how often a study that size misses a difference that size: about 31% of the time. A study that misses a real effect nearly a third of the time is a poor use of its participants. How many older adults would the study have needed, and could that have been worked out before anyone was measured?

It could. The power of a test is the probability that it rejects the null hypothesis when a particular alternative is true. It can be computed in advance from the planned sample size, and the sample size can then be chosen to make the power as high as the study needs. This article is the course's home for the power function, the noncentral $t$ distribution and the sample size formula for a test. Sample Size for Comparing Two Means builds on them.

Here is the route. First we derive the power of the $z$ test and draw power curves. Then we extend it to the $t$ test through the noncentral $t$ distribution. Next we turn power around into the sample size formula, checked against the exact calculation and simulation. Finally we look at what moves the answer, and at power computed after the study. The article assumes the error rates of Type I and Type II Errors.

The power function of the z test

Here is the idea in one sentence. If the true mean is not $\mu_0$, the test statistic's distribution slides sideways, and the power is the share of it that lands beyond the critical values.

The formal version. Consider the two-sided $z$ test of $H_0: \mu = \mu_0$ against $H_A: \mu \ne \mu_0$ at level $\alpha$, with $\sigma$ known. It rejects when $|z| \ge z_{1-\alpha/2}$, where

$$z = \frac{\bar{x} - \mu_0}{\sigma/\sqrt{n}}.$$

Suppose the true mean is $\mu_1 = \mu_0 + \delta$. Then $\bar{X}$ is centered on $\mu_1$, not $\mu_0$, and the statistic can be split into a random part and a fixed shift:

$$Z = \frac{\bar{X} - \mu_1}{\sigma/\sqrt{n}} + \frac{\mu_1 - \mu_0}{\sigma/\sqrt{n}} = Z_0 + \lambda, \qquad \lambda = \frac{\delta\sqrt{n}}{\sigma},$$

where $Z_0$ has a standard normal distribution. So under the alternative, $Z$ is normal with mean $\lambda$ and standard deviation 1: the null distribution shifted by $\lambda$ standard errors. The power is the probability that this shifted variable lands in the rejection region:

$$1 - \beta = P(Z \ge z_{1-\alpha/2}) + P(Z \le -z_{1-\alpha/2}) = \Phi(\lambda - z_{1-\alpha/2}) + \Phi(-\lambda - z_{1-\alpha/2}),$$

where $\Phi$ is the standard normal distribution function. A one-sided test of $H_A: \mu > \mu_0$ rejects when $z \ge z_{1-\alpha}$, so only the first kind of term appears: $1 - \beta = \Phi(\lambda - z_{1-\alpha})$.

The quantity $\lambda$ carries everything that matters. The power depends on the difference $\delta$, the standard deviation $\sigma$ and the sample size $n$ only through $\delta\sqrt{n}/\sigma$, the true difference measured in standard errors of the mean.

Check your understanding. Which raises the power more: doubling the sample size, or doubling the difference to be detected?

Doubling the difference. It doubles $\lambda$, while doubling $n$ multiplies $\lambda$ only by $\sqrt{2} \approx 1.41$. For the older adults below, going from 5 to 10 observations moves $\lambda$ from 1.116 to 1.578. Doubling $\delta$ has the same effect as quadrupling $n$.

A worked case

Back to the older adults. The null value is $\mu_0 = 126.36$, the all-adult mean, and the true mean is $\mu_1 = 136.85$, the mean of the 1,923 adults aged 60 and over. So $\delta = 10.49$ mm Hg, and their standard deviation is $\sigma = 21.02$ mm Hg. With $n = 10$,

$$\lambda = \frac{10.49\sqrt{10}}{21.02} = 1.578,$$

$$1 - \beta = \Phi(1.578 - 1.960) + \Phi(-1.578 - 1.960) = 0.3511 + 0.0002 = 0.3513.$$

The second term is the chance of rejecting in the wrong direction, concluding that older adults have lower pressure than average. It is tiny here and becomes tinier as $\lambda$ grows: 0.000004 at $n = 25$.

Power curves

Computed for every possible true mean, the power is a curve, the power function of the test. The figure shows it for the $t$ test (derived in the next section) at three sample sizes.

Power of the two-sided t test at alpha = 0.05 against the true difference between the population mean and the hypothesized mean, with sigma = 21.0 mm Hg, for samples of 10, 25 and 50. Notice that at a true difference of zero the power equals alpha, 0.05 (dotted line): rejection there is a Type I error. The curves rise with the size of the difference and steepen with n. At the 10.5 mm Hg difference of adults aged 60 and over (vertical line), power is 0.29 with 10 observations, 0.67 with 25 and 0.93 with 50.
Power of the two-sided t test at alpha = 0.05 against the true difference between the population mean and the hypothesized mean, with sigma = 21.0 mm Hg, for samples of 10, 25 and 50. Notice that at a true difference of zero the power equals alpha, 0.05 (dotted line): rejection there is a Type I error. The curves rise with the size of the difference and steepen with n. At the 10.5 mm Hg difference of adults aged 60 and over (vertical line), power is 0.29 with 10 observations, 0.67 with 25 and 0.93 with 50.

Three properties of the curves hold for every test of this kind.

  • At a true difference of zero the "power" is $\alpha$, since rejecting there is a Type I error.
  • The curves are symmetric for a two-sided test, because a difference of $-\delta$ is as easy to detect as $+\delta$.
  • For every difference other than zero, power rises toward 1 as $n$ grows. A consistent test eventually detects any real difference, however small. That is the reason behind the ASA's warning that statistical significance does not measure the size of an effect (Wasserstein and Lazar, 2016).

The NIST/SEMATECH e-Handbook of Statistical Methods (§7.1.3) calls the curve of $\beta$ against the true value, one minus the power function, the operating characteristic curve.

The curves in the figure are for the $t$ test, which is what a real study would run. But the $z$ formula assumed $\sigma$ known. What changes when $s$ stands in for it?

Power of the t test

Here is the plan of the derivation. The same split into a random part and a shift works for $t$. The shift no longer simply slides a known curve, because the denominator is random too, and the resulting distribution has its own name.

The formal version. When $\sigma$ is estimated, the test statistic is $t = (\bar{x} - \mu_0)/(s/\sqrt{n})$, and the same split as above gives

$$T = \frac{\bar{X} - \mu_0}{S/\sqrt{n}} = \frac{Z_0 + \lambda}{S/\sigma} = \frac{Z_0 + \lambda}{\sqrt{V/(n-1)}}, \qquad V = \frac{(n-1)S^2}{\sigma^2}.$$

For a normal population, $Z_0$ is standard normal, $V$ is chi-squared with $n - 1$ degrees of freedom, and the two are independent (the three facts of Confidence Intervals for a Mean). When $\lambda = 0$ this is the definition of the $t$ distribution. When $\lambda \ne 0$, the ratio $(Z_0 + \lambda)/\sqrt{V/\nu}$ has the noncentral $t$ distribution with $\nu$ degrees of freedom and noncentrality parameter $\lambda$. That is how scipy defines it (scipy.stats.nct). The power of the two-sided $t$ test is then

$$1 - \beta = P(T' \ge t_{n-1,\,1-\alpha/2}) + P(T' \le -t_{n-1,\,1-\alpha/2}),$$

with $T'$ noncentral $t$ with $n - 1$ degrees of freedom and noncentrality $\lambda = \delta\sqrt{n}/\sigma$.

The noncentral $t$ has no simple closed form, but its distribution function is a one-dimensional integral. Write $W = \sqrt{V}$. Given $W = w$, the event $T' \le x$ is the event $Z_0 \le xw/\sqrt{\nu} - \lambda$, which has probability $\Phi(xw/\sqrt{\nu} - \lambda)$. Averaging over the distribution of $W$,

$$P(T' \le x) = \int_0^\infty \Phi\!\left(\frac{xw}{\sqrt{\nu}} - \lambda\right) g(w)\,dw, \qquad g(w) = \frac{w^{\nu-1}e^{-w^2/2}}{2^{\nu/2-1}\,\Gamma(\nu/2)},$$

where $g$ is the density of the square root of a chi-squared variable with $\nu$ degrees of freedom. The article's script evaluates this integral numerically, and its results agree with scipy's nct to within $10^{-10}$.

Normal theory against real data

Check your understanding. For the same $n$ and the same true difference, will the $t$ test have more power than the $z$ test, or less?

Less. The $t$ test's critical value is larger, to allow for the uncertainty in $s$. The table shows by how much, for the older-adult test. It compares three things:

  • the $z$ test's power, as if $\sigma$ were known;
  • the $t$ test's power for a normal population;
  • the share of samples drawn from the actual 1,923 older adults in which the $t$ test rejected (10,000 samples for each of 30 seeds, median).
$n$ $\lambda$ Power, $z$ test Power, $t$ test (normal theory) Power, $t$ test (NHANES, simulated)
5 1.116 0.200 0.140 0.096
10 1.578 0.351 0.292 0.250
25 2.494 0.703 0.668 0.688
50 3.528 0.942 0.933 0.956

The $t$ test always has less power than the $z$ test. The gap is large for tiny samples and nearly gone by $n = 50$. The real population, skewed to the right, departs from normal theory by a few points in either direction, as Type I and Type II Errors discussed. For planning, the normal-theory calculation is the standard tool, and simulation from realistic data is a check on it.

We can now compute the power of any planned design. Planning usually asks the reverse question.

From power to sample size

Turning the power calculation around answers the planning question. How large must $n$ be for the test to have power $1 - \beta$ against a difference $\delta = \mu_1 - \mu_0$?

The formula has a picture, and it is worth seeing first. The critical value must sit $z_{1-\alpha/2}$ standard errors above $\mu_0$, so that the null distribution has only $\alpha/2$ beyond it. It must also sit $z_{1-\beta}$ standard errors below $\mu_1$, so that the alternative distribution has $1 - \beta$ beyond it. Both can hold only if $\mu_1 - \mu_0$ spans $z_{1-\alpha/2} + z_{1-\beta}$ standard errors, $\delta = (z_{1-\alpha/2} + z_{1-\beta})\,\sigma/\sqrt{n}$. Solving for $n$ gives the formula.

The formal version. Start with the one-sided $z$ test, whose power is $\Phi(\lambda - z_{1-\alpha})$. Setting this equal to $1 - \beta$ means $\lambda - z_{1-\alpha}$ must equal the point with probability $1 - \beta$ below it, $z_{1-\beta}$:

$$\frac{\delta\sqrt{n}}{\sigma} - z_{1-\alpha} = z_{1-\beta} \quad\Longrightarrow\quad \sqrt{n} = \frac{(z_{1-\alpha} + z_{1-\beta})\,\sigma}{\delta}.$$

For the two-sided test, the power is $\Phi(\lambda - z_{1-\alpha/2}) + \Phi(-\lambda - z_{1-\alpha/2})$. The second term, the chance of rejecting in the wrong direction, is negligible for any design with useful power. It was 0.0002 in the worked case above, with power only 0.35. Dropping it and solving the same way gives the two-sided formula. Squaring:

$$n = \left[\frac{(z_{1-\alpha} + z_{1-\beta})\,\sigma}{\mu_1 - \mu_0}\right]^2 \ \text{(one-sided)}, \qquad n = \left[\frac{(z_{1-\alpha/2} + z_{1-\beta})\,\sigma}{\mu_1 - \mu_0}\right]^2 \ \text{(two-sided)},$$

rounded up to the next whole number. The same formulas serve a paired design: a paired test is a one-sample test on the within-pair differences (The Paired Samples t-Test), so $\delta$ becomes the mean difference to be detected and $\sigma$ the standard deviation of the differences, as in the sleep trial below. The NIST handbook (§7.2.2.2) gives the same formulas, writing the two-sided one as $N = (z_{1-\alpha/2} + z_{1-\beta})^2(\sigma/\delta)^2$ with $\delta$ the shift to be detected.

Power and the width of an interval

Compare the sample size for a confidence interval of half-width $E$, from One-Sided Confidence Bounds and Sample Size: $n = (z_{1-\alpha/2}\,\sigma/E)^2$. It is the power formula with $z_{1-\beta} = 0$.

Check your understanding. A study is sized so that its 95% interval has half-width $\delta$. If the true difference is exactly $\delta$, what is the chance of a significant result?

About 50%. Setting $z_{1-\beta} = 0$ means $\beta = 0.5$. Half the time the sample mean falls short of the true mean, and the interval then reaches back past $\mu_0$. Sizing for power adds $z_{1-\beta}$ standard errors of margin; for 80% power, $z_{0.80} = 0.8416$.

Planning a study of older adults

Suppose a study of adults aged 60 and over wants an 80% chance of detecting a mean systolic pressure 5 mm Hg away from the all-adult mean. It will use a two-sided test at $\alpha = 0.05$. It plans with the older adults' standard deviation unrounded, $\sigma = 21.024$ mm Hg. Then $\delta/\sigma = 5/21.024 = 0.2378$ and

$$n = \left[\frac{(1.9600 + 0.8416) \times 21.024}{5}\right]^2 = 138.77, \quad\text{so } n = 139.$$

For 90% power, $z_{0.90} = 1.2816$ replaces 0.8416 and $n = 185.78$, so 186.

Correcting for t

The formula assumes $\sigma$ known. The finished study will use the $t$ test, which is slightly less powerful, so the formula slightly undershoots. Three ways to correct it agree here.

  • Compute the $t$ test's power at the formula's $n$ with the noncentral $t$: at $n = 139$ it is 0.7951, just short of 0.80.
  • Iterate with $t$ quantiles, as the NIST handbook (§7.2.2.2) suggests. Replace $z_{1-\alpha/2}$ and $z_{1-\beta}$ by $t_{n-1,\,1-\alpha/2}$ and $t_{n-1,\,1-\beta}$ at the current $n$ and recompute. From 139, $t_{138,\,0.975} = 1.9773$ and $t_{138,\,0.80} = 0.8442$ give 140.76, so 141, and a second step stays at 141.
  • Search for the exact answer: the smallest $n$ whose noncentral-$t$ power reaches 0.80. It is 141 (power 0.8008; at 140, 0.7980).

For 90% power the same corrections move 186 to 188. A simulation drew samples of 141 from the older-adult population, shifted so that its mean is exactly 5 mm Hg above the all-adult mean. It found power 0.816, close to the planned 0.80.

Planning a new sleep trial

The second running example is Student's ten sleep patients. Suppose a new crossover trial of the two sleeping drugs wants 80% power, two-sided at 0.05, to detect a mean difference of 0.5 hours of sleep. It plans with the standard deviation of Student's ten differences, $\sigma_d = 1.23$ hours (Student, 1908). The paired design reduces to a one-sample test on the differences, so the same formula applies with $\sigma_d$:

$$n = \left[\frac{(1.9600 + 0.8416) \times 1.23}{0.5}\right]^2 = 47.50, \quad\text{so } n = 48.$$

At 48 patients the $t$ test's power is 0.7877. The NIST iteration moves to 50, and 50 is also the exact answer (power 0.8045; 0.7962 at 49). Simulating normal differences with mean 0.5 and standard deviation 1.23 gave power 0.8052 at $n = 50$ and 0.7886 at $n = 48$. For 90% power, the $z$ formula gives 64 and the exact answer 66. The paired design is developed in The Paired Samples t-Test. The larger samples that two independent groups require are in Sample Size for Comparing Two Means.

The NIST example

The NIST handbook (§7.2.2.2) works a one-sided test at $\alpha = 0.05$ with $\beta = 0.10$ against a shift of one standard deviation. It finds $N = (1.645 + 1.282)^2 = 8.567$, so 9, then one $t$ iteration to $(1.860 + 1.397)^2 = 10.6$, so 11. The script reproduces this: 8.564 with unrounded quantiles, and 10.604 after one iteration, stable at 11 after the next. It also finds that 11 is the exact answer: power 0.9245 at $n = 11$ and 0.8975 at $n = 10$.

A table and a rule of thumb

$\delta/\sigma$ $n$ for 80% power, $z$ formula exact $t$ $n$ for 90% power, $z$ formula exact $t$
0.20 197 199 263 265
0.25 126 128 169 171
0.50 32 34 43 44
0.75 14 16 19 21
1.00 8 10 11 13

The table is for two-sided tests at $\alpha = 0.05$. Across it, the exact $t$ answer is the $z$ formula's answer plus one or two. So adding 2 to the formula is a fair correction for a one-sample test at these settings. The multipliers worth remembering are $(z_{0.975} + z_{0.80})^2 = 7.85$ and $(z_{0.975} + z_{0.90})^2 = 10.51$: for 80% power, $n \approx 7.85\,(\sigma/\delta)^2$.

Power of the two-sided t test at 0.05 as the sample size grows, sigma = 21.0 mm Hg. Solid: a true difference of 10.5 mm Hg, the gap between adults aged 60 and over and all adults; power reaches 0.80 at n = 34. Dashed: a true difference of 5 mm Hg; notice that power is still below 0.75 at n = 120 and reaches 0.80 only at n = 141. Halving the difference to be detected roughly quadruples the sample.
Power of the two-sided t test at 0.05 as the sample size grows, sigma = 21.0 mm Hg. Solid: a true difference of 10.5 mm Hg, the gap between adults aged 60 and over and all adults; power reaches 0.80 at n = 34. Dashed: a true difference of 5 mm Hg; notice that power is still below 0.75 at n = 120 and reaches 0.80 only at n = 141. Halving the difference to be detected roughly quadruples the sample.

What moves the sample size

The formula has four inputs, and each moves the answer in a way worth knowing before a study's budget is set. The baseline is the older-adult design with $n = 141$ (80% power for 5 mm Hg).

Change Effect
One-sided test at 0.05 instead of two-sided Power at 141 rises to 0.878; 111 suffice for 80%
$\alpha = 0.01$ instead of 0.05 Power at 141 falls to 0.585; 80% needs 210
True $\sigma$ 10% larger than planned Power falls to 0.722
True $\sigma$ 25% larger than planned Power falls to 0.612
True difference 2.5 mm Hg instead of 5 Power falls to 0.289; 80% needs 558
15% of participants expected to drop out Recruit $141/(1 - 0.15) = 165.9$, so 166

The difference $\delta$ and the standard deviation $\sigma$ enter squared, so they dominate. The planning value of $\sigma$ usually comes from an earlier study or a pilot, and it carries uncertainty of its own. One-Sided Confidence Bounds and Sample Size showed how variable a standard deviation estimated from a small sample is. Planning with a cautious, larger $\sigma$ protects the power.

The difference $\delta$ is a choice, and the most consequential one. It should be the smallest difference that would matter, clinically or practically. It should not be the difference the investigators hope or expect to find. A study powered for an optimistic $\delta$ is underpowered for every smaller effect that would still be worth knowing about. The table shows how quickly the requirement grows: detecting half the difference with the same power takes about four times the sample, 558 instead of 141.

Power after the study

Power is a property of a design, computed before the data exist. A common practice after a nonsignificant result is to compute the power of the test at the effect that was observed, and to report it as if it said something about the result. It does not.

For the $z$ test, the "observed power" is the power formula with the observed $|z|$ in place of $\lambda$: $\Phi(|z| - 1.96) + \Phi(-|z| - 1.96)$. Since $|z|$ is determined by the two-sided $p$-value, so is the observed power.

Check your understanding. A study reports $p = 0.05$ exactly. What is its "observed power"?

One half. At $p = 0.05$, $|z| = 1.960$, so the first term is $\Phi(0) = 0.5$ and the second is negligible. The table shows the whole correspondence.

Two-sided $p$ 0.5 0.2 0.1 0.05 0.01 0.005 0.001
$\lvert z \rvert$ 0.674 1.282 1.645 1.960 2.576 2.807 3.291
"Observed power" 0.10 0.25 0.38 0.50 0.73 0.80 0.91

Every nonsignificant result has observed power of at most about 0.50. That is because $p > 0.05$ means $|z| < 1.96$, which makes $\Phi(|z| - 1.96)$ less than one half. Saying that a nonsignificant study "had low power to detect the observed effect" therefore restates that it was nonsignificant. Greenland and colleagues (2016) make the same point in general. Power calculated from the observed data "is a direct (if obscure) transformation of the null P value," and so provides no test of the alternatives.

What a reader of a nonsignificant result needs is the confidence interval. It shows directly which effects the data can and cannot exclude. For the older-adult sample of 25, the interval ran from 0.9 mm Hg below to 14.6 mm Hg above the all-adult mean, and no power figure would add to that.

Pre-study power does have a use in reading results: it explains why significance often fails to replicate. Suppose two studies of a real effect each have power 0.80. The chance that both are significant is $0.80^2 = 0.64$. The chance that exactly one is, inviting the false conclusion that they conflict, is $2 \times 0.80 \times 0.20 = 0.32$ (Greenland et al., 2016). At power 0.50 the two outcomes are 0.25 and 0.50: a coin toss decides which study looks "positive."

Summary

Quantity Formula
Noncentrality (standard errors between $\mu_1$ and $\mu_0$) $\lambda = (\mu_1 - \mu_0)\sqrt{n}/\sigma$
Power, two-sided $z$ test $\Phi(\lambda - z_{1-\alpha/2}) + \Phi(-\lambda - z_{1-\alpha/2})$
Power, one-sided $z$ test ($H_A: \mu > \mu_0$) $\Phi(\lambda - z_{1-\alpha})$
Power, two-sided $t$ test $P(\lvert T' \rvert \ge t_{n-1,\,1-\alpha/2})$, $T'$ noncentral $t$ with $n - 1$ df and noncentrality $\lambda$
Sample size, two-sided $n = [(z_{1-\alpha/2} + z_{1-\beta})\,\sigma/(\mu_1 - \mu_0)]^2$, rounded up; check or correct with $t$
Sample size, one-sided $n = [(z_{1-\alpha} + z_{1-\beta})\,\sigma/(\mu_1 - \mu_0)]^2$, rounded up

The same reasoning sizes other studies. The information retrieval wiki's Significance Testing uses the formula to ask how many test queries are needed to detect a given improvement in a search engine. The library's notes on adding error bars to evals apply power analysis to the number of questions a language model evaluation needs.

What we have learned

Power depends on the design only through $\lambda$, the true difference in standard errors, and for the $t$ test it comes from the noncentral $t$ distribution. Solving for a target power gives $n = [(z_{1-\alpha/2} + z_{1-\beta})\,\sigma/\delta]^2$, which a $t$ correction raises by one or two. That meant 141 older adults for 80% power against 5 mm Hg, and 50 sleep patients against half an hour. The difference to detect and the planning $\sigma$ dominate the answer. Power computed from the observed effect only restates the $p$-value.

This closes the topic on hypothesis testing. Every test so far compared one mean with a fixed value. Yet Student's ten patients were really a comparison of two drugs, made one sample by taking differences. The next topic, Comparison of Two Means, starts there, with The Paired Samples t-Test: why the pairing matters, and what is lost when it is ignored.

In Python

import math
from statistics import NormalDist
from scipy import stats

def t_power(d_over_sigma, n, alpha=0.05):
    """Power of the two-sided one-sample t test (normal population)."""
    df, nc = n - 1, d_over_sigma * math.sqrt(n)
    c = stats.t.ppf(1 - alpha / 2, df)
    return stats.nct.sf(c, df, nc) + stats.nct.cdf(-c, df, nc)

z = NormalDist().inv_cdf
d = 5 / 21.02
n_z = math.ceil(((z(0.975) + z(0.80)) / d) ** 2)
print(n_z, t_power(d, n_z))          # 139, about 0.795

n = n_z
while t_power(d, n) < 0.80:
    n += 1
print(n, t_power(d, n))              # 141, about 0.801

Every number in this article is computed by tools/experiments/applied-statistics/power-and-sample-size-for-a-mean.py in the site's repository, which implements the noncentral $t$ distribution by the integral above and was checked against scipy 1.17.1.

References