A study of 25 adults aged 60 and over, in Hypothesis Tests for a Mean, tested whether their mean systolic blood pressure differed from the all-adult mean. It found $p = 0.081$, although the true difference in the NHANES population is 10.5 mm Hg. Type I and Type II Errors measured how often a study that size misses a difference that size: about 31% of the time. A study that misses a real effect nearly a third of the time is a poor use of its participants. How many older adults would the study have needed, and could that have been worked out before anyone was measured?
It could. The power of a test is the probability that it rejects the null hypothesis when a particular alternative is true. It can be computed in advance from the planned sample size, and the sample size can then be chosen to make the power as high as the study needs. This article is the course's home for the power function, the noncentral $t$ distribution and the sample size formula for a test. Sample Size for Comparing Two Means builds on them.
Here is the route. First we derive the power of the $z$ test and draw power curves. Then we extend it to the $t$ test through the noncentral $t$ distribution. Next we turn power around into the sample size formula, checked against the exact calculation and simulation. Finally we look at what moves the answer, and at power computed after the study. The article assumes the error rates of Type I and Type II Errors.
The power function of the z test
Here is the idea in one sentence. If the true mean is not $\mu_0$, the test statistic's distribution slides sideways, and the power is the share of it that lands beyond the critical values.
The formal version. Consider the two-sided $z$ test of $H_0: \mu = \mu_0$ against $H_A: \mu \ne \mu_0$ at level $\alpha$, with $\sigma$ known. It rejects when $|z| \ge z_{1-\alpha/2}$, where
$$z = \frac{\bar{x} - \mu_0}{\sigma/\sqrt{n}}.$$
Suppose the true mean is $\mu_1 = \mu_0 + \delta$. Then $\bar{X}$ is centered on $\mu_1$, not $\mu_0$, and the statistic can be split into a random part and a fixed shift:
$$Z = \frac{\bar{X} - \mu_1}{\sigma/\sqrt{n}} + \frac{\mu_1 - \mu_0}{\sigma/\sqrt{n}} = Z_0 + \lambda, \qquad \lambda = \frac{\delta\sqrt{n}}{\sigma},$$
where $Z_0$ has a standard normal distribution. So under the alternative, $Z$ is normal with mean $\lambda$ and standard deviation 1: the null distribution shifted by $\lambda$ standard errors. The power is the probability that this shifted variable lands in the rejection region:
$$1 - \beta = P(Z \ge z_{1-\alpha/2}) + P(Z \le -z_{1-\alpha/2}) = \Phi(\lambda - z_{1-\alpha/2}) + \Phi(-\lambda - z_{1-\alpha/2}),$$
where $\Phi$ is the standard normal distribution function. A one-sided test of $H_A: \mu > \mu_0$ rejects when $z \ge z_{1-\alpha}$, so only the first kind of term appears: $1 - \beta = \Phi(\lambda - z_{1-\alpha})$.
The quantity $\lambda$ carries everything that matters. The power depends on the difference $\delta$, the standard deviation $\sigma$ and the sample size $n$ only through $\delta\sqrt{n}/\sigma$, the true difference measured in standard errors of the mean.
Check your understanding. Which raises the power more: doubling the sample size, or doubling the difference to be detected?
Doubling the difference. It doubles $\lambda$, while doubling $n$ multiplies $\lambda$ only by $\sqrt{2} \approx 1.41$. For the older adults below, going from 5 to 10 observations moves $\lambda$ from 1.116 to 1.578. Doubling $\delta$ has the same effect as quadrupling $n$.
A worked case
Back to the older adults. The null value is $\mu_0 = 126.36$, the all-adult mean, and the true mean is $\mu_1 = 136.85$, the mean of the 1,923 adults aged 60 and over. So $\delta = 10.49$ mm Hg, and their standard deviation is $\sigma = 21.02$ mm Hg. With $n = 10$,
$$\lambda = \frac{10.49\sqrt{10}}{21.02} = 1.578,$$
$$1 - \beta = \Phi(1.578 - 1.960) + \Phi(-1.578 - 1.960) = 0.3511 + 0.0002 = 0.3513.$$
The second term is the chance of rejecting in the wrong direction, concluding that older adults have lower pressure than average. It is tiny here and becomes tinier as $\lambda$ grows: 0.000004 at $n = 25$.
Power curves
Computed for every possible true mean, the power is a curve, the power function of the test. The figure shows it for the $t$ test (derived in the next section) at three sample sizes.
Three properties of the curves hold for every test of this kind.
- At a true difference of zero the "power" is $\alpha$, since rejecting there is a Type I error.
- The curves are symmetric for a two-sided test, because a difference of $-\delta$ is as easy to detect as $+\delta$.
- For every difference other than zero, power rises toward 1 as $n$ grows. A consistent test eventually detects any real difference, however small. That is the reason behind the ASA's warning that statistical significance does not measure the size of an effect (Wasserstein and Lazar, 2016).
The NIST/SEMATECH e-Handbook of Statistical Methods (§7.1.3) calls the curve of $\beta$ against the true value, one minus the power function, the operating characteristic curve.
The curves in the figure are for the $t$ test, which is what a real study would run. But the $z$ formula assumed $\sigma$ known. What changes when $s$ stands in for it?
Power of the t test
Here is the plan of the derivation. The same split into a random part and a shift works for $t$. The shift no longer simply slides a known curve, because the denominator is random too, and the resulting distribution has its own name.
The formal version. When $\sigma$ is estimated, the test statistic is $t = (\bar{x} - \mu_0)/(s/\sqrt{n})$, and the same split as above gives
$$T = \frac{\bar{X} - \mu_0}{S/\sqrt{n}} = \frac{Z_0 + \lambda}{S/\sigma} = \frac{Z_0 + \lambda}{\sqrt{V/(n-1)}}, \qquad V = \frac{(n-1)S^2}{\sigma^2}.$$
For a normal population, $Z_0$ is standard normal, $V$ is chi-squared with $n - 1$ degrees of freedom, and the two are independent (the three facts of Confidence Intervals for a Mean). When $\lambda = 0$ this is the definition of the $t$ distribution. When $\lambda \ne 0$, the ratio $(Z_0 + \lambda)/\sqrt{V/\nu}$ has the noncentral $t$ distribution with $\nu$ degrees of freedom and noncentrality parameter $\lambda$. That is how scipy defines it (scipy.stats.nct). The power of the two-sided $t$ test is then
$$1 - \beta = P(T' \ge t_{n-1,\,1-\alpha/2}) + P(T' \le -t_{n-1,\,1-\alpha/2}),$$
with $T'$ noncentral $t$ with $n - 1$ degrees of freedom and noncentrality $\lambda = \delta\sqrt{n}/\sigma$.
The noncentral $t$ has no simple closed form, but its distribution function is a one-dimensional integral. Write $W = \sqrt{V}$. Given $W = w$, the event $T' \le x$ is the event $Z_0 \le xw/\sqrt{\nu} - \lambda$, which has probability $\Phi(xw/\sqrt{\nu} - \lambda)$. Averaging over the distribution of $W$,
$$P(T' \le x) = \int_0^\infty \Phi\!\left(\frac{xw}{\sqrt{\nu}} - \lambda\right) g(w)\,dw, \qquad g(w) = \frac{w^{\nu-1}e^{-w^2/2}}{2^{\nu/2-1}\,\Gamma(\nu/2)},$$
where $g$ is the density of the square root of a chi-squared variable with $\nu$ degrees of freedom. The article's script evaluates this integral numerically, and its results agree with scipy's nct to within $10^{-10}$.
Normal theory against real data
Check your understanding. For the same $n$ and the same true difference, will the $t$ test have more power than the $z$ test, or less?
Less. The $t$ test's critical value is larger, to allow for the uncertainty in $s$. The table shows by how much, for the older-adult test. It compares three things:
- the $z$ test's power, as if $\sigma$ were known;
- the $t$ test's power for a normal population;
- the share of samples drawn from the actual 1,923 older adults in which the $t$ test rejected (10,000 samples for each of 30 seeds, median).
| $n$ | $\lambda$ | Power, $z$ test | Power, $t$ test (normal theory) | Power, $t$ test (NHANES, simulated) |
|---|---|---|---|---|
| 5 | 1.116 | 0.200 | 0.140 | 0.096 |
| 10 | 1.578 | 0.351 | 0.292 | 0.250 |
| 25 | 2.494 | 0.703 | 0.668 | 0.688 |
| 50 | 3.528 | 0.942 | 0.933 | 0.956 |
The $t$ test always has less power than the $z$ test. The gap is large for tiny samples and nearly gone by $n = 50$. The real population, skewed to the right, departs from normal theory by a few points in either direction, as Type I and Type II Errors discussed. For planning, the normal-theory calculation is the standard tool, and simulation from realistic data is a check on it.
We can now compute the power of any planned design. Planning usually asks the reverse question.
From power to sample size
Turning the power calculation around answers the planning question. How large must $n$ be for the test to have power $1 - \beta$ against a difference $\delta = \mu_1 - \mu_0$?
The formula has a picture, and it is worth seeing first. The critical value must sit $z_{1-\alpha/2}$ standard errors above $\mu_0$, so that the null distribution has only $\alpha/2$ beyond it. It must also sit $z_{1-\beta}$ standard errors below $\mu_1$, so that the alternative distribution has $1 - \beta$ beyond it. Both can hold only if $\mu_1 - \mu_0$ spans $z_{1-\alpha/2} + z_{1-\beta}$ standard errors, $\delta = (z_{1-\alpha/2} + z_{1-\beta})\,\sigma/\sqrt{n}$. Solving for $n$ gives the formula.
The formal version. Start with the one-sided $z$ test, whose power is $\Phi(\lambda - z_{1-\alpha})$. Setting this equal to $1 - \beta$ means $\lambda - z_{1-\alpha}$ must equal the point with probability $1 - \beta$ below it, $z_{1-\beta}$:
$$\frac{\delta\sqrt{n}}{\sigma} - z_{1-\alpha} = z_{1-\beta} \quad\Longrightarrow\quad \sqrt{n} = \frac{(z_{1-\alpha} + z_{1-\beta})\,\sigma}{\delta}.$$
For the two-sided test, the power is $\Phi(\lambda - z_{1-\alpha/2}) + \Phi(-\lambda - z_{1-\alpha/2})$. The second term, the chance of rejecting in the wrong direction, is negligible for any design with useful power. It was 0.0002 in the worked case above, with power only 0.35. Dropping it and solving the same way gives the two-sided formula. Squaring:
$$n = \left[\frac{(z_{1-\alpha} + z_{1-\beta})\,\sigma}{\mu_1 - \mu_0}\right]^2 \ \text{(one-sided)}, \qquad n = \left[\frac{(z_{1-\alpha/2} + z_{1-\beta})\,\sigma}{\mu_1 - \mu_0}\right]^2 \ \text{(two-sided)},$$
rounded up to the next whole number. The same formulas serve a paired design: a paired test is a one-sample test on the within-pair differences (The Paired Samples t-Test), so $\delta$ becomes the mean difference to be detected and $\sigma$ the standard deviation of the differences, as in the sleep trial below. The NIST handbook (§7.2.2.2) gives the same formulas, writing the two-sided one as $N = (z_{1-\alpha/2} + z_{1-\beta})^2(\sigma/\delta)^2$ with $\delta$ the shift to be detected.
Power and the width of an interval
Compare the sample size for a confidence interval of half-width $E$, from One-Sided Confidence Bounds and Sample Size: $n = (z_{1-\alpha/2}\,\sigma/E)^2$. It is the power formula with $z_{1-\beta} = 0$.
Check your understanding. A study is sized so that its 95% interval has half-width $\delta$. If the true difference is exactly $\delta$, what is the chance of a significant result?
About 50%. Setting $z_{1-\beta} = 0$ means $\beta = 0.5$. Half the time the sample mean falls short of the true mean, and the interval then reaches back past $\mu_0$. Sizing for power adds $z_{1-\beta}$ standard errors of margin; for 80% power, $z_{0.80} = 0.8416$.
Planning a study of older adults
Suppose a study of adults aged 60 and over wants an 80% chance of detecting a mean systolic pressure 5 mm Hg away from the all-adult mean. It will use a two-sided test at $\alpha = 0.05$. It plans with the older adults' standard deviation unrounded, $\sigma = 21.024$ mm Hg. Then $\delta/\sigma = 5/21.024 = 0.2378$ and
$$n = \left[\frac{(1.9600 + 0.8416) \times 21.024}{5}\right]^2 = 138.77, \quad\text{so } n = 139.$$
For 90% power, $z_{0.90} = 1.2816$ replaces 0.8416 and $n = 185.78$, so 186.
Correcting for t
The formula assumes $\sigma$ known. The finished study will use the $t$ test, which is slightly less powerful, so the formula slightly undershoots. Three ways to correct it agree here.
- Compute the $t$ test's power at the formula's $n$ with the noncentral $t$: at $n = 139$ it is 0.7951, just short of 0.80.
- Iterate with $t$ quantiles, as the NIST handbook (§7.2.2.2) suggests. Replace $z_{1-\alpha/2}$ and $z_{1-\beta}$ by $t_{n-1,\,1-\alpha/2}$ and $t_{n-1,\,1-\beta}$ at the current $n$ and recompute. From 139, $t_{138,\,0.975} = 1.9773$ and $t_{138,\,0.80} = 0.8442$ give 140.76, so 141, and a second step stays at 141.
- Search for the exact answer: the smallest $n$ whose noncentral-$t$ power reaches 0.80. It is 141 (power 0.8008; at 140, 0.7980).
For 90% power the same corrections move 186 to 188. A simulation drew samples of 141 from the older-adult population, shifted so that its mean is exactly 5 mm Hg above the all-adult mean. It found power 0.816, close to the planned 0.80.
Planning a new sleep trial
The second running example is Student's ten sleep patients. Suppose a new crossover trial of the two sleeping drugs wants 80% power, two-sided at 0.05, to detect a mean difference of 0.5 hours of sleep. It plans with the standard deviation of Student's ten differences, $\sigma_d = 1.23$ hours (Student, 1908). The paired design reduces to a one-sample test on the differences, so the same formula applies with $\sigma_d$:
$$n = \left[\frac{(1.9600 + 0.8416) \times 1.23}{0.5}\right]^2 = 47.50, \quad\text{so } n = 48.$$
At 48 patients the $t$ test's power is 0.7877. The NIST iteration moves to 50, and 50 is also the exact answer (power 0.8045; 0.7962 at 49). Simulating normal differences with mean 0.5 and standard deviation 1.23 gave power 0.8052 at $n = 50$ and 0.7886 at $n = 48$. For 90% power, the $z$ formula gives 64 and the exact answer 66. The paired design is developed in The Paired Samples t-Test. The larger samples that two independent groups require are in Sample Size for Comparing Two Means.
The NIST example
The NIST handbook (§7.2.2.2) works a one-sided test at $\alpha = 0.05$ with $\beta = 0.10$ against a shift of one standard deviation. It finds $N = (1.645 + 1.282)^2 = 8.567$, so 9, then one $t$ iteration to $(1.860 + 1.397)^2 = 10.6$, so 11. The script reproduces this: 8.564 with unrounded quantiles, and 10.604 after one iteration, stable at 11 after the next. It also finds that 11 is the exact answer: power 0.9245 at $n = 11$ and 0.8975 at $n = 10$.
A table and a rule of thumb
| $\delta/\sigma$ | $n$ for 80% power, $z$ formula | exact $t$ | $n$ for 90% power, $z$ formula | exact $t$ |
|---|---|---|---|---|
| 0.20 | 197 | 199 | 263 | 265 |
| 0.25 | 126 | 128 | 169 | 171 |
| 0.50 | 32 | 34 | 43 | 44 |
| 0.75 | 14 | 16 | 19 | 21 |
| 1.00 | 8 | 10 | 11 | 13 |
The table is for two-sided tests at $\alpha = 0.05$. Across it, the exact $t$ answer is the $z$ formula's answer plus one or two. So adding 2 to the formula is a fair correction for a one-sample test at these settings. The multipliers worth remembering are $(z_{0.975} + z_{0.80})^2 = 7.85$ and $(z_{0.975} + z_{0.90})^2 = 10.51$: for 80% power, $n \approx 7.85\,(\sigma/\delta)^2$.
What moves the sample size
The formula has four inputs, and each moves the answer in a way worth knowing before a study's budget is set. The baseline is the older-adult design with $n = 141$ (80% power for 5 mm Hg).
| Change | Effect |
|---|---|
| One-sided test at 0.05 instead of two-sided | Power at 141 rises to 0.878; 111 suffice for 80% |
| $\alpha = 0.01$ instead of 0.05 | Power at 141 falls to 0.585; 80% needs 210 |
| True $\sigma$ 10% larger than planned | Power falls to 0.722 |
| True $\sigma$ 25% larger than planned | Power falls to 0.612 |
| True difference 2.5 mm Hg instead of 5 | Power falls to 0.289; 80% needs 558 |
| 15% of participants expected to drop out | Recruit $141/(1 - 0.15) = 165.9$, so 166 |
The difference $\delta$ and the standard deviation $\sigma$ enter squared, so they dominate. The planning value of $\sigma$ usually comes from an earlier study or a pilot, and it carries uncertainty of its own. One-Sided Confidence Bounds and Sample Size showed how variable a standard deviation estimated from a small sample is. Planning with a cautious, larger $\sigma$ protects the power.
The difference $\delta$ is a choice, and the most consequential one. It should be the smallest difference that would matter, clinically or practically. It should not be the difference the investigators hope or expect to find. A study powered for an optimistic $\delta$ is underpowered for every smaller effect that would still be worth knowing about. The table shows how quickly the requirement grows: detecting half the difference with the same power takes about four times the sample, 558 instead of 141.
Power after the study
Power is a property of a design, computed before the data exist. A common practice after a nonsignificant result is to compute the power of the test at the effect that was observed, and to report it as if it said something about the result. It does not.
For the $z$ test, the "observed power" is the power formula with the observed $|z|$ in place of $\lambda$: $\Phi(|z| - 1.96) + \Phi(-|z| - 1.96)$. Since $|z|$ is determined by the two-sided $p$-value, so is the observed power.
Check your understanding. A study reports $p = 0.05$ exactly. What is its "observed power"?
One half. At $p = 0.05$, $|z| = 1.960$, so the first term is $\Phi(0) = 0.5$ and the second is negligible. The table shows the whole correspondence.
| Two-sided $p$ | 0.5 | 0.2 | 0.1 | 0.05 | 0.01 | 0.005 | 0.001 |
|---|---|---|---|---|---|---|---|
| $\lvert z \rvert$ | 0.674 | 1.282 | 1.645 | 1.960 | 2.576 | 2.807 | 3.291 |
| "Observed power" | 0.10 | 0.25 | 0.38 | 0.50 | 0.73 | 0.80 | 0.91 |
Every nonsignificant result has observed power of at most about 0.50. That is because $p > 0.05$ means $|z| < 1.96$, which makes $\Phi(|z| - 1.96)$ less than one half. Saying that a nonsignificant study "had low power to detect the observed effect" therefore restates that it was nonsignificant. Greenland and colleagues (2016) make the same point in general. Power calculated from the observed data "is a direct (if obscure) transformation of the null P value," and so provides no test of the alternatives.
What a reader of a nonsignificant result needs is the confidence interval. It shows directly which effects the data can and cannot exclude. For the older-adult sample of 25, the interval ran from 0.9 mm Hg below to 14.6 mm Hg above the all-adult mean, and no power figure would add to that.
Pre-study power does have a use in reading results: it explains why significance often fails to replicate. Suppose two studies of a real effect each have power 0.80. The chance that both are significant is $0.80^2 = 0.64$. The chance that exactly one is, inviting the false conclusion that they conflict, is $2 \times 0.80 \times 0.20 = 0.32$ (Greenland et al., 2016). At power 0.50 the two outcomes are 0.25 and 0.50: a coin toss decides which study looks "positive."
Summary
| Quantity | Formula |
|---|---|
| Noncentrality (standard errors between $\mu_1$ and $\mu_0$) | $\lambda = (\mu_1 - \mu_0)\sqrt{n}/\sigma$ |
| Power, two-sided $z$ test | $\Phi(\lambda - z_{1-\alpha/2}) + \Phi(-\lambda - z_{1-\alpha/2})$ |
| Power, one-sided $z$ test ($H_A: \mu > \mu_0$) | $\Phi(\lambda - z_{1-\alpha})$ |
| Power, two-sided $t$ test | $P(\lvert T' \rvert \ge t_{n-1,\,1-\alpha/2})$, $T'$ noncentral $t$ with $n - 1$ df and noncentrality $\lambda$ |
| Sample size, two-sided | $n = [(z_{1-\alpha/2} + z_{1-\beta})\,\sigma/(\mu_1 - \mu_0)]^2$, rounded up; check or correct with $t$ |
| Sample size, one-sided | $n = [(z_{1-\alpha} + z_{1-\beta})\,\sigma/(\mu_1 - \mu_0)]^2$, rounded up |
The same reasoning sizes other studies. The information retrieval wiki's Significance Testing uses the formula to ask how many test queries are needed to detect a given improvement in a search engine. The library's notes on adding error bars to evals apply power analysis to the number of questions a language model evaluation needs.
What we have learned
Power depends on the design only through $\lambda$, the true difference in standard errors, and for the $t$ test it comes from the noncentral $t$ distribution. Solving for a target power gives $n = [(z_{1-\alpha/2} + z_{1-\beta})\,\sigma/\delta]^2$, which a $t$ correction raises by one or two. That meant 141 older adults for 80% power against 5 mm Hg, and 50 sleep patients against half an hour. The difference to detect and the planning $\sigma$ dominate the answer. Power computed from the observed effect only restates the $p$-value.
This closes the topic on hypothesis testing. Every test so far compared one mean with a fixed value. Yet Student's ten patients were really a comparison of two drugs, made one sample by taking differences. The next topic, Comparison of Two Means, starts there, with The Paired Samples t-Test: why the pairing matters, and what is lost when it is ignored.
In Python
import math
from statistics import NormalDist
from scipy import stats
def t_power(d_over_sigma, n, alpha=0.05):
"""Power of the two-sided one-sample t test (normal population)."""
df, nc = n - 1, d_over_sigma * math.sqrt(n)
c = stats.t.ppf(1 - alpha / 2, df)
return stats.nct.sf(c, df, nc) + stats.nct.cdf(-c, df, nc)
z = NormalDist().inv_cdf
d = 5 / 21.02
n_z = math.ceil(((z(0.975) + z(0.80)) / d) ** 2)
print(n_z, t_power(d, n_z)) # 139, about 0.795
n = n_z
while t_power(d, n) < 0.80:
n += 1
print(n, t_power(d, n)) # 141, about 0.801
Every number in this article is computed by tools/experiments/applied-statistics/power-and-sample-size-for-a-mean.py in the site's repository, which implements the noncentral $t$ distribution by the integral above and was checked against scipy 1.17.1.
References
- Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., and Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. record · 5 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J) and Blood Pressure (BPX_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
- NIST/SEMATECH. e-Handbook of Statistical Methods, §7.1.3 (What are statistical tests?) and §7.2.2.2 (Sample sizes required). National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/ record · 30 pages
- SciPy documentation.
scipy.stats.nct: a non-central Student's t continuous random variable. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.nct.html record · 2 pages - Student (1908). The probable error of a mean. Biometrika, 6(1), 1–25. record · 11 pages
- Wasserstein, R. L., and Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108 record · 6 pages