Among US adults aged 18 to 39, an estimated 22.4% had hypertension in 2017 to 2018 by the definition of the 2017 ACC/AHA guideline (Ostchega et al., 2020). Is the proportion among young men different? A random sample of 60 men aged 18 to 39 from the National Health and Nutrition Examination Survey (NHANES) contains 15 with hypertension. That is $\hat{p} = 0.250$, close to 0.224, and a test finds no evidence of a difference. Should we conclude that young men are like young adults in general?
We should not. The sample came from the 822 men aged 18 to 39 defined in The Normal Approximation to the Binomial, and their true proportion is 0.3175, almost ten points higher. The test was not wrong in any formal sense. It was a test that could see a difference of that size only about 44% of the time. A test for a proportion asks whether the data are consistent with a stated value $p_0$, and how well it can answer depends on how many observations it has.
First we set out the one-sample $z$ test on the course's formula sheet, and find the interval that matches it. Then comes its exact binomial counterpart, and the actual error rates of both, computed exactly. Finally we answer the question the opening raises: how many observations a study of a proportion needs, whether it aims to test a hypothesis or to estimate the proportion to a given precision. The logic of tests, $p$-values and errors is that of Hypothesis Tests for a Mean and Type I and Type II Errors, and is not repeated here.
The one-sample z test
To test $H_0: p = p_0$ against $H_A: p \ne p_0$, the formula sheet uses
$$z = \frac{\hat{p} - p_0}{\operatorname{SE}_0}, \qquad \operatorname{SE}_0 = \sqrt{\frac{p_0(1 - p_0)}{n}},$$
and refers $z$ to the standard normal distribution. The two-sided $p$-value is $2P(Z \ge |z|)$, and the test at level $\alpha$ rejects when $|z| \ge z_{1-\alpha/2}$. For a one-sided alternative such as $H_A: p > p_0$, the $p$-value is $P(Z \ge z)$ and the test rejects when $z \ge z_{1-\alpha}$. The NIST/SEMATECH e-Handbook of Statistical Methods (§7.2.4) gives the same statistic. Its rule for when the normal approximation may be used is $\min\{np_0,\ n(1 - p_0)\} \ge 5$.
The standard error uses $p_0$, not $\hat{p}$. The formula sheet's table of cautions makes the point explicitly: for a test use $p_0$, for an interval use $\hat{p}$.
Check your understanding. A test of a mean must estimate the standard deviation from the sample. Why can the test of a proportion compute its standard error without looking at the sample at all?
Because a test computes probabilities on the assumption that $H_0$ is true. If $p = p_0$, the variance of $\hat{p}$ is known exactly, $p_0(1-p_0)/n$, and there is nothing to estimate. For a mean, the null hypothesis fixes the mean but says nothing about the standard deviation, which must be estimated from the sample. For a proportion, the null hypothesis fixes both.
The worked test
Back to the sample of 60 men. It has $x = 15$, so $\hat{p} = 15/60 = 0.2500$. Against $p_0 = 0.224$, with $np_0 = 13.4$ and $n(1-p_0) = 46.6$ both above 5,
$$\operatorname{SE}_0 = \sqrt{\frac{0.224 \times 0.776}{60}} = 0.05382, \qquad z = \frac{0.2500 - 0.224}{0.05382} = 0.483,$$
and the two-sided $p$-value is $2P(Z \ge 0.483) = 0.629$. The data are entirely consistent with $p = 0.224$. The 95% Wilson interval, $(0.158,\ 0.372)$, contains 0.224. It also contains 0.3175, the true value: the sample cannot distinguish the two.
The NIST example
The handbook (§7.2.4) works a one-sided test in manufacturing: 26 defective wafers among 200 made by a new process, against a historical defect rate of at most $p_0 = 0.10$. The script reproduces its statistic,
$$z = \frac{0.13 - 0.10}{\sqrt{0.10 \times 0.90/200}} = 1.414.$$
This is below $z_{0.95} = 1.645$, so the test does not conclude that the new process is worse; the one-sided $p$-value is 0.079. The script also reproduces the handbook's one-sided 95% Wilson lower bound for the defect rate, 0.09577. It lies below 0.10, the same conclusion reached from the interval.
That agreement between a test and an interval is not automatic, as the next section shows.
Tests and intervals: which ones match
A test at level $\alpha$ and a $1 - \alpha$ confidence interval correspond when the interval contains exactly the null values the test does not reject. For a mean, the $t$ test and the $t$ interval correspond in this way, as Hypothesis Tests for a Mean shows. For a proportion, the $z$ test and the textbook (Wald) interval do not, because they use different standard errors.
The interval that matches the $z$ test is Wilson's. Confidence Intervals for a Proportion derives it as the set of $p$ satisfying $|\hat{p} - p| \le z\sqrt{p(1-p)/n}$. That is precisely the set of $p_0$ for which the $z$ test does not reject. The script confirmed this for every possible count from 0 to 60 with $p_0 = 0.224$: the test rejects if and only if 0.224 lies outside the 95% Wilson interval. With the Wald interval the agreement fails for some counts. The consistent pairing, then, is the $z$ test with the Wilson interval, one more reason to prefer the Wilson interval in reports.
Some texts and software test with the estimated standard error $\sqrt{\hat{p}(1-\hat{p})/n}$ instead, which is the test the Wald interval inverts. For the worked sample the two statistics are nearly equal ($0.465$ against $0.483$). Over repeated samples the difference matters, as we will see.
The exact binomial test
When $n$ is small or $p_0$ is near 0 or 1, the normal approximation is unreliable. The formula sheet's alternative is then the exact binomial test. Under $H_0$ the count $X$ is binomial with parameters $n$ and $p_0$, so the probability of any result can be computed exactly. A one-sided $p$-value is a binomial tail probability, for example $P_{p_0}(X \ge x)$ for $H_A: p > p_0$.
For a two-sided test there is no single obvious rule, because the binomial is not symmetric. The rule scipy's binomtest uses, and the one used here, adds the probabilities of every outcome that is no more likely under $H_0$ than the one observed:
$$p\text{-value} = \sum_{k:\ P_{p_0}(X = k) \le P_{p_0}(X = x)} P_{p_0}(X = k).$$
For the worked sample, the counts no more likely than 15 under $p_0 = 0.224$ with $n = 60$ have total probability 0.642. That is the exact $p$-value, close to the $z$ test's 0.629. For the NIST wafers, the exact one-sided $p$-value $P(X \ge 26)$ with $p_0 = 0.10$ and $n = 200$ is 0.1005, against the normal approximation's 0.079. Even at $n = 200$ the two differ noticeably in the tail, for the reasons explained in The Normal Approximation to the Binomial.
The test is built from binomial probabilities, so its rejection region contains only outcomes whose total probability under $H_0$ is at most $\alpha$. Its Type I error rate can never exceed $\alpha$. Like the Clopper–Pearson interval, though, it usually falls short of $\alpha$, which costs power.
The actual Type I error rates
A test's stated level is a promise about its Type I error rate. Is the promise kept? For any test of a proportion this can be computed exactly, by adding the binomial probabilities under $p_0$ of all the counts the test rejects. At $n = 60$ and $p_0 = 0.224$, the $z$ test rejects when $x \le 7$ or $x \ge 20$, and the exact test when $x \le 7$ or $x \ge 21$. Their actual Type I error rates are 0.0612 and 0.0446.
Check your understanding. The test with the estimated standard error is the test the Wald interval inverts. Given how the Wald interval's coverage fell short, should this test's actual error rate be above or below 0.05?
Above. At $n = 60$ it rejects when $x \le 8$ or $x \ge 21$, with an error rate of 0.0749. Over all sample sizes from 10 to 200 it averages 0.0634, and it reaches 0.1892 at $n = 13$. A test that rejects too often is the mirror image of an interval that misses too often.
| Test ($p_0 = 0.224$, $\alpha = 0.05$, $10 \le n \le 200$) | Lowest | Highest | Mean | Share of $n$ above 0.05 |
|---|---|---|---|---|
| $z$ test, null SE | 0.0203 | 0.0705 ($n = 29$) | 0.0492 | 45.5% |
| Exact binomial test | 0.0114 | 0.0498 ($n = 85$) | 0.0422 | 0% |
| $z$ test, estimated SE | 0.0439 | 0.1892 ($n = 13$) | 0.0634 | 85.9% |
The $z$ test with the null standard error keeps its promise on average, with oscillations of a point or two in either direction. The version with the estimated standard error is too liberal on average, and badly so at small $n$. The exact test is conservative, as its construction guarantees. None of the three can hit 0.05 exactly at most sample sizes. With a whole-number count only a few error rates are achievable, and a nonrandomized test must choose one of them.
So much for errors when $H_0$ is true. The opening example was about the other kind: a real difference that the test missed.
Power
Here is the idea. The rejection region is fixed by the null distribution of $\hat{p}$, but the chance of landing in it is computed under the alternative, and the two distributions have different standard deviations.
Suppose the true proportion is $p_1 \ne p_0$. Consider the upper half of the rejection region, $\hat{p} \ge p_0 + z_{1-\alpha/2}\operatorname{SE}_0$. The lower half has negligible probability when $p_1 > p_0$, as for means. Under the alternative, $\hat{p}$ is approximately normal with mean $p_1$ and a different standard deviation, $\sqrt{p_1(1-p_1)/n}$. Standardizing with it,
$$1 - \beta \approx P\!\left(\hat{p} \ge p_0 + z_{1-\alpha/2}\sqrt{\frac{p_0(1-p_0)}{n}}\right) = \Phi\!\left(\frac{(p_1 - p_0)\sqrt{n} - z_{1-\alpha/2}\sqrt{p_0(1-p_0)}}{\sqrt{p_1(1-p_1)}}\right),$$
after multiplying numerator and denominator by $\sqrt{n}$. For $p_1 < p_0$ the same expression holds with $|p_1 - p_0|$. The structure is that of the power of a test of a mean, with one difference. The standard deviation under the alternative, $\sqrt{p_1(1-p_1)}$, is not the one under the null, $\sqrt{p_0(1-p_0)}$, because for a proportion the variance moves with the mean.
Back to the 60 young men, with $p_0 = 0.224$, $p_1 = 0.3175$ and $n = 60$:
$$1 - \beta \approx \Phi\!\left(\frac{0.0935\sqrt{60} - 1.96\sqrt{0.224 \times 0.776}}{\sqrt{0.3175 \times 0.6825}}\right) = 0.421.$$
The exact power is the binomial probability under $p_1$ of the rejection region $x \le 7$ or $x \ge 20$, and it is 0.444. Drawing 10,000 samples of 60 from the 822 men for each of 30 seeds gave a median rejection rate of 0.4423 (interquartile range 0.4405 to 0.4445). The exact binomial test, whose upper rejection region starts one count later, has power 0.339. The sample of 60 in the opening was, in effect, a coin toss weighted slightly against detection.
Sample size for a test
How many men would have given the test a fair chance? Setting the power expression equal to $1 - \beta$ means its argument must equal $z_{1-\beta}$:
$$(p_1 - p_0)\sqrt{n} - z_{1-\alpha/2}\sqrt{p_0(1-p_0)} = z_{1-\beta}\sqrt{p_1(1-p_1)},$$
and solving for $n$ gives the formula on the course's formula sheet:
$$n = \left(\frac{z_{1-\alpha/2}\sqrt{p_0(1-p_0)} + z_{1-\beta}\sqrt{p_1(1-p_1)}}{p_1 - p_0}\right)^2 \ \text{(two-sided)},$$
with $z_{1-\alpha}$ in place of $z_{1-\alpha/2}$ for a one-sided test, rounded up. The NIST handbook (§7.2.4.2) gives the same formulas, citing Fleiss, Levin and Paik for the derivation. Each standard deviation is attached to the quantile it belongs to:
- $z_{1-\alpha/2}$ places the critical value, using the null distribution;
- $z_{1-\beta}$ measures the distance from the critical value to $p_1$, using the alternative distribution.
For a study of young men powered to detect $p_1 = 0.3175$ against $p_0 = 0.224$:
| Test | Power | $n$ from the formula |
|---|---|---|
| Two-sided, $\alpha = 0.05$ | 80% | 167.11, so 168 |
| Two-sided, $\alpha = 0.05$ | 90% | 228.53, so 229 |
| One-sided, $\alpha = 0.05$ | 80% | 132.77, so 133 |
| One-sided, $\alpha = 0.05$ | 90% | 188.03, so 189 |
The script then drew a sample of 168 men. Of them, 57 had hypertension: $\hat{p} = 0.339$, $z = 3.58$, $p = 0.0003$ (exact test, $p = 0.0006$), with Wilson interval $(0.272,\ 0.414)$, which excludes 0.224. Repeated 10,000 times for each of 30 seeds, the $z$ test rejected with median rate 0.818. That is above the planned 0.80, although the binomial calculation below gives 0.788 at this $n$. The reason is that 168 is a fifth of the 822 men, and sampling without replacement from a small population makes $\hat{p}$ less variable. (The finite population correction of the normal approximation article is $\sqrt{654/821} = 0.89$ here.)
The sawtooth in exact power
Check your understanding. The formula says 168 men give 80% power. Computed exactly from the binomial, will the power of the $z$ test at $n = 168$ be at least 0.80? And does adding one more man always add power?
The answer to both is no. For a binomial count, the exact power of a test does not rise smoothly with $n$. At 168 it is 0.788, and the table below shows 167 men doing better than 168.
| $n$ | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 |
|---|---|---|---|---|---|---|---|---|---|
| $z$ test | 0.777 | 0.792 | 0.807 | 0.820 | 0.788 | 0.802 | 0.816 | 0.829 | 0.798 |
| Exact test | 0.777 | 0.792 | 0.757 | 0.773 | 0.788 | 0.802 | 0.768 | 0.783 | 0.798 |
Each tooth comes from the critical count. As $n$ grows with the critical count fixed, the rejection region covers more of the alternative distribution and power rises. When the critical count steps up by one, power drops. For the $z$ test the formula's 168 itself falls short, with exact power 0.788, while 166, 167 and 169 exceed 0.80. The power stays above 0.80 for every larger $n$ only from 176 on. For the exact test, which is more conservative, the corresponding values are 169 and 186. Fleiss, Levin and Paik's continuity correction, which NIST reports as $n + 1/|p_1 - p_0|$, gives $167.11 + 1/0.0935 = 177.8$. Here that is a reasonable allowance for planning an analysis with the exact test or with a continuity-corrected statistic.
The practical lesson: the normal formula is a sound starting point. But for a study that will be analyzed with an exact test, or where the margin matters, the exact power should be computed at the chosen $n$ and at a few values around it.
Sample size for estimating a proportion
Many studies do not test a hypothesis but estimate a prevalence. The design question is then how many observations give a 95% interval of half-width at most $E$, the margin of error. Setting the Wald margin equal to $E$,
$$z_{1-\alpha/2}\sqrt{\frac{p(1-p)}{n}} = E \quad\Longrightarrow\quad n = \frac{z_{1-\alpha/2}^2\,p(1 - p)}{E^2},$$
rounded up. The formula needs a planning value for $p$, the very quantity the study is meant to estimate. Because $p(1-p)$ is at most $1/4$, at $p = 0.5$, the choice $p = 0.5$ gives a sample size that is enough whatever $p$ turns out to be: $n = z_{1-\alpha/2}^2/(4E^2)$.
| Margin $E$ | Planning $p$ | $n$ (95%) |
|---|---|---|
| 0.05 | 0.5 | 384.15, so 385 |
| 0.03 | 0.5 | 1,067.07, so 1,068 |
| 0.01 | 0.5 | 9,603.65, so 9,604 |
| 0.05 | 0.23 | 272.13, so 273 |
| 0.03 | 0.23 | 755.91, so 756 |
| 0.02 | 0.23 | 1,700.81, so 1,701 |
Halving the margin quadruples the sample, and a planning value well away from 0.5 saves a good deal. For one more planning case, take $p = 0.20$ and $E = 0.03$. With $z = 1.96$ the formula gives $1.96^2 \times 0.16/0.0009 = 682.95$, so 683 participants.
The formula has the same blind spot as its counterpart for means in One-Sided Confidence Bounds and Sample Size. It sets the margin equal to $E$ at the planning value of $p$, so about half of all studies will see a wider one.
Check your understanding. Planning with the true prevalence of young men, 0.3175, and $E = 0.05$ gives $n = 333$. Planning with $p = 0.5$ gives 385. What do the extra 52 participants buy?
A guarantee. With 333 men, the realized Wald half-width is at most 0.05 with probability 0.492. With 385, the half-width is at most 0.05 in every possible sample, with an expected value of 0.046. The cautious planning value removes the coin toss from the design.
The estimation formula is the test formula with $z_{1-\beta} = 0$, that is, with power one half. Power and Sample Size for a Test of a Mean found the same relation for means. In both forms the sample size should then be increased for expected nonresponse or dropout: divide by the expected response rate, or by one minus the dropout rate.
Summary
| Quantity | Formula |
|---|---|
| Test statistic | $z = (\hat{p} - p_0)/\sqrt{p_0(1-p_0)/n}$ |
| Validity check | $np_0 \ge 5$ and $n(1-p_0) \ge 5$; otherwise the exact binomial test |
| Interval that matches the $z$ test | Wilson |
| Power (two-sided, approximate) | $\Phi\big((\lvert p_1 - p_0\rvert\sqrt{n} - z_{1-\alpha/2}\sqrt{p_0(1-p_0)})/\sqrt{p_1(1-p_1)}\big)$ |
| Sample size for a test | $n = \big[(z_{1-\alpha/2}\sqrt{p_0(1-p_0)} + z_{1-\beta}\sqrt{p_1(1-p_1)})/(p_1 - p_0)\big]^2$, rounded up; check exact power nearby |
| Sample size for a margin $E$ | $n = z_{1-\alpha/2}^2\,p(1-p)/E^2$; $p = 0.5$ when nothing is known |
What we have learned
The test of a proportion knows its standard error under the null hypothesis, $\sqrt{p_0(1-p_0)/n}$, so it estimates nothing. Inverted, it gives Wilson's interval, not the textbook one. Computed exactly, its actual error rate oscillates around 0.05. The exact binomial test never exceeds 0.05 but averages 0.042, and the test with the estimated standard error runs too high. Power and sample size follow the pattern for means, with a different standard deviation under each hypothesis. The 60 young men of the opening had only 44% power; 168 men would have had about 80%, give or take the sawtooth of exact power. The sign test of the nonparametric unit is this exact binomial test with $p_0 = 0.5$, applied to the signs of paired differences.
So far one group has been compared with a fixed number. In practice the comparison is more often between two groups, both sampled: young men against young women, treatment against control. Comparing Two Proportions extends the test, the interval and the sample size to that case.
In Python
from math import sqrt, ceil
from statistics import NormalDist
from scipy import stats
x, n, p0 = 15, 60, 0.224
z = (x / n - p0) / sqrt(p0 * (1 - p0) / n)
print(z, 2 * stats.norm.sf(abs(z))) # about 0.483, 0.629
print(stats.binomtest(x, n, p0).pvalue) # exact test: about 0.642
q = NormalDist().inv_cdf
p1 = 0.3175
n_req = ((q(0.975) * sqrt(p0 * (1 - p0)) + q(0.80) * sqrt(p1 * (1 - p1))) / (p1 - p0)) ** 2
print(ceil(n_req)) # 168
Every number in this article is computed by tools/experiments/applied-statistics/tests-and-sample-size-for-a-proportion.py in the site's repository; its exact binomial test agrees with scipy's binomtest to within $10^{-13}$.
References
- Brown, L. D., Cai, T. T., and DasGupta, A. (2001). Interval estimation for a binomial proportion. Statistical Science, 16(2), 101–133. record · 4 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: DEMO_J, BPX_J and BPQ_J data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
- NIST/SEMATECH. e-Handbook of Statistical Methods, §7.2.4 (Does the proportion of defectives meet requirements?), §7.2.4.1 (Confidence intervals) and §7.2.4.2 (Sample sizes required). National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/ record · 30 pages
- Ostchega, Y., Fryar, C. D., Nwankwo, T., and Nguyen, D. T. (2020). Hypertension Prevalence Among Adults Aged 18 and Over: United States, 2017–2018. NCHS Data Brief No. 364. National Center for Health Statistics. record · 6 pages
- Pagano, M., Gauvreau, K., and Mattie, H. (2022). Principles of Biostatistics (3rd ed.). CRC Press. The course text. record · 35 pages
- SciPy documentation.
scipy.stats.binomtest. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.binomtest.html record - Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158), 209–212. record · 3 pages