In the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018, 31.8% of the 822 men aged 18 to 39 had hypertension, against 14.6% of the 874 women of the same ages. A study samples 80 young men and 80 young women. It finds 26 hypertensive men and 9 hypertensive women: 32.5% against 11.3%. Is the difference in the samples evidence of a difference between the populations, and how large might the population difference be?
Comparing two proportions is the most common comparison in medical research. It is the design of nearly every trial with a yes-or-no outcome, such as recovered or not, or readmitted or not, in a treatment group and a control group. The single-proportion methods it builds on are in Confidence Intervals for a Proportion and Tests and Sample Size for a Proportion. The outcome, hypertension by the 2017 ACC/AHA definition, is described in The Normal Approximation to the Binomial.
First we derive the two-proportion $z$ test and the interval for the difference on the course's formula sheet. Then we show that the chi-square test of a $2 \times 2$ table is the same test in a different form. Next come the exact alternative for small samples and three ways to measure the size of a difference. Finally we derive the sample size each group needs, and look back over the course.
The setting
The notation follows the formula sheet. From population 1, with proportion $p_1$ having the outcome, a random sample of $n_1$ contains $x_1$ with the outcome. From population 2, independently, a sample of $n_2$ contains $x_2$. The sample proportions are
$$\hat{p}_1 = \frac{x_1}{n_1}, \qquad \hat{p}_2 = \frac{x_2}{n_2}.$$
The hypotheses are $H_0: p_1 = p_2$ (equivalently $p_1 - p_2 = 0$) and, for a two-sided test, $H_A: p_1 \ne p_2$.
The natural estimate of $p_1 - p_2$ is $\hat{p}_1 - \hat{p}_2$. Its mean is $p_1 - p_2$. Since the samples are independent, its variance is the sum of the two variances from the normal approximation article:
$$\operatorname{Var}(\hat{p}_1 - \hat{p}_2) = \frac{p_1(1-p_1)}{n_1} + \frac{p_2(1-p_2)}{n_2}.$$
For large samples each $\hat{p}_i$ is approximately normal, and so is their difference. The test and the interval both standardize the difference by an estimate of this standard deviation. They differ in which estimate they use, for the same reason the one-sample test and interval differ.
The pooled z test
Under $H_0$, the two proportions are equal to a common value $p$, and the variance becomes
$$p(1 - p)\left(\frac{1}{n_1} + \frac{1}{n_2}\right).$$
The null hypothesis does not say what $p$ is, so it must be estimated, and under $H_0$ both samples estimate it. The best estimate combines them. It is the pooled proportion,
$$\hat{p} = \frac{x_1 + x_2}{n_1 + n_2} = \frac{n_1\hat{p}_1 + n_2\hat{p}_2}{n_1 + n_2},$$
the overall proportion with the outcome when the two samples are put together. (The formula sheet writes it $\hat{p}$ without a subscript; the subscripted $\hat{p}_1$ and $\hat{p}_2$ are the separate groups.) The test statistic is
$$z = \frac{\hat{p}_1 - \hat{p}_2}{\sqrt{\hat{p}(1-\hat{p})\left(\dfrac{1}{n_1} + \dfrac{1}{n_2}\right)}},$$
referred to the standard normal distribution, with two-sided $p$-value $2P(Z \ge |z|)$. The NIST/SEMATECH e-Handbook of Statistical Methods (§7.3.3) gives the same statistic for comparing the defect rates of two processes.
Pooling is the counterpart of the pooled variance in the two-sample $t$-test, with one difference. There, pooling is an assumption that may fail. Here, equal variances follow from the null hypothesis itself, since equal proportions imply equal Bernoulli variances.
The worked test
Back to the 80 men and 80 women:
$$\hat{p}_1 = \frac{26}{80} = 0.3250, \qquad \hat{p}_2 = \frac{9}{80} = 0.1125, \qquad \hat{p} = \frac{26 + 9}{160} = 0.2188,$$
$$\operatorname{SE}_0 = \sqrt{0.2188 \times 0.7813 \times \left(\frac{1}{80} + \frac{1}{80}\right)} = 0.06536, \qquad z = \frac{0.3250 - 0.1125}{0.06536} = 3.251,$$
and the two-sided $p$-value is 0.00115. The samples give strong evidence that hypertension is more common among young men than young women. The usual check for the normal approximation is that each group has at least 5 expected outcomes and 5 expected non-outcomes under $H_0$, that is, $n_i\hat{p}$ and $n_i(1 - \hat{p})$. Here they are 17.5 and 62.5 in each group, so the check is comfortably met.
The test says the difference is real. How large is it?
The interval for the difference
Check your understanding. The test pooled the two samples to estimate one common proportion. Should an interval for $p_1 - p_2$ pool them too?
No. An interval for $p_1 - p_2$ cannot assume the proportions are equal, since its purpose is to say how far apart they are. It uses the unpooled standard error, each group's variance estimated from its own proportion:
$$\operatorname{SE} = \sqrt{\frac{\hat{p}_1(1-\hat{p}_1)}{n_1} + \frac{\hat{p}_2(1-\hat{p}_2)}{n_2}}, \qquad (\hat{p}_1 - \hat{p}_2) \pm z_{1-\alpha/2}\operatorname{SE}.$$
The formula sheet lists both standard errors: pooled for the test, unpooled for the interval. It is the two-sample form of its caution to use $p_0$ for a test and $\hat{p}$ for an interval. For the worked samples,
$$\operatorname{SE} = \sqrt{\frac{0.3250 \times 0.6750}{80} + \frac{0.1125 \times 0.8875}{80}} = 0.06317,$$
and the 95% interval is $0.2125 \pm 1.96 \times 0.06317 = (0.089,\ 0.336)$. The population difference, 0.1711, lies inside it. The sample happened to exaggerate the difference, and the interval is wide enough to say so. With 80 per group, the data are consistent with a difference anywhere from 9 to 34 percentage points.
This is a Wald interval. It shares the weakness of the one-sample Wald interval, though in a milder form, because the errors of two estimates partly offset. The script computed its exact coverage at the true proportions, 0.3175 and 0.1465, by summing the probabilities of all pairs of counts whose interval contains 0.1711. The coverage is 0.9366 with 20 per group, 0.9448 with 40 and 0.9455 with 80. For small samples, or proportions near 0 or 1, intervals built on Wilson's method for each group are preferred, along the lines of the one-sample case.
Unlike the one-sample case, the pooled test and the unpooled interval do not correspond exactly. Near the boundary, the test can reject while the interval just includes 0, or the reverse. They agree in the worked example, where $z = 3.25$ and the interval is far from 0.
Why not test with the unpooled standard error, so that test and interval match? That test gives $z = 3.36$ here, but its actual Type I error rate runs above $\alpha$ in small samples. Computed exactly at $p_1 = p_2 = 0.2294$, it is 0.0675 with 20 per group, against 0.0507 for the pooled test. At 40 and 80 per group the two are 0.0525 against 0.0505 and 0.0517 against 0.0513. The pooled statistic keeps the stated level more closely, which is why it is the test on the formula sheet.
The same test as a 2 x 2 table
Many reports present the same comparison as a table and a chi-square test. Is that a different test? The data can be written as a contingency table with one row per group and one column per outcome:
| Hypertension | No hypertension | Total | |
|---|---|---|---|
| Men | $a = 26$ | $b = 54$ | $n_1 = 80$ |
| Women | $c = 9$ | $d = 71$ | $n_2 = 80$ |
| Total | $m_1 = 35$ | $m_2 = 125$ | $N = 160$ |
The chi-square test, which goes back to Karl Pearson, asks whether the rows and columns are independent, that is, whether the chance of hypertension is the same in both rows. If it is, the best estimate of that chance is the overall share $m_1/N$. The expected count in each cell is then its row total times its column share. For men with hypertension it is $E_{11} = n_1 m_1/N = 80 \times 35/160 = 17.5$, and similarly 62.5, 17.5 and 62.5 for the other cells. The statistic adds up the squared differences between observed and expected counts, each scaled by the expected count:
$$X^2 = \sum_{\text{cells}} \frac{(O - E)^2}{E} = \frac{(26 - 17.5)^2}{17.5} + \frac{(54 - 62.5)^2}{62.5} + \frac{(9 - 17.5)^2}{17.5} + \frac{(71 - 62.5)^2}{62.5} = 10.569.$$
It is referred to the chi-squared distribution with one degree of freedom, giving $p = 0.00115$.
Check your understanding. That $p$-value is exactly the $z$ test's. How is $X^2 = 10.569$ related to the $z$ statistic, 3.251?
It is its square: $3.2510^2 = 10.569$. The agreement is not a coincidence, and the proof shows why.
Proof that X² = z²
Here is the plan of the proof. Every cell of a $2 \times 2$ table turns out to deviate from its expected count by the same amount. That reduces $X^2$ to a short formula, and the square of $z$ reduces to the same formula.
The formal version. For the first cell,
$$a - \frac{n_1 m_1}{N} = \frac{a(a + b + c + d) - (a + b)(a + c)}{N} = \frac{ad - bc}{N}.$$
The observed and expected counts have the same row and column totals, so the deviations in a row or a column must sum to zero. The other cells therefore deviate by $-(ad - bc)/N$, $-(ad - bc)/N$ and $+(ad - bc)/N$. Hence
$$X^2 = \frac{(ad - bc)^2}{N^2}\sum_{\text{cells}}\frac{1}{E}.$$
The sum of reciprocals of the expected counts simplifies by grouping cells in the same row:
$$\sum \frac{1}{E} = \frac{N}{n_1}\left(\frac{1}{m_1} + \frac{1}{m_2}\right) + \frac{N}{n_2}\left(\frac{1}{m_1} + \frac{1}{m_2}\right) = \frac{N}{m_1 m_2}\cdot N\left(\frac{1}{n_1} + \frac{1}{n_2}\right) = \frac{N^3}{n_1 n_2 m_1 m_2},$$
using $m_1 + m_2 = N$ and $n_1 + n_2 = N$. So
$$X^2 = \frac{N(ad - bc)^2}{n_1 n_2 m_1 m_2},$$
the shortcut formula for a $2 \times 2$ table. Now the $z$ statistic. Its numerator is
$$\hat{p}_1 - \hat{p}_2 = \frac{a}{n_1} - \frac{c}{n_2} = \frac{a(c + d) - c(a + b)}{n_1 n_2} = \frac{ad - bc}{n_1 n_2},$$
and since the pooled proportion is $\hat{p} = m_1/N$, its squared denominator is
$$\frac{m_1}{N}\cdot\frac{m_2}{N}\cdot\frac{N}{n_1 n_2} = \frac{m_1 m_2}{N n_1 n_2}.$$
Dividing,
$$z^2 = \frac{(ad - bc)^2}{n_1^2 n_2^2}\cdot\frac{N n_1 n_2}{m_1 m_2} = \frac{N(ad - bc)^2}{n_1 n_2 m_1 m_2} = X^2.$$
The two statistics are equal for every $2 \times 2$ table. Their reference distributions match as well. The chi-squared distribution with one degree of freedom is by definition the distribution of the square of a standard normal variable (Confidence Intervals for a Mean defines the chi-squared family). So $P(\chi^2_1 \ge z^2) = P(|Z| \ge |z|)$, and the two-sided $z$ test and the chi-square test give identical $p$-values. The same relation holds for means: with two groups, the $F$ statistic of one-way analysis of variance is the square of the pooled $t$ statistic.
Each form has its uses:
- the $z$ form shows the direction of the difference, and it allows a one-sided test;
- the chi-square form generalizes to tables with more rows or columns, where several proportions are compared at once.
Small samples: Yates' correction and Fisher's exact test
Both forms rest on a normal approximation. What if the samples are small? Like the one-sample test, the chi-square test approximates a discrete distribution by a continuous one. Yates' continuity correction reduces each $|O - E|$ by one half before squaring,
$$X^2_{\text{Yates}} = \sum \frac{(|O - E| - 0.5)^2}{E},$$
which for the worked table gives 9.362 and $p = 0.00221$. scipy's chi2_contingency applies this correction by default for a $2 \times 2$ table. The correction moves the test toward the exact test described next, and like the exact test it tends to be conservative.
Fisher's exact test avoids the approximation altogether. It treats the row and column totals as fixed. Then, if group and outcome are independent, the count $a$ in the first cell has the hypergeometric distribution. It is the distribution of the number of hypertensive people among $n_1$ drawn without replacement from the $N$ people, of whom $m_1$ are hypertensive:
$$P(a) = \frac{\dbinom{m_1}{a}\dbinom{m_2}{n_1 - a}}{\dbinom{N}{n_1}}.$$
The NIST handbook (§7.3.3) gives this hypergeometric probability as the basis of the test. The two-sided $p$-value, as scipy's fisher_exact computes it, adds the probabilities of all tables with the same margins that are no more likely than the one observed. For the worked table it is 0.00193.
With 80 per group, all four methods agree that the difference is real. The $p$-value is 0.00115 for the $z$ and chi-square tests, 0.00221 with Yates' correction and 0.00193 for Fisher's test. They part ways when expected counts are small, and then Fisher's test is the safe choice, at the cost of some power.
Measuring the difference: risk difference, relative risk, odds ratio
The test asks whether the proportions differ. Three measures say by how much, and each answers a different question:
| Measure | Definition | Population | Sample |
|---|---|---|---|
| Risk difference | $p_1 - p_2$ | 0.171 | 0.213 |
| Relative risk | $p_1/p_2$ | 2.17 | 2.89 |
| Odds ratio | $\dfrac{p_1/(1-p_1)}{p_2/(1-p_2)}$ | 2.71 | 3.80 |
The difference is what this article's interval estimates. The relative risk says young men are about twice as likely as young women to have hypertension. The odds ratio is the measure that case-control studies and logistic regression produce. The difference is zero, and the two ratios are one, exactly when $p_1 = p_2$, so the test of this article tests them all. Intervals for the ratios, and when each measure is appropriate, are the subject of Relative Risk and the Odds Ratio in the probability unit. The library's Logistic Regression estimates odds ratios that adjust for other variables.
Sample size for comparing two proportions
The worked study found its difference with 80 per group. Was that a good design, and what should a new study plan for? The power calculation follows the one-sample case in Tests and Sample Size for a Proportion. Here is the idea: the critical value is placed with the pooled standard deviation that holds under $H_0$, and the chance of passing it is computed with the unpooled one that holds under the alternative.
Take equal groups of $n$, and write $\bar{p} = (p_1 + p_2)/2$ for the common proportion the pooled estimate approaches when the groups are of equal size. Under $H_0$ the standard deviation of $\hat{p}_1 - \hat{p}_2$ is about $\sqrt{2\bar{p}(1-\bar{p})/n}$. So the test rejects (in the direction of $p_1 > p_2$) when the difference exceeds $z_{1-\alpha/2}\sqrt{2\bar{p}(1-\bar{p})/n}$. Under the alternative the difference is approximately normal with mean $p_1 - p_2$ and standard deviation $\sqrt{[p_1(1-p_1) + p_2(1-p_2)]/n}$, so
$$1 - \beta \approx \Phi\!\left(\frac{|p_1 - p_2|\sqrt{n} - z_{1-\alpha/2}\sqrt{2\bar{p}(1-\bar{p})}}{\sqrt{p_1(1-p_1) + p_2(1-p_2)}}\right),$$
and setting the argument equal to $z_{1-\beta}$ gives the sample size per group:
$$n = \left(\frac{z_{1-\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_{1-\beta}\sqrt{p_1(1-p_1) + p_2(1-p_2)}}{p_1 - p_2}\right)^2.$$
A simpler and widely used version uses the alternative's variance in both places,
$$n = \frac{(z_{1-\alpha/2} + z_{1-\beta})^2\,[p_1(1-p_1) + p_2(1-p_2)]}{(p_1 - p_2)^2},$$
This is the corresponding approximation for a test with the unpooled standard error. It is close to the pooled formula when the proportions are not far apart. It is also the two-proportion form of the formula for two means in Sample Size for Comparing Two Means, with the Bernoulli variances in place of $\sigma_1^2 + \sigma_2^2$.
For a study designed to detect the population difference between young men and young women, 0.3175 against 0.1465, with a two-sided test at 0.05:
| Power | Pooled formula | Exact power at that $n$ | Simpler formula | Exact power at that $n$ |
|---|---|---|---|---|
| 80% | 94.39, so 95 per group | 0.810 | 91.65, so 92 | 0.796 |
| 90% | 125.86, so 126 | 0.903 | 122.69, so 123 | 0.900 |
The exact power is the probability, summed over every pair of counts, that the pooled test rejects. The pooled formula's 95 per group reaches 0.81, and the smallest group size with exact power of at least 0.80 is 93. The simpler formula, here, falls just short. The script also drew 95 men and 95 women from the NHANES populations, 10,000 times for each of 30 seeds. The pooled test rejected at a median rate of 0.8235. That is a little above the planned power, because the samples are a large fraction of the small populations.
So the worked study, with 80 per group, had about a three-in-four chance of finding the difference it found.
Check your understanding. Across all ages, 58.6% of men and 51.1% of women in the NHANES population have hypertension, a difference of 7.5 points. That is less than half the 17-point difference among young adults. Will detecting it with 80% power take more or less than four times the 95 per group?
More. It takes $n = 693.6$, so 694 per group (690.9 by the simpler formula), more than seven times the sample for young adults. Shrinking the difference to less than half would by itself more than quadruple $n$. The Bernoulli variances add the rest: they are larger for proportions near one half, 0.586 and 0.511, than for 0.318 and 0.147. The difference to be detected dominates the answer, as it did for means.
Summary
| Quantity | Formula |
|---|---|
| Pooled proportion | $\hat{p} = (x_1 + x_2)/(n_1 + n_2)$ |
| Test statistic | $z = (\hat{p}_1 - \hat{p}_2)\big/\sqrt{\hat{p}(1-\hat{p})(1/n_1 + 1/n_2)}$ |
| Interval for $p_1 - p_2$ | $(\hat{p}_1 - \hat{p}_2) \pm z_{1-\alpha/2}\sqrt{\hat{p}_1(1-\hat{p}_1)/n_1 + \hat{p}_2(1-\hat{p}_2)/n_2}$ |
| Chi-square for a $2 \times 2$ table | $X^2 = N(ad - bc)^2/(n_1 n_2 m_1 m_2) = z^2$, 1 degree of freedom |
| Small expected counts | Fisher's exact test |
| Sample size per group | $n = \big[z_{1-\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_{1-\beta}\sqrt{p_1(1-p_1) + p_2(1-p_2)}\big]^2/(p_1 - p_2)^2$ |
What we have learned
The test of two proportions uses the pooled standard error, because under the null hypothesis the groups share one variance; the interval uses the unpooled one, because it must not assume what it measures. For the 80 young men and 80 young women, $p = 0.00115$, and the difference lay between 9 and 34 percentage points. The chi-square test of the $2 \times 2$ table is the same test, since $X^2 = z^2$, and Fisher's exact test takes over when expected counts are small. Detecting the 17-point difference with 80% power needs about 95 per group.
This article ends the unit and the course. The course began with Descriptive Statistics: tables, graphs and summary numbers for the sample in hand. Rates and Standardization put rates and proportions to work in public health. Probability gave a language for chance, which Screening and Diagnostic Tests applied to diagnosis, still before any inference. Theoretical Probability Distributions supplied the binomial, Poisson and normal models, and Sampling Distributions showed how a statistic varies from sample to sample. Every unit after that used one move: compare an estimate with its sampling variability, against a reference distribution, and turn the result into an interval or a test. It served for means in Confidence Intervals, Hypothesis Testing and Comparison of Two Means, and for several means in Analysis of Variance. Nonparametric Methods replaced the normal reference distribution with one found by counting. This unit applied the move to counts of yes-or-no outcomes.
The unit's main caution reaches back over all of them. A stated confidence level or significance level is a claim about a method, and it can be checked. For proportions the check is exact, and it showed the textbook interval falling short. Regression, the natural next step, is in the library's Simple Linear Regression and Logistic Regression.
In Python
from math import sqrt
from scipy import stats
x1, n1, x2, n2 = 26, 80, 9, 80
p1, p2, p = x1 / n1, x2 / n2, (x1 + x2) / (n1 + n2)
z = (p1 - p2) / sqrt(p * (1 - p) * (1 / n1 + 1 / n2))
print(z, 2 * stats.norm.sf(abs(z))) # about 3.251, 0.00115
table = [[x1, n1 - x1], [x2, n2 - x2]]
print(stats.chi2_contingency(table, correction=False)) # statistic 10.569 = z**2
print(stats.chi2_contingency(table).pvalue) # Yates: about 0.00221
print(stats.fisher_exact(table).pvalue) # about 0.00193
Every number in this article is computed by tools/experiments/applied-statistics/comparing-two-proportions.py in the site's repository; its $z^2$ equals scipy's uncorrected chi-square statistic to ten decimal places.
References
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: DEMO_J, BPX_J and BPQ_J data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
- NIST/SEMATECH. e-Handbook of Statistical Methods, §7.3.3 (How can we determine whether two processes produce the same proportion of defectives?). National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/ record · 30 pages
- Ostchega, Y., Fryar, C. D., Nwankwo, T., and Nguyen, D. T. (2020). Hypertension Prevalence Among Adults Aged 18 and Over: United States, 2017–2018. NCHS Data Brief No. 364. National Center for Health Statistics. record · 6 pages
- Pagano, M., Gauvreau, K., and Mattie, H. (2022). Principles of Biostatistics (3rd ed.). CRC Press. The course text. record · 35 pages
- SciPy documentation.
scipy.stats.chi2_contingencyandscipy.stats.fisher_exact. https://docs.scipy.org/doc/scipy/reference/stats.html record