McNemar's Test for Paired Proportions

How to compare two proportions measured on the same people. Why only the discordant pairs inform the test, proved by factoring the multinomial likelihood; the exact conditional binomial test, its mid-p, asymptotic and continuity-corrected versions and their exact error rates; the variance of a difference of dependent proportions and what ignoring the pairing costs; intervals for the difference and for the conditional odds ratio; the number of pairs a study needs; and a test of symmetry for more than two categories. Worked on repeated blood pressure readings of 4,782 NHANES adults, with worked problems on a published small study, a sample of 300 adults, the goodness-of-fit form of the statistic and the likelihood ratio alternative.

The NHANES examination of 2017 to 2018 measured each participant's blood pressure up to three times at one sitting. Among the 4,782 adults with a first and a third systolic reading, 36.0% had a first reading of 130 mm Hg or more and 34.9% had a third reading that high. The same people supply both shares. Is a difference of about one percentage point evidence that the first reading tends to run higher than the third, or the noise of 4,782 people?

McNemar's test answers questions of this kind: two proportions measured on the same units, or on matched pairs of units, compared with each other. McNemar (1947) presented formulas "for judging the significance of the difference between correlated proportions" (his abstract), and Fagerland, Lydersen and Laake (2013) call the test "a simple and frequently used test for binary matched-pairs data". For the blood pressure readings it gives $z = 2.70$ and a two-sided p-value of 0.0069, while a test that ignores the pairing finds nothing.

First we set out the paired table and prove, by factoring its likelihood, that only the pairs whose two classifications differ carry information about the difference. Then we build the exact test and its three approximations and compute their error rates exactly. Next we find the variance of a difference of dependent proportions, which explains why the unpaired analysis fails, and we turn it into intervals and a sample size formula. Finally we extend the test to tables with more than two categories. The article uses the multinomial distribution and its covariances from The Chi-Square Goodness-of-Fit Test and the sign test of The Sign Test and the Wilcoxon Signed-Rank Test.

The paired table

Each adult contributes a pair of classifications: high or not on the first reading, high or not on the third. Cross-classifying the 4,782 pairs gives four counts. Write $n_{ij}$ for the number of pairs with first classification $i$ and second classification $j$, where 1 means "130 mm Hg or more" and 2 means "below 130":

Third reading high Third reading not high Total
First reading high $n_{11} = 1{,}496$ $n_{12} = 227$ $n_{1+} = 1{,}723$
First reading not high $n_{21} = 173$ $n_{22} = 2{,}886$ $n_{2+} = 3{,}059$
Total $n_{+1} = 1{,}669$ $n_{+2} = 3{,}113$ $n = 4{,}782$

This is not the $2 \times 2$ table of Comparing Two Proportions. There the rows were two independent groups of people. Here each person appears once, in the cell for their pair of outcomes, and the two proportions being compared are the row margin and the column margin of the same table. The share high on the first reading is $\hat{p}_1 = n_{1+}/n = 1{,}723/4{,}782$ and the share high on the third is $\hat{p}_2 = n_{+1}/n = 1{,}669/4{,}782$.

Their difference simplifies, because the 1,496 adults high on both readings are counted in both shares:

$$\hat{p}_1 - \hat{p}_2 = \frac{n_{11} + n_{12}}{n} - \frac{n_{11} + n_{21}}{n} = \frac{n_{12} - n_{21}}{n} = \frac{227 - 173}{4{,}782} = 0.0113,$$

a difference of 1.13 percentage points. The concordant pairs, $n_{11}$ and $n_{22}$, cancel. Only the discordant pairs remain: 227 adults high on the first reading and not on the third, and 173 adults with the reverse, 400 in all.

In the population that the pairs are drawn from, write $\pi_{ij}$ for the probability that a pair falls in cell $(i, j)$, and $\pi_{1+} = \pi_{11} + \pi_{12}$ and $\pi_{+1} = \pi_{11} + \pi_{21}$ for the two marginal probabilities. The same cancellation gives

$$\pi_{1+} - \pi_{+1} = \pi_{12} - \pi_{21}.$$

So the hypothesis that the two proportions are equal, called marginal homogeneity, is the hypothesis $\pi_{12} = \pi_{21}$: discordance in one direction is as likely as discordance in the other.

Why only the discordant pairs carry information

The cancellation shows that the estimate of the difference uses only $n_{12} - n_{21}$. A stronger statement is true: once the number of discordant pairs is known, the concordant pairs carry no information at all about whether the hypothesis holds. Here is the idea in one sentence. The likelihood of the table splits into a factor for "how many pairs are discordant" and a factor for "which way they go", and the hypothesis concerns only the second.

The formal version. Suppose the $n$ pairs are independent and each falls in cell $(i, j)$ with probability $\pi_{ij}$. Then the counts are multinomial, and the probability of the observed table is

$$P(n_{11}, n_{12}, n_{21}, n_{22}) = \frac{n!}{n_{11}!\,n_{12}!\,n_{21}!\,n_{22}!}\,\pi_{11}^{n_{11}}\pi_{12}^{n_{12}}\pi_{21}^{n_{21}}\pi_{22}^{n_{22}}.$$

Write $\delta = \pi_{12} + \pi_{21}$ for the probability that a pair is discordant, assumed positive, and $\theta = \pi_{12}/\delta$ for the probability that a discordant pair is discordant in the first direction. Then $\pi_{12} = \delta\theta$ and $\pi_{21} = \delta(1 - \theta)$. Write $m = n_{12} + n_{21}$ for the number of discordant pairs. Substituting, and multiplying and dividing by $m!$,

$$P = \underbrace{\frac{n!}{n_{11}!\,m!\,n_{22}!}\,\pi_{11}^{n_{11}}\,\delta^{m}\,\pi_{22}^{n_{22}}}_{\text{trinomial: } (n_{11},\ m,\ n_{22})} \times \underbrace{\binom{m}{n_{12}}\theta^{n_{12}}(1 - \theta)^{n_{21}}}_{\text{binomial: } n_{12} \text{ given } m}.$$

The first factor is the probability that the pairs split into $n_{11}$ concordant-high, $m$ discordant and $n_{22}$ concordant-low, a trinomial distribution with probabilities $\pi_{11}$, $\delta$ and $\pi_{22}$ (which sum to 1). The second is the binomial probability of $n_{12}$ successes in $m$ trials. Since the first factor is the probability of $(n_{11}, m, n_{22})$, the second is the conditional probability of $n_{12}$ given those three counts:

$$n_{12} \mid (n_{11}, m, n_{22}) \sim \text{Binomial}(m, \theta).$$

Three consequences follow, and the rest of the article uses them.

  • The hypothesis $\pi_{12} = \pi_{21}$ is $\theta = \tfrac12$. The parameter $\theta$ appears only in the second factor, and the conditional distribution of $n_{12}$ given $m$ does not involve $\pi_{11}$, $\pi_{22}$ or $\delta$. A test built on it has the same distribution under the hypothesis whatever the concordant pairs are doing.
  • The concordant counts inform only $\pi_{11}$, $\pi_{22}$ and $\delta$. Two studies with the same $n_{12}$ and $n_{21}$ but different numbers of concordant pairs give the same conditional test.
  • The concordant pairs still matter for the size of the difference. The difference $\pi_{12} - \pi_{21} = \delta(2\theta - 1)$ involves $\delta$, so an estimate of it on the probability scale needs $n$, as the section on estimation shows.

Example (are 4,382 adults wasted?). A reader may object that the test ignores 4,382 of the 4,782 adults. It does, and it loses nothing by doing so. Their readings agree, so they say nothing about which reading tends to be higher. Their number does affect the precision of an estimated difference, but not the test of whether one exists.

The test

Under the hypothesis, each discordant pair is a fair coin: high-then-not with probability $\tfrac12$ and not-then-high with probability $\tfrac12$, independently of the others. That is the sign test of The Sign Test and the Wilcoxon Signed-Rank Test, with each discordant pair a nonzero difference and each concordant pair a zero difference, dropped. So McNemar's exact test is the sign test, and its two-sided p-value is

$$p_{\text{exact}} = \min\Big\{1,\ 2\sum_{x=0}^{\min(n_{12},\,n_{21})}\binom{m}{x}\frac{1}{2^m}\Big\}.$$

For large $m$ the binomial is approximated by the normal with the same mean $m/2$ and variance $m/4$ (The Normal Approximation to the Binomial). Standardizing $n_{12}$,

$$z = \frac{n_{12} - m/2}{\sqrt{m/4}} = \frac{2n_{12} - m}{\sqrt{m}} = \frac{n_{12} - n_{21}}{\sqrt{n_{12} + n_{21}}},$$

since $2n_{12} - m = n_{12} - n_{21}$. Its square, $z^2 = (n_{12} - n_{21})^2/(n_{12} + n_{21})$, is referred to the chi-squared distribution with one degree of freedom. McNemar's (1947) abstract points out "the chi square equivalent of one of the developed formulas". Fagerland et al. (2013) note that $z$ is the score statistic for marginal homogeneity; Problem 4 checks this against the likelihood ratio statistic.

Two refinements are in use.

  • The continuity correction. Edwards (1948) proposed correcting McNemar's formula for continuity. The count $n_{12}$ moves in steps of 1, so the usual correction subtracts $\tfrac12$ from $|n_{12} - m/2|$. Since $n_{12} - n_{21} = 2(n_{12} - m/2)$, that is 1 subtracted from $|n_{12} - n_{21}|$: $z_{\text{cc}} = (|n_{12} - n_{21}| - 1)/\sqrt{m}$. Breslow and Day (1980, §5.2, p. 165, equation 5.5) give the corrected statistic as $\chi^2 = (|n_{10} - n_{01}| - 1)^2/(n_{10} + n_{01})$ in their notation.
  • The mid-p value. Subtract the probability of the observed $n_{12}$ from the exact two-sided p-value: $p_{\text{mid}} = p_{\text{exact}} - \binom{m}{n_{12}}2^{-m}$, and $p_{\text{mid}} = 1 - \tfrac12\binom{m}{n_{12}}2^{-m}$ when $n_{12} = n_{21}$ (Fagerland et al., 2013, equations 5 and 6). It counts half of the observed outcome as "at least as extreme" instead of all of it.

For the blood pressure readings, $m = 400$ and

$$z = \frac{227 - 173}{\sqrt{400}} = \frac{54}{20} = 2.70, \qquad z^2 = 7.29.$$

The four two-sided p-values are:

Version p-value
Asymptotic, $z = 2.70$ 0.0069
Continuity-corrected, $z_{\text{cc}} = 2.65$ 0.0080
Exact conditional 0.0080
Mid-p 0.0069

All four reject the hypothesis at the 1% level. Among these adults, a discordant classification went from high to not high more often than the reverse, and the first reading was classified high more often than the third.

The normal approximation is a statement about the conditional distribution given $m$, and it improves as $m$ grows. With 400 discordant pairs it is close: the asymptotic and mid-p values agree to four places here, and the corrected value matches the exact one.

How the four versions behave

Which version should be reported? The answer depends on how often each rejects a true hypothesis, and that can be computed exactly. For $n$ pairs, the number of discordant pairs $m$ is binomial with $n$ trials and probability $\delta$. Given $m$, under the hypothesis, $n_{12}$ is binomial with $m$ trials and probability $\tfrac12$. The probability that a test rejects is therefore a double sum over $m$ and $n_{12}$ of the product of the two binomial probabilities, counted when the test's p-value is below 0.05. The script evaluates it for each version.

The exact test never exceeds its level. For a fixed $m$, let $F$ be the distribution function of the binomial with $m$ trials and probability $\tfrac12$, and let $c$ be the largest integer with $2F(c) \le \alpha$. The exact test rejects when $\min(n_{12}, n_{21}) \le c$, that is, when $n_{12} \le c$ or $n_{12} \ge m - c$. If $c < m/2$ these two sets are disjoint, and by the symmetry of the binomial with probability $\tfrac12$ each has probability $F(c)$, so the rejection probability is $2F(c) \le \alpha$. (If no such $c$ exists the test never rejects; $c \ge m/2$ is impossible, since $2F(c) \ge 2 \cdot \tfrac12 = 1 > \alpha$ there.) Averaging over $m$ keeps the bound: the unconditional rejection probability is at most $\alpha$.

The bound is reached only when some $2F(c)$ equals $\alpha$ exactly, which a discrete distribution rarely allows. So the exact test usually rejects less often than its nominal level, and with few discordant pairs much less often. With the discordance probability of the blood pressure readings, $\delta = 400/4{,}782$, or 8.4%, the exact null rejection rates at the 5% level are:

Pairs $n$ Asymptotic Corrected Exact Mid-p
25 0.0152 0.0004 0.0004 0.0028
50 0.0438 0.0056 0.0056 0.0184
100 0.0442 0.0191 0.0192 0.0387
200 0.0493 0.0248 0.0284 0.0463
500 0.0493 0.0335 0.0351 0.0487

With 100 pairs and half of them expected to be discordant ($\delta = 0.5$), the rates are 0.0494 (asymptotic), 0.0357 (corrected), 0.0367 (exact) and 0.0494 (mid-p).

What matters is the expected number of discordant pairs, $n\delta$, not $n$. With 25 pairs at $\delta = 8.4\%$ a study expects about two discordant pairs, and no version can reject often. Even at 500 pairs, about 42 discordant, the exact test rejects a true hypothesis 3.5% of the time instead of 5%. That conservatism is paid for in power.

These calculations agree with a much larger study. Fagerland et al. (2013) computed the rejection rates of the same four tests, and of an exact unconditional test, over 9,595 null scenarios with 10 to 100 pairs. In their Table 4 the exact conditional test had a mean rate of 0.0201 and a maximum of 0.0367. The asymptotic test exceeded 5% in 29.4% of the scenarios, but its largest rate was 0.0537. The mid-p test "did not violate the nominal level in any of the 9595 scenarios", although nothing guarantees that it cannot. They recommend the mid-p test, or the asymptotic test "if small but frequent violations of the nominal level is acceptable", and recommend against the exact conditional test and the corrected test "in any situation". The tables above, at the readings' discordance rate, show the same ordering.

Example (when the versions disagree). Fagerland et al. (2013, Tables 1 and 6) give a study of 21 children, of whom 1 had airway hyper-responsiveness before stem cell transplantation and not after, and 7 had it after and not before. The asymptotic p-value is 0.0339 and the mid-p value 0.0391, while the exact and corrected values are 0.0703 and 0.0771. At the 5% level the conclusion depends on the version chosen. Problem 1 works the four values by hand.

Analyzing the pairs as two samples

A common error is to treat the 4,782 first readings and the 4,782 third readings as two independent samples and compare them with the pooled two-proportion test of Comparing Two Proportions. The pooled share is 35.5%, and the test gives $z = 1.15$ and $p = 0.25$. The paired test on the same data gives $z = 2.70$. The difference lies entirely in the variance.

The variance of a difference of dependent proportions. The difference is $(n_{12} - n_{21})/n$, a linear combination of two multinomial counts. The Chi-Square Goodness-of-Fit Test proves that a multinomial count has variance $n\pi_{ij}(1 - \pi_{ij})$ and that two different counts have covariance $-n\pi_{ij}\pi_{kl}$. So

$$\operatorname{Var}(n_{12} - n_{21}) = n\pi_{12}(1 - \pi_{12}) + n\pi_{21}(1 - \pi_{21}) + 2n\pi_{12}\pi_{21} = n\big[\pi_{12} + \pi_{21} - (\pi_{12} - \pi_{21})^2\big],$$

and

$$\operatorname{Var}(\hat{p}_1 - \hat{p}_2) = \frac{\pi_{12} + \pi_{21} - (\pi_{12} - \pi_{21})^2}{n}.$$

What the unpaired test assumes. Two independent samples of size $n$ with these proportions would give a difference with variance $[\pi_{1+}(1 - \pi_{1+}) + \pi_{+1}(1 - \pi_{+1})]/n$. The two variances are related by

$$\pi_{12} + \pi_{21} - (\pi_{12} - \pi_{21})^2 = \pi_{1+}(1 - \pi_{1+}) + \pi_{+1}(1 - \pi_{+1}) - 2(\pi_{11}\pi_{22} - \pi_{12}\pi_{21}).$$

Proof. Since $1 - \pi_{1+} = \pi_{21} + \pi_{22}$ and $1 - \pi_{+1} = \pi_{12} + \pi_{22}$, expand the two products:

$$\pi_{1+}(1 - \pi_{1+}) = (\pi_{11} + \pi_{12})(\pi_{21} + \pi_{22}) = \pi_{11}\pi_{21} + \pi_{11}\pi_{22} + \pi_{12}\pi_{21} + \pi_{12}\pi_{22},$$

$$\pi_{+1}(1 - \pi_{+1}) = (\pi_{11} + \pi_{21})(\pi_{12} + \pi_{22}) = \pi_{11}\pi_{12} + \pi_{11}\pi_{22} + \pi_{21}\pi_{12} + \pi_{21}\pi_{22}.$$

Their sum is $2\pi_{11}\pi_{22} + 2\pi_{12}\pi_{21} + (\pi_{11} + \pi_{22})(\pi_{12} + \pi_{21})$. Write $\delta = \pi_{12} + \pi_{21}$, so that $\pi_{11} + \pi_{22} = 1 - \delta$. Subtracting $2(\pi_{11}\pi_{22} - \pi_{12}\pi_{21})$ leaves $4\pi_{12}\pi_{21} + \delta(1 - \delta) = \delta - (\delta^2 - 4\pi_{12}\pi_{21}) = \delta - (\pi_{12} - \pi_{21})^2$. $\blacksquare$

The term $\pi_{11}\pi_{22} - \pi_{12}\pi_{21}$ is the covariance of the two indicators "first reading high" and "third reading high" for one person: the expected product is $\pi_{11}$, and $\pi_{11} - \pi_{1+}\pi_{+1} = \pi_{11}\pi_{22} - \pi_{12}\pi_{21}$ by the same expansion. Two readings on one person are strongly and positively correlated, and the positive covariance is subtracted. This is the identity of The Paired Samples t-Test for means, $\operatorname{Var}(\bar{d}) = (\sigma_1^2 + \sigma_2^2 - 2\sigma_{12})/n$, applied to variables that take only the values 0 and 1.

For the readings, the correlation of the two indicators (the phi coefficient) is 0.818. With the cell shares in place of the probabilities, the paired standard error of the difference is 0.418 percentage points and the unpaired one 0.978. The unpaired variance is 5.5 times too large, and its standard deviation 2.34 times, which is why the unpaired $z$ is less than half the paired one. The unpaired test is not merely weak. It uses the wrong variance, too large when the pairs are positively correlated and too small when they are negatively correlated, so its error rates are not the ones it states.

A sampling experiment. To see the cost in samples of a realistic size, the script treats the 4,782 adults as a population and draws 30 samples of 1,000 without replacement (seeds 18,000,000 to 18,000,029). Over the 30 samples, McNemar's $z$ had median 1.37 (interquartile range 0.68 to 2.06) and the unpaired $z$ had median 0.59 (0.29 to 0.92). The paired test rejected at the 5% level in 9 of the 30 samples and the unpaired test in 0. In one sample the two shares were equal and both statistics were 0. Over the other 29, the ratio of McNemar's $z$ to the unpaired $z$ had median 2.29 (interquartile range 2.23 to 2.39), close to the ratio of standard deviations among all 4,782 adults, 2.34.

Estimating the difference

A test says whether the readings differ. An interval says by how much. Replacing the probabilities in the variance by the sample shares gives the standard error

$$\widehat{\operatorname{SE}}(\hat{p}_1 - \hat{p}_2) = \frac{1}{n}\sqrt{n_{12} + n_{21} - \frac{(n_{12} - n_{21})^2}{n}} = \frac{1}{4{,}782}\sqrt{400 - \frac{54^2}{4{,}782}} = 0.00418,$$

or 0.418 percentage points, and the 95% interval $1.13 \pm 1.96 \times 0.418$, from 0.31 to 1.95 percentage points. The interval is a Wald interval, justified by the multivariate central limit theorem for the multinomial counts and the delta method (The Delta Method). Unlike the test, it uses $n$: the same 227 and 173 discordant pairs among 400 adults instead of 4,782 would describe a difference twelve times as large.

The conditional odds ratio. The parameter of the binomial factor is $\theta$, the share of discordant pairs that run high-then-not. Its estimate is $227/400 = 0.568$, and the exact Clopper–Pearson interval of Confidence Intervals for a Proportion gives 0.517 to 0.617. The odds $\theta/(1 - \theta) = \pi_{12}/\pi_{21}$ are the ratio of the two kinds of discordance, estimated by

$$\hat\psi = \frac{n_{12}}{n_{21}} = \frac{227}{173} = 1.312.$$

Because $\theta \mapsto \theta/(1 - \theta)$ is increasing, the interval for $\theta$ maps to the interval 1.072 to 1.609 for $\psi$. Breslow and Day (1980, §5.2, pp. 165–166, equations 5.6, 5.7 and 5.9) give the same estimate and the same construction for matched case-control pairs. There $n_{12}$ counts pairs in which only the case was exposed and $n_{21}$ pairs in which only the control was, and $\psi$ is the odds ratio for disease given exposure. They note that $n_{12}/n_{21}$ is also the Mantel–Haenszel estimate applied to the pairs, which The Mantel–Haenszel Test derives.

Power and the number of pairs

How many pairs does a study need to detect a difference of the size seen here? The answer turns on the discordance rate, not on the two marginal proportions. Here is the plan: find the distribution of the estimated difference under the alternative, find where the test's boundary falls, and require the boundary to be crossed with probability $1 - \beta$.

Write $\Delta = \pi_{12} - \pi_{21}$ and $\delta = \pi_{12} + \pi_{21}$. Under the alternative, the estimate $D = (n_{12} - n_{21})/n$ has mean $\Delta$ and, by the variance formula above, variance $(\delta - \Delta^2)/n$, and it is approximately normal. The asymptotic test rejects when $|n_{12} - n_{21}| > z_{1-\alpha/2}\sqrt{m}$. Replacing $m$ by its expected value $n\delta$ and dividing by $n$, it rejects approximately when $|D| > z_{1-\alpha/2}\sqrt{\delta/n}$. Ignoring the far tail, for $\Delta > 0$,

$$1 - \beta \approx P\big(D > z_{1-\alpha/2}\sqrt{\delta/n}\big) = \Phi\!\left(\frac{\sqrt{n}\,\Delta - z_{1-\alpha/2}\sqrt{\delta}}{\sqrt{\delta - \Delta^2}}\right).$$

Setting the argument equal to $z_{1-\beta}$ and solving,

$$n = \left(\frac{z_{1-\alpha/2}\sqrt{\delta} + z_{1-\beta}\sqrt{\delta - \Delta^2}}{\Delta}\right)^2.$$

Two approximations were made, the normal one and the replacement of $m$ by $n\delta$, so the formula is checked by exact calculation. At the shares of the 4,782 adults, $\Delta = 54/4{,}782$ and $\delta = 400/4{,}782$, with $\alpha = 0.05$ and $1 - \beta = 0.80$, the formula gives 5,146 pairs, rounded up to 5,147. The exact power with 5,147 pairs, summed over the trinomial distribution of the three kinds of pair, is 0.802 for the asymptotic test, 0.801 for the mid-p test and 0.788 for the exact test.

An unpaired design with the same two proportions needs far more. The formula of Comparing Two Proportions for two independent groups gives 28,175 people per group. The pairing turns a difference too small to see into one that a study of the size of NHANES can detect.

More than two categories: symmetry

Blood pressure is often reported in bands rather than above or below one threshold. With four systolic bands, below 120, 120 to 129, 130 to 139, and 140 or more, the first and third readings of the 4,782 adults give a $4 \times 4$ table:

First \ Third <120 120–129 130–139 ≥140 Total
<120 1,807 184 7 0 1,998
120–129 228 667 164 2 1,061
130–139 6 200 408 103 717
≥140 3 18 138 847 1,006
Total 2,044 1,069 717 952 4,782

Collapsing the bands at 130 recovers the $2 \times 2$ table: the upper right block holds $7 + 0 + 164 + 2 = 173$ pairs and the lower left $6 + 200 + 3 + 18 = 227$.

With $k$ categories, the natural generalization of $\pi_{12} = \pi_{21}$ is symmetry: $\pi_{ij} = \pi_{ji}$ for every pair of categories. Bowker (1948) posed the test of symmetry for a $k \times k$ table whose rows and columns are the same categories. Symmetry implies marginal homogeneity, since $\pi_{i+} = \sum_j \pi_{ij} = \sum_j \pi_{ji} = \pi_{+i}$. For $k = 2$ the two are the same hypothesis; for $k \ge 3$ marginal homogeneity is weaker. Stuart (1955) and Maxwell (1970) treated the test of marginal homogeneity itself.

A test of symmetry, derived. The factorization of the $2 \times 2$ likelihood extends to $k$ categories with one change: group the cells into the $k$ diagonal cells and the $k(k-1)/2$ pairs $\{(i, j), (j, i)\}$ with $i < j$. Write $m_{ij} = n_{ij} + n_{ji}$ and $\theta_{ij} = \pi_{ij}/(\pi_{ij} + \pi_{ji})$. Exactly as before, the multinomial probability is the probability of the diagonal counts and the pair totals $m_{ij}$ (a multinomial over $k + k(k-1)/2$ outcomes), times a product of binomial factors $\binom{m_{ij}}{n_{ij}}\theta_{ij}^{n_{ij}}(1 - \theta_{ij})^{n_{ji}}$, one per pair. So, given the diagonal and the pair totals, the counts $n_{ij}$ for $i < j$ are independent binomials, and under symmetry each has probability $\tfrac12$. Each standardized count $(n_{ij} - n_{ji})/\sqrt{m_{ij}}$ is then approximately standard normal when $m_{ij}$ is large, and they are independent, so

$$S = \sum_{i < j} \frac{(n_{ij} - n_{ji})^2}{n_{ij} + n_{ji}}$$

is approximately a sum of $k(k-1)/2$ independent squared standard normals, which is chi-squared with $k(k-1)/2$ degrees of freedom (The Chi-Squared and t Distributions). A pair with $m_{ij} = 0$ contributes nothing and is dropped, and the degrees of freedom fall by one for each pair dropped. For $k = 2$, $S$ is McNemar's $z^2$.

For the four bands, the six contributions are:

Bands $n_{ij}$ vs $n_{ji}$ Contribution
<120 and 120–129 184 vs 228 4.70
<120 and 130–139 7 vs 6 0.08
<120 and ≥140 0 vs 3 3.00
120–129 and 130–139 164 vs 200 3.56
120–129 and ≥140 2 vs 18 12.80
130–139 and ≥140 103 vs 138 5.08

Their sum is $S = 29.22$ on 6 degrees of freedom, with $p = 0.0001$. Every adjacent pair of bands leans the same way: more adults moved down a band from the first reading to the third than moved up. The largest single contribution comes from 20 adults whose readings differed by two bands, 18 of them moving down from 140 or more to 120 to 129.

The approximation needs every pair total to be large, and here two are small, 13 and 3 pairs. Dropping those two pairs leaves four terms with sum $S_4 = 26.14$ on 4 degrees of freedom and $p = 0.00003$, so the conclusion does not rest on the small pairs.

The margins tell the same story: 1,998 first readings below 120 against 2,044 third readings, and 1,006 first readings of 140 or more against 952 third readings. This article does not ask why the readings fall. The tests establish only that, among these adults, they did.

What is proved, what is cited, what the experiment shows

Proved here. The factorization of the multinomial likelihood of a paired table into a trinomial factor and a conditional binomial factor, and its consequence that the conditional test does not depend on the concordant pairs; the asymptotic statistic and its continuity correction as the standardized binomial count; the bound on the exact test's rejection probability; the variance of a difference of dependent proportions and its relation to the unpaired variance; the sample size formula, as an approximation; the factorization for $k$ categories and the conditional independence behind the symmetry statistic.

Cited. The normal approximation to the binomial, the multinomial covariances, the sum of independent chi-squared variables, and the Clopper–Pearson interval, from the linked pages; the multivariate central limit theorem and delta method behind the Wald interval, through The Delta Method; Fagerland et al. (2013) for the mid-p definition, their evaluation over 9,595 scenarios and their recommendations; Breslow and Day (1980) for the matched-pairs estimate and interval.

Computed. The tables of rejection rates and the power at 5,147 pairs are exact sums over binomial and trinomial distributions. The comparison of paired and unpaired statistics in samples of 1,000 is a seeded experiment on these adults, and its numbers describe that experiment, not paired designs in general.

Worked problems

These problems were written for this article. Problem 1 uses the published table of Fagerland et al. (2013), the others the NHANES readings. The script tools/experiments/statistics/mcnemars-test.py recomputes every number.

Problem 1 (four p-values by hand). Fagerland et al. (2013, Table 1) report 21 children classified before and after stem cell transplantation: 1 with airway hyper-responsiveness at both times, 1 before only, 7 after only and 12 at neither. Compute the four two-sided p-values for marginal homogeneity.

Solution. Asymptotic 0.0339, corrected 0.0771, exact 0.0703, mid-p 0.0391, the values of their Table 6.

The discordant counts are $n_{12} = 1$ (before only) and $n_{21} = 7$ (after only), so $m = 8$.

  • Asymptotic. $z = (1 - 7)/\sqrt{8} = -2.121$, and $2\{1 - \Phi(2.121)\} = 0.0339$.
  • Corrected. $z_{\text{cc}} = (|1 - 7| - 1)/\sqrt{8} = 5/\sqrt{8} = 1.768$, and $2\{1 - \Phi(1.768)\} = 0.0771$.
  • Exact. Under the hypothesis $n_{12}$ is binomial with 8 trials and probability $\tfrac12$. The smaller count is 1, and $P(X \le 1) = (1 + 8)/256 = 9/256$, so $p = 2 \times 9/256 = 0.0703$.
  • Mid-p. The probability of the observed $n_{12} = 1$ is $8/256$, so $p_{\text{mid}} = 18/256 - 8/256 = 10/256 = 0.0391$.

The exact p-value counts the whole probability of the observed outcome and of its mirror image, $n_{12} = 7$, as evidence against the hypothesis. The mid-p value counts half of each. With only 8 discordant pairs, those two outcomes carry most of the tail, which is why the two values differ by a factor of almost two.

Problem 2 (a sample that sees nothing). A study of 300 adults drawn at random from the 4,782 (seed 18,100,000) finds $n_{11} = 89$, $n_{12} = 15$, $n_{21} = 15$ and $n_{22} = 181$. Test marginal homogeneity, give a 95% interval for the difference in the shares high on the two readings, and say what the study shows.

Solution. The tests find no difference, $z = 0.000$ with $p = 1.0000$, and the interval is $0 \pm 3.58$ percentage points, which contains the difference of 1.13 points among all 4,782 adults.

The discordant pairs split 15 and 15, so $z = 0$, the asymptotic and exact p-values are 1, and the mid-p value is $1 - \tfrac12\binom{30}{15}2^{-30} = 1 - \tfrac12 \times 0.1445 = 0.9278$. The estimated difference is 0, with standard error

$$\frac{1}{300}\sqrt{30 - \frac{0^2}{300}} = 0.0183,$$

or 1.83 percentage points, so the interval is $0 \pm 1.96 \times 1.83 = 0 \pm 3.58$ points. The study cannot distinguish a difference of zero from one of 1.13 points or of 3 points. A nonsignificant test is not evidence that the readings agree. To have 80% power for a difference of 1.13 points with this sample's discordance share of $30/300 = 0.10$, the formula of the section on power asks for 6,153 pairs, about twenty times as many.

Problem 3 (the goodness-of-fit form). Show that McNemar's $z^2$ equals Pearson's statistic for testing that the $m$ discordant pairs split evenly between the two directions, and compute it for the readings.

Solution. Pearson's statistic for two categories with observed counts $n_{12}$ and $n_{21}$ and expected counts $m/2$ each is

$$X^2 = \frac{(n_{12} - m/2)^2}{m/2} + \frac{(n_{21} - m/2)^2}{m/2}.$$

Since $n_{21} - m/2 = -(n_{12} - m/2)$, the two numerators are equal and $X^2 = 4(n_{12} - m/2)^2/m = (2n_{12} - m)^2/m = (n_{12} - n_{21})^2/m = z^2$. For the readings the expected counts are 200.0 each and $X^2 = (27^2 + 27^2)/200 = 7.29$. McNemar's test is the goodness-of-fit test of The Chi-Square Goodness-of-Fit Test for two categories, applied to the discordant pairs, which that article shows is the square of the binomial $z$.

Problem 4 (score and likelihood ratio). For the binomial factor $n_{12} \mid m \sim \text{Binomial}(m, \theta)$, show that McNemar's $z$ is the score statistic for $\theta = \tfrac12$, and compute the likelihood ratio statistic for the readings.

Solution. The likelihood ratio statistic is $G^2 = 7.31$, with $p = 0.0068$, against $z^2 = 7.29$ and $p = 0.0069$.

Likelihood Ratio, Wald and Score Tests shows that for a binomial proportion $\hat\theta = x/m$ the score statistic for $\theta_0$ is $m(\hat\theta - \theta_0)^2/[\theta_0(1 - \theta_0)]$, the square of the standardized count $(x - m\theta_0)/\sqrt{m\theta_0(1 - \theta_0)}$, whose variance is taken at the null value. At $\theta_0 = \tfrac12$ the standardized count is $(n_{12} - m/2)/\sqrt{m/4}$, which the section on the test showed to be $z$. The likelihood ratio statistic compares the log-likelihood at $\hat\theta = n_{12}/m$ with that at $\tfrac12$:

$$G^2 = 2\Big[n_{12}\ln\frac{n_{12}}{m/2} + n_{21}\ln\frac{n_{21}}{m/2}\Big] = 2\Big[227\ln\frac{227}{200} + 173\ln\frac{173}{200}\Big] = 7.31.$$

Both statistics are referred to the chi-squared distribution with one degree of freedom, and with 400 discordant pairs they agree closely.

Summary

Quantity Formula
Difference of paired proportions $\hat{p}_1 - \hat{p}_2 = (n_{12} - n_{21})/n$
Null distribution $n_{12} \mid m \sim \text{Binomial}(m, \tfrac12)$, $m = n_{12} + n_{21}$
McNemar statistic $z = (n_{12} - n_{21})/\sqrt{n_{12} + n_{21}}$; corrected: $(\lvert n_{12} - n_{21}\rvert - 1)/\sqrt{m}$
Mid-p exact two-sided p-value minus $\binom{m}{n_{12}}2^{-m}$
Variance of the difference $[\pi_{12} + \pi_{21} - (\pi_{12} - \pi_{21})^2]/n$
Conditional odds ratio $\hat\psi = n_{12}/n_{21}$
Pairs for power $1 - \beta$ $n = \big[(z_{1-\alpha/2}\sqrt{\delta} + z_{1-\beta}\sqrt{\delta - \Delta^2})/\Delta\big]^2$
Symmetry, $k$ categories $S = \sum_{i<j}(n_{ij} - n_{ji})^2/(n_{ij} + n_{ji})$, $k(k-1)/2$ degrees of freedom

What we have learned

Two proportions measured on the same people are the two margins of one paired table, and their difference is the difference of the two discordant counts divided by $n$. The likelihood of the table factors into a part that counts the discordant pairs and a binomial part that says which way they went, and the hypothesis of equal proportions concerns only the second. McNemar's test is therefore the sign test applied to the discordant pairs. Its exact version is conservative, often far below its nominal level when discordant pairs are few; the mid-p and asymptotic versions are close to the level. For 4,782 adults the first systolic reading was classified as 130 mm Hg or more 1.13 percentage points more often than the third, $z = 2.70$, a difference that an unpaired analysis cannot detect because it ignores a correlation of 0.818 between the two readings.

A matched pair is the smallest possible stratum: two people who share everything the matching controlled. The Mantel–Haenszel Test combines evidence across strata of any size, and shows that with one pair per stratum it reduces to McNemar's test.

References