Paired Comparisons of Models

How to compare two language models scored on the same questions. Why the paired standard error is smaller, with the shared part of the variance identified as the covariance of question difficulty; the paired z statistic and McNemar's statistic related exactly for 0/1 scores; the paired difference with clustered questions; the paired bootstrap, whose variance is proved to equal the plug-in paired variance and whose "share of resamples won" is shown to be a one-sided test at half the level it is often read as; and the effect of answer randomness on a comparison. Worked on all 378 pairs of 28 models on the 14,042 MMLU questions published by HELM, where pairing typically halves the variance, with an experiment on error rates in samples of questions and a second example on Cranfield retrieval runs; worked problems on two near-identical models, a paired table, a correlation formula, a bootstrap reading and a tie.

On HELM's 14,042 MMLU questions, Llama 3 (70B) scored 78.68% and Qwen1.5 (72B) 77.42%. Standard Errors for Benchmark Scores gave each score a standard error of 0.35 percentage points. Treating the two scores as independent measurements, the difference of 1.25 points would have a standard error of 0.49 points. But the two models answered the same questions, and on most questions they agree: both right on 9,822, both wrong on 1,944. Is Llama 3 (70B) better on questions of this kind, and how precisely is the difference measured?

A paired comparison analyzes the question-by-question differences between two models scored on the same questions, rather than the two overall scores. Here it gives a standard error of 0.34 points for the difference, not 0.49, and a z statistic of 3.69 instead of 2.54. The gain comes from the questions both models find easy or hard, which cancel in the difference.

First we identify what pairing removes and relate the paired z statistic to McNemar's test exactly. Then we measure the gain over all 378 pairs of the 28 models and combine pairing with the clustering of questions into subjects. Next we prove what the paired bootstrap estimates and what its "share of resamples won" means as a test. Finally we check error rates by sampling questions, measure how answer randomness enters a comparison, and repeat the analysis on Cranfield retrieval runs. The variance of a paired difference is proved in The Paired Samples t-Test for means and in McNemar's Test for Paired Proportions for 0/1 outcomes; this article states it and builds on it. Notation follows Standard Errors for Benchmark Scores.

What pairing removes

For question $i$, let $s_{Ai}$ and $s_{Bi}$ be the two models' scores and $d_i = s_{Ai} - s_{Bi}$. The difference of the benchmark scores is the mean difference, $\bar d = \bar s_A - \bar s_B$. When the questions are independent draws, the $d_i$ are independent and identically distributed, so Theorem 1 of the first article applies to them: $\bar d$ estimates $\mu_A - \mu_B$ without bias, with standard error $S_d/\sqrt n$, where $S_d$ is the standard deviation of the $d_i$. This is Miller's paired standard error (2024, Equation 7). The variance of a single difference is

$$\operatorname{Var}(d) = \sigma_A^2 + \sigma_B^2 - 2\sigma_{AB},$$

with $\sigma_{AB} = \operatorname{Cov}(s_A, s_B)$, proved in The Paired Samples t-Test. An analysis that treats the two scores as independent uses $\sigma_A^2 + \sigma_B^2$ instead. When the covariance is positive, that overstates the variance of the difference, and the unpaired test loses power. When it is negative, it understates the variance, and the unpaired test rejects too often.

What is the covariance made of? Recall the decomposition of the first article: $s_{Ai} = x_{Ai} + \epsilon_{Ai}$, where $x_{Ai}$ is model A's conditional mean on question $i$ and $\epsilon_{Ai}$ the randomness of one answer.

Theorem 1 (the shared part is question difficulty). Suppose that, given the question, the two models' answer deviations $\epsilon_{Ai}$ and $\epsilon_{Bi}$ are uncorrelated with each other, and each has conditional mean 0. Then

$$\operatorname{Cov}(s_A, s_B) = \operatorname{Cov}(x_A, x_B), \qquad \operatorname{Var}(d) = \operatorname{Var}(x_A - x_B) + E[\sigma_{A,i}^2] + E[\sigma_{B,i}^2].$$

Proof. By the law of total covariance (Conditional Expectation and the Laws of Total Expectation and Variance), $\operatorname{Cov}(s_A, s_B) = E[\operatorname{Cov}(s_A, s_B \mid \text{question})] + \operatorname{Cov}(E[s_A \mid \text{question}], E[s_B \mid \text{question}])$. Given the question, $x_A$ and $x_B$ are constants, so the conditional covariance is the conditional covariance of the $\epsilon$'s, which is 0 by assumption. The conditional means are $x_A$ and $x_B$. For the second formula, apply the law of total variance to $d$: its conditional mean is $x_A - x_B$, and its conditional variance is $\sigma_{A,i}^2 + \sigma_{B,i}^2$, because the cross term vanishes. $\blacksquare$

So pairing removes exactly the part of the variance due to questions that both models find easy or hard. Under the theorem's assumption, answer randomness is not shared between the two models and is never removed by pairing. The assumption can fail when both models see the same random prompt. In HELM Classic both models of a pair saw the same three orderings of the same in-context examples; there the pooled correlation of the two models' answer deviations (the average within-question covariance over the three orderings, divided by the geometric mean of the two average within-question variances) was 0.028 for text-davinci-002 and text-davinci-003 and −0.007 for davinci and text-davinci-002, and ranged from −0.100 to 0.157 over the 28 pairs of the eight models (median 0.025). The assumption nearly holds there. Miller (2024, Section 4.2) states the same reduction as $\operatorname{Var}(\hat\mu_{A-B,\text{paired}}) = \operatorname{Var}(\hat\mu_{A-B,\text{unpaired}}) - 2\operatorname{Cov}(x_A, x_B)/n$. In HELM's MMLU runs each question was asked once, at temperature 0 and with one fixed set of examples, so the conditional variances given that prompt cannot be estimated and are small if the serving is deterministic. But the Classic runs show that the order of the same five examples alone leaves 14% to 60% of a single score's variance as answer randomness (Standard Errors for Benchmark Scores), and that part is not removed by pairing.

Llama 3 (70B) against Qwen1.5 (72B). The correlation of the two models' 0/1 scores over the 14,042 questions is 0.527. The paired standard error of the difference is 0.34 points, against 0.49 points unpaired, so the ratio of the variances is 0.473: pairing is worth more than doubling the number of questions. The 95% interval for the difference is 0.59 to 1.92 points. (The difference is 1.25 points; the two rounded accuracies differ by 1.26 because each was rounded separately.)

Paired z and McNemar's statistic

For 0/1 scores each question falls in one cell of a $2 \times 2$ table: $n_{11}$ questions both models answer correctly, $n_{12}$ that only A answers correctly, $n_{21}$ that only B answers correctly, and $n_{22}$ that neither does. For the running pair these are 9,822, 1,226, 1,050 and 1,944. Only the $m = n_{12} + n_{21} = 2{,}276$ discordant questions carry information about the difference, as McNemar's Test for Paired Proportions proves by factoring the likelihood. That page derives the exact conditional test and its normal form, McNemar's statistic $z_M = (n_{12} - n_{21})/\sqrt{m}$. The paired z statistic of Theorem 1 of the first article is a different formula on the same table. The two are related exactly.

Theorem 2 (paired z and McNemar's z). For 0/1 scores with $D = n_{12} - n_{21}$ and $m = n_{12} + n_{21} > 0$, and with the $d_i$ not all equal (equivalently $nm > D^2$), the paired statistic $z_P = \bar d/(S_d/\sqrt n)$ satisfies

$$z_P^2 = \frac{(n - 1)\,z_M^2}{n - z_M^2}.$$

Proof. Each $d_i$ is $+1$ on $n_{12}$ questions, $-1$ on $n_{21}$ and 0 otherwise, so $\sum_i d_i = D$ and $\sum_i d_i^2 = m$. Then $(n - 1)S_d^2 = \sum_i d_i^2 - n\bar d^2 = m - D^2/n$, and

$$z_P^2 = \frac{n\bar d^2}{S_d^2} = \frac{D^2/n}{(m - D^2/n)/(n - 1)} = \frac{(n - 1)D^2}{nm - D^2}.$$

Dividing numerator and denominator by $m$ and writing $z_M^2 = D^2/m$ gives the result. $\blacksquare$

The two statistics differ in the variance they use. McNemar's uses the variance under the hypothesis of no difference, $m/n^2$ for $\bar d$. The paired z uses the estimated variance, which subtracts $D^2/n$. Whenever $z_M^2$ is small compared with $n$, the factor $(n - 1)/(n - z_M^2)$ is close to 1. For the running pair, $D = 176$, both statistics are 3.69 to two decimals, and the exact two-sided McNemar p-value is 0.00024. For 0/1 scores McNemar's test is also the sign test on the nonzero differences (The Sign Test and the Wilcoxon Signed-Rank Test): each discordant question is a "plus" for A or a "minus".

Dietterich (1998) studied five tests for comparing two learning algorithms and found that McNemar's test has "low type I error" in his simulations. The "test for difference of two proportions" (the unpaired test) was one of two that have "high probability of type I error in certain situations" (his abstract). His Section 7 explains the failure: the unpaired test "will fail in cases where the two learning algorithms have very different 'regions' of poor performance and where the error rates are close to 0.5," which he did not encounter in his experiments on realistic data, where it had acceptable Type I error but low power. That matches the experiment below, where the two models' scores are positively correlated and the unpaired test is conservative. Dietterich nonetheless concludes that "there can be no justification for ever employing the uncorrected difference-of-proportions test", given McNemar's better performance and the incorrect assumptions of the unpaired version. He also noted a limit that carries over to language models: McNemar's test "does not measure variation due to the choice of training set." Comparing two fixed, already trained models is the situation where that limit does not matter, his question of which of two given classifiers "will be more accurate on new test examples."

How much pairing helps

The running pair is typical. Over all 378 pairs of the 28 models, the correlation between two models' question scores had:

  • minimum −0.06;
  • quartiles 0.37, 0.46 and 0.50;
  • maximum 0.68, for GPT-4 (0613) and GPT-4 Turbo (1106 preview).

The low end belongs to OLMo (7B), whose accuracy of 28.74% is near guessing: its largest correlation with any other model is 0.10. Without OLMo, the smallest correlation is 0.14. Anthropic's summary post on Miller's paper (Anthropic, 2024) reports correlations of question scores between 0.3 and 0.7 between frontier models on popular evaluations, as also recorded in the library note on that paper; the HELM pairs fall mostly in that range.

When the two models have equal variances, the ratio of the paired to the unpaired variance is $(2\sigma^2 - 2r\sigma^2)/(2\sigma^2) = 1 - r$, where $r$ is the correlation. Over the 378 pairs the median ratio was 0.54 and the smallest 0.32. Pairing typically halves the variance, which is worth as much as doubling the benchmark.

Example (overlapping intervals). "The two models' 95% intervals overlap, so the difference is not significant." Llama 3 (70B)'s interval is 78.00% to 79.36% and Qwen1.5 (72B)'s 76.73% to 78.12%; they overlap. Yet the paired comparison rejects equality with $z = 3.69$. Overlap of two separate intervals is a statement about the unpaired comparison, and even for that it is conservative: the intervals fail to overlap only if $|\bar s_A - \bar s_B| > 1.96(\mathrm{SE}_A + \mathrm{SE}_B)$, and since $\mathrm{SE}_A + \mathrm{SE}_B \ge \sqrt{\mathrm{SE}_A^2 + \mathrm{SE}_B^2}$, non-overlap implies that the unpaired test rejects, but not the converse. The paired analysis is the one the data support.

Many pairs at once. With 378 comparisons, some will look significant by chance. At the 5% level, 365 of the 378 pairs had paired $|z| > 1.96$ and 360 had unpaired $|z| > 1.96$. A Bonferroni correction for 378 tests requires $|z| > 3.82$, which 347 pairs met. With the cluster-robust standard error of the next section, 349 pairs had $|z| > 1.96$. Such counts are no substitute for a procedure designed for many models; Multiple Comparisons and the Bonferroni Correction covers family-wise control, and Comparing Systems Across Many Tasks treats rankings of many systems.

Paired differences with clustered questions

The questions of MMLU come in 57 subjects. Clustered Standard Errors showed that, if the subjects are regarded as a sample of subjects, a single model's standard error grows by a median factor of 7.4. The cluster-robust estimator applies equally to the differences $d_i$: compute residuals $d_i - \bar d$, add them within each subject, square, sum and divide by $n^2$. This is Miller's Equation 8.

Pairing removes most of the subject effect, because the models agree on which subjects are hard. For the running pair, the cluster-robust standard error of the difference is 0.68 points, 1.99 times the paired one. Treating the two models' clustered scores as independent would give 3.74 points. Over the 378 pairs, the median ratio of the cluster-robust to the paired standard error of the difference was 2.38 (largest 6.93), against 7.4 for a single model's score.

The clustering still matters for close comparisons. With subjects as clusters, the running pair's z statistic is $1.25/0.68 = 1.85$, and the CR1 interval with the t quantile on 56 degrees of freedom runs from −0.11 to 2.62 points, which includes 0. A claim that Llama 3 (70B) is better at academic questions in general, of which MMLU's subjects are a selection, is therefore not supported at the 5% level by these data, while the claim for MMLU's own subjects is. With the subjects as fixed strata (Theorem 5 of Clustered Standard Errors), the stratified standard error of the difference is 0.34 points, the same as the paired one to two decimals. GPT-4 (0613) answers 872 questions correctly that GPT-4 Turbo (1106 preview) misses, and the reverse holds for 452. Their difference of 2.99 points has a paired standard error of 0.26 points ($z = 11.6$) and a cluster-robust one of 1.06 points ($z = 2.81$): significant on either view.

The paired bootstrap

A common way to compare two systems in natural language processing is the paired bootstrap. Koehn (2004, Section 6) describes it for machine translation: "we repeatedly (say, 1000 times) create new virtual test sets by drawing sentences with replacement from the collection. For each set, we compute the evaluation metric score for both systems. We note, which system performs better." The essential word is paired: each resampled test set contains the same items for both systems.

For a metric that is a mean over items, the bootstrap needs no simulation to analyze. The resampling notation is that of The Plug-In Principle and Why the Bootstrap Works: $P_*$, $E_*$ and $\operatorname{Var}_*$ are taken over resampling with the data held fixed.

Theorem 3 (variance of the paired and unpaired bootstrap). Let $d_1^*, \ldots, d_n^*$ be drawn with replacement from $d_1, \ldots, d_n$. Then $E_*[\bar d^*] = \bar d$ and

$$\operatorname{Var}_*(\bar d^*) = \frac{\hat\sigma_d^2}{n}, \qquad \hat\sigma_d^2 = \frac1n\sum_i (d_i - \bar d)^2 = \frac{n-1}{n}S_d^2.$$

If instead the scores of A and of B are resampled independently of each other, the variance of the bootstrap difference is $(\hat\sigma_A^2 + \hat\sigma_B^2)/n$, the plug-in version of the unpaired variance.

Proof. The first part is the exact bootstrap mean and variance of a sample mean, Theorem 4 of The Plug-In Principle and Why the Bootstrap Works, applied to the differences; in brief: each $d_j^*$ is one of the $d_i$ with probability $1/n$ each, so $E_*[d_j^*] = \bar d$ and $\operatorname{Var}_*(d_j^*) = \frac1n\sum_i (d_i - \bar d)^2 = \hat\sigma_d^2$. The draws are independent given the data, so $\bar d^*$ has mean $\bar d$ and variance $\hat\sigma_d^2/n$. In the unpaired scheme $\bar s_A^*$ and $\bar s_B^*$ are independent given the data, with variances $\hat\sigma_A^2/n$ and $\hat\sigma_B^2/n$ by the same argument, and the variance of their difference is the sum. $\blacksquare$

So the paired bootstrap standard error is the paired standard error times $\sqrt{(n-1)/n}$, up to Monte Carlo error, and an unpaired bootstrap reproduces the unpaired standard error with all its loss. For the running pair, 10,000 paired resamples (seed 20,261,118) gave a standard error of 0.339 points, against 0.340 from the formula, and the unpaired resampling gave 0.495. The 95% percentile interval was 0.59 to 1.92 points, the same as the normal interval to two decimals, as expected for a mean of 14,042 terms. When to prefer other bootstrap intervals is the subject of Bootstrap Confidence Intervals.

The share of resamples won. Koehn reads the bootstrap as a test: "If, say, one system outperforms the other system 95% of the time, we draw the conclusion that it is better with 95% statistical significance." This site's search evaluation reports the same quantity as P(<=0), "the share of resamples in which B is no better than A (a one-sided bootstrap p-value)" (tools/search/eval/README.md). What level does it test at?

Theorem 4 (share won and the percentile interval). Let $\bar d^{*1}, \ldots, \bar d^{*B}$ be the bootstrap replicates, $\bar d^{*(1)} \le \cdots \le \bar d^{*(B)}$ their order statistics, and take as the lower end of the percentile interval at level $1 - \alpha$ the order statistic $\bar d^{*(k)}$ with $k = \lceil B\alpha/2 \rceil$. Then the lower end is above 0 if and only if fewer than $k$ replicates are at or below 0.

Proof. The $k$-th smallest replicate exceeds 0 exactly when at most $k - 1$ replicates are $\le 0$. $\blacksquare$

The theorem is elementary, and its consequence is easy to miss. "A wins in more than 95% of resamples" means that fewer than 5% of replicates are at or below 0. That is the condition for the lower end of the 90% percentile interval to be above 0, a one-sided test at 5% or a two-sided test at 10%, not a two-sided test at 5%. A two-sided test at 5% needs A to win in more than 97.5% of resamples. (With exactly 95% wins out of 10,000 resamples, 500 replicates are at or below 0 and $k = 500$, so even the 90% interval contains 0.) For the running pair, 2 of the 10,000 replicates were at or below 0. The bootstrap distribution is centered at the observed difference, not at 0, so this share is the percentile method's p-value, not a test of a resampled null hypothesis. A test that does resample under the null, flipping the sign of each $d_i$ at random, is the permutation test of Permutation Tests, which is exact under its hypothesis of exchangeability.

Error rates on samples of questions

The formulas above are asymptotic. How do paired and unpaired analyses behave on benchmarks of realistic size?

The experiment. The script treats the 14,042 questions, with both models' results, as a population in which Llama 3 (70B) leads Qwen1.5 (72B) by 1.25 points. It draws benchmarks of $n$ questions with replacement and computes the paired and unpaired 95% intervals for the difference. Each cell is the median over 30 seeds (41,000,000 + $n$ to 41,000,029 + $n$) of the share of 2,000 benchmarks, with the interquartile range in parentheses.

Questions Paired interval covers Unpaired interval covers Paired test rejects Unpaired test rejects
200 0.949 (0.946 to 0.952) 0.996 (0.995 to 0.997) 0.076 (0.073 to 0.080) 0.008 (0.007 to 0.009)
1,000 0.950 (0.949 to 0.952) 0.996 (0.995 to 0.997) 0.167 (0.163 to 0.175) 0.032 (0.029 to 0.035)
5,000 0.948 (0.946 to 0.953) 0.996 (0.995 to 0.997) 0.594 (0.591 to 0.601) 0.258 (0.249 to 0.265)

To measure the error rate when there is no difference, the script also builds a null population: in each sampled question, the two models' scores are swapped with probability 1/2, so that the expected difference is exactly 0. The rejection rates of a nominal 5% test were:

Questions Paired test rejects Unpaired test rejects
200 0.051 (0.049 to 0.054) 0.005 (0.004 to 0.006)
1,000 0.050 (0.046 to 0.053) 0.005 (0.004 to 0.005)
5,000 0.049 (0.045 to 0.053) 0.004 (0.003 to 0.006)

In this experiment the paired interval covered at its nominal rate at every size, and the paired test rejected a true null about 5% of the time. The unpaired analysis was not anti-conservative: it covered 99.6% of the time and rejected a true null about 0.5% of the time, because it overstates the variance when the correlation is positive. Its cost was power. With 1,000 questions the paired test detected the 1.25-point difference in 16.7% of benchmarks, the unpaired one in 3.2%. Even with 5,000 questions, a difference of this size was detected only 59.4% of the time. How many questions a comparison needs is the subject of Power Analysis for Evaluations.

Answer randomness in a comparison

Theorem 1 says that answer randomness adds $E[\sigma_{A,i}^2] + E[\sigma_{B,i}^2]$ to the variance of a difference and that, when the two models' deviations are uncorrelated, pairing cannot remove it. Averaging $K$ answers per question divides each term by $K$, as Theorem 4 of the first article shows for one model. Miller's sample-size formula (2024, Equation 9) is built from exactly these three terms.

HELM Classic's three orderings of the same five in-context examples give a real case. On the 514 questions of the five Classic MMLU subjects, text-davinci-002 minus text-davinci-003 was:

  • +0.78 points with the first ordering of the examples, with a paired standard error of 1.74;
  • −1.17 points with the second ordering;
  • −0.19 points with the third.

The sign of the comparison depended on the order in which the same five examples were shown. Averaging each question's three scores, the difference is −0.19 points with a standard error of 1.49, and the correlation between the two models' question scores rises from 0.684 to 0.745. No single draw supports a claim about which model is better on these questions, and the average does not either.

A second example: retrieval runs on Cranfield

The same reasoning governs the comparison of retrieval systems over a set of queries, where Comparing Systems with Significance Tests works through the paired tests in detail. Over the 225 Cranfield queries of Comparing Ranking Models, BM25's average precision exceeds TF-IDF's by 0.0124 on average. The two runs' per-query average precisions have correlation 0.903, much higher than any pair of language models on MMLU: queries that are hard for one run are hard for the other. The paired standard error is 0.0069 and the unpaired one 0.0219, a ratio of variances of 0.099. For retrieval runs, the unpaired analysis would waste nine tenths of the information.

What is proved here, what is cited, what the experiment shows

Proved here. That the covariance removed by pairing is the covariance of the models' conditional means, when answer randomness is uncorrelated between models (Theorem 1); the exact relation between the paired z statistic and McNemar's statistic (Theorem 2); the variance of the paired and unpaired bootstrap (Theorem 3); the correspondence between the share of bootstrap resamples won and the percentile interval (Theorem 4); the ratio $1 - r$ for equal variances.

Cited. The variance of a paired difference, from The Paired Samples t-Test and McNemar's Test for Paired Proportions, with the exact McNemar test from the latter; the law of total covariance, from the linked article; the cluster-robust estimator and the stratified variance, from Clustered Standard Errors; Miller's Equations 7 to 9; Dietterich's findings on McNemar's test and the unpaired test; Koehn's description of paired bootstrap resampling; the correlations reported in Anthropic's summary post on Miller's paper.

Computed. Every count, difference, correlation and standard error from the HELM per-question files and the Cranfield per-query values; the bootstrap with a stated seed. The error-rate tables are a seeded experiment on one pair of models, and the null population was built by random swapping; their numbers describe that experiment.

Worked problems

These problems were written for this article; they use the HELM results and small exact cases.

Problem 1 (two near-identical models). Qwen1.5 (72B) and Mixtral (8x22B) differ by 0.08 points. The discordant counts are 1,124 questions only Qwen1.5 (72B) answers correctly and 1,113 only Mixtral (8x22B) does. Test the difference and give an interval.

McNemar's statistic is $z_M = (1{,}124 - 1{,}113)/\sqrt{2{,}237} = 11/47.30 = 0.23$, and the exact two-sided p-value is 0.83. The paired standard error is 0.34 points, so the 95% interval is $0.08 \pm 1.96 \times 0.34$, from −0.58 to 0.74 points. The data are consistent with either model being better by up to about two thirds of a point. "No significant difference" here is a statement about precision, not a finding of equality: the interval is what to report.

Problem 2 (from the paired table to the standard error). For Llama 3 (70B) against Qwen1.5 (72B), compute the paired standard error from the four counts.

From Theorem 2's proof, $(n - 1)S_d^2 = m - D^2/n$ with $m = 2{,}276$ and $D = 176$. Then $D^2/n = 30{,}976/14{,}042 = 2.206$, and $S_d^2 = (2{,}276 - 2.206)/14{,}041 = 0.16194$. The standard error is $\sqrt{0.16194/14{,}042} = 0.0034$, or 0.34 points. The concordant counts 9,822 and 1,944 enter only through $n$.

Problem 3 (Miller's paired example). Two models have scores with equal variances $1/12$ and correlation $1/2$. By what fraction does pairing reduce the variance of the difference?

The unpaired variance per question is $\tfrac{1}{12} + \tfrac{1}{12} = \tfrac16$. The covariance is $\tfrac12 \cdot \tfrac{1}{12} = \tfrac{1}{24}$, so the paired variance is $\tfrac16 - 2 \cdot \tfrac{1}{24} = \tfrac{1}{12}$. The reduction is one half, as the formula $1 - r$ gives directly. Miller (2024, Section 4.2) states a reduction of one third (from 1/6 to 1/9) for this example; the library note on his paper records the discrepancy, and the arithmetic here confirms the note.

Problem 4 (reading a bootstrap). A paired bootstrap with 10,000 resamples finds model A ahead in 9,600 of them. Is A better at the two-sided 5% level?

400 replicates, 4%, are at or below 0. For a two-sided test at 5% the lower end of the 95% percentile interval is the order statistic $k = \lceil 10{,}000 \times 0.025 \rceil = 250$. Since 400 replicates are at or below 0, the 250th smallest is at or below 0, and by Theorem 4 the interval includes 0: not significant at 5%. For the 90% interval, $k = 500$, and fewer than 500 replicates are at or below 0, so that interval excludes 0. "A wins 96% of the time" supports a one-sided claim at 5%, or a two-sided one at 10%, but not "95% significance" in the usual two-sided sense.

Problem 5 (a near tie on the discordant questions). Claude 3 Sonnet and Yi (34B) differ by 0.06 points, with 1,289 and 1,281 discordant questions and a score correlation of 0.496. What do the data say?

The difference is $8/14{,}042$ of the questions, and the exact two-sided McNemar p-value is 0.89. Each model answers more than 1,280 questions the other misses, so the two models are far from interchangeable question by question, yet their accuracies agree almost exactly. A paired analysis distinguishes "the same score" from "the same answers": the 2,570 discordant questions show that the models differ in which questions they get right, not in how many.

Summary

Question Answer
Paired standard error $S_d/\sqrt n$ from the per-question differences (Miller's Equation 7)
What pairing removes the covariance of question difficulty, $\operatorname{Cov}(x_A, x_B)$; never answer randomness (Theorem 1)
0/1 scores McNemar's test on the discordant questions; $z_P^2 = (n-1)z_M^2/(n - z_M^2)$ (Theorem 2)
Typical gain on MMLU median variance ratio 0.54 over 378 pairs (correlations 0.37 to 0.50 in the middle half)
Clustered questions cluster-robust estimator on the $d_i$; median inflation 2.38, against 7.4 for one model
Paired bootstrap variance $\hat\sigma_d^2/n$ (Theorem 3); "wins in more than 95% of resamples" is a one-sided 5% test (Theorem 4)
Llama 3 (70B) minus Qwen1.5 (72B) 1.25 points, paired SE 0.34, $z = 3.69$; cluster-robust SE 0.68

What we have learned

Two models scored on the same questions should be compared through their per-question differences. The covariance of the two models' question difficulties cancels in the difference, which on MMLU typically halves the variance and on Cranfield removes nine tenths of it. For 0/1 scores the paired z statistic and McNemar's statistic agree closely and can be computed from the four counts of the paired table. Clustering still matters for paired differences, though far less than for single scores. The paired bootstrap estimates the same variance as the formula, and "wins in more than 95% of resamples" is a one-sided test at 5%, a two-sided test at 10%, not the two-sided 5% test it is often read as.

A comparison needs enough questions to detect the difference it is looking for. With 1,000 questions, the paired test above detected a 1.25-point lead only one time in six. Power Analysis for Evaluations turns the paired variance of this article into the number of questions a comparison needs.

References