On the 14,042 questions of the MMLU test set, as run by Stanford's HELM project, Claude 3 Opus answered 85.04% correctly and GPT-4 (0613) 84.01%, a difference of 1.04 percentage points. With all 14,042 questions that difference is clearly larger than chance. Suppose instead that the two models had been compared on an evaluation of 1,000 questions drawn from the same population. The comparison would then detect a true difference of 1.04 points only about one time in six. How many questions does a comparison of two models need, and what is the smallest difference that an evaluation of a given size can be expected to detect?
The power of a test is the probability that it rejects the null hypothesis when a stated alternative is true. For an evaluation, the alternative is a true difference $\Delta$ between two models' expected scores on the population of questions the evaluation samples from. The minimum detectable difference (also called the minimum detectable effect, MDE) of an evaluation with $n$ questions is the true difference that it detects with a chosen power, conventionally 80%, at a chosen significance level, conventionally 5%. This article derives both for evaluations of language models, where questions are the sampled units, the two models answer the same questions, answers may themselves be random, and questions are often written in groups. For the running pair the answers are 8,317 questions for 80% power, and a minimum detectable difference of 3.0 points at 1,000 questions.
Here is the route. First we derive the variance that governs every comparison: that of the per-question difference between two models, including the case where each model answers each question several times. Then we turn it into the sample size and minimum detectable difference, check both against simulated evaluations drawn from the 14,042 questions, and compare them with McNemar's formula. Next we measure what pairing buys across all 378 pairs of HELM's 28 models, choose the number of answers per question when answers cost money, and correct the plan for questions written in clusters. Finally we ask how far a plan built on a small pilot can be trusted, and what a significant result from an underpowered evaluation says. The article assumes the power function of the $z$ test from Power and Sample Size for a Test of a Mean, which it does not re-derive, and the paired comparison of two models from Paired Comparisons of Models.
The data and the model of an evaluation
HELM (Holistic Evaluation of Language Models; Liang et al., 2023) publishes every per-question result it computes. The unit's running data, loaded by evaldata.py (see Statistics of Model Evaluation), are HELM's runs of 28 models on all 57 subjects of the MMLU test split (Hendrycks et al., 2021): five examples in the prompt, the model writes an answer letter, and the answer is scored 1 if it matches the key and 0 otherwise. A second, smaller set, HELM Classic's five MMLU subjects, records each question in three runs that use the same five examples in three different orders, and is used below where repeated answers are needed.
The statistical model is the one of Standard Errors for Benchmark Scores and of Miller (2024, Section 2). The questions of an evaluation are treated as a random sample from a large population of questions that could have been written to the same specification. The quantity of interest is a model's expected score over that population, not its score on the particular questions sampled. For two models $A$ and $B$ answering the same questions, the per-question difference is
$$d_i = s_{Ai} - s_{Bi}, \qquad i = 1, \dots, n,$$
where $s_{Ai}$ is model $A$'s score on question $i$. For scores of 0 or 1, $d_i$ is $+1$ when only $A$ is right, $-1$ when only $B$ is right and 0 otherwise. The true difference is $\Delta = E[d_i]$, and the evaluation estimates it by $\bar{d}$, the mean of the $d_i$.
To check formulas against data we need a population whose truth is known. For that purpose this article treats the 14,042 MMLU questions themselves as the population. A simulated evaluation of $n$ questions draws $n$ of them at random with replacement, so that its questions are independent draws from a population whose difference is known exactly. That population is the real one for the two models' results on MMLU. It is not a claim about questions MMLU does not contain.
For Claude 3 Opus ($A$) and GPT-4 (0613) ($B$) the 14,042 questions fall into the four cells of the paired table of McNemar's Test for Paired Proportions:
| GPT-4 right | GPT-4 wrong | |
|---|---|---|
| Opus right | $n_{11} = 11{,}064$ | $n_{12} = 878$ |
| Opus wrong | $n_{21} = 732$ | $n_{22} = 1{,}368$ |
So 878 questions were answered correctly only by Opus and 732 only by GPT-4, and $\Delta = (878 - 732)/14{,}042 = 0.0104$, or 1.04 points.
The variance that sets the sample size
How many questions a comparison needs depends on how much the difference $d_i$ varies from question to question. Here is the idea in one sentence. A mean of $n$ independent differences has variance $\operatorname{Var}(d)/n$, so everything that follows is a statement about $\operatorname{Var}(d)$, and about how the design of the evaluation changes it.
One answer per question. For scores of 0 or 1, $d_i^2$ is 1 on a discordant question and 0 otherwise, so $E[d_i^2]$ is the discordant share $\pi_{12} + \pi_{21}$ and
$$\operatorname{Var}(d) = \pi_{12} + \pi_{21} - \Delta^2,$$
the variance of a difference of dependent proportions derived in McNemar's Test for Paired Proportions. For the running pair the discordant share is $1{,}610/14{,}042 = 0.1147$ and $\operatorname{Var}(d) = 0.1147 - 0.0104^2 = 0.1145$, a standard deviation of 0.3384.
The same variance can be written in terms of the two models' own variances and the correlation $r_{AB}$ between their scores, as Paired Comparisons of Models proves:
$$\operatorname{Var}(d) = \sigma_A^2 + \sigma_B^2 - 2\,r_{AB}\,\sigma_A\sigma_B.$$
For 0/1 scores $\sigma_A^2 = p_A(1 - p_A)$, here $0.1272$ for Opus and $0.1344$ for GPT-4. The correlation of the two models' scores across questions is $r_{AB} = 0.562$. Two models that find the same questions hard have positively correlated scores, and the covariance term removes most of the variance. Comparing the two accuracies as if they came from separate sets of questions would use $\sigma_A^2 + \sigma_B^2 = 0.2616$ instead, 2.28 times the paired variance.
Example (what pairing is worth when the variances are equal). If $\sigma_A = \sigma_B = \sigma$, the paired variance is $2\sigma^2(1 - r_{AB})$ and the unpaired one $2\sigma^2$, so pairing divides the number of questions needed by $1/(1 - r_{AB})$. For the running pair $1/(1 - 0.562) = 2.28$, the same as the ratio of the actual variances to two decimals, because the two models' variances differ little.
Several answers per question. A model sampled at a positive temperature may answer the same question differently on different occasions, and an evaluation may also vary the examples placed in the prompt, or only their order. Then a question's score is itself random. Write $x_{Ai}$ for model $A$'s expected score on question $i$ over that randomness, and $\sigma_{A,i}^2$ for the variance of one answer about it, as Standard Errors for Benchmark Scores does. That article splits the variance of one model's score into the spread of the $x_{Ai}$ and the average answer variance, with $K$ answers per question (its Theorem 4). Paired Comparisons of Models does the same for a difference with one answer each, assuming the two models' answers are uncorrelated given the question (its Theorem 1). Planning needs both at once, and HELM's repeated runs violate the assumption, because in each run both models see the same five examples in the same order. The following theorem covers that case.
Theorem 1 (variance of a paired comparison with repeated answers). Suppose questions are drawn independently from a population. On each question $i$, $K$ independent draws $k = 1, \dots, K$ are made, and draw $k$ yields scores for both models and their difference $d_{ik} = s_{Aik} - s_{Bik}$. Given the question, the draws are independent with common mean $\delta_i = x_{Ai} - x_{Bi}$ and common variance $v_i$. Let $\bar{d}_i$ be the mean of the $K$ differences on question $i$ and $\bar{d}$ the mean of the $\bar{d}_i$ over the $n$ questions. Then
$$\operatorname{Var}(\bar{d}) = \frac{1}{n}\left(\omega^2 + \frac{\bar{v}}{K}\right), \qquad \omega^2 = \operatorname{Var}(\delta_i), \quad \bar{v} = E[v_i].$$
If the two models' draws are independent of each other given the question, $v_i = \sigma_{A,i}^2 + \sigma_{B,i}^2$, the sum of the two models' answer variances.
Here is the idea. A question's averaged difference varies for two reasons: questions differ in which model they favor, and answers on one question vary about their mean. Averaging $K$ answers divides only the second part by $K$.
Proof. Condition on the question. Given question $i$, $\bar{d}_i$ is a mean of $K$ independent draws with mean $\delta_i$ and variance $v_i$, so $E[\bar{d}_i \mid i] = \delta_i$ and $\operatorname{Var}(\bar{d}_i \mid i) = v_i/K$. The law of total variance, proved in Conditional Expectation and the Laws of Total Expectation and Variance, gives
$$\operatorname{Var}(\bar{d}_i) = \operatorname{Var}\big(E[\bar{d}_i \mid i]\big) + E\big[\operatorname{Var}(\bar{d}_i \mid i)\big] = \omega^2 + \frac{\bar{v}}{K}.$$
The questions are drawn independently, so the $\bar{d}_i$ are independent and identically distributed, and the variance of their mean is $\operatorname{Var}(\bar{d}_i)/n$. For the last statement, if given the question $s_{Aik}$ and $s_{Bik}$ are independent with variances $\sigma_{A,i}^2$ and $\sigma_{B,i}^2$, then $\operatorname{Var}(d_{ik} \mid i) = \sigma_{A,i}^2 + \sigma_{B,i}^2$, since the variance of a difference of independent variables is the sum of their variances. In general $v_i = \sigma_{A,i}^2 + \sigma_{B,i}^2 - 2\operatorname{Cov}(s_{Aik}, s_{Bik} \mid i)$. $\square$
With independent draws for the two models and the same $K$ for both, this is the variance in Miller's (2024, Section 5) sample size formula, $\omega^2 + \sigma_A^2/K_A + \sigma_B^2/K_B$ in his notation, where $\sigma_A^2$ is the average answer variance of model $A$. When both models see the same prompt examples on a draw, $v_i$ also contains their conditional covariance, which Miller's form leaves out.
Sample size and the minimum detectable difference
Write $V$ for the variance of one question's (averaged) difference, $V = \omega^2 + \bar{v}/K$, or $\operatorname{Var}(d)$ when $K = 1$. (On this page $V$ is the variance of one question's difference; Standard Errors for Benchmark Scores uses $V$ for the variance of a mean, which here is $V/n$.) The test of $H_0: \Delta = 0$ rejects at level $\alpha$ when $|\bar{d}|/\sqrt{V/n} > z_{1-\alpha/2}$, where $z_{1-\alpha/2}$ is the standard normal point with probability $1 - \alpha/2$ below it. By the central limit theorem $\bar{d}$ is approximately normal with mean $\Delta$ and standard deviation $\sqrt{V/n}$, so the statistic is approximately normal with standard deviation 1 and mean
$$\lambda = \frac{\Delta\sqrt{n}}{\sqrt{V}},$$
the true difference measured in standard errors. That is exactly the setting of the $z$ test in Power and Sample Size for a Test of a Mean, which derives the power $\Phi(\lambda - z_{1-\alpha/2}) + \Phi(-\lambda - z_{1-\alpha/2})$, with $\Phi$ the standard normal distribution function, and the sample size at which the power is $1 - \beta$. Applied here:
Theorem 2 (sample size and minimum detectable difference). For a two-sided test at level $\alpha$ to have power $1 - \beta$ against a true difference $\Delta \ne 0$, to the approximation of that article (normality, and neglect of rejections in the wrong direction, which only adds power), the evaluation needs
$$n = \frac{(z_{1-\alpha/2} + z_{1-\beta})^2\,V}{\Delta^2}$$
questions, rounded up. An evaluation with $n$ questions has minimum detectable difference
$$\text{MDE} = (z_{1-\alpha/2} + z_{1-\beta})\sqrt{\frac{V}{n}}.$$
Proof. The first formula is the sample size formula of the linked article with $\sigma^2$ replaced by $V$ and the difference $\mu_1 - \mu_0$ by $\Delta$, since $\bar{d}$ is a mean of $n$ independent variables with variance $V$. Solving it for $\Delta$ gives the second. $\square$
Miller (2024, Section 5) gives the same two formulas for evaluations, deriving them in his Appendix B. With $\alpha = 0.05$ and power 0.80, $z_{0.975} = 1.96$, $z_{0.80} = 0.8416$, and $(z_{1-\alpha/2} + z_{1-\beta})^2 = 7.849$. Back to Opus and GPT-4:
- Questions needed. $n = 7.849 \times 0.1145 / 0.0104^2$, which with unrounded values is 8,317 questions. Comparing the two accuracies as unpaired proportions, with variance 0.2616, would need 18,990.
- Power of MMLU itself. At $n = 14{,}042$, $\lambda = 3.64$ and the power is 0.954. Its minimum detectable difference is 0.80 points.
- Smaller evaluations. At 1,000 questions the power against the true 1.04 points is 0.163, and at 2,000 it is 0.279.
The minimum detectable difference at other sizes, with the running pair's variance:
| Questions $n$ | 100 | 164 | 500 | 1,000 | 14,042 |
|---|---|---|---|---|---|
| MDE (points) | 9.5 | 7.4 | 4.2 | 3.0 | 0.80 |
It shrinks like $1/\sqrt{n}$, so halving it takes four times the questions.
Checking the formulas against the data
Two approximations went into Theorem 2: the normal distribution for $\bar{d}$, and a known $V$, where a real test estimates it. The script checks both by drawing evaluations from the 14,042 questions. For each $n$ and each of 30 seeds (31,000,000 to 31,000,029) it draws 2,000 evaluations of $n$ questions with replacement, computes $z = \bar{d}/(s_d/\sqrt{n})$ with the sample standard deviation $s_d$, and counts the evaluations with $|z| > 1.96$.
| Questions $n$ | Power, Theorem 2 | Simulated power, median (IQR over 30 seeds) |
|---|---|---|
| 1,000 | 0.163 | 0.165 (0.157 to 0.170) |
| 2,000 | 0.279 | 0.282 (0.275 to 0.289) |
| 4,000 | 0.493 | 0.490 (0.486 to 0.500) |
| 8,317 | 0.800 | 0.797 (0.793 to 0.802) |
| 14,042 | 0.954 | 0.953 (0.948 to 0.957) |
In this experiment the formula's power lies inside the interquartile range of the simulated power at every size. At these sizes, with a difference that takes only three values, the normal approximation and the estimated variance cost nothing visible.
McNemar's formula gives the same answer
McNemar's Test for Paired Proportions derives a sample size for the same comparison by a slightly different route. Its test standardizes by the variance under the null hypothesis, which for 0/1 scores is the discordant share $\pi_{12} + \pi_{21}$, and it reaches
$$n_{\text{McN}} = \left(\frac{z_{1-\alpha/2}\sqrt{\pi_{12} + \pi_{21}} + z_{1-\beta}\sqrt{\pi_{12} + \pi_{21} - \Delta^2}}{\Delta}\right)^2.$$
Theorem 2 uses $V = \pi_{12} + \pi_{21} - \Delta^2$ in both places. Since $\pi_{12} + \pi_{21} \ge \pi_{12} + \pi_{21} - \Delta^2$, McNemar's $n$ is never smaller, and the two agree to the extent that $\Delta^2$ is small beside the discordant share. For the running pair McNemar's formula asks for 8,323 questions, a ratio of 1.0007 to Theorem 2's 8,317. For differences of a few points between models that disagree on a tenth of the questions or more, the two formulas are interchangeable.
How much pairing buys across 378 pairs of models
Is the running pair typical? HELM's 28 models make 378 pairs. Across them the correlation $r_{AB}$ of the two models' 0/1 scores has median 0.46, with interquartile range 0.37 to 0.50, a minimum of $-0.06$ and a maximum of 0.68. By the example above, a correlation of 0.46 roughly halves the questions a comparison needs. Miller's summary post reports correlations between 0.3 and 0.7 for frontier models on popular evaluations, as the library's note on the paper records, and HELM's MMLU results fall mostly in that range.
How many of the 378 differences can MMLU itself detect? For each pair, Theorem 2 with that pair's own variance and its own observed difference gives the number of questions needed for 80% power. For 359 pairs that number is at most 14,042, and for 19 it is larger. In all, 76 pairs would need more than 1,000 questions, and 39 pairs are closer than 2 points. For those 39 close pairs the variance of $d$ has median 0.192, larger than the running pair's, because the models are weaker and disagree more often.
Example (a nonsignificant difference is not evidence of equality). An evaluation of 1,000 questions finds no significant difference between Opus and GPT-4, and a reader concludes that the models are equally capable on MMLU. The conclusion does not follow. At 1,000 questions the evaluation has power 0.163 against the true 1.04 points, so it fails to detect that difference more than four times in five. What the evaluation can say is that differences larger than about its minimum detectable difference, 3.0 points, would probably have been detected. Computing the power "at the observed difference" after the fact does not help either: Power and Sample Size for a Test of a Mean shows that this "observed power" is a function of the p-value and adds nothing to it.
Answers that are themselves random
When a question can be answered more than once, should an evaluation spend its budget on more questions or on more answers per question? Theorem 1 says that more answers shrink only the part $\bar{v}/K$, while more questions shrink everything. To use it we need $\omega^2$ and $\bar{v}$, and both can be estimated from an evaluation that asked each question $K \ge 2$ times.
Theorem 3 (unbiased estimates of the two parts). Under the conditions of Theorem 1 with $K \ge 2$, let $w_i$ be the sample variance (divisor $K - 1$) of the $K$ differences on question $i$, and $S_b^2$ the sample variance (divisor $n - 1$) of the question means $\bar{d}_1, \dots, \bar{d}_n$. Then
$$\hat{v} = \frac{1}{n}\sum_{i=1}^{n} w_i \quad\text{and}\quad \hat{\omega}^2 = S_b^2 - \frac{\hat{v}}{K}$$
are unbiased for $\bar{v}$ and $\omega^2$.
Proof. Given question $i$, the $K$ differences are independent with variance $v_i$, so their sample variance has conditional expectation $v_i$ (the divisor $K - 1$ makes the sample variance unbiased for independent, identically distributed draws). By the law of total expectation $E[w_i] = E[v_i] = \bar{v}$, and so $E[\hat{v}] = \bar{v}$. The $\bar{d}_i$ are independent and identically distributed with variance $\omega^2 + \bar{v}/K$ by Theorem 1, so $E[S_b^2] = \omega^2 + \bar{v}/K$. Subtracting $E[\hat{v}]/K = \bar{v}/K$ leaves $\omega^2$. $\square$
The estimate $\hat{\omega}^2$ can come out negative when $\omega^2$ is small, since it is a difference of two estimates. Miller (2024, Appendix C) estimates the answer variance in the same way, dividing the within-question sum of squares by $n(K - 1)$ rather than $nK$; his estimate of $\omega^2$ from the question means does not subtract $\hat{v}/K$.
The HELM Classic runs. HELM Classic asked each of its 514 MMLU questions three times ($K = 3$). Each MMLU subject's development split has exactly five questions, and every run used those same five as the prompt examples, in a different order each time; the three orders are the same for every model. HELM's code fixes the order once per run, so in a given run every question of a subject, for both models, saw the same five examples in the same order. The only thing that varies between runs is the order of a fixed set of five examples, and each order is shared by all the questions of its subject: three orders per subject, fifteen in all, each shared by 100 to 114 questions. That is a crossed design, in which runs cut across questions, not the nested design of Theorem 1, in which each question gets its own independently drawn order. Theorem 1 allows the two models' answers in one run to be dependent; it does not allow the questions of one run to share an order. This subsection first analyzes the data as if they were nested, which is the design that Theorems 3 and 4 plan for, and the next subsection measures what the sharing changes. It also treats the three fixed orders as if drawn at random, which is an assumption, not a fact about how HELM chose them. For text-davinci-003 against text-davinci-002 the accuracies, averaged over the three runs, are 0.5668 and 0.5649, a difference of 0.19 points. The two estimates are $\hat{v} = 0.0681$ and $\hat{\omega}^2 = 0.0910$, from $S_b^2 = 0.1137$. So the order of the five prompt examples accounts for a share $\hat{v}/(\hat{\omega}^2 + \hat{v}) = 0.43$ of the variance of a single-run difference. Each model's own answer variance is 0.0350, and the two models' answers in the same run are nearly uncorrelated given the question (covariance 0.0010), so $\hat{v}$ is close to the sum of the two.
With the 514 questions fixed, more runs per question lower the minimum detectable difference only down to the floor set by $\omega^2$. Five examples have $5! = 120$ orders, so "unlimited" here means averaging over all of them, under the same assumption that the orders behave like random draws. For a nested design with these components:
| Runs per question $K$ | 1 | 3 | 10 | unlimited |
|---|---|---|---|---|
| MDE at 514 questions, nested (points) | 4.9 | 4.2 | 3.9 | 3.7 |
Going from one run to three lowers it by 0.8 points; going from three to infinitely many lowers it by only 0.4 more (differences of the unrounded values). Miller (2024, Section 3.1) reaches the same diminishing return for answer randomness from sampling.
Theorem 4 (the cost-optimal number of answers per question). Suppose a new question costs $c_q$ (writing it, checking it, and any fixed cost of asking it) and each answer costs $c_a$, and the evaluation must reach a target variance $T$ for $\bar{d}$. Among designs with $K$ answers per question, the total cost is minimized over real $K > 0$ at
$$K^* = \sqrt{\frac{c_q\,\bar{v}}{c_a\,\omega^2}},$$
and the best whole number of answers is $\lfloor K^* \rfloor$ or $\lceil K^* \rceil$ (and 1 if $K^* < 1$).
Here is the idea. Answers are worth buying until the answer noise they remove, $\bar{v}/K$, is small beside the question-to-question spread $\omega^2$, and the exchange rate is the relative price of a question and an answer.
Proof. By Theorem 1 the target $\operatorname{Var}(\bar{d}) = T$ requires $n = (\omega^2 + \bar{v}/K)/T$ questions, at a cost of $n(c_q + Kc_a)$. So the cost is $f(K)/T$ with
$$f(K) = (c_q + Kc_a)\left(\omega^2 + \frac{\bar{v}}{K}\right) = c_q\omega^2 + c_a\bar{v} + \frac{c_q\bar{v}}{K} + Kc_a\omega^2.$$
Its derivative is $f'(K) = -c_q\bar{v}/K^2 + c_a\omega^2$, zero only at $K = K^*$, and its second derivative $2c_q\bar{v}/K^3$ is positive for $K > 0$, so $f$ is strictly convex and $K^*$ is its minimum. A convex function of one variable decreases up to its minimum and increases after it, so among whole numbers the minimum is at one of the two nearest to $K^*$, or at 1 if $K^* < 1$. $\square$
This is the classical optimum for two-stage sampling, where the stages are questions and answers. For the Classic pair an extra "answer" is one more run of the question with another order of the same five examples (there are at most 120 distinct orders), and $\hat{v}/\hat{\omega}^2 = 0.75$, so $K^* = 0.86$ when a question costs as much as an answer (one answer per question is best), 2.74 when it costs ten times as much, and 8.65 at a hundred times. When answers come from a model and questions from expert writers, the second and third cases are the realistic ones. Theorem 4 plans for the nested design, in which every new answer comes with its own order; the next subsection shows what changes when one order serves many questions.
When one order is shared by many questions
In HELM's design an order that happens to favor one model favors it on every question of the subject in that run. Such an effect is not averaged away by more questions, only by more runs or more groups. Here is the model that makes this precise. Write the difference on question $i$ in run $k$ as
$$d_{ik} = \delta_i + \gamma_{g(i)k} + e_{ik},$$
where $g(i)$ is the group (here the subject) of question $i$, $\delta_i$ is the question's mean difference, with mean $\Delta$ and variance $\omega^2$, $\gamma_{gk}$ is the effect of group $g$'s order in run $k$ on the difference, with mean 0 and variance $\tau_r^2$, and $e_{ik}$ is the remaining answer noise, with mean 0 and variance $v_e$. All three kinds of term are independent of each other, the $\delta_i$ across questions, the $\gamma_{gk}$ across groups and runs, and the $e_{ik}$ across questions and runs. Write $w_g = n_g/n$ for group $g$'s share of the $n$ questions. The model is an assumption: it says the order shifts every question of its group by the same amount on average, and nothing more. It also takes the $\delta_i$ to be independent across questions even within a group, so it leaves out any clustering of the questions' own differences by subject, the subject of the next section; such clustering, with variance $\tau_\delta^2$ of the subjects' mean differences, would add a term $\tau_\delta^2\sum_g w_g^2$ of the same form, which also does not fall with $n$. For the Classic pair no such clustering is visible: the variance of the five subjects' mean differences, less the part that sampling noise within subjects explains, comes out at $-0.0009$, below zero, so the page uses no such term.
Theorem 5 (variance with an order shared within groups). Under this model, with $K$ runs,
$$\operatorname{Var}(\bar{d}) = \frac{\omega^2 + v_e/K}{n} + \frac{\tau_r^2}{K}\sum_g w_g^2.$$
The second term does not fall as questions are added within the same groups; it falls only with more runs or with more, smaller groups.
Proof. Averaging over runs and then questions, $\bar{d} = \frac{1}{n}\sum_i \delta_i + \sum_g w_g\,\bar{\gamma}_g + \frac{1}{nK}\sum_{i,k} e_{ik}$, where $\bar{\gamma}_g$ is the mean of group $g$'s $K$ run effects, because each of group $g$'s $n_g$ questions carries the same $\gamma_{gk}$ in run $k$. The three sums are independent, so their variances add: $\omega^2/n$; $\sum_g w_g^2\,\tau_r^2/K$, since the $\bar{\gamma}_g$ are independent with variance $\tau_r^2/K$; and $nKv_e/(nK)^2 = v_e/(nK)$. $\square$
The same model says what Theorem 3's estimates measure here. Within one question, the $K$ differences vary through both $\gamma$ and $e$, so $\hat{v}$ estimates $\tau_r^2 + v_e$, not $v_e$. And the question means $\bar{d}_i$ of one group share $\bar{\gamma}_g$, so they are correlated. A calculation like that of Theorem 3 gives $E[S_b^2] = \omega^2 + v_e/K + (\tau_r^2/K)\,n(1 - \sum_g w_g^2)/(n - 1)$, because $\sum_i \operatorname{Var}(\bar{d}_i) = n(\omega^2 + v_e/K + \tau_r^2/K)$ and $n\operatorname{Var}(\bar{d})$ is given by Theorem 5, and $E\sum_i(\bar{d}_i - \bar{d})^2$ is their difference. So the nested estimate $\hat{\omega}^2 = S_b^2 - \hat{v}/K$ has expectation $\omega^2 - (\tau_r^2/K)(n\sum_g w_g^2 - 1)/(n - 1)$, slightly too small when $\tau_r^2 > 0$.
Measuring the shared effect. Within one subject, the three run means of $d$ have variance $\tau_r^2 + v_e/n_g$ in expectation, while the average within-question variance has expectation $\tau_r^2 + v_e$. Under the nested model ($\tau_r^2 = 0$) the ratio of the first to the second divided by $n_g$ should be near 1. For the Classic pair the five subjects give ratios 1.4, 1.5, 0.7, 1.7 and 0.4, each on two degrees of freedom. Solving the two expectations subject by subject gives estimates of $\tau_r^2$ from $-0.00017$ to $0.00047$, mean $\hat{\tau}_r^2 = 0.00019$, and $\hat{v}_e = 0.0678$. On five subjects with three runs each, these data cannot pin $\tau_r^2$ down; what they show is no large shared effect for this pair. With $\hat{\tau}_r^2$ and $\sum_g w_g^2 = 0.201$, the crossed estimate of $\omega^2$ is 0.0911, almost the nested 0.0910 (they differ by 0.000060, of which the bias derived above accounts for 0.000013 at the estimated components; the rest is the sampling difference between the two estimators, since the crossed one estimates $v_e$ and $\tau_r^2$ as unweighted means over subjects), and Theorem 5 gives:
| Runs per question $K$ | 1 | 3 | 10 | unlimited |
|---|---|---|---|---|
| MDE at 514 questions, crossed (points) | 5.2 | 4.3 | 3.9 | 3.7 |
As in the nested table, "unlimited" means averaging over all 120 orders, which removes both order terms and leaves $2.8016\sqrt{\hat{\omega}^2/514}$.
The shared term matters more as questions are added within the same five subjects. If $\tau_r^2$ is as large as estimated, then with one run it is a share 0.11 of $\operatorname{Var}(\bar{d})$ at 514 questions, 0.56 at 5,140 and 0.93 at 51,400. Beyond a few thousand questions in the same subjects, further questions reduce the variance little (going from 5,140 to 51,400 questions cuts the standard error by only 22%), and more runs (orders) or more subjects are what reduce it. If $\tau_r^2$ is smaller, as the two negative subject estimates allow, the point where questions stop helping moves further out. For Theorem 4 the consequence is that with a shared order, extra runs are worth more than the nested formula says, since each one also averages the shared term; Worked problem 3 repeats its calculation under Theorem 5.
Example (temperature zero does not remove the problem). Setting the sampling temperature to zero removes the answer variance from sampling, so it might seem to make $K = 1$ free of noise. It does not remove the variance from the prompt: in the Classic data, merely reordering the same five examples accounts for 43% of the single-run variance, and Miller (2024, Section 3.3) argues that changing the temperature changes what is being measured.
Questions written in clusters
Many evaluations are written in groups: several questions about one passage, one program or one topic. Questions within a group tend to favor the same model, so they are positively correlated and carry less information than the same number of independent questions. The effect on planning is a factor.
The Binomial Distribution and Conditional Expectation and the Laws of Total Expectation and Variance derive the design effect: if clusters are drawn at random and $m$ questions are drawn from each, the variance of the mean is $1 + (m - 1)\rho$ times that of a simple random sample of the same total size, where $\rho$ is the intraclass correlation, the correlation between two questions of the same cluster. Clustered Standard Errors derives the cluster-robust variance that estimates it from data. For planning, Theorem 2 applies with $n$ multiplied by the design effect.
Those derivations are for 0/1 outcomes, and $d$ takes the values $-1$, 0 and 1, so here is the general version in one step. Suppose $c$ clusters are drawn at random and, given cluster $g$, $m$ questions are drawn independently (with replacement, or from a cluster much larger than $m$) with mean $\mu_g$ and variance $\sigma_g^2$. Write $\tau^2 = \operatorname{Var}(\mu_g)$ for the variance of the cluster means of $d$ and $\sigma_w^2 = E[\sigma_g^2]$ for the average variance within a cluster. By the law of total variance a single question drawn this way has variance $\tau^2 + \sigma_w^2$, and by the law of total covariance two questions of one cluster have covariance $\operatorname{Var}(\mu_g) = \tau^2$, since given the cluster they are independent. So a cluster's mean of $m$ questions has variance $\tau^2 + \sigma_w^2/m$, and the mean over $c$ independent clusters has
$$\operatorname{Var}(\bar{d}) = \frac{\tau^2 + \sigma_w^2/m}{c} = \frac{\tau^2 + \sigma_w^2}{cm}\,\big[1 + (m - 1)\rho\big], \qquad \rho = \frac{\tau^2}{\tau^2 + \sigma_w^2},$$
where the second form follows by multiplying out $(\tau^2 + \sigma_w^2)[1 + (m - 1)\rho] = \tau^2 + \sigma_w^2 + (m - 1)\tau^2 = m\tau^2 + \sigma_w^2$ and dividing by $cm$. The bracket is the design effect, and $\rho$ is the correlation of two questions of one cluster.
MMLU's questions were not written around shared passages, so the only grouping available is the subject (as the unit's topic page notes). Treating the 57 subjects, weighted equally, as the clusters of a benchmark written topic by topic:
- For the paired difference between Opus and GPT-4, $\tau^2 = 0.001229$, $\sigma_w^2 = 0.1169$ and $\rho = 0.0104$. The design effect is 1.198 with 20 questions per topic and 1.510 with 50.
- For Opus's own scores, the intraclass correlation is 0.093, and the design effect with 50 questions per topic would be 5.54.
These are the variances of MMLU's own 57 subject means, exact with the 14,042 questions as the population, which is how the simulation below uses them. As estimates for new topics written to the same pattern they include the sampling noise of those means. For the difference that noise is about 0.0008 of the 0.0012: corrected for it, $\rho$ is 0.0040 and the design effect at 50 questions per topic is 1.20, not 1.51. The design effects here, and Worked problem 4's answer, are therefore on the conservative side for new topics.
For this pair the difference clusters far less than either score. Is that typical? Across HELM's 28 models the intraclass correlation of a model's scores over the 57 subjects has median 0.113, and across the 378 pairs the intraclass correlation of the difference has median 0.015 and maximum 0.097. In 351 of the 378 pairs the difference clusters less than both models' scores. In these data, then, subject difficulty moves the two models' scores together and the paired difference cancels most of it, so a comparison of two models suffers much less from clustering than a single model's score does. That is an observation on MMLU's subjects and these models, not a theorem; for two models whose strengths lie in different subjects the difference would cluster strongly, since $\tau^2$ for $d$ is the variance across subjects of the difference between the two models' subject means.
The script checks the design effect by simulation: 20 subjects drawn at random with replacement, 50 questions drawn with replacement from each, 4,000 evaluations for each of 30 seeds (32,000,000 to 32,000,029). The variance of $\bar{d}$, divided by that of a simple random sample of 1,000 questions from the equal-weight mixture, had median 1.505 (interquartile range 1.485 to 1.531), against the formula's 1.510.
Where the planning values come from
Every number above used the variance of the full 14,042 questions, which a planner does not have. In practice $V$ comes from a pilot or from earlier runs of similar models, and the plan inherits the pilot's error. How much does that matter?
Here is the exact relation. Suppose the plan aims at a difference $\Delta_0$ and uses an estimate $\hat{V}$, so $n = (z_{1-\alpha/2} + z_{1-\beta})^2\hat{V}/\Delta_0^2$. With the true variance $V$, the evaluation's $\lambda$ at $\Delta_0$ is $\Delta_0\sqrt{n/V} = (z_{1-\alpha/2} + z_{1-\beta})\sqrt{\hat{V}/V}$, and its power is approximately
$$\Phi\left((z_{1-\alpha/2} + z_{1-\beta})\sqrt{\hat{V}/V} - z_{1-\alpha/2}\right).$$
The realized power depends only on the ratio $\hat{V}/V$. A variance underestimated by 20% gives power 0.71 instead of 0.80; overestimated by 20%, 0.87.
The script plans a comparison of Opus and GPT-4 for $\Delta_0 = 2$ points, which with the true variance needs 2,248 questions, from pilots of $m$ questions drawn from the 14,042 (30 seeds, 33,000,000 to 33,000,029), and computes the power each plan actually achieves:
| Pilot questions $m$ | Realized power, median | IQR | Lowest of 30 |
|---|---|---|---|
| 100 | 0.75 | 0.70 to 0.87 | 0.51 |
| 300 | 0.80 | 0.75 to 0.86 | 0.63 |
| 1,000 | 0.80 | 0.77 to 0.81 | 0.74 |
A pilot of 100 questions leaves a real chance of a plan with little more than even odds of detecting the difference it was built for. The variance of a 0/1 difference is driven by the discordant share, about one question in nine here, and a pilot of 100 questions is expected to contain only 11.5 discordant ones. Planning from earlier runs of models of similar strength, or from a larger pilot, is the remedy.
What a significant result from a small evaluation says
Low power has a second cost besides missed differences. When an underpowered evaluation does reach significance, the estimate that got it there tends to be too large. Gelman and Carlin (2014) call the factor by which a significant estimate overstates the true effect the exaggeration ratio (the expected Type M, or magnitude, error), and the probability that a significant estimate has the wrong sign the Type S (sign) error rate.
For an estimate that is normal and unbiased, both have closed forms. Let $Z = \hat{\Delta}/\operatorname{SE}$ be normal with mean $\lambda > 0$ and standard deviation 1, and call the result significant when $|Z| > z = z_{1-\alpha/2}$. The power is $\Phi(\lambda - z) + \Phi(-\lambda - z)$, and the Type S rate is the share of it from the wrong side, $\Phi(-\lambda - z)$ divided by the power. For the exaggeration ratio we need $E[|Z|;\ |Z| > z]$. Substituting $u = t - \lambda$ and using $\int_a^\infty u\,\phi(u)\,du = \phi(a)$ and $\int_{-\infty}^a u\,\phi(u)\,du = -\phi(a)$, where $\phi$ is the standard normal density,
$$\int_z^\infty t\,\phi(t - \lambda)\,dt = \phi(\lambda - z) + \lambda\,\Phi(\lambda - z), \qquad -\int_{-\infty}^{-z} t\,\phi(t - \lambda)\,dt = \phi(\lambda + z) - \lambda\,\Phi(-\lambda - z).$$
Adding them and dividing by $\lambda$ times the power,
$$\text{exaggeration ratio} = \frac{\lambda\,[\Phi(\lambda - z) - \Phi(-\lambda - z)] + \phi(\lambda - z) + \phi(\lambda + z)}{\lambda\,[\Phi(\lambda - z) + \Phi(-\lambda - z)]}.$$
The script checks the closed form against numerical integration. At $\lambda = 2.8$, the design with 80% power, it gives power 0.80, Type S rate $1.2 \times 10^{-6}$ and exaggeration ratio 1.125. Gelman and Carlin (2014, p. 643), who estimate the ratio by simulation, report 0.80, $1.2 \times 10^{-6}$ and 1.12.
For the running pair at 1,000 questions, $\lambda = 0.97$. The formulas give a Type S rate of 0.010 and an exaggeration ratio of 2.56: a significant estimate from such an evaluation is on average about two and a half times the true 1.04 points. The simulated evaluations above agree. Among those that reached significance, the mean of $|\bar{d}|/\Delta$ had median 2.55 over the 30 seeds (interquartile range 2.54 to 2.57), and a median share of 0.011 had the wrong sign. At 2,000 questions ($\lambda = 1.37$) the formulas give a Type S rate of 0.002 and a ratio of 1.88; in the simulation the ratio had median 1.88 (interquartile range 1.87 to 1.88) and the share with the wrong sign 0.001.
Example (the winner's margin). A leaderboard built on a 1,000-question evaluation reports that one model beats another "significantly", by 2.7 points. If the true difference is about 1 point, as for Opus and GPT-4, a margin of that size is what significance at this sample size selects for. Card et al. (2020) make the same point for natural language processing, observing that underpowered comparisons are common there and that they "increase the chances of exaggerated findings".
Several comparisons at once
An evaluation report usually compares a model with several others, or on several benchmarks. If each comparison is tested at 5%, the chance of at least one false positive grows with their number. The Bonferroni correction of Multiple Comparisons and the Bonferroni Correction tests each of $k$ comparisons at $\alpha/k$. For planning, that replaces $z_{1-\alpha/2}$ by $z_{1-\alpha/(2k)}$ in Theorem 2, and Worked problem 2 measures what it costs. The library note on Miller's paper observes that the paper's running example reads significance from three evaluations without such an adjustment.
What is proved, what is cited, what the experiment shows
Proved here. The variance of a paired comparison with $K$ draws per question, allowing the models' answers on one draw to be dependent (Theorem 1); unbiased estimators of its two components (Theorem 3); the cost-optimal number of draws (Theorem 4); the variance when one order is shared by all questions of a group, and what Theorem 3's estimates then measure (Theorem 5); the design effect for a difference that is not 0/1; the comparison of Theorem 2's sample size with McNemar's; the realized power of a plan built on an estimated variance; the closed forms for the Type S rate and exaggeration ratio of a normal estimate.
Cited or linked. The power function and sample size formula of the $z$ test, from Power and Sample Size for a Test of a Mean; the variance of a difference of dependent proportions and McNemar's sample size formula, from McNemar's Test for Paired Proportions; the paired variance, from Paired Comparisons of Models; the law of total variance and the design effect, from Conditional Expectation and the Laws of Total Expectation and Variance and The Binomial Distribution; the central limit theorem, from The Central Limit Theorem; Miller (2024) for the same formulas in the evaluation setting; Gelman and Carlin (2014) for the definitions of Type S and Type M errors and their numbers at 80% power.
What the experiments show. All the data are HELM's MMLU results, with the 14,042 questions standing in for a population. Within that setting, Theorem 2's power matched simulated evaluations at every size tried, the design effect matched simulation, pilots of 100 questions often produced underpowered plans, and significant results at 1,000 questions overstated the true difference by a factor of about 2.5. These are observations about these two models, these questions and these designs. The subject clusters are MMLU's subjects, not passages shared by questions, and the three runs in the Classic data vary only the order of a fixed set of five prompt examples, not the examples themselves or the sampling temperature. Each order is shared by every question of its subject, a crossed design; for the Classic pair the measured shared effect is small and imprecisely estimated (five subjects, three runs).
Worked problems
These problems were written for this article and are posed on the unit's HELM data. The script tools/experiments/statistics/power-analysis-for-evaluations.py recomputes every number.
Problem 1 (a benchmark of 164 questions). A benchmark has 164 questions. Two models with the same per-question variance as Opus and GPT-4 are compared on it. Find the minimum detectable difference at 80% power and the 5% level, and the power against a true difference of 1.04 points.
Solution. The minimum detectable difference is 7.4 points and the power 0.068.
By Theorem 2 with $V = 0.1145$ and $n = 164$,
$$\text{MDE} = 2.8016\sqrt{0.1145/164} = 2.8016 \times 0.02643 = 0.0740.$$
Against 1.04 points, $\lambda = 0.0104\sqrt{164}/0.3384 = 0.39$, and the power is $\Phi(0.39 - 1.96) + \Phi(-0.39 - 1.96) = 0.068$, barely above the 0.05 that a test has when there is no difference at all. A benchmark of this size can separate models that differ by several points, not by one. Against 3 points the power is only 0.21, so two models 3 points apart are declared "not significantly different" on it about four times in five, and a model reported as better by 3 points has not been shown to be better by any particular amount.
Problem 2 (five comparisons at once). A report will compare one model with five others, each by a paired test with a Bonferroni-corrected level, and wants 80% power against a 2-point difference for each comparison. With the running pair's variance, how many questions does each comparison need, and how does that compare with a single comparison?
Solution. Each comparison needs 3,345 questions, against 2,248 for a single comparison, 1.49 times as many.
With $k = 5$ comparisons each is tested at $0.05/5 = 0.01$, two-sided, so the critical value is $z_{1 - 0.01/2} = z_{0.995} = 2.576$. Theorem 2 gives
$$n = \frac{(2.576 + 0.8416)^2 \times 0.1145}{0.02^2} = 3{,}345,$$
against $(1.96 + 0.8416)^2 \times 0.1145/0.02^2 = 2{,}248$ for one comparison. The ratio is $[(2.576 + 0.8416)/(1.96 + 0.8416)]^2 = 1.49$. Since the same questions serve all five comparisons, the evaluation needs 3,345 questions in all, not five times that. The correction costs about half again as many questions, not five times as many, because the critical value grows only slowly with $k$.
Problem 3 (buying answers). For the Classic pair, $\hat{\omega}^2 = 0.0910$ and $\hat{v} = 0.0681$. A new question costs 20 times as much as one answer, where an answer is one run of a question with the five prompt examples in a new order. The evaluation must detect a 5-point difference with 80% power at the 5% level. Find the cost-optimal number of answers per question, the questions needed for $K = 1$ to 6, and the cost in units of one answer.
Solution. Four answers per question is cheapest, with 340 questions at a cost of 8,160 answer-units.
By Theorem 4, $K^* = \sqrt{20 \times 0.0681/0.0910} = 3.87$, so the best whole number is 3 or 4. For each $K$, Theorem 2 with $V = \hat{\omega}^2 + \hat{v}/K$ gives the questions needed, and the cost is $n(20 + K)$:
| $K$ | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
| Questions $n$ | 500 | 393 | 357 | 340 | 329 | 322 |
| Cost | 10,500 | 8,646 | 8,211 | 8,160 | 8,225 | 8,372 |
The costs fall to a minimum at $K = 4$ and rise after it, as the convexity in Theorem 4 requires. One answer per question would cost 29% more. The saving depends on answers being cheap relative to questions; at equal prices $K^* = 0.86$ and one answer is best.
These numbers plan for the nested design. If instead new questions come in five equal subjects and each run uses one order per subject, as in HELM, Theorem 5 applies with $\hat{\omega}^2 = 0.0911$, $\hat{v}_e = 0.0678$, $\hat{\tau}_r^2 = 0.00019$ and $\sum_g w_g^2 = 0.2$: the questions needed for a target variance $T$ are $(\omega^2 + v_e/K)/(T - \tau_r^2 \cdot 0.2/K)$. With one run, the shared term alone uses a share 0.12 of the target variance. Searching $K$ from 1 to 12, the cheapest design is again 4 runs, now with 350 questions at a cost of 8,400. For this pair the estimated shared effect is too small to change the answer, and the estimate itself is imprecise.
Problem 4 (a benchmark written topic by topic). A new benchmark will be written as 50 questions on each of a number of topics, with topics behaving like MMLU's subjects for the Opus and GPT-4 difference. How many topics are needed for 80% power against a 2-point difference?
Solution. 71 topics, or 3,550 questions.
A single question from a randomly chosen topic has variance $\tau^2 + \sigma_w^2 = 0.1182$, so a simple random sample would need $n = 7.8489 \times 0.11816/0.02^2 = 2{,}318.5$ questions (2,319 rounded up; all products here use unrounded values). With 50 questions per topic the design effect is $1 + 49 \times 0.0104 = 1.5097$, so the clustered design needs $2{,}318.5 \times 1.5097 = 3{,}500.3$ questions' worth of information, that is $3{,}500.3/50 = 70.005$ topics, rounded up to 71. This treats MMLU's observed subject means as the truth for new topics; with the sampling noise of those means removed, as in the section on clusters, the same calculation gives 56 topics. The 71 is the conservative answer. The answer depends heavily on what is compared. For a single model's score against a fixed threshold, with intraclass correlation 0.093, the design effect at 50 per topic would be 5.54, and writing fewer questions on more topics would be far more efficient.
Summary
| Quantity | Formula |
|---|---|
| Variance of a paired difference, one answer, 0/1 scores | $\operatorname{Var}(d) = \pi_{12} + \pi_{21} - \Delta^2$ |
| With $K$ draws per question | $\operatorname{Var}(\bar{d}) = (\omega^2 + \bar{v}/K)/n$ |
| Questions needed | $n = (z_{1-\alpha/2} + z_{1-\beta})^2\,V/\Delta^2$ |
| Minimum detectable difference | $(z_{1-\alpha/2} + z_{1-\beta})\sqrt{V/n}$ |
| Cost-optimal draws per question | $K^* = \sqrt{c_q\bar{v}/(c_a\omega^2)}$ |
| One order shared within groups (Theorem 5) | $\operatorname{Var}(\bar{d}) = (\omega^2 + v_e/K)/n + (\tau_r^2/K)\sum_g w_g^2$ |
| Clustered questions, $m$ per cluster | multiply $n$ by $1 + (m - 1)\rho$ |
| Realized power with estimated $\hat{V}$ | $\Phi\big((z_{1-\alpha/2} + z_{1-\beta})\sqrt{\hat{V}/V} - z_{1-\alpha/2}\big)$ |
What we have learned
The number of questions a comparison needs is set by the variance of the per-question difference between the two models, and pairing makes that variance small: for Claude 3 Opus and GPT-4 on MMLU it is 2.28 times smaller than the unpaired variance, and across 378 pairs of models the median correlation of 0.46 roughly halves the questions needed. Even so, a difference of 1.04 points needs 8,317 questions for 80% power, and an evaluation of 1,000 questions can reliably detect only differences of about 3 points. Repeated answers help only until the answer noise is small beside the spread between questions; when one prompt order is shared by many questions, as in HELM, only more runs or more groups reduce its part of the variance. Questions written in clusters cost a design effect that, on MMLU's subjects, is far smaller for differences than for scores, and plans built on small pilots often fall short. When an underpowered comparison does reach significance, its estimate is likely to be several times too large.
Power concerns whether a difference in accuracy can be seen. A model's answers also carry probabilities, and whether those probabilities can be trusted is a separate question, the subject of Calibration and Proper Scoring Rules.
References
- Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., and Jurafsky, D. (2020). With little power comes great responsibility. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 9263–9274). https://doi.org/10.18653/v1/2020.emnlp-main.745 record
- Gelman, A., and Carlin, J. (2014). Beyond power calculations: Assessing Type S (sign) and Type M (magnitude) errors. Perspectives on Psychological Science, 9(6), 641–651. https://doi.org/10.1177/1745691614551642 record
- Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021). Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR). arXiv:2009.03300. record · 7 pages
- Liang, P., Bommasani, R., Lee, T., et al. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research. arXiv:2211.09110. Raw results: public bucket
crfm-helm-public(MMLU project v1.0.0 and v1.1.0; Classic v0.3.0). record · 6 pages - Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640v1. record · 7 pages