Standard Errors for Benchmark Scores

What the standard error of a language model's benchmark score measures and how to compute it. The benchmark as a sample of questions from a larger population; the standard error of a mean of question scores, with the finite-pool correction proved; the bound Var(s) <= mu(1 - mu) for scores between 0 and 1, which makes the 0/1 formula conservative for graded scores; the split of the variance into question difficulty and answer randomness, with the exact bias of a standard error pooled over repeated answers; and which interval to use for an accuracy near 0 or 1, by exact coverage. Worked on per-question results of 28 models on the 14,042 MMLU questions published by HELM, on three orderings of the same in-context examples for eight older models, and on Cranfield average precision, with worked problems on two close models, a three-valued score, repeated answers and a short benchmark.

In the Holistic Evaluation of Language Models (HELM) run of the MMLU benchmark, Llama 3 (70B) answered 11,048 of the 14,042 test questions correctly, an accuracy of 78.68%. Mixtral (8x22B) scored 77.35% and Qwen1.5 (72B) 77.42% on the same questions. A leaderboard prints such numbers to two decimals and ranks the models by them. How precisely does 78.68% measure Llama 3 (70B), and what would a "plus or minus" after it refer to?

The standard error of a benchmark score is the standard deviation of the score over the benchmarks that could have been drawn, under a stated model of how the questions were drawn. It is not a property of the number 78.68% alone. It depends on what the score is taken to estimate, and the first job of this article is to make that explicit. With it, the familiar formulas follow in a few lines, and so do their limits.

First we say what a benchmark score estimates and prove the standard error of a mean of question scores, including the correction for a benchmark drawn from a finite pool. Then we prove that a score between 0 and 1 has variance at most $\mu(1 - \mu)$, so the 0/1 formula overstates the error of graded scores. Next we split the variance into the part due to which questions were drawn and the part due to the randomness of answers. We also find the exact bias of a standard error computed from repeated answers as if they were independent. Finally we choose among intervals when the accuracy is near 0 or 1, by exact coverage, and check the normal interval for graded scores on small samples. The article assumes the sampling distribution of a mean (Sampling Distributions), the central limit theorem (The Central Limit Theorem) and the intervals of Confidence Intervals for a Proportion. It is the first article of Statistics of Model Evaluation, and the later articles build on its notation.

The data

HELM, from Stanford's Center for Research on Foundation Models, publishes the raw per-question output of every run in a public storage bucket that "allows public unauthenticated access" (HELM documentation, "Downloading Raw Results"). Two sets are used throughout this unit, loaded by tools/experiments/statistics/evaldata.py, which pins every file by its storage generation and SHA-256 hash. HELM's code is Apache-2.0 and MMLU's questions are MIT-licensed; no licence for HELM's result files was found. These pages use HELM's files only to compute summary statistics, and no question text or per-question HELM result is republished.

  • HELM MMLU. The MMLU benchmark of Hendrycks et al. (2021) has four-option multiple-choice questions in 57 subjects, from abstract algebra to world religions. The paper describes a test set of 14,079 questions; the HELM runs contain 14,042. HELM's MMLU releases v1.0.0 and v1.1.0 cover 28 models, each prompted with five examples from the subject and asked for the letter of the answer at temperature 0. A question scores 1 if the letter matches and 0 otherwise. Accuracies run from 28.74% for OLMo (7B), near the 25% of guessing, to 85.04% for Claude 3 Opus.
  • HELM Classic MMLU. Liang et al. (2023) evaluated older models on five of the subjects (514 test questions). Each question was asked three times, again at temperature 0. Liang et al. describe such trials as "three different random sets of in-context examples" (their Figure 31 caption), but for MMLU the pool of examples is the subject's development split of five questions, and five examples are shown, so every trial shows the same five examples in a different order. Within a trial, all questions of a subject and all models see the same order. (HELM source at the code tag v0.3.0, mmlu_scenario.py and in_context_learning_adapter.py; the script confirms it in the published prompts of all five subjects for two models.) Eight of these models were served with token probabilities; the next articles use those for calibration.

Throughout, $n$ is the number of questions, $s_i$ the score on question $i$ (here 0 or 1), and the benchmark score is the mean $\bar s = \frac1n\sum_i s_i$, written $\hat p$ when the scores are 0 or 1.

What a benchmark score estimates

Before computing an error, we must say what the score is an estimate of. Three answers are in use, and each gives a different standard error.

The benchmark itself. If the claim is "Llama 3 (70B) answers 78.68% of these 14,042 questions correctly under HELM's prompt," there is nothing to estimate. At temperature 0 the run is deterministic up to the serving system, so rerunning it gives the same number, and the standard error is zero. This reading is exact and of little use, because nobody cares about these particular questions for their own sake.

A super-population of questions. Miller (2024, Section 2) proposes to treat the questions as "drawn at random from a (hypothetical, infinite, unseen) super-population of questions." The target is the model's mean score $\mu$ over that population, its skill at questions of this kind. Each question's score is then a random variable, and

$$\mu = E[s], \qquad \sigma^2 = \operatorname{Var}(s).$$

A finite pool. Sometimes the questions really are a sample from a finite list: a 1,000-question subset of MMLU drawn to save compute, or a test split cut from a larger collection. Then the target is the mean over the list, and the standard error carries a correction for sampling without replacement.

The super-population reading is the one behind every standard error reported for language model benchmarks, and the rest of this unit adopts it unless a section says otherwise. It is an assumption, not a fact about the benchmark. MMLU's questions were collected by hand from sources the authors chose, not drawn by a random mechanism. The standard error then measures how much the score would vary over benchmarks assembled the same way, if they behaved like random draws. Miller's own discussion stops at the supposition; the limitations section of the library note on his paper records that it is not tested there.

Example (what the standard error is not). "The standard error says how much the score would change if we ran the model again." At temperature 0 the score would not change at all, and Llama 3 (70B)'s standard error below is 0.35 points. The standard error describes the variation over sets of questions. The variation over repeated answers to the same questions is a separate quantity, measured later in this article from HELM's three orderings of the in-context examples.

The standard error of a mean of question scores

Here is the idea in one sentence. A benchmark score is a sample mean, so its variance is the variance of one question's score divided by the number of questions, and the central limit theorem makes the mean approximately normal.

Theorem 1 (standard error of a benchmark score). Let $s_1, \ldots, s_n$ be independent and identically distributed with mean $\mu$ and variance $0 < \sigma^2 < \infty$, and let $S^2 = \frac{1}{n-1}\sum_i (s_i - \bar s)^2$. Then

  1. $E[\bar s] = \mu$ and $\operatorname{Var}(\bar s) = \sigma^2/n$;
  2. $E[S^2] = \sigma^2$, so $S^2/n$ is an unbiased estimate of $\operatorname{Var}(\bar s)$;
  3. as $n \to \infty$, $(\bar s - \mu)/(S/\sqrt n)$ converges in distribution to the standard normal, and $\bar s \pm z_{0.975}\,S/\sqrt n$, with $z_{0.975} = 1.96$, is an approximate 95% confidence interval for $\mu$.

Proof. (1) By linearity, $E[\bar s] = \frac1n\sum_i E[s_i] = \mu$. The $s_i$ are independent, so their covariances are 0 and the variance of the sum is the sum of the variances: $\operatorname{Var}(\bar s) = \frac{1}{n^2}\cdot n\sigma^2 = \sigma^2/n$. Both rules are proved in Expectation, Variance and Covariance. Parts 1 and 2 use only independence and a common mean and variance; part 3 uses the common distribution.

(2) Expand $\sum_i (s_i - \bar s)^2 = \sum_i s_i^2 - n\bar s^2$. Taking expectations, $E[s_i^2] = \sigma^2 + \mu^2$ and $E[\bar s^2] = \operatorname{Var}(\bar s) + \mu^2 = \sigma^2/n + \mu^2$. So $E\big[\sum_i (s_i - \bar s)^2\big] = n\sigma^2 + n\mu^2 - \sigma^2 - n\mu^2 = (n-1)\sigma^2$, and dividing by $n - 1$ gives $E[S^2] = \sigma^2$.

(3) The central limit theorem for independent, identically distributed terms with finite positive variance (Lindeberg–Lévy) gives $\sqrt n(\bar s - \mu)/\sigma \to N(0, 1)$. The weak law of large numbers, applied to the $s_i$ and to the $s_i^2$ (which have the finite mean $\sigma^2 + \mu^2$), gives $\frac1n\sum_i s_i^2 - \bar s^2 \to \sigma^2$ in probability (The Laws of Large Numbers); since $S^2$ is this times $n/(n-1)$, $S \to \sigma$ by the continuous mapping theorem; since $\sigma > 0$, $\sigma/S \to 1$, and Slutsky's theorem (Convergence in Distribution and the Proof of the Central Limit Theorem) allows $\sigma$ to be replaced by $S$. The interval is the set of $\mu$ for which the standardized mean lies between $-1.96$ and $1.96$. $\blacksquare$

For scores of 0 and 1, $s_i^2 = s_i$, so $\sum_i(s_i - \bar s)^2 = n\hat p - n\hat p^2 = n\hat p(1 - \hat p)$ and $S^2 = \frac{n}{n-1}\hat p(1 - \hat p)$. The standard error is then the familiar $\sqrt{\hat p(1 - \hat p)/n}$, apart from the factor $\sqrt{n/(n-1)}$, which is 1.00004 at $n = 14{,}042$. Miller's Equations 1 and 2 are these two forms.

Llama 3 (70B). With $\hat p = 11{,}048/14{,}042$,

$$\widehat{\operatorname{SE}} = \sqrt{\frac{0.7868 \times 0.2132}{14{,}042}} = 0.0035,$$

or 0.35 percentage points, and the 95% interval is 78.00% to 79.36%. The two Mixtral and Qwen models have the same standard error, 0.35 points, to two decimals. No 0/1 benchmark of 14,042 questions can have a standard error above 0.42 points, the value at $\hat p = 1/2$, because $p(1 - p) \le 1/4$.

The interval says that benchmarks assembled like MMLU, of this size, would put Llama 3 (70B) within about 0.7 points of its skill on such questions in 95% of draws. It does not by itself say whether Llama 3 (70B) is better than Mixtral (8x22B): that is a question about a difference, and the two scores were measured on the same questions. Paired Comparisons of Models answers it.

One assumption in Theorem 1 does not hold for MMLU: independence. The questions come in 57 subjects, and a model that is weak in professional law is weak on all 1,534 of its questions. Clustered Standard Errors shows that this raises Llama 3 (70B)'s standard error from 0.35 to 2.78 points if the subjects themselves are regarded as a sample, and explains when that is the right view. The present article keeps the independent case, which every later formula builds on.

A benchmark drawn from a finite pool

Suppose the 14,042 questions are the population and an evaluator scores a model on a random 1,000 of them, drawn without replacement. The draws are no longer independent: once a hard question is drawn, it cannot be drawn again, so the remaining pool is slightly easier. Here is what the correction does in one sentence. The draws are negatively correlated, so the variance of their mean is smaller than $\sigma^2/n$, by the factor $(N - n)/(N - 1)$.

Theorem 2 (sampling without replacement). Let a list of $N$ numbers have mean $\mu$ and variance $\sigma^2 = \frac1N\sum_{j=1}^N (v_j - \mu)^2$, and let $s_1, \ldots, s_n$ be a simple random sample of $n$ of them drawn without replacement. Then $E[\bar s] = \mu$ and

$$\operatorname{Var}(\bar s) = \frac{\sigma^2}{n}\cdot\frac{N - n}{N - 1}.$$

Proof. Each draw $s_i$ is equally likely to be any entry of the list, so $E[s_i] = \mu$ and $\operatorname{Var}(s_i) = \sigma^2$. For two different draws $i \ne k$, the pair $(s_i, s_k)$ is equally likely to be any ordered pair of distinct entries. Since $\sum_j (v_j - \mu) = 0$,

$$\operatorname{Cov}(s_i, s_k) = \frac{1}{N(N-1)}\sum_{j \ne l}(v_j - \mu)(v_l - \mu) = \frac{1}{N(N-1)}\Big[\Big(\sum_j (v_j - \mu)\Big)^2 - \sum_j (v_j - \mu)^2\Big] = -\frac{\sigma^2}{N - 1}.$$

The variance of the sum of the $n$ draws is $n\sigma^2 + n(n-1)\big(-\sigma^2/(N - 1)\big) = n\sigma^2\,\frac{N - n}{N - 1}$, and dividing by $n^2$ gives the result. $\blacksquare$

For 1,000 of the 14,042 questions the correction factor on the standard error is $\sqrt{(14{,}042 - 1{,}000)/14{,}041} = 0.9638$. At Llama 3 (70B)'s accuracy the standard error of the subset's accuracy, as an estimate of its accuracy on all 14,042, is 1.25 points instead of 1.30. When $n = N$ the variance is 0, the first reading above.

The correction answers a different question from the uncorrected formula. With it, the target is the accuracy on the full 14,042, which a full run would give exactly. Without it, the target is the super-population accuracy, and a subset of 1,000 and the full set are both samples from it. The evaluator must decide which target the subset stands for; most reports intend the second and should omit the factor.

Scores between 0 and 1

Not every benchmark scores 0 or 1. Reading-comprehension benchmarks give partial credit by word overlap (F1), code benchmarks report the share of tests passed, and retrieval evaluations average a precision per query. For such scores the general form $S^2/n$ of Theorem 1 applies. The 0/1 form $\sqrt{\bar s(1 - \bar s)/n}$ is sometimes used anyway. Miller (2024, Section 2.1) states that the Llama 3 report did so for scores such as F1, where the 0/1 form "will tend to be conservative (too wide)." The next theorem shows that the 0/1 form is always at least as large, and by how much.

Theorem 3 (variance of a bounded score). If $0 \le s \le 1$ and $E[s] = \mu$, then

$$\operatorname{Var}(s) = \mu(1 - \mu) - E[s(1 - s)] \le \mu(1 - \mu),$$

with equality if and only if $s$ takes only the values 0 and 1 (with probability 1). For the sample, $\frac1n\sum_i (s_i - \bar s)^2 \le \bar s(1 - \bar s)$ whenever every $s_i$ lies in $[0, 1]$.

Proof. $\operatorname{Var}(s) = E[s^2] - \mu^2$, and $s^2 = s - s(1 - s)$, so $E[s^2] = \mu - E[s(1 - s)]$ and $\operatorname{Var}(s) = \mu - \mu^2 - E[s(1 - s)]$. On $[0, 1]$ the product $s(1 - s)$ is nonnegative, so its expectation is nonnegative, which gives the inequality. A nonnegative variable has expectation 0 only if it is 0 with probability 1, and $s(1 - s) = 0$ exactly when $s$ is 0 or 1. The sample statement is the same argument applied to the distribution that puts mass $1/n$ on each $s_i$. $\blacksquare$

So the 0/1 formula never understates the standard error of a mean of bounded scores. It overstates it by the factor $\sqrt{\mu(1 - \mu)/\operatorname{Var}(s)}$, which grows as more of the scores sit strictly between 0 and 1.

Cranfield average precision. The second running example of this unit is the Cranfield retrieval collection of Comparing Ranking Models, where a query's score is its average precision, a number between 0 and 1. For BM25 over the 225 queries, the mean average precision is 0.3066 and the standard deviation of the per-query values is 0.2394. Only 4 queries score exactly 0 and 3 score exactly 1. The bound of Theorem 3 is $\sqrt{0.3066 \times 0.6934} = 0.4611$. The standard error of the mean is $0.2394/\sqrt{225} = 0.0160$, while the 0/1 formula gives 0.0307, larger by a factor of 1.93. An interval built from the 0/1 formula would be almost twice as wide as it should be.

Two sources of randomness

At a temperature above 0, a model asked the same question twice may answer differently. Even at temperature 0, the answer can depend on incidental choices such as which examples are shown in the prompt, or in what order. Miller (2024, Section 2.1) writes the score on question $i$ as

$$s_i = x_i + \epsilon_i,$$

where $x_i$ is the question's conditional mean, the average score the model would get over many answers to question $i$, and $\epsilon_i$ is the deviation of one answer from it, with mean 0 and variance $\sigma_i^2$ given the question. Questions are drawn from the super-population, so $x_i$ is random too. Two variances describe the model on the benchmark: $\operatorname{Var}(x)$, the spread of difficulty across questions, and $E[\sigma_i^2]$, the average randomness of a single answer.

Here is the idea. Averaging $K$ answers per question shrinks the second part by $K$ and leaves the first part alone, and the standard error must be computed from the question averages, not from the $Kn$ answers as if they were separate questions.

Theorem 4 (variance with repeated answers). Let each of $n$ independently drawn questions be answered $K$ times, with answers independent given the question, and let $\bar s_i$ be the mean of question $i$'s $K$ scores. Then the benchmark score $\bar s = \frac1n\sum_i \bar s_i$ has

$$\operatorname{Var}(\bar s) = \frac{1}{n}\Big(\operatorname{Var}(x) + \frac{E[\sigma_i^2]}{K}\Big).$$

Proof. Given question $i$, its $K$ scores are independent with mean $x_i$ and variance $\sigma_i^2$, so $E[\bar s_i \mid \text{question}] = x_i$ and $\operatorname{Var}(\bar s_i \mid \text{question}) = \sigma_i^2/K$. The law of total variance (Conditional Expectation and the Laws of Total Expectation and Variance) gives $\operatorname{Var}(\bar s_i) = E[\sigma_i^2/K] + \operatorname{Var}(x_i)$. The $\bar s_i$ are independent across questions, so Theorem 1 applies to them. $\blacksquare$

With $K = 1$ this is Miller's decomposition $\operatorname{Var}(\hat\mu) = (\operatorname{Var}(x) + E[\sigma_i^2])/n$ (his Section 3). The standard error of Theorem 1, computed from the $n$ question means $\bar s_i$, estimates this variance correctly for any $K$, because the $\bar s_i$ are independent and identically distributed. A common mistake is to compute it from all $Kn$ answers as if they were $Kn$ separate questions. Miller notes that such a pooled standard error is inconsistent. The next result says exactly how wrong it is.

Theorem 5 (the pooled standard error). Under the conditions of Theorem 4, let $S^2_{\text{pool}}$ be the sample variance of all $Kn$ scores, with divisor $Kn - 1$, and $\tau^2 = \operatorname{Var}(x) + E[\sigma_i^2]$ the variance of a single answer's score. Write $v = \operatorname{Var}(\bar s)$. Then

$$E\Big[\frac{S^2_{\text{pool}}}{Kn}\Big] = \frac{\tau^2 - v}{Kn - 1}.$$

The pooled standard error is $\sqrt{S^2_{\text{pool}}/(Kn)}$. As $n \to \infty$ with $K$ fixed, $n$ times its square, $S^2_{\text{pool}}/K$, converges in probability to $\tau^2/K$, while $n$ times the true variance is $nv = \operatorname{Var}(x) + E[\sigma_i^2]/K$. The ratio of the true variance to the pooled one therefore tends to

$$\frac{nv}{\tau^2/K} = \frac{K\operatorname{Var}(x) + E[\sigma_i^2]}{\operatorname{Var}(x) + E[\sigma_i^2]} \ge 1,$$

with equality only when $\operatorname{Var}(x) = 0$ or $K = 1$.

Proof. Write $L = Kn$ and let $y_1, \ldots, y_L$ be the $L$ scores, each with mean $\mu$ and variance $\tau^2$ (by the law of total variance, as in Theorem 4 with $K = 1$), and $\bar y = \bar s$ their mean. Then $\sum_j (y_j - \bar y)^2 = \sum_j y_j^2 - L\bar y^2$. Taking expectations, $E[y_j^2] = \tau^2 + \mu^2$ and $E[\bar y^2] = v + \mu^2$, so $E\big[\sum_j (y_j - \bar y)^2\big] = L\tau^2 - Lv$. Dividing by $L - 1$ and then by $L$ gives the first statement. For the limit, group the scores by question: $\sum_j y_j^2/L$ is the mean of the $n$ independent, identically distributed per-question averages of $y^2$, and $\bar y$ is the mean of the $\bar s_i$. By the weak law of large numbers they converge in probability to $\tau^2 + \mu^2$ and $\mu$, so $S^2_{\text{pool}} = \frac{L}{L-1}\big(\sum_j y_j^2/L - \bar y^2\big) \to \tau^2$. Dividing $nv = \operatorname{Var}(x) + E[\sigma_i^2]/K$ by $\tau^2/K$ gives the ratio. It exceeds 1 by $(K - 1)\operatorname{Var}(x)/\tau^2$, which is positive when $K > 1$ and $\operatorname{Var}(x) > 0$. $\blacksquare$

The pooled standard error treats the $K$ answers to one question as $K$ fresh questions. They are not: they share the question's difficulty $x_i$. The pooled estimate is too small whenever questions differ in difficulty, which on any real benchmark they do.

Three orderings of the in-context examples. The HELM Classic runs give a real case. Each of the 514 questions was asked three times, at temperature 0, with the same five examples in three different orders, so the only source of answer randomness is the order of the examples, which is set once per trial and shared by all questions of a subject, not set separately for each question. For text-davinci-002 the three accuracies were 56.42%, 56.03% and 57.00%, and 54 of the 514 questions were answered correctly under some draws and incorrectly under others.

The two variances can be estimated from the three runs. For each question, the sample variance $w_i$ of its three scores (divisor 2) has conditional expectation $\sigma_i^2$, so the mean of the $w_i$ estimates $E[\sigma_i^2]$ without bias. The sample variance of the question means $\bar s_i$ estimates $\operatorname{Var}(x) + E[\sigma_i^2]/3$ by Theorem 4, and subtracting a third of the first estimate leaves an estimate of $\operatorname{Var}(x)$. For text-davinci-002 these are $E[\sigma_i^2] \approx 0.0350$ and $\operatorname{Var}(x) \approx 0.2112$, so answer randomness is 14.2% of the variance of a single score. The estimate of $E[\sigma_i^2]$ stays unbiased under the shared design, because each $w_i$ uses only question $i$'s own three scores.

The standard errors follow:

  • from one ordering ($K = 1$), 2.19 points;
  • from the average of the three orderings ($K = 3$), 2.08 points;
  • pooled over all 1,542 answers as if independent, 1.26 points.

Averaging the three orderings reduces the variance of the benchmark score by 9.5%, and no number of orderings could reduce it by more than 14.2%, the share of answer randomness. The pooled figure is not a better standard error obtained from more data; it understates the correct one by about two fifths. Across the eight models, answer randomness was between 14.2% and 60.4% of the variance of a single score. The largest share belongs to GPT-NeoX (20B), whose accuracy is near the 25% of guessing. Nearness to guessing does not explain it: curie and GPT-J (6B), also near 25%, have shares of 22.8% and 19.4%, below davinci (26.1%) and OPT (175B) (28.8%).

The shared ordering. Theorem 4 assumes that answers are independent across questions given the questions. With one ordering per trial shared by a whole subject, each trial can add a common shift to all its questions, whose variance is not divided by $n$, and the $K = 3$ standard error of 2.08 points leaves that term out. Three trials cannot estimate it. Across the eight models, the variance of the three trial accuracies averaged 1.04 times what independent answer randomness alone would give ($E[\sigma_i^2]/n$), ranging from 0.35 (text-davinci-002) to 2.6 (GPT-J (6B)). With two degrees of freedom per model, these ratios are consistent with no shared effect, but they cannot rule out one as large as the independent part: pooled over the eight models, the one-sided 95% upper limit for the ratio is 2.1. Even a shared effect at that limit would add only about 6% to the variance of text-davinci-002's $K = 3$ score. Treating the orderings as independent across questions is therefore an approximation here.

Miller (2024, Section 3.3) argues against setting the temperature to 0 merely to remove answer randomness. Rounding each question's conditional mean to 0 or 1 can move the variance from the part that resampling reduces into $\operatorname{Var}(x)$, which it does not. HELM's runs are at temperature 0, and the three orderings show that randomness remains even then, through the prompt.

Accuracy near 0 or 1

Theorem 1's interval relies on the normal approximation, and for a 0/1 score near 0 or 1 that approximation is poor unless $n$ is large. Confidence Intervals for a Proportion computes the coverage of the Wald interval $\hat p \pm 1.96\sqrt{\hat p(1 - \hat p)/n}$, Wilson's score interval and the Clopper–Pearson interval exactly and explains why the Wald interval fails. That page establishes the intervals; here we apply them to benchmark sizes.

When questions are drawn independently, the number correct is exactly binomial with $n$ trials and success probability $\mu$, so the coverage of each interval is an exact sum over the binomial distribution. At GPT-4 (0613)'s HELM accuracy, $\mu = 0.8401$, the exact coverage of nominal 95% intervals is:

Questions Wald Wilson Clopper–Pearson
100 0.926 0.961 0.961
164 0.938 0.946 0.968
500 0.946 0.956 0.956
1,000 0.947 0.948 0.953

The 164 row is the size of HumanEval, which Miller's examples use (his Table 2). At 1,000 questions and this accuracy the three intervals agree to within a point of coverage. Closer to 1 the Wald interval breaks down even with 100 questions:

Accuracy Wald Wilson Clopper–Pearson
0.95 0.877 0.966 0.983
0.98 0.866 0.949 0.985
0.99 0.633 0.921 0.982

(100 questions; exact binomial sums.) At an accuracy of 0.99 the nominal 95% Wald interval covers the true accuracy only 63% of the time: with probability $0.99^{100} = 0.366$ every answer is right, and then the Wald interval is the single point 1.

Per-subject scores. Such sizes and accuracies are common in practice whenever a benchmark is broken down by subject. GPT-4 (0613) answered 95 of the 100 questions in US foreign policy correctly. The three 95% intervals for its accuracy on such questions are:

  • Wald: 90.7% to 99.3%;
  • Wilson: 88.8% to 97.8%;
  • Clopper–Pearson: 88.7% to 98.4%.

The Wald interval is symmetric about 95%, although only five points remain above it and the accuracy cannot pass 100%. The Wilson interval is pulled toward 50%, and the Clopper–Pearson interval is the widest. Bowyer, Aitchison and Ivanova (2025) reach the same recommendation for small benchmarks from simulations: CLT-based intervals "fail to provide adequate uncertainty estimates" with small, specialized benchmarks (their abstract), and they recommend the Wilson score interval or Bayesian intervals instead. In the cases computed above, Wilson's coverage stayed between 0.92 and 0.97, far closer to nominal than Wald's, and it costs nothing over the Wald interval; Confidence Intervals for a Proportion computes its coverage over every $p$, including the dips near 0 and 1 for small $n$.

Graded scores on few questions

Wilson's interval is specific to 0/1 scores. For graded scores, such as average precision, the usual interval is $\bar s \pm t_{n-1,\,0.975}\,S/\sqrt n$, the t interval of Confidence Intervals for a Mean. It is exact for normal scores, and graded benchmark scores are far from normal. How well does it do?

The experiment. The script treats the 225 Cranfield queries as a population whose BM25 mean average precision, 0.3066, is the truth. It draws samples of $n$ queries with replacement, so that the queries are independent draws, and computes the 95% t interval and the interval from the 0/1 formula. Each cell is the median over 30 seeds (20,261,118 to 20,261,147) of the share of 10,000 samples whose interval covers 0.3066, with the interquartile range in parentheses. The middle column is the median share of samples whose interval lies entirely below the truth.

Queries t interval covers t interval below the truth 0/1-formula interval covers
10 0.935 (0.933 to 0.936) 0.054 0.999 (0.998 to 0.999)
25 0.944 (0.942 to 0.945) 0.042 1.000 (0.999 to 1.000)
50 0.947 (0.945 to 0.948) 0.036 1.000 (1.000 to 1.000)

In this experiment the t interval came close to 95% from 25 queries on, but its misses were lopsided: most fell on the low side. Average precision is skewed to the right, with many queries near 0 and a few near 1. In these samples the mean and the standard deviation were positively correlated (median correlation 0.57 over the 30 seeds at each of the three sizes), so samples that missed the few high queries had both a low mean and a narrow interval. The 0/1 formula covered almost always, as Theorem 3 predicts: its intervals were about twice as wide as needed. These are observations on one population of 225 queries and one metric; a different metric or benchmark can behave differently, and bootstrap intervals designed for skewed statistics are the subject of Bootstrap Confidence Intervals.

What is proved here, what is cited, what the experiment shows

Proved here. The mean and variance of a benchmark score under independent sampling of questions, the unbiasedness of $S^2$, and the asymptotic interval (Theorem 1); the variance under sampling without replacement from a finite pool (Theorem 2); the bound $\operatorname{Var}(s) \le \mu(1 - \mu)$ for scores in $[0, 1]$ and its equality case (Theorem 3); the variance with $K$ repeated answers (Theorem 4) and the exact expectation and limit of the pooled standard error (Theorem 5).

Cited. The central limit theorem, the weak law of large numbers, Slutsky's theorem and the law of total variance, from the linked articles; the definitions and exact-coverage analysis of the Wald, Wilson and Clopper–Pearson intervals, from Confidence Intervals for a Proportion; the super-population framing, the conditional-mean decomposition and the temperature argument, from Miller (2024); the recommendation of Wilson and Bayesian intervals for small benchmarks, from Bowyer et al. (2025).

Computed. Every accuracy, standard error and interval from the HELM per-question files and the Cranfield per-query values; the coverage tables are exact binomial sums. The variance components for the three orderings of the in-context examples are estimates from 514 questions; that the orderings differ only in order is checked in the published prompts. The t-interval experiment is seeded, and its numbers describe that experiment on one population, not graded scores in general.

Worked problems

These problems were written for this article; they use the HELM results and small exact cases.

Problem 1 (two close models, taken one at a time). Mixtral (8x22B) scored 77.35% and Qwen1.5 (72B) 77.42% on the 14,042 questions. Give each a 95% interval, and say what the two intervals do and do not show.

Both standard errors are $\sqrt{\hat p(1 - \hat p)/14{,}042}$, which is 0.35 points for each. The intervals are 76.65% to 78.04% for Mixtral (8x22B) and 76.73% to 78.12% for Qwen1.5 (72B). They show that each model's skill on MMLU-like questions is pinned down to within about 0.7 points. They overlap almost entirely, but overlap of two intervals is not a test of the difference. The two scores were measured on the same questions, so they are correlated, and the standard error of their difference is not $\sqrt{0.35^2 + 0.35^2}$. Paired Comparisons of Models works this comparison with the paired standard error.

Problem 2 (how many questions). How many independent 0/1 questions guarantee a standard error of at most 1 point, whatever the model's accuracy? At most half a point?

The standard error is $\sqrt{\mu(1 - \mu)/n} \le \sqrt{1/(4n)} = 1/(2\sqrt n)$, with equality at $\mu = 1/2$. Requiring $1/(2\sqrt n) \le 0.01$ gives $\sqrt n \ge 50$, so $n \ge 2{,}500$. Halving the error to 0.005 requires $\sqrt n \ge 100$, so $n \ge 10{,}000$: four times the questions for half the error. The guarantee holds for every accuracy because the bound is attained at $\mu = 1/2$. With clustered questions these numbers are too small, by the design effect of Clustered Standard Errors.

Problem 3 (a three-valued score). A benchmark gives 0, 1/2 or 1 point per question. For one model, the three scores have probabilities 1/4, 1/4 and 1/2. Find $\mu$, $\operatorname{Var}(s)$ and the overstatement of the 0/1 formula, and check Theorem 3.

$\mu = 0 \cdot \tfrac14 + \tfrac12 \cdot \tfrac14 + 1 \cdot \tfrac12 = \tfrac58$. $E[s^2] = \tfrac14 \cdot \tfrac14 + \tfrac12 = \tfrac{9}{16}$, so $\operatorname{Var}(s) = \tfrac{9}{16} - \tfrac{25}{64} = \tfrac{11}{64}$. The 0/1 formula uses $\mu(1 - \mu) = \tfrac{15}{64}$. Theorem 3 says the difference is $E[s(1 - s)]$; only the score 1/2 contributes, $\tfrac12 \cdot \tfrac12 \cdot \tfrac14 = \tfrac{1}{16} = \tfrac{4}{64}$, and indeed $\tfrac{15}{64} - \tfrac{11}{64} = \tfrac{4}{64}$. The 0/1 formula overstates the standard error by the factor $\sqrt{15/11}$, about 1.17.

Problem 4 (pooling repeated answers). In Miller's example (his Section 3.1), scores are 0 or 1 and question difficulty $x$ is uniform on $[0, 1]$, so that $\operatorname{Var}(x) = 1/12$ and $E[\sigma_i^2] = E[x(1 - x)] = 1/6$. Each question is answered $K = 3$ times. By what factor does the pooled standard error understate the correct one, for many questions?

By Theorem 4, $n\operatorname{Var}(\bar s) = \tfrac{1}{12} + \tfrac{1}{6}/3 = \tfrac{5}{36}$. By Theorem 5, the pooled estimate behaves like $\tau^2/(Kn)$ with $\tau^2 = \tfrac{1}{12} + \tfrac{1}{6} = \tfrac14$, so $n$ times it is $\tfrac{1}{12} = \tfrac{3}{36}$. The ratio of variances is $\tfrac53$, so the pooled standard error is $\sqrt{3/5} = 0.775$ times the correct one: it understates it by about 23%. The two values of $E[\sigma_i^2]$ and $\operatorname{Var}(x)$ come from integrating $x(1 - x)$ and $(x - \tfrac12)^2$ over $[0, 1]$, which give $\tfrac12 - \tfrac13 = \tfrac16$ and $\tfrac{1}{12}$.

Problem 5 (a short benchmark near the ceiling). A model answers all 100 questions of a benchmark correctly. Give a 95% interval for its accuracy, and say what the Wald interval would report.

With $\hat p = 1$, the Wald standard error is $\sqrt{1 \cdot 0/100} = 0$, so the Wald interval is the single point 100%, which claims certainty. Wilson's interval at $x = n$ has upper end 1 and lower end $1/(1 + z^2/n) = 1/(1 + 3.8415/100) = 0.963$. The Clopper–Pearson lower end solves $p^{100} = 0.025$, so $p = 0.025^{1/100} = 0.964$. Either interval says that 100 out of 100 is consistent with an accuracy as low as about 96%. The second table above shows the same failure from the other side: at a true accuracy of 0.99, all 100 answers are right with probability 0.366, and every such run reports a zero-width Wald interval.

Summary

Question Answer
What does the score estimate? under the usual assumption, the mean score $\mu$ over a super-population of questions
Standard error, independent questions $S/\sqrt n$; for 0/1 scores $\sqrt{\hat p(1 - \hat p)/n}$ up to $\sqrt{n/(n-1)}$
Llama 3 (70B) on 14,042 MMLU questions 78.68%, standard error 0.35 points, interval 78.00% to 79.36%
Subset of a finite pool multiply the variance by $(N - n)/(N - 1)$ only if the pool's mean is the target
Graded scores use $S$; the 0/1 formula overstates (1.93 times for Cranfield average precision)
Repeated answers average them per question; never pool them (Theorem 5)
Accuracy near 0 or 1 Wilson's interval; the Wald interval can cover far less than 95%

What we have learned

A benchmark score has a standard error only relative to a model of how its questions were drawn. Under the super-population model it is a mean of independent scores, and its standard error is the score's standard deviation over $\sqrt n$, about a third of a point for 14,042 MMLU questions. That figure has to be read with three cautions proved here: a finite pool changes the target, graded scores need their own variance, and repeated answers to one question are not new questions. Near the ends of the scale the normal interval itself fails, and Wilson's interval should replace it.

One assumption was left standing: that the questions are independent. MMLU's questions come in 57 subjects, and for Llama 3 (70B) that grouping multiplies the standard error by 8.0 when the subjects are treated as a sample. Clustered Standard Errors derives the cluster-robust variance and shows when the larger figure is the right one.

References

  • Bowyer, S., Aitchison, L., and Ivanova, D. R. (2025). Position: Don't use the CLT in LLM evals with fewer than a few hundred datapoints. Proceedings of the 42nd International Conference on Machine Learning, PMLR 267, 81143–81184. arXiv:2503.01747. record · 2 pages
  • Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021). Measuring massive multitask language understanding. International Conference on Learning Representations (ICLR 2021). arXiv:2009.03300. record · 7 pages
  • Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., et al. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research, August 2023. arXiv:2211.09110. record · 6 pages
  • Miller, E. (2024). Adding error bars to evals: A statistical approach to language model evaluations. arXiv:2411.00640. record · 7 pages
  • Stanford Center for Research on Foundation Models. HELM documentation: Downloading raw results. https://github.com/stanford-crfm/helm/blob/main/docs/downloading_raw_results.md (raw results in the public bucket crfm-helm-public; files used here pinned on 18 November 2026). record