Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations

evaluationstatisticsstandard errorsconfidence intervalspower analysismethodology

Evan Miller treats eval questions as a random sample from an unseen population of questions and derives how to report language model evaluations as experiments, with standard errors (clustered where questions come in groups), variance reduction by resampling or next-token probabilities, paired comparisons between models, and power analysis; worked examples show clustering can triple standard errors and a 3-point difference needs about 1,000 questions to detect.

Evan Miller (Anthropic) · 2024-11-19 · original

When one language model scores 65.5% on a benchmark and another 63.0%, is the first model better, or did it happen to get easier questions? The paper observes that benchmark results for language models were commonly reported as single numbers, with the state-of-the-art result in bold and no test of whether differences are larger than chance (Section 1). This paper by Evan Miller argues that evaluations ("evals") are experiments and should be analyzed with the tools other sciences use for experiments. It presents formulas for the uncertainty of a single eval score, for comparing two models, and for planning how many questions an eval needs, and turns them into reporting recommendations (Abstract; Section 1). It was posted to arXiv on 1 November 2024 and summarized in an Anthropic post on 19 November 2024 titled "A statistical approach to model evaluations". It is a methods paper: its examples use fictional models, apart from one table of real standard errors.

The paper notes exceptions to the absence of error bars at the time: Chatbot Arena's confidence intervals on its ratings, and simple confidence intervals in the Llama 3 technical report (Section 1).

Terms used in this note

  • Super-population. A hypothetical, infinite, unseen population of questions from which an eval's questions are assumed to be drawn at random. The quantity of interest is the model's mean score over that population, its underlying skill, rather than its score on the particular questions sampled (Section 2).
  • Conditional mean and conditional variance. The score on question $i$ is written $s_i = x_i + \epsilon_i$: $x_i$ is the average score the model would get if asked question $i$ many times, and $\epsilon_i$ is the zero-mean randomness of a single answer, with variance $\sigma_i^2$ (Section 2.1).
  • Standard error of the mean (SEM). The standard deviation of an average across repeated samples, which measures how precisely the eval score estimates the population mean.
  • Clustered standard errors. Standard errors that account for questions drawn in groups, such as several questions about one reading passage (Section 2.2).
  • Paired analysis. Comparing two models question by question on the same questions, rather than comparing their overall averages (Section 4.2).
  • Power and Minimum Detectable Effect (MDE). Power is the probability that an experiment detects a difference of a given size when it exists; the MDE is the smallest difference an eval of a given size can reliably detect (Section 5).

The framework and its recommendations

The paper makes five recommendations (Section 1). Each is summarized below with its formula.

1. Report standard errors from the Central Limit Theorem

With $n$ independently drawn questions and observed mean $\bar{s}$, the Central Limit Theorem gives the standard error

$$\mathrm{SE}_{\text{C.L.T.}} = \sqrt{\frac{1}{n-1}\sum_i (s_i - \bar{s})^2 \Big/ n},$$

which for scores of 0 or 1 reduces to $\sqrt{\bar{s}(1-\bar{s})/n}$ (Equations 1 and 2). A 95% confidence interval is $\bar{s} \pm 1.96 \times \mathrm{SE}$ (Equation 3). The paper regards bootstrapping as unnecessary unless the sampling scheme or estimator is complicated. It suggests reporting the number of questions and the standard error in parentheses beneath each score (Table 2). It also states that the Llama 3 report used the 0/1 formula even for fractional scores such as F1, which tends to make those intervals too wide, and that the Inspect evaluation framework computes the general formula correctly (Section 2.1).

2. Cluster standard errors when questions come in groups

Reading-comprehension evals such as DROP, QuAC, RACE and SQuAD ask several questions about each passage, and MGSM translates the same question into many languages. Such questions are not independent, so the plain formula understates uncertainty. The clustered version adds the cross-products of residuals within each cluster $c$:

$$\mathrm{SE}_{\text{clustered}} = \left(\mathrm{SE}_{\text{C.L.T.}}^2 + \frac{1}{n^2}\sum_c \sum_i \sum_{j \neq i} (s_{i,c} - \bar{s})(s_{j,c} - \bar{s})\right)^{1/2}$$

(Equation 4, derived in Appendix A). It slides between treating each cluster as one observation, when scores within a cluster are perfectly correlated, and the unclustered case, when they are uncorrelated. The paper's one set of real numbers, computed on Anthropic models, shows the adjustment is "far from trivial": on DROP the clustered standard error is about three times the naive one (ratio 3.05), with smaller increases on MGSM and RACE-H (Table 4). The paper concludes that the Llama 3 report's confidence intervals for reading-comprehension evals are "likely anti-conservative (too narrow)" (Section 2.2), and suggests reporting the number of clusters alongside the number of questions (Table 3).

3. Reduce the variance within questions

By the law of total variance, the variance of the estimated mean splits into the spread of question difficulty and the average randomness of answers: $\mathrm{Var}(\hat{\mu}) = (\mathrm{Var}(x) + \mathbb{E}[\sigma_i^2])/n$ (Section 3). The first part is fixed by the population of questions; the second can be reduced in two ways.

  • Resampling. Answer each question $K$ times and use the average, so the per-question randomness falls to $\sigma_i^2/K$. In a worked example with 0/1 scores and question difficulty uniform between 0 and 1, $\mathrm{Var}(x) = 1/12$ and $\mathbb{E}[\sigma_i^2] = 1/6$. Going from $K = 1$ to $K = 2$ reduces total variance by 1/3, to $K = 4$ by 1/2, and to $K = 6$ by 5/9, with an upper limit of 2/3 (Section 3.1). The standard error must be computed over the $n$ question averages, not over all $Kn$ answers, which are not independent.
  • Next-token probabilities. For evals without chain-of-thought reasoning, where the answer is the first token, the probability the model assigns to the correct token can serve as the score, which removes answer randomness entirely; in the same example this achieves the full 2/3 reduction (Section 3.2). The summary post adds that the author was "not aware of an open-source evals framework" that implemented this.
  • "Don't touch the thermostat!" Lowering the sampling temperature to make answers deterministic is advised against unless the purpose is to study the model at that temperature (Section 3.3). It changes the model's behavior and can move variance into the part that cannot be reduced. In one example, rounding uniformly distributed per-question means to 0 or 1 at temperature zero triples the variance of the conditional means, from 1/12 to 1/4. In another, the expected score shifts from 2/3 to 3/4 and that variance rises roughly fivefold, from 1/27 to 3/16.

4. Compare models with paired differences

If two models answer the same questions, the standard error of their difference should be computed from the per-question differences (Equation 7). Its variance is lower than the unpaired one by $2\,\mathrm{Cov}(x_A, x_B)/n$, so pairing helps whenever the models tend to find the same questions easy or hard (Section 4.2). With uniformly distributed scores and a correlation of 0.5, pairing reduces the variance from 1/6 to 1/9, a relative reduction of 1/3. Because eval scores are likely to correlate even across unrelated models, the paper calls pairing a "free" reduction in variance, and recommends that reports include pairwise differences, their standard errors and score correlations. The summary post adds that on popular evals the question-score correlations between frontier models fall between 0.3 and 0.7.

The paper's running example shows what this changes. The fictional model "Galleon" beats "Dreadnought" on MATH (65.5% against 63.0%) but trails on HumanEval and MGSM by slightly wider margins, which at face value favors Dreadnought on two of three evals (Section 1, Table 1). With paired, appropriately clustered standard errors, only the MATH difference is statistically significant (95% confidence interval +1.2% to +3.8%); the intervals for HumanEval and MGSM include zero (Table 5). So "the careful analyst" would lean toward the opposite of the superficial conclusion (Section 4.2). HumanEval has only 164 questions (Table 2).

5. Plan evals with power analysis

For a paired comparison with significance level $\alpha$, power $1-\beta$ and minimum detectable effect $\delta$, the number of independent questions needed is

$$n = \frac{(z_{\alpha/2} + z_\beta)^2\,(\omega^2 + \sigma_A^2/K_A + \sigma_B^2/K_B)}{\delta^2},$$

where $\omega^2 = \mathrm{Var}(x_A) + \mathrm{Var}(x_B) - 2\,\mathrm{Cov}(x_A, x_B)$ is the variance of the per-question difference in means, $\sigma_A^2$ and $\sigma_B^2$ are the mean answer randomness of each model, and $K_A$, $K_B$ are the resamples per question (Equation 9, derived in Appendix B; cluster-adjusted versions in Appendix C).

Worked example (Section 5): with no answer randomness, $\omega^2 = 1/9$, a 3-point difference ($\delta = 0.03$), 80% power and a 5% false-positive rate, $n \approx 969$. The paper calls these parameters fictional but reasonable, and suggests that new evals "should contain at least 1,000 questions". Inverting the formula gives the MDE for a fixed number of questions (Equation 10): with 198 questions, $\omega^2 = 1/9$, answer variances of 1/6 and a correlation of 0.5, raising $K$ from 1 to 10 lowers the MDE from 13.2% to 7.5%.

Limitations and open questions

  • The sampling assumption. The framework rests on treating eval questions as a random draw from a population of possible questions. The paper presents this as a supposition that lets the analysis reach "the underlying skill" (Section 2), and the summary post adds that conclusions should then hold for a new eval with questions of the same difficulty distribution; neither tests the assumption empirically.
  • Quantitative only. The paper sets aside the qualitative merits of particular evals, on which it stays agnostic (Section 1), and refers to Challenges in evaluating AI systems for those issues.
  • Example data. Apart from Table 4, the numbers are fictional or derived from idealized distributions, so the size of the gains on real evals depends on quantities, such as score correlations and answer variance, that must be estimated from previous runs (Appendix C).
  • Scope. The formulas are developed for evals scored as an average over questions; preference-based ratings such as Chatbot Arena's are mentioned only in the introduction.

Research questions

  1. How often do published comparisons survive paired testing? Re-analyzing question-level results from released evaluations with the paper's paired, clustered standard errors would show how many reported state-of-the-art margins are within noise.
  2. How large are clustering effects across evals? Table 4 covers three evals. Computing clustered and naive standard errors for other grouped benchmarks would show how general the up-to-threefold ratio is.
  3. How far do correlations between models vary by eval? The summary post gives 0.3 to 0.7. Measuring correlations across many model pairs and evals would indicate where pairing helps most.
  4. Do later Anthropic reports adopt these practices? Checking whether subsequent system cards and papers report standard errors, question counts and paired differences would show the paper's uptake.
  5. What does temperature zero do to real eval scores? The thermostat argument uses idealized distributions. Running the same evals at several temperatures with resampling would measure the bias and variance changes in practice.
  6. Does the super-population assumption hold for benchmarks built from narrow sources? Comparing scores on held-out questions written to the same specification with the original confidence intervals would test whether intervals from one question set predict another.

How it connects

References

  • Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640v1. https://arxiv.org/abs/2411.00640
  • Anthropic. (2024, November 19). A statistical approach to model evaluations [summary post]. https://www.anthropic.com/research/statistical-approach-to-model-evals
  • Ganguli, D., Schiefer, N., Favaro, M., and Clark, J. (2023). Challenges in evaluating AI systems. Anthropic. https://www.anthropic.com/research/evaluating-ai-systems