Statistics of Model Evaluation

The statistics behind reporting and comparing the scores of language models and other systems on benchmarks. Eight articles, on the standard error of a benchmark score, questions that come in groups, paired comparisons of two models, the number of questions a comparison needs, calibration and proper scoring rules, rates of rare failures, ratings from pairwise verdicts, and comparisons across many tasks, with every interval and test derived and the running examples computed on per-question results of real models published by HELM, together with Cranfield retrieval runs and MT-bench human judgments.

A language model's benchmark score is usually printed as a single number, and two models are ranked by comparing such numbers. Behind each number is a sample: a few hundred or a few thousand questions, standing for the much larger set of questions the benchmark is meant to represent. This unit treats evaluation as the statistics of that sample. It asks how precisely a score measures a model, how grouped questions, repeated answers and the choice of prompt change that precision, how to tell whether one model is better than another, how many questions a comparison needs, whether a model's stated confidence can be trusted, how to bound the rate of a failure never observed, and how to combine many pairwise verdicts or many tasks into one ranking.

The unit follows the framework of Miller (2024), summarized in the library note Adding Error Bars to Evals, in which the questions of a benchmark are treated as a random sample from a larger population of questions. Its articles prove what that paper states, check it on real data, and go beyond it where the paper stops: to the choice between clusters and strata, the exact relation between the paired z test and McNemar's test, what the paired bootstrap's share of resamples means as a test, calibration, rare events and rankings.

What the unit assumes

The sampling distribution of a mean and the central limit theorem (Sampling Distributions, The Central Limit Theorem); confidence intervals for a proportion and their exact coverage (Confidence Intervals for a Proportion); the paired t-test and the variance of a difference (The Paired Samples t-Test); power and sample size for a mean (Power and Sample Size for a Mean); and from the collection's earlier units, the laws of total expectation and variance (Conditional Expectation and the Laws of Total Expectation and Variance), Slutsky's theorem (Convergence in Distribution and the Proof of the Central Limit Theorem), McNemar's test (McNemar's Test for Paired Proportions), the likelihood and its asymptotics (Likelihood and Estimation) and the Kullback–Leibler divergence (Cross-Entropy and Kullback–Leibler Divergence). Bootstrap intervals and permutation tests come from the parallel unit Resampling Methods.

What is proved and what is cited

Proved in the unit: the standard error of a benchmark score under sampling of questions, with the finite-pool correction; the bound $\operatorname{Var}(s) \le \mu(1 - \mu)$ that makes the 0/1 formula conservative for graded scores; the variance with repeated answers per question and the exact bias of a standard error pooled over them; the variance of a mean of clustered scores and the design effect for clusters of unequal size; the expectation, consistency and normal limit of the cluster-robust variance estimator; the stratified variance and its relation to the independent one; what pairing removes, the relation between the paired z statistic and McNemar's statistic, the variance of the paired bootstrap and the meaning of its share of resamples won. The later articles derive the sample size and minimum detectable difference formulas, the properness of the Brier and logarithmic scores, the zero-failure bounds, the Bradley–Terry likelihood equations and Elo's rule as a stochastic gradient step, and the rank-based tests for many tasks; each article lists what it proves.

Cited: Lyapunov's central limit theorem and other limit theorems, through the articles that state them; the history and finite-cluster behavior of the cluster-robust estimator (Cameron and Miller, 2015); the sampling account of when to cluster (Abadie et al., 2023); and published results of evaluations, reported as the papers' results. Coverage and error rates are checked by seeded experiments, as in the course, and their numbers describe those experiments.

The data

The unit's running data are the per-question results that Stanford's HELM project publishes in a public bucket, loaded and pinned by tools/experiments/statistics/evaldata.py:

  • HELM MMLU: 28 models, from Llama 2 (7B) and OLMo (7B) to GPT-4 and Claude 3 Opus, on all 14,042 test questions of MMLU's 57 subjects, each question scored right or wrong. The subjects serve as clusters or strata.
  • HELM Classic MMLU: eight older models on 514 questions in five subjects, each question asked under three orderings of the same five in-context examples, with the probability the model gave to its chosen answer. These give answer randomness and calibration.

The per-question "details" of the Open LLM Leaderboard were considered first and not used: the current ones require a logged-in account, and the older ones are published only as Parquet files, which need a library (pyarrow) that neither of the site's Python environments has. HELM's code is Apache-2.0 and MMLU's questions are MIT-licensed; no licence was found for HELM's result files, and only summary statistics derived from them are published here, with no question text or per-question HELM result. (The unit's own run of Qwen2.5 (0.5B) on MMLU, made by tools/experiments/statistics/mmlu_small_model.py for later articles, is stored per question; MMLU is MIT-licensed.) The second examples come from this site's own experiments: the per-query average precision of retrieval runs on the Cranfield collection (Comparing Ranking Models), and, for ratings, the expert pairwise judgments of MT-bench.

Notation

A benchmark has $n$ questions, $i = 1, \ldots, n$, with score $s_i$ (0 or 1, or a value in $[0, 1]$). The benchmark score is the mean $\bar s$, written $\hat p$ for 0/1 scores; its target is the mean $\mu$ over a population of questions, and $\sigma^2 = \operatorname{Var}(s)$. Following Miller (2024), $s_i = x_i + \epsilon_i$, with $x_i$ the question's conditional mean and $\sigma_i^2$ the variance of one answer given the question; $K$ is the number of answers per question. Clusters are $g = 1, \ldots, G$ with sizes $n_g$, scores $s_{gi}$, intraclass correlation $\rho$ and design effect $1 + (m^* - 1)\rho$. Two models A and B on the same questions have differences $d_i = s_{Ai} - s_{Bi}$ with mean $\bar d$, and, for 0/1 scores, the paired table $n_{11}$, $n_{12}$ (A right, B wrong), $n_{21}$ (A wrong, B right), $n_{22}$, as on McNemar's Test for Paired Proportions. Intervals are at level $1 - \alpha$ with $z_{1-\alpha/2}$ the normal quantile ($z_{0.975} = 1.96$). Models are named as HELM names them, in readable form, such as Llama 3 (70B) or GPT-4 (0613).

Articles

  1. Standard Errors for Benchmark Scores: what a benchmark score estimates and the standard error of a mean of question scores, with the finite-pool correction, the bound that makes the 0/1 formula conservative for graded scores, the variance with repeated answers and the bias of pooling them, and the choice of interval near 0 or 1 by exact coverage.
  2. Clustered Standard Errors: questions that come in groups. Derives the variance of a clustered mean and the design effect for unequal clusters, proves the expectation, consistency and normal limit of the cluster-robust estimator, and shows that whether MMLU's subjects are clusters or strata decides whether grouping multiplies the standard error by eight or lowers it.
  3. Paired Comparisons of Models: two models on the same questions. Identifies what pairing removes, relates the paired z statistic to McNemar's exactly, combines pairing with clustering, proves what the paired bootstrap estimates and what its share of resamples won tests, and measures the gain over 378 pairs of models.
  4. Power Analysis for Evaluations: how many questions a comparison needs, and the smallest difference an evaluation of a given size can detect, with repeated answers, clustered questions, the realized power of a plan built on a pilot and the exaggeration of significant results from small evaluations.
  5. Calibration and Proper Scoring Rules: whether a model's stated confidence matches how often it is right. Reliability diagrams, the expected calibration error and its bias, and the Brier and logarithmic scores, with their properness proved.
  6. Estimating Rare Event Rates: bounding the rate of a failure that was never or seldom observed, from the rule of three to extrapolation from more common events.
  7. Rating Models from Pairwise Comparisons: the Bradley–Terry model as a logistic regression, existence and uniqueness of its estimate, Zermelo's iteration, standard errors, ties, and Elo's rule as a stochastic gradient step on the same likelihood.
  8. Comparing Systems Across Many Tasks: rank-based comparison of several systems over many tasks, following Demšar's procedure, and where averaging scores over benchmarks misleads.

Where the unit leads

Evaluations on this site's own search use the same tools: Comparing Systems with Significance Tests applies paired tests over queries, and the search evaluation in tools/search/eval/ reports paired bootstrap intervals over a stratified sample of queries. Choosing among models by their estimated error on new data, rather than comparing fixed models, is the subject of the planned unit Prediction Error and Model Choice, and testing many comparisons at once that of Multiple Testing.

References