Clustered Standard Errors

How to compute the standard error of a benchmark score when questions come in groups, such as several questions on one passage or one subject. The variance of a mean of clustered scores and the design effect for clusters of unequal size, derived; the cluster-robust variance estimator, its identity with Miller's formula, its exact expectation (unbiased after the factor G/(G - 1) when clusters are equal), and its consistency and normal limit as the number of clusters grows, proved; and the choice between clusters and strata, which decides whether grouping raises or lowers the standard error. Worked on the 57 subjects of MMLU for 28 models, where treating subjects as a sample multiplies the standard error by 2.3 to 9.0, with a coverage experiment that samples subjects; worked problems on a small exact case, the effective number of questions, three reporting claims and Miller's DROP figures.

In HELM's run of MMLU, Llama 3 (70B) answered 78.68% of 14,042 questions correctly, and Standard Errors for Benchmark Scores gave that score a standard error of 0.35 percentage points by treating the questions as independent. But the questions come in 57 subjects, and the model's accuracy varies from 43.0% in abstract algebra to 97.9% in its best subject. A model that is weak in professional law is weak on all 1,534 of its questions at once. How much does this grouping change the standard error, and when should it be counted?

A cluster is a group of questions whose scores may be dependent: questions about one passage, one problem translated into several languages, or one subject of an exam. A clustered standard error (also called cluster-robust) is a standard error computed under the assumption that clusters are drawn independently while questions within a cluster may be correlated in any way. For Llama 3 (70B) it is 2.78 points, eight times the independent figure.

First we derive the variance of a mean of clustered scores and the design effect for clusters of unequal size. Then we build the cluster-robust estimator, show that it is the formula Miller (2024) gives, and prove its exact expectation, its consistency and its normal limit as the number of clusters grows. Next we compute it for the 28 models on MMLU's subjects and ask the question that decides whether it applies: are the subjects a sample, or fixed strata? Finally we check its coverage by sampling subjects. The article assumes Standard Errors for Benchmark Scores, whose notation it uses, and the design effect for equal clusters derived in The Binomial Distribution.

Questions that come in groups

Miller (2024, Section 2.2) names the common cases. DROP, QuAC, RACE and SQuAD ask "multiple related questions about independently selected passages of text," and MGSM consists of "the same question translated into many languages." MMLU groups its questions by subject. In each case the questions of a cluster share something that affects the score: a passage the model misreads, a problem it cannot solve in any language, a subject it knows poorly.

The model. There are $G$ clusters, $g = 1, \ldots, G$, with $n_g$ questions in cluster $g$ and $n = \sum_g n_g$ in all. The score of question $i$ in cluster $g$ is $s_{gi}$, and the cluster total is $S_g = \sum_{i=1}^{n_g} s_{gi}$. Clusters are independent of each other. Within a cluster the scores have a common mean $\mu$ and may be correlated in any way. The sizes $n_g$ are treated as fixed, which amounts to conditioning on them; the article returns to this choice in the experiment. The benchmark score is again $\bar s = \frac1n\sum_g S_g$.

Theorem 1 (variance of a clustered mean). Under this model, $E[\bar s] = \mu$ and

$$\operatorname{Var}(\bar s) = \frac{1}{n^2}\sum_{g=1}^G \operatorname{Var}(S_g) = \frac{1}{n^2}\sum_{g=1}^G\Big(\sum_{i}\operatorname{Var}(s_{gi}) + \sum_{i \ne j}\operatorname{Cov}(s_{gi}, s_{gj})\Big).$$

Proof. $E[\bar s] = \frac1n\sum_g\sum_i \mu = \mu$. The cluster totals are independent, so the variance of their sum is the sum of their variances, and $\operatorname{Var}(\bar s) = \frac{1}{n^2}\sum_g \operatorname{Var}(S_g)$. The variance of the sum $S_g$ is the sum of the variances of its terms plus the covariances of every ordered pair of different terms (Expectation, Variance and Covariance). $\blacksquare$

With independent questions every covariance is 0 and the formula reduces to $\sigma^2/n$. Positive covariances within clusters add to it, and there are $n_g(n_g - 1)$ of them in a cluster of size $n_g$: large clusters multiply even a small correlation.

The design effect

How large is the inflation? A simple model answers in one number. Suppose every score has variance $\sigma^2$ and any two scores in the same cluster have correlation $\rho$, the intraclass correlation. This is the exchangeable model: within a cluster the questions are interchangeable.

Theorem 2 (design effect for clusters of unequal size). Under the exchangeable model,

$$\operatorname{Var}(\bar s) = \frac{\sigma^2}{n}\big[1 + (m^* - 1)\rho\big], \qquad m^* = \frac{1}{n}\sum_{g} n_g^2 = \bar m + \frac{V_m}{\bar m},$$

where $\bar m = n/G$ is the mean cluster size and $V_m = \frac1G\sum_g (n_g - \bar m)^2$ their variance.

Proof. Each covariance within a cluster is $\rho\sigma^2$, so $\operatorname{Var}(S_g) = n_g\sigma^2 + n_g(n_g - 1)\rho\sigma^2$. Summing over clusters with Theorem 1,

$$\operatorname{Var}(\bar s) = \frac{\sigma^2}{n^2}\Big(n + \rho\sum_g n_g^2 - \rho n\Big) = \frac{\sigma^2}{n}\Big(1 + \rho\Big(\frac{\sum_g n_g^2}{n} - 1\Big)\Big).$$

For the second form, $\sum_g n_g^2 = G(V_m + \bar m^2)$, and dividing by $n = G\bar m$ gives $\bar m + V_m/\bar m$. $\blacksquare$

The factor $1 + (m^* - 1)\rho$ is the design effect, "the ratio of the actual variance of a sample to the variance of a simple random sample of the same number of elements" in Kish's definition (Kish, 1965, p. 258, as quoted by Cameron and Miller, 2015). With equal clusters $m^* = \bar m$, and the factor is the $1 + (n - 1)\rho$ of The Binomial Distribution for one cluster of size $n$. Unequal sizes make it larger than the mean size alone suggests, because $m^*$ is the size of the cluster that a randomly chosen question belongs to, on average. Cameron and Miller (2015, Section II.B) give the same replacement of the mean size by $V_m/\bar m + \bar m$ for regression, citing Moulton (1986, p. 387).

For MMLU, the subjects have between 100 and 1,534 questions, a mean of 246.4 and a standard deviation of 234.2, so $m^* = 468.9$. A correlation of only 0.01 between two questions of the same subject would give a design effect of $1 + 467.9 \times 0.01 = 5.68$.

The cluster-robust estimator

Theorem 2 needs $\rho$ and a model as rigid as exchangeability. The cluster-robust estimator needs neither. Here is the idea in one sentence. Theorem 1 says that only the variances of the cluster totals matter, and each cluster's squared deviation of its total from its expected value is an unbiased estimate of that variance, so we add up the squared residual totals.

The estimator. With residuals $e_{gi} = s_{gi} - \bar s$ and residual totals $R_g = \sum_i e_{gi} = S_g - n_g\bar s$,

$$\hat V_{\text{CR}} = \frac{1}{n^2}\sum_{g=1}^G R_g^2, \qquad \widehat{\operatorname{SE}}_{\text{CR}} = \sqrt{\hat V_{\text{CR}}}.$$

This is the sandwich estimator of Liang and Zeger (1986) and White (1984) applied to a regression on a constant, as Cameron and Miller (2015, Section II.C) present it, and as Miller (2024, Appendix A) derives it for evaluations. When every cluster has one question, $R_g = s_g - \bar s$ and the estimator is $\frac{1}{n^2}\sum_g (s_g - \bar s)^2$, the independent-question variance with divisor $n$ instead of $n - 1$.

Miller's formula is the same estimator. Miller's Equation 4 writes the clustered standard error as the independent one plus the cross-products within clusters:

$$\operatorname{SE}_{\text{clustered}}^2 = \operatorname{SE}_{\text{C.L.T.}}^2 + \frac{1}{n^2}\sum_g\sum_i\sum_{j \ne i} e_{gi}e_{gj}, \qquad \operatorname{SE}_{\text{C.L.T.}}^2 = \frac{1}{n(n-1)}\sum_{g,i} e_{gi}^2.$$

Expanding the square, $R_g^2 = \sum_i e_{gi}^2 + \sum_{i \ne j} e_{gi}e_{gj}$, so

$$\hat V_{\text{CR}} = \frac{1}{n^2}\sum_{g,i} e_{gi}^2 + \frac{1}{n^2}\sum_g\sum_{i \ne j} e_{gi}e_{gj} = \operatorname{SE}_{\text{clustered}}^2 - \frac{1}{n}\operatorname{SE}_{\text{C.L.T.}}^2,$$

using $\frac{1}{n^2} = \frac{1}{n(n-1)} - \frac{1}{n^2(n-1)}$. The two differ by one $n$-th of the independent variance, a relative difference of less than 0.01% for MMLU.

How good an estimate is $\hat V_{\text{CR}}$? Each $R_g^2$ is a poor estimate of $\operatorname{Var}(S_g)$ on its own, one squared number. Cameron and Miller (2015, Section II.C) find it striking that averaging these poor estimates across clusters still gives a consistent estimate, and point out that the averaging is over clusters, so its accuracy depends on the number of clusters $G$. The next two theorems make both halves of that remark precise.

Theorem 3 (expectation of the cluster-robust estimator). Under the model above, write $V_g = \operatorname{Var}(S_g)$ and $w_g = n_g/n$. Then

$$E[\hat V_{\text{CR}}] = \frac{1}{n^2}\Big[\sum_g V_g(1 - 2w_g) + \Big(\sum_g w_g^2\Big)\sum_h V_h\Big].$$

If all clusters have the same size, $E[\hat V_{\text{CR}}] = \big(1 - \tfrac1G\big)\operatorname{Var}(\bar s)$, so $\frac{G}{G-1}\hat V_{\text{CR}}$ is unbiased.

Proof. Let $T_g = S_g - n_g\mu$, which has mean 0 and variance $V_g$, and the $T_g$ are independent. Since $\bar s - \mu = \frac1n\sum_h T_h$, the residual total is $R_g = S_g - n_g\bar s = T_g - w_g\sum_h T_h$. Squaring and taking expectations, with $E[T_gT_h] = 0$ for $g \ne h$,

$$E[R_g^2] = V_g - 2w_gV_g + w_g^2\sum_h V_h.$$

Summing over $g$ and dividing by $n^2$ gives the formula. With equal sizes, $w_g = 1/G$ and $\sum_g w_g^2 = 1/G$, so the bracket is $\sum_g V_g(1 - 2/G + 1/G) = (1 - 1/G)\sum_g V_g$, and $\frac{1}{n^2}\sum_g V_g = \operatorname{Var}(\bar s)$ by Theorem 1. $\blacksquare$

The bias comes from estimating $\mu$ by $\bar s$: residuals around the sample mean are on average smaller than deviations around $\mu$, exactly as $\sum(x_i - \bar x)^2$ underestimates $(n-1)\sigma^2$ by one degree of freedom out of $n$. Here the degrees of freedom are clusters, not questions. The factor $G/(G-1)$ is the correction SAS uses and Stata uses in a slightly different form (Cameron and Miller, 2015, Section II.C); this article calls the corrected version CR1. With unequal sizes the first term penalizes large clusters, and no single factor removes the bias in general.

Theorem 4 (consistency and normal limit). Suppose the scores lie in $[0, 1]$, every cluster has at most $M$ questions, and $\operatorname{Var}(S_g) \ge c$ for some $c > 0$ and every $g$. As the number of clusters $G \to \infty$,

  1. $\hat V_{\text{CR}}/\operatorname{Var}(\bar s) \to 1$ in probability;
  2. $(\bar s - \mu)/\widehat{\operatorname{SE}}_{\text{CR}}$ converges in distribution to the standard normal.

Proof. Let $W = \sum_g V_g$, so $\operatorname{Var}(\bar s) = W/n^2$, and $\hat V_{\text{CR}}/\operatorname{Var}(\bar s) = \sum_g R_g^2/W$. The assumptions give $|T_g| \le M$, $V_g \le M^2$, $W \ge cG$, and $w_g \le M/n \le M/G$ (since $n \ge G$). Expand

$$\sum_g R_g^2 = \sum_g T_g^2 - 2\Big(\sum_g w_gT_g\Big)\Big(\sum_h T_h\Big) + \Big(\sum_g w_g^2\Big)\Big(\sum_h T_h\Big)^2.$$

First term. $E[\sum_g T_g^2] = W$, and the $T_g^2$ are independent with $\operatorname{Var}(T_g^2) \le E[T_g^4] \le M^4$, so $\operatorname{Var}(\sum_g T_g^2/W) \le GM^4/(cG)^2 = M^4/(c^2G) \to 0$. By Chebyshev's inequality (Probability Inequalities), $\sum_g T_g^2/W \to 1$ in probability.

Second term. By the Cauchy–Schwarz inequality, $E\big|\sum_g w_gT_g\sum_h T_h\big| \le \sqrt{E[(\sum_g w_gT_g)^2]\,E[(\sum_h T_h)^2]} = \sqrt{\sum_g w_g^2V_g \cdot W}$. Here $\sum_g w_g^2V_g \le G(M/G)^2M^2 = M^4/G$. Dividing by $W \ge cG$, the expected absolute value of the second term over $W$ is at most $2\sqrt{M^4/(GW)} \le 2M^2/(G\sqrt c) \to 0$, so by Markov's inequality the term over $W$ tends to 0 in probability.

Third term. $\sum_g w_g^2 \le \max_g w_g \sum_g w_g \le M/G$, and $E[(\sum_h T_h)^2] = W$, so the expected value of the third term over $W$ is at most $M/G \to 0$, and by Markov's inequality it tends to 0 in probability.

Together, $\sum_g R_g^2/W \to 1$ in probability, which is part 1.

For part 2, $(\bar s - \mu)/\sqrt{\operatorname{Var}(\bar s)} = \sum_g T_g/\sqrt W$ is a standardized sum of independent, mean-zero, non-identically distributed terms. Lyapunov's theorem, stated in The Sign Test and the Wilcoxon Signed-Rank Test (Billingsley, 1995), gives a standard normal limit when $\sum_g E|T_g|^3/W^{3/2} \to 0$. Since $|T_g| \le M$, $E|T_g|^3 \le M\,E[T_g^2] = MV_g$, so the ratio is at most $MW/W^{3/2} = M/\sqrt W \le M/\sqrt{cG} \to 0$. By part 1 and the continuous mapping theorem, $\widehat{\operatorname{SE}}_{\text{CR}}/\sqrt{\operatorname{Var}(\bar s)} \to 1$ in probability, and Slutsky's theorem (Convergence in Distribution and the Proof of the Central Limit Theorem) completes the proof. $\blacksquare$

The theorem is about many clusters. Nothing in it requires many questions per cluster, and with a fixed number of clusters no number of questions makes the estimator consistent: it is still an average of $G$ squared totals. For example, take $G$ clusters of equal size whose totals $T_g$ are normal with a common variance $V$. Then $R_g = T_g - \bar T$ with $\bar T = \frac1G\sum_h T_h$, and $\sum_g R_g^2/V$ is the sum of squared deviations of $G$ independent standard normal variables from their mean, which has the chi-squared distribution with $G - 1$ degrees of freedom (Sampling Distributions from the Normal). So $\hat V_{\text{CR}}/\operatorname{Var}(\bar s) = \sum_g R_g^2/(GV)$ has the distribution of $\chi^2_{G-1}/G$ however many questions each cluster holds, and it does not converge to 1. Cameron and Miller (2015, Section VI) treat the case of few clusters. For a test of one coefficient they report that it is common to compare the statistic with the t distribution with $G - 1$ degrees of freedom, a practice they trace to the survey sampling literature (Section IV.B). They also report that there is no specific point at which clusters become too few: "Current consensus appears to be that $G = 50$ is enough for state-year panel data." The experiment below measures what happens with fewer.

MMLU's subjects as clusters

For Llama 3 (70B), with each of the 57 subjects as a cluster:

  • the independent-question standard error is 0.35 points;
  • the cluster-robust standard error is 2.78 points, and 2.80 after the factor $57/56$;
  • Miller's Equation 4 gives 2.78 points, as the identity above requires.

The ratio is 8.04, a design effect of 64.6. Theorem 2 then implies an intraclass correlation of $(64.6 - 1)/(468.9 - 1) = 0.136$, if the exchangeable model is taken as a summary. The 95% interval becomes 73.2% to 84.1% with the normal quantile, or 73.1% to 84.3% with CR1 and the t quantile with 56 degrees of freedom, 2.003. The source of the inflation is visible in the subject accuracies, which range from 43.0% to 97.9% with a standard deviation of 14.3 points across subjects. Which subjects a benchmark happens to contain matters far more than which questions within them.

Six of the 28 models:

Model Accuracy (%) Stratified SE Independent SE Cluster-robust SE Implied $\rho$
Claude 3 Opus 85.04 0.29 0.30 1.92 0.084
GPT-4 (0613) 84.01 0.30 0.31 1.88 0.077
Llama 3 (70B) 78.68 0.33 0.35 2.78 0.136
Mixtral (8x22B) 77.35 0.34 0.35 2.59 0.113
Llama 2 (7B) 45.35 0.41 0.42 2.44 0.070
OLMo (7B) 28.74 0.38 0.38 0.89 0.009

(Standard errors in percentage points. The stratified column is explained in the next section. The implied $\rho$ is $(\text{design effect} - 1)/(m^* - 1)$.)

Across all 28 models the ratio of the cluster-robust to the independent standard error ranges from 2.3 to 9.0, with a median of 7.4, and the implied correlation from 0.009 to 0.170. The smallest belongs to OLMo (7B), at 28.74% close to the 25% of guessing: its 57 subject accuracies have a standard deviation of 5.6 points (they range only from 22% to 45%), little more than the 3.5 points that sampling alone would give in subjects of these sizes at its accuracy, so the subjects hardly differ for it. The largest, 9.0 with a correlation of 0.170, belongs to PaLM 2 (text-bison). For comparison, Miller's Table 4 reports ratios of 3.05 for DROP, 1.10 for RACE-H and 1.88 for MGSM, computed on Anthropic models. Those clusters are passages and translated problems; his illustrative Table 3, whose scores are fictional, lists them with about 3 to 16 questions per cluster on average, far fewer than MMLU's subjects, and by Theorem 2 the design effect grows with cluster size.

Clusters or strata?

The figure of 2.78 points rests on an assumption that deserves a second look: that the 57 subjects are a random sample from a population of subjects, as the passages of DROP are a sample of passages. But every version of MMLU has the same 57 subjects. A rerun of the benchmark with fresh questions would draw new questions within the same subjects, in roughly the same proportions. Under that design the subjects are strata, not clusters, and the standard error is no larger than the independent one when the questions are allocated to subjects in proportion to their shares (Theorem 5 below), not eight times larger.

Abadie, Athey, Imbens and Wooldridge (2023) put the distinction in terms of sampling. Clustering, they write, "can be needed to address sampling issues if sampling follows a two stage process where in the first stage, a subset of clusters are sampled from a population of clusters, and in the second stage, units are sampled from the sampled clusters." Then "clustered standard errors account for the existence of clusters in the population that we do not see in the sample" (abstract of the arXiv version, 1710.02926). If every cluster of the population is in the sample, there are no unseen clusters to account for. Abadie et al. analyze regression and treatment-effect estimators; for a mean, their sampling argument reduces to classical stratified sampling, which Theorem 5 states.

Theorem 5 (stratified sampling). Suppose the population is divided into $G$ fixed strata, stratum $g$ having mean $\mu_g$ and within-stratum variance $\sigma_g^2$, and the benchmark draws $n_g$ questions independently from stratum $g$, with $n_g = w_gn$ for fixed shares $w_g$ that match the strata's shares of the population. Then $\bar s$ estimates $\mu = \sum_g w_g\mu_g$ without bias, and

$$\operatorname{Var}(\bar s) = \sum_g w_g^2\frac{\sigma_g^2}{n_g} = \frac{1}{n}\sum_g w_g\sigma_g^2 \le \frac{\sigma^2}{n},$$

where $\sigma^2 = \sum_g w_g\sigma_g^2 + \sum_g w_g(\mu_g - \mu)^2$ is the variance of a question drawn from the whole population. The difference from the independent-question variance is $\frac1n\sum_g w_g(\mu_g - \mu)^2$, the spread of the stratum means.

Proof. $\bar s = \sum_g w_g\bar s_g$, where $\bar s_g$ is the mean of the $n_g$ questions from stratum $g$. These means are independent, with mean $\mu_g$ and variance $\sigma_g^2/n_g$ by Theorem 1 of Standard Errors for Benchmark Scores. So $E[\bar s] = \sum_g w_g\mu_g$ and $\operatorname{Var}(\bar s) = \sum_g w_g^2\sigma_g^2/n_g$. Substituting $n_g = w_gn$ gives $\frac1n\sum_g w_g\sigma_g^2$. A question drawn from the whole population lands in stratum $g$ with probability $w_g$. The law of total variance (Conditional Expectation and the Laws of Total Expectation and Variance) splits its variance into the mean of the within-stratum variances and the variance of the stratum means, which is the stated $\sigma^2$. The difference follows. $\blacksquare$

So the same data support three standard errors for Llama 3 (70B), and each answers a different question:

  • 0.33 points (stratified): how much the score would vary over new questions drawn within these 57 subjects, in MMLU's proportions. The estimate uses each subject's sample variance (divisor $n_g - 1$) for $\sigma_g^2$. It still assumes independent questions within a subject; if a subject's questions come in groups, such as several questions from one source exam or sharing one case description, 0.33 points is too low.
  • 0.35 points (independent): how much it would vary if the questions were drawn without regard to subject, so that the subject mix could change by chance.
  • 2.78 points (cluster-robust): how much it would vary over benchmarks built from a fresh sample of 57 subjects from some larger population of academic subjects.

A claim about "MMLU" as defined is a claim about these subjects, and the stratified figure fits it. A claim about general academic knowledge, of which MMLU's subjects are one selection, needs the clustered figure, and even that assumes the 57 were selected as if at random, which they were not. For reading-comprehension benchmarks the choice is easier: a new benchmark would have new passages, so the passages are clusters. Abadie et al. (2023) also propose estimators for designs between the two extremes, which this article does not cover.

The same question for averages over subjects. HELM's per-question files also allow an average that gives each subject equal weight, the macro-average: 79.34% for Llama 3 (70B), against the 78.68% of the question-weighted average. Its standard error is 0.40 points with subjects as strata (the square root of $\sum_g \hat\sigma_g^2/n_g$, divided by 57) and 1.90 points with subjects as a sample (the standard deviation of the 57 subject accuracies divided by $\sqrt{57}$). The weighting changes the target, and the design changes the error.

The same distinction governs this site's own search evaluation. Its known-item queries were drawn by stratum, with every page of the site in one stratum by course or collection and targets allocated in proportion to stratum size by largest remainder, with at least one per stratum (tools/search/eval/README.md), so the allocation is only approximately proportional. Under that design the strata are fixed, and a stratified standard error, not a clustered one, describes the variation over draws; since the targets are drawn without replacement from a finite set of pages, the finite-pool correction of Standard Errors for Benchmark Scores (Theorem 2) applies within each stratum.

How well the cluster-robust interval covers

Theorem 4 is a statement about many clusters. With MMLU's 57 subjects we can see how the interval behaves with few.

The experiment. The script takes the 57 subjects, with Llama 3 (70B)'s results, as a population of clusters. A benchmark is built by drawing $G$ subjects with replacement and keeping every question of each, so its size varies with the subjects drawn. The target is the population accuracy, 78.68%, the limit of the ratio $\sum_g S_g/\sum_g n_g$ under this sampling. Three nominal 95% intervals are compared: the independent-question interval, the cluster-robust interval with the normal quantile, and CR1 with the $t_{G-1}$ quantile. Each cell is the median over 30 seeds (31,000,000 + 100 $G$ to 31,000,029 + 100 $G$) of the share of 2,000 benchmarks whose interval covers the target, with the interquartile range in parentheses.

Subjects drawn Independent Cluster-robust, normal CR1, $t_{G-1}$
5 0.202 (0.197 to 0.209) 0.667 (0.662 to 0.673) 0.801 (0.796 to 0.807)
10 0.178 (0.173 to 0.184) 0.744 (0.740 to 0.749) 0.807 (0.803 to 0.813)
20 0.184 (0.177 to 0.188) 0.803 (0.794 to 0.806) 0.830 (0.828 to 0.836)
40 0.190 (0.185 to 0.194) 0.860 (0.852 to 0.863) 0.871 (0.865 to 0.875)

The independent interval, nominally 95%, covered the target about a fifth of the time at every size: it is far too narrow once subjects vary. The cluster-robust intervals did much better, but in this experiment they fell short of 95% even with 40 subjects. The unequal sizes are the reason. The two largest subjects, professional law (1,534 questions, 10.9% of all) and moral scenarios (895 questions), are among the model's weakest, at 61.5% and 59.8%. Together they hold 17.3% of the questions. Where do the misses come from? The script splits the CR1 intervals by how many of the two subjects a draw contains. The misses were mostly on the high side: with 5 subjects, 16.6% of the intervals lay entirely above the target and 3.3% entirely below; with 40 subjects, 10.7% and 2.3%. They came mostly from the draws that contained neither subject. With 40 subjects, CR1 covered 0.98 of the draws that contained exactly one of the two and 0.56 of the draws that contained neither (24% of all draws); with 5 subjects, 84% of the draws contained neither, and those covered 0.80. A benchmark without the two subjects overstates the accuracy, and the residual totals of the smaller, stronger subjects it does contain do not reveal what is missing: a few large, low cluster totals carry much of the variance, and a small sample of clusters usually holds none of them. That last sentence is an interpretation of these measurements, not a result proved here.

To separate the two effects, the script repeats the experiment on a balanced population, keeping only the first 100 questions of each subject in HELM's order (target 79.37%), with seeds 32,000,000 + 100 $G$ to 32,000,029 + 100 $G$:

Subjects drawn Independent Cluster-robust, normal CR1, $t_{G-1}$
5 0.400 (0.392 to 0.405) 0.803 (0.798 to 0.808) 0.884 (0.880 to 0.887)
10 0.408 (0.398 to 0.412) 0.886 (0.882 to 0.892) 0.926 (0.923 to 0.931)
20 0.413 (0.408 to 0.420) 0.921 (0.916 to 0.924) 0.940 (0.936 to 0.943)
40 0.415 (0.405 to 0.420) 0.935 (0.932 to 0.939) 0.945 (0.942 to 0.949)

With equal clusters, CR1 with the t quantile came within about a point of 95% from 20 subjects on. The factor $G/(G-1)$, which Theorem 3 shows to be exact for the expectation in this case, made a visible difference with 5 and 10. These are observations of one model's results on one benchmark. They agree with the warning of Cameron and Miller (2015, Section VI) that few clusters lead to over-rejection, and they add that strongly unequal cluster sizes made it worse here. With few clusters, Cameron and Miller (2015, Section VI.C) discuss the pairs cluster bootstrap and recommend the wild cluster bootstrap in several of the cases they consider; Resampling Dependent and Clustered Data proves how the cluster bootstrap relates to the estimator of this article.

When two models are compared on the same clustered questions, the cluster-robust estimator applies to the per-question differences. Paired Comparisons of Models shows that pairing removes most of the subject effect, because both models find the same subjects hard.

What is proved here, what is cited, what the experiment shows

Proved here. The variance of a mean of clustered scores (Theorem 1); the design effect for unequal clusters under the exchangeable model, with $m^* = \bar m + V_m/\bar m$ (Theorem 2); the identity between the cluster-robust estimator and Miller's Equation 4; the exact expectation of the estimator and the unbiasedness of the factor $G/(G-1)$ for equal clusters (Theorem 3); its consistency and the normal limit of the studentized mean as the number of clusters grows, for bounded scores and cluster sizes (Theorem 4); the variance under stratified sampling and its relation to the independent-question variance (Theorem 5).

Cited. Lyapunov's central limit theorem (Billingsley, 1995, through the linked page), Chebyshev's and Markov's inequalities, Slutsky's theorem and the law of total variance from the linked articles; Kish's definition of the design effect and Moulton's unequal-size form, through Cameron and Miller (2015); their account of the estimator's history (White, 1984; Liang and Zeger, 1986), of finite-cluster corrections and of the practice of using $t_{G-1}$; the sampling account of when to cluster, from Abadie et al. (2023); the ratios of Miller's Table 4.

Computed. Every standard error and correlation from the HELM per-question files. The coverage tables are a seeded experiment that resamples the subjects of one model on one benchmark, and their numbers describe that experiment.

Worked problems

These problems were written for this article; they use small exact cases, the HELM results and the figures of Miller (2024).

Problem 1 (a small exact case). A benchmark has four clusters of two questions. A model's scores are $(1, 1)$, $(1, 0)$, $(0, 0)$ and $(1, 1)$. Compute the independent standard error, the cluster-robust one and CR1.

The mean is $\bar s = 5/8$. Five scores are 1, with residual $3/8$, and three are 0, with residual $-5/8$, so $\sum e^2 = 5 \cdot \tfrac{9}{64} + 3 \cdot \tfrac{25}{64} = \tfrac{120}{64}$. The independent variance is $\tfrac{120/64}{7 \cdot 8} = \tfrac{15}{448}$, a standard error of 0.183. The residual totals are $2 - \tfrac{10}{8} = \tfrac34$, $1 - \tfrac58 \cdot 2 = -\tfrac14$, $-\tfrac54$ and $\tfrac34$. Their squares sum to $\tfrac{9 + 1 + 25 + 9}{16} = \tfrac{11}{4}$, so $\hat V_{\text{CR}} = \tfrac{11/4}{64} = \tfrac{11}{256}$, a standard error of 0.207. CR1 multiplies by $\tfrac43$: $\tfrac{11}{192}$, a standard error of 0.239. Pairs that agree, $(1,1)$ and $(0,0)$, carry the positive within-cluster covariance that raises the clustered figures.

Problem 2 (the effective number of questions). With MMLU's subject sizes, what intraclass correlation would make the 14,042 questions worth only as much as 2,472 independent ones?

The effective number of questions is $n$ divided by the design effect. With $m^* = 468.9$ (Theorem 2), a correlation of $\rho = 0.01$ gives a design effect of $1 + 467.9 \times 0.01 = 5.68$, and $14{,}042/5.68 = 2{,}472$. A within-subject correlation of 0.01 is small enough to be invisible in any pair of questions, yet it would cost more than four fifths of the benchmark's information about a population of subjects. The models' implied correlations, 0.009 to 0.170, are mostly far above it.

Problem 3 (which standard error). Choose the standard error for each claim: (a) a model's accuracy on a reading-comprehension benchmark of 600 passages with 5 questions each, as a measure of its reading comprehension; (b) "Llama 3 (70B) scores 78.68% on MMLU"; (c) "Llama 3 (70B) answers about 79% of academic exam questions correctly."

(a) Passages: a new benchmark would draw new passages, so the passages are a sample of clusters, and the cluster-robust standard error with $G = 600$ applies. With 600 clusters, Theorem 4's large-$G$ approximation is reasonable. (b) If the claim is about the defined benchmark, a rerun would draw new questions within the same subjects, so the stratified standard error, 0.33 points, fits; if it is about these exact questions, the standard error is zero. (c) The claim reaches beyond MMLU's subjects, so the subjects play the role of a sample from a larger population, and the cluster-robust figure, 2.78 points, is the relevant order of magnitude. Even that figure assumes the subjects were drawn at random; they were chosen, and no standard error accounts for how the choice was made.

Problem 4 (Miller's DROP figures). Miller's illustrative Table 3, whose scores are fictional ("fictional models and numbers"), lists DROP with 9,622 questions in 588 clusters, and his Table 4 gives a ratio of 3.05 between the clustered and naive standard errors, computed on real models. If Table 4's ratio was computed on a set of that size, what intraclass correlation does it allow under the exchangeable model?

At most 0.54. The design effect is the squared ratio, $3.05^2 = 9.30$. The mean cluster size is $9{,}622/588 = 16.4$. If the clusters were equal, $\rho = (9.30 - 1)/(16.4 - 1) = 0.54$. Unequal sizes make $m^*$ larger than 16.4 (Theorem 2), so $\rho$ is at most 0.54 under this model. That bound allows a within-passage correlation that could be large: if it were near 0.54, a passage's questions would behave more like one question than like 16 separate ones. The table does not give $m^*$, so the value itself cannot be recovered.

Problem 5 (when the correction is exact). Show that with $G = 10$ equal clusters the cluster-robust variance is on average 10% too small, and find the correction.

Theorem 3 with equal sizes gives $E[\hat V_{\text{CR}}] = (1 - 1/G)\operatorname{Var}(\bar s) = 0.9\operatorname{Var}(\bar s)$. Multiplying by $G/(G-1) = 10/9$ makes the expectation exactly $\operatorname{Var}(\bar s)$. The standard error, a square root, is still biased slightly downward after the correction, because the square root is concave (Jensen's inequality in Probability Inequalities). The balanced experiment above shows the correction's effect with 10 subjects: coverage rose from 0.886 to 0.926 when CR1 and the t quantile replaced the uncorrected normal interval.

Summary

Question Answer
Variance of a clustered mean $\frac{1}{n^2}\sum_g \operatorname{Var}(S_g)$ (Theorem 1)
Design effect, unequal clusters $1 + (m^* - 1)\rho$ with $m^* = \sum_g n_g^2/n$ (Theorem 2)
Estimator $\hat V_{\text{CR}} = \frac{1}{n^2}\sum_g R_g^2$, equal to Miller's Equation 4 up to $\frac1n\operatorname{SE}_{\text{C.L.T.}}^2$
Bias factor $1 - 1/G$ with equal clusters; CR1 multiplies by $G/(G-1)$ (Theorem 3)
Validity consistent and asymptotically normal as $G \to \infty$ (Theorem 4); use $t_{G-1}$ with few clusters
Llama 3 (70B) on MMLU 0.33 (strata), 0.35 (independent), 2.78 (clusters) points
Clusters or strata cluster when the population has groups the benchmark does not contain; stratify when it contains them all (Theorem 5)

What we have learned

Grouped questions add the covariances within groups to the variance of a benchmark score, and large groups multiply even small correlations. The cluster-robust estimator measures that variance from the residual totals of the groups without any model of the correlation, and it is consistent as the number of groups grows. For MMLU it multiplies the standard error by up to nine. But it answers a question about unseen groups, and whether that is the right question depends on the claim: for passages it usually is, for a fixed list of subjects usually not, and then, with proportional allocation, the grouping makes the standard error no larger than the independent one.

Comparing two models on the same questions removes much of what makes questions differ, including the subject effect. Paired Comparisons of Models works out how much, and applies the cluster-robust estimator to the paired differences.

References