Six language models scored within 2.6 points of each other on the 14,042 questions of the MMLU benchmark in Stanford's HELM evaluation (HELM's MMLU releases v1.0.0 and v1.1.0, whose raw results are public; Stanford Center for Research on Foundation Models, HELM documentation), from 78.68% for Llama 3 (70B) to 76.16% for Yi (34B). Suppose an evaluation draws 200 of those questions at random, scores the six models on them, and announces the winner with its score. Over 30,000 such evaluations, the winner's announced score exceeded its accuracy on all 14,042 questions by 2.24 points on average, and the usual 95% interval around it covered that accuracy 88% of the time. Nothing in the arithmetic was wrong. The number was reported because it was the largest, and being the largest is evidence of luck. What does choosing a result after seeing the data do to its estimate, its p-value and its interval, and how can each be corrected?
Selective inference is inference about a quantity that the data themselves selected for attention: the best model, the significant effect, the subgroup that looked interesting, the analysis that worked. Its central idea, in Fithian, Sun and Taylor's (2014) phrase, is that "the answer must be valid, given that the question was asked".
First we prove that a selected estimate is biased upward, and measure the bias on the HELM data. Then we turn to p-values: the file-drawer effect, and the garden of forking paths, in which an analysis that looks single is one of many that could have been run. Next we prove the result that makes exact correction possible, that a normal estimate conditioned on its own selection is a truncated normal, and build from it selective p-values, intervals and estimates. Finally we compare three valid repairs on the HELM winner and say where the same ideas reach regression. The article assumes Multiple Comparisons and the Bonferroni Correction for the union bound and Standard Errors for Benchmark Scores for the HELM data.
The winner's curse
Back to the six models. Each evaluation of 200 questions gives six accuracies $X_1, \dots, X_6$, each an unbiased estimate of the model's accuracy on the whole benchmark, $\theta_1, \dots, \theta_6$: a question drawn at random is answered correctly with probability $\theta_i$. The evaluation reports $X_S$, where $S$ is the index of the largest $X_i$. Here is the idea of the theorem in one sentence: the maximum of several estimates is at least each of them, so its average is at least each of their averages, while the target it is compared with, $\theta_S$, is at most the largest target.
Theorem 1 (the winner's curse). Let $X_1, \dots, X_K$ have finite means $E[X_i] = \theta_i$, and let $S$ be an index with $X_S = \max_i X_i$, chosen in any way among ties. Then
$$E[X_S] \ \ge\ \max_i \theta_i \ \ge\ E[\theta_S], \qquad \text{so} \qquad E[X_S - \theta_S] \ge 0.$$
Proof. For each $j$, $X_S = \max_i X_i \ge X_j$, so $E[X_S] \ge E[X_j] = \theta_j$; taking the largest $j$ gives the first inequality. For every outcome, $\theta_S$ is one of the $\theta_i$ and so at most $\max_i\theta_i$; taking expectations gives the second. $\blacksquare$
The theorem needs no distributional assumption and no independence. It says the reported score is too high on average, and also that it is too high as an estimate of the best model's accuracy, whichever model that is. Efron and Hastie (2016, Section 20.3) describe the same effect under the name of selection bias, with the winner's curse as its "more picturesque name".
How large is it? Here is the plan: when the estimates have equal means, the bias is the expected maximum of the errors, and for normal errors that has a closed form for two estimates and a simple bound for any number.
Theorem 2 (the size of the curse). Let $X_i = \theta + \sigma\big(\sqrt{\rho}\,W + \sqrt{1 - \rho}\,Z_i\big)$ for $i = 1, \dots, K$, with $W, Z_1, \dots, Z_K$ independent standard normal variables and $0 \le \rho < 1$, so that the $X_i$ have a common mean $\theta$, standard deviation $\sigma$ and correlation $\rho$. Then
$$E[X_S - \theta] = \sigma\sqrt{1 - \rho}\;E\big[\max_i Z_i\big],$$
with $E[\max(Z_1, Z_2)] = 1/\sqrt{\pi}$ and, for every $K \ge 2$, $E[\max_i Z_i] \le \sqrt{2\ln K}$.
Proof. The term $\sqrt{\rho}\,W$ is shared by all the $X_i$, so the largest $X_i$ belongs to the largest $Z_i$, and $X_S - \theta = \sigma(\sqrt{\rho}\,W + \sqrt{1-\rho}\,\max_i Z_i)$. Since $E[W] = 0$, the first statement follows.
For two variables, $\max(a, b) = \tfrac{1}{2}(a + b) + \tfrac{1}{2}\lvert a - b\rvert$. Here $Z_1 - Z_2$ is normal with mean 0 and variance 2, and for a normal variable $D$ with standard deviation $s$, $E\lvert D\rvert = 2\int_0^\infty x\,e^{-x^2/(2s^2)}\,dx/(s\sqrt{2\pi}) = s\sqrt{2/\pi}$. So $E[\max(Z_1, Z_2)] = \tfrac12\sqrt{2}\sqrt{2/\pi} = 1/\sqrt{\pi}$.
For the bound, fix $t > 0$. The function $e^{tx}$ is convex, so by Jensen's inequality (Probability Inequalities) and then because the largest of $K$ positive numbers is at most their sum,
$$e^{t\,E[\max_i Z_i]} \le E\big[e^{t\max_i Z_i}\big] = E\big[\max_i e^{tZ_i}\big] \le \sum_{i=1}^K E\big[e^{tZ_i}\big] = K e^{t^2/2},$$
using the standard normal moment generating function $E[e^{tZ}] = e^{t^2/2}$ (Moment Generating Functions and Slutsky's Theorem). Taking logarithms, $E[\max_i Z_i] \le (\ln K)/t + t/2$ for every $t > 0$, and $t = \sqrt{2\ln K}$ gives $\sqrt{2\ln K}$. $\blacksquare$
The script computes the expected maximum by numerical integration of $x\,K\phi(x)\Phi(x)^{K-1}$, the density of the maximum times $x$ (Random Variables and Distribution Functions derives the distribution of a maximum):
| $K$ | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|
| $E[\max_i Z_i]$ | 0.564 | 0.846 | 1.029 | 1.163 | 1.267 |
| $\sqrt{2\ln K}$ | 1.177 | 1.482 | 1.665 | 1.794 | 1.893 |
The bound is loose for few estimates and tightens slowly: for $K = 1{,}000$ the expected maximum is 3.241 and the bound 3.717. Theorem 2 bounds the bias by $\sigma\sqrt{1 - \rho}\sqrt{2\ln K}$, so it can grow at most like the square root of the logarithm of $K$.
The six models. On the full benchmark the six accuracies are 78.68%, 77.99%, 77.42%, 77.35%, 76.21% and 76.16%. Two models' 0/1 scores on the same questions are correlated, between 0.50 and 0.56 for these pairs, mean 0.524, because some questions are hard for every model. On 200 questions one accuracy near the six models' mean of 77.3% has standard deviation $\sqrt{0.773 \times 0.227/200} = 2.96$ points. If the six accuracies were equal, Theorem 2 would put the winner's bias at $2.96 \times \sqrt{1 - 0.524} \times 1.267 = 2.59$ points. The accuracies are not equal, so the leader wins more often on merit, and the measured bias is smaller. The script drew 1,000 benchmarks of 200 questions for each of 30 seeds (questions drawn without replacement from the 14,042):
- the winner's score exceeded its full-benchmark accuracy by a median of 2.24 points over the seeds (interquartile range 2.16 to 2.28);
- the winner was the model that is best on the full benchmark, Llama 3 (70B), in 39.9% of the benchmarks (39.1% to 40.8%);
- the bias depended on who won: 1.60 points when Llama 3 (70B) won, rising to 3.76 points when Yi (34B), the sixth model, won. A weaker model wins only with a large lucky error.
The bias shrinks with the size of the benchmark, and faster than the standard error: from 100 to 2,000 questions the standard error falls by a factor of 4.81 ($\sqrt{20} = 4.5$, times a small finite-population factor, because the questions are drawn without replacement from 14,042) and the bias by a factor of about 11. With more questions the strongest model wins on merit more often: Llama 3 (70B) won 35.5% of the benchmarks of 100 questions and 76.3% of those of 2,000 (medians over seeds). The run used 200 benchmarks per seed, seeds 1,001 to 1,030:
| Questions | 100 | 200 | 500 | 1,000 | 2,000 |
|---|---|---|---|---|---|
| Winner's bias (points) | 3.20 | 2.16 | 1.20 | 0.62 | 0.30 |
Example (choosing the best prompt). The same curse applies to choosing among prompts. In the older HELM Classic run (Liang et al., 2023), eight models answered 514 MMLU questions under three prompts that show the same five worked examples in different orders (the data are described in Standard Errors for Benchmark Scores). On the full 514 questions, a model's best and worst orderings differ by 1.90 points on average. The script split the questions in half at random (30 seeds), chose each model's best ordering on the first half, and scored it on the second. The chosen ordering scored a median of 1.00 points higher on the half used to choose it than on the other half (interquartile range 0.17 to 1.99), averaged over the eight models; a fixed ordering showed $-0.34$ points ($-1.34$ to $0.56$). Part of an apparent prompt improvement chosen this way is the luck of the questions it was chosen on. The same holds for any score used both to choose and to report, as Training, Validation and Test Sets shows for validation error.
Low power makes the curse worse, because only a large error lets a small true effect reach significance. Power Analysis for Evaluations derives the exaggeration ratio and the Type S error rate of Gelman and Carlin (2014) for that case, and the section on the significance filter below gives the conditional interval that corrects it.
What selection does to a p-value
A p-value is valid when it is computed for a question fixed in advance. Fithian, Sun and Taylor (2014, Example 1) make the failure concrete with a stylized model of publication. Many groups each measure an effect $Y_i \sim N(\mu_i, 1)$, and only those with $\lvert Y_i\rvert > 1$ go on to test $\mu_i = 0$ at level 0.05 with the usual cutoff 1.96. Among true nulls that are tested, the share rejected is
$$P(\lvert Y\rvert > 1.96 \mid \lvert Y\rvert > 1) = \frac{2\Phi(-1.96)}{2\Phi(-1)} = \frac{0.05}{0.317} = 0.158,$$
three times the nominal 0.05, as the paper finds (0.16). The test is still valid unconditionally, since most nulls are never tested; it is invalid among the tests actually performed. The paper's remedy is to raise the cutoff to the $c$ with $P(\lvert Y\rvert > c \mid \lvert Y\rvert > 1) = 0.05$, which is $c = 2.41$ (Worked problem 3). It calls the target the selective type I error, the error rate of a test given that it was performed.
The same arithmetic covers testing several hypotheses and reporting the best. If $K$ valid p-values are computed and the smallest is reported, the union bound gives $P(\min_i p_i \le \alpha) \le K\alpha$ under the null, with equality only when no two can be small together; Multiple Comparisons and the Bonferroni Correction proves it and measures the rate for pairwise comparisons.
The garden of forking paths
Gelman and Loken (2013) argue that the problem arises even when only one analysis is run. They distinguish four ways of testing:
- a single test chosen in advance;
- a test chosen in advance from a set of possible ones, the choice preregistered;
- a single test whose details (which outcome, which subgroup, which exclusions) are chosen after seeing the data, "in an environment where a different test would have been performed given different data";
- "fishing": running many tests and reporting the best.
Their claim is that researchers usually do the third, and that it carries the multiple-comparisons problem of the fourth. The frequentist error rate of a p-value refers to what would have been done in other samples, and in other samples the analyst would have taken a different, equally reasonable path. Simmons, Nelson and Simonsohn (2011) describe the same choices as "flexibility in data collection, analysis, and reporting", and report from computer simulations and a pair of experiments that it "dramatically increases actual false-positive rates". A slightly edited version of Gelman and Loken's manuscript appeared in American Scientist (Gelman and Loken, 2014).
How many paths does a modest analysis have? The script built a world with no effects at all from 5,107 NHANES 2017–2018 adults (nonpregnant, aged 18 and over, with blood pressure, body mass index, diabetes and smoking status recorded), using their mean systolic and diastolic pressures and their body mass index. A study draws 100 adults at random and assigns 50 of them at random to a "treatment" label that, by construction, does nothing. It then compares the two groups with Welch's $t$-test (Welch's t-Test) along one of these paths:
- outcome: systolic pressure, diastolic pressure or body mass index;
- subgroup: everyone, men, women, under 50, or 50 and over;
- outliers: keep all values, or drop values more than 3 standard deviations from the subgroup's mean;
- scale: the raw values or their logarithms.
That is $3 \times 5 \times 2 \times 2 = 60$ paths, each a defensible analysis. Every path's test is valid: across all paths and studies, 5.06% of the tests were significant at 0.05. The table gives the share of studies in which at least one path in a growing set was significant, as if the analyst had found the significant one (30 seeds of 1,000 studies, medians):
| Paths open to the analyst | Studies with some $p \le 0.05$ |
|---|---|
| 1 (systolic, everyone, all values, raw) | 0.051 |
| 3 (choice of outcome) | 0.140 |
| 15 (and of subgroup) | 0.414 |
| 30 (and of the outlier rule) | 0.477 |
| 60 (and of the scale) | 0.509 |
Half of the null studies offer a significant result somewhere. The paths are far from independent, so the rate is well below the $1 - 0.95^{60} = 0.954$ that 60 independent tests would give; logging a variable or dropping values beyond three standard deviations rarely changes a conclusion. Subgroups do most of the damage, because a subgroup is a different sample. A Bonferroni correction over all 60 paths, a threshold of 0.05/60, held the rate to 0.014 (interquartile range 0.012 to 0.015), at the price of power that a study of 100 cannot afford. The honest alternatives are to fix the path in advance, to report every path, or to use a method that accounts for the choice, which the rest of this article develops for the cases where the choice can be written down.
The truncated-normal pivot
The corrections that follow all rest on one fact. Here is the idea. If an estimate is normal and it was reported only because it fell in some interval, then its distribution given that it was reported is the normal distribution cut off outside that interval and rescaled. That distribution is known up to the unknown mean, so the usual construction of tests and intervals goes through with it in place of the normal.
Write $\Phi$ for the standard normal distribution function and $\bar\Phi = 1 - \Phi$. For an interval $[a, b]$ with $a < b$ (the end $b$ may be $+\infty$, the end $a$ may be $-\infty$), the normal distribution with mean $\mu$ and standard deviation $\sigma$ truncated to $[a, b]$ has distribution function, for $a \le x \le b$,
$$F_\mu(x) = \frac{\Phi\big((x - \mu)/\sigma\big) - \Phi\big((a - \mu)/\sigma\big)}{\Phi\big((b - \mu)/\sigma\big) - \Phi\big((a - \mu)/\sigma\big)},$$
the share of the normal's probability in $[a, b]$ that lies at or below $x$.
Theorem 3 (the pivot). Let $X \sim N(\mu, \sigma^2)$ with $\sigma$ known, and suppose $X$ is reported only when $X \in [a, b]$, with $P(X \in [a, b]) > 0$ and $a$, $b$ fixed. Given that it is reported, $X$ has the truncated distribution $F_\mu$, and $F_\mu(X)$ is uniform on $[0, 1]$.
Proof. For $a \le x \le b$, $P(X \le x \mid a \le X \le b) = P(a \le X \le x)/P(a \le X \le b)$, which is $F_\mu(x)$. This distribution function is continuous, so by the probability integral transform (Random Variables and Distribution Functions) the variable $F_\mu(X)$ is uniform under it. $\blacksquare$
A quantity whose distribution is known even though it involves the unknown $\mu$ is a pivot, and inverting a pivot gives an interval. That needs one more fact: the pivot moves in one direction as $\mu$ moves.
Lemma 1 (continuity and monotonicity). For fixed $a < x < b$, $F_\mu(x)$ is continuous and strictly decreasing in $\mu$, tends to 1 as $\mu \to -\infty$ and to 0 as $\mu \to +\infty$.
Proof. Take $\mu_1 < \mu_2$, with truncated densities $f_1, f_2$ on $[a, b]$, each proportional to $\phi((y - \mu_k)/\sigma)$. Their ratio is proportional to $\exp\big((\mu_2 - \mu_1)y/\sigma^2\big)$, a strictly increasing function $r(y)$. Write $A_k = \int_a^x f_k$ and $B_k = \int_x^b f_k$, so that $F_{\mu_k}(x) = A_k/(A_k + B_k)$. Since $f_2 = r f_1/C$ for a constant $C > 0$, and $r < r(x)$ on $[a, x)$ while $r > r(x)$ on $(x, b]$,
$$\frac{B_2}{A_2} = \frac{\int_x^b r f_1}{\int_a^x r f_1} > \frac{r(x)\int_x^b f_1}{r(x)\int_a^x f_1} = \frac{B_1}{A_1},$$
so $F_{\mu_2}(x) = 1/(1 + B_2/A_2) < 1/(1 + B_1/A_1) = F_{\mu_1}(x)$.
For the limits, first let $a$ be finite and standardize: with $d = (x - a)/\sigma > 0$ and $u = (a - \mu)/\sigma$, the normal tail satisfies $\bar\Phi(u + d) = \int_u^\infty \phi(s + d)\,ds = \int_u^\infty \phi(s)\,e^{-sd - d^2/2}\,ds \le e^{-ud}\,\bar\Phi(u)$ for $u \ge 0$. As $\mu \to -\infty$, $u \to \infty$, and $1 - F_\mu(x) = [\bar\Phi(u + d) - \bar\Phi((b - \mu)/\sigma)]/[\bar\Phi(u) - \bar\Phi((b - \mu)/\sigma)] \le \bar\Phi(u + d)/\bar\Phi(u) \le e^{-ud} \to 0$, using that subtracting the same positive amount from the numerator and the denominator of a ratio below 1 lowers it. If $a = -\infty$, $F_\mu(x) = \Phi((x - \mu)/\sigma)/\Phi((b - \mu)/\sigma) \ge \Phi((x - \mu)/\sigma)$, which tends to 1 as $\mu \to -\infty$. As $\mu \to +\infty$ with $b = \infty$, $F_\mu(x) = 1 - \bar\Phi((x - \mu)/\sigma)/\bar\Phi((a - \mu)/\sigma)$, and both tails tend to 1, so $F_\mu(x) \to 0$. With $b$ finite, let $v = (b - \mu)/\sigma$ and $\delta = (b - x)/\sigma > 0$. The tail inequality with $u = -v$ and $d = \delta$, together with $\Phi(y) = \bar\Phi(-y)$, gives $\Phi(v - \delta) = \bar\Phi(-v + \delta) \le \exp(v\delta)\,\bar\Phi(-v) = \exp(v\delta)\,\Phi(v)$. By the same fact about ratios, $F_\mu(x) \le \Phi(v - \delta)/\Phi(v) \le \exp(v\delta)$, which tends to 0 as $\mu \to +\infty$, since then $v \to -\infty$. Finally, $F_\mu(x)$ is continuous in $\mu$, as a ratio of continuous functions of $\mu$ whose denominator is positive. $\blacksquare$
The ratio argument is the statement that the truncated normal family has a monotone likelihood ratio in $\mu$, which Lee, Sun, Sun and Taylor (2016, Theorem 6.1) use for the same purpose.
Theorem 4 (selective inference for one selected estimate). Under the assumptions of Theorem 3, let $L$ and $U$ solve $F_L(X) = 1 - \alpha/2$ and $F_U(X) = \alpha/2$, and let $M$ solve $F_M(X) = 1/2$. Then, given that $X$ is reported:
- $P(L \le \mu \le U) = 1 - \alpha$;
- $P(M \le \mu) = P(M \ge \mu) = 1/2$, so $M$ is median-unbiased;
- for testing $\mu = \mu_0$ against $\mu > \mu_0$, the p-value $1 - F_{\mu_0}(X)$ is uniform when $\mu = \mu_0$.
Proof. $X$ lies in the open interval $(a, b)$ with probability 1, so Lemma 1 applies at $x = X$: $\mu \mapsto F_\mu(X)$ is continuous, strictly decreasing, and runs from 1 to 0, so by the intermediate value theorem each equation has a solution, unique by strict monotonicity, with $L < M < U$; and for any $\mu$, $\mu \ge L$ exactly when $F_\mu(X) \le F_L(X) = 1 - \alpha/2$, and $\mu \le U$ exactly when $F_\mu(X) \ge \alpha/2$. So $\{L \le \mu \le U\} = \{\alpha/2 \le F_\mu(X) \le 1 - \alpha/2\}$, which by Theorem 3 has conditional probability $1 - \alpha$. The same step with $1/2$ gives (2), and (3) is Theorem 3 again, since $1 - F_{\mu_0}(X)$ is uniform because $F_{\mu_0}(X)$ is. $\blacksquare$
A result reported because it was significant
Take the simplest filter: a study reports its standardized estimate $X \sim N(\mu, 1)$ only if it is significant and positive, $X > z_{0.975}$, where $z_{0.975} = 1.95996$ (written 1.96 below; the computations use the exact value). Then $[a, b] = [z_{0.975}, \infty)$. Suppose the reported value is $X = 2.2$.
- Naive inference. The one-sided p-value is 0.0139 (two-sided 0.028) and the 95% interval $2.2 \pm 1.96 = [0.24, 4.16]$.
- Selective p-value. For $\mu = 0$, $P(X \ge 2.2 \mid X > 1.96) = \bar\Phi(2.2)/\bar\Phi(1.96) = 0.0139/0.025 = 0.556$.
- Selective interval and estimate. Solving Theorem 4's equations gives $[-13.22, 3.74]$ and a median-unbiased estimate of $-0.49$.
Example (what a barely significant result says). A reader may think that a published $z = 2.2$ is decent evidence of a positive effect, and that selection only trims the estimate a little. Given that it was published because it crossed 1.96, it is almost no evidence at all. A null effect produces a value between 1.96 and 2.2 in 44% of the cases that pass the filter, and a large negative effect occasionally does too, which is why the interval reaches far below zero. A larger value is informative: for $X = 4$ the selective interval is $[1.65, 5.96]$ and the median-unbiased estimate 3.97, close to the naive estimate, because a value far above the cutoff was not much helped by selection (Worked problem 2).
How much the filter inflates the reported value depends on the truth. If $\mu = 1$, the filter passes 16.9% of studies, and the passed values average 2.49, two and a half times the truth; if $\mu = 0$, they average 2.34. The script checked the coverage of the two intervals among the studies that pass, with 2,000 passing studies for each of 30 seeds:
| True $\mu$ | Share of studies reported | Naive interval covers | Selective interval covers |
|---|---|---|---|
| 0 | 0.025 | 0.000 | 0.950 |
| 1 | 0.169 | 0.852 | 0.949 |
| 2 | 0.516 | 0.952 | 0.951 |
| 3 | 0.851 | 0.972 | 0.950 |
The naive interval never covers $\mu = 0$ among reported studies, since every reported value exceeds 1.96. When selection is nearly certain ($\mu = 3$) the naive interval is fine, even conservative. The selective interval covers 95% in every row, as Theorem 4 requires. Fithian, Sun and Taylor (2014, Example 2 and Figure 3) draw the same picture for the filter $Y > 3$: the selective interval is long and shifted down when $Y$ is near the cutoff, and close to the nominal one far above it.
The best of several correlated estimates
The winner of the six models is selected by an event that involves all six estimates, not one. Lee, Sun, Sun and Taylor (2016, Lemma 5.1) show that such events can be reduced to an interval for one estimate, with endpoints that are independent of it. Here is the idea: split the vector of estimates into the winner's estimate and a remainder uncorrelated with it; given the remainder, "the winner beats every other" is a set of linear inequalities in the winner's estimate alone, that is, an interval.
Lemma 2 (the polyhedral lemma, for the winner). Let $X = (X_1, \dots, X_K)$ be multivariate normal with mean $\theta$ and known covariance $\Sigma$, and fix an index $w$ with $\Sigma_{ww} > 0$. Let $c_j = \Sigma_{jw}/\Sigma_{ww}$ and $Z_j = X_j - c_jX_w$. Then $Z = (Z_1, \dots, Z_K)$ is independent of $X_w$, and
$$\{X_w \ge X_j \text{ for all } j\} = \{V^-(Z) \le X_w \le V^+(Z)\} \cap \{V^0(Z) \le 0\},$$
where $V^-(Z) = \max_{j:\,c_j < 1} \dfrac{Z_j}{1 - c_j}$, $V^+(Z) = \min_{j:\,c_j > 1} \dfrac{Z_j}{1 - c_j}$, and $V^0(Z) = \max_{j:\,c_j = 1} Z_j$ (a maximum over no indices is $-\infty$, a minimum $+\infty$).
Proof. $\operatorname{Cov}(Z_j, X_w) = \Sigma_{jw} - c_j\Sigma_{ww} = 0$. The pair $(Z, X_w)$ is a linear function of $X$, hence jointly normal, and jointly normal variables that are uncorrelated are independent (Lee et al., 2016, Section 5.1, use the same step). For the event, $X_j = Z_j + c_jX_w$, so $X_w \ge X_j$ is $(1 - c_j)X_w \ge Z_j$. When $c_j < 1$ this says $X_w \ge Z_j/(1 - c_j)$; when $c_j > 1$, dividing by a negative number reverses it, $X_w \le Z_j/(1 - c_j)$; when $c_j = 1$ it says $Z_j \le 0$. All the inequalities hold together exactly when $X_w$ is at least the largest lower bound and at most the smallest upper bound. ($j = w$ gives $c_w = 1$ and $Z_w = 0$, which is always satisfied.) $\blacksquare$
Given $Z = z$, the selection event is a fixed interval for $X_w$, and $X_w$ is still $N(\theta_w, \Sigma_{ww})$ because it is independent of $Z$. So Theorem 3 and Theorem 4 apply with $[a, b] = [V^-(z), V^+(z)]$. The resulting $L$, $U$ have conditional coverage $1 - \alpha$ given $Z = z$ and $w$ winning, for every $z$, and therefore given only that $w$ won (Lee et al., 2016, Theorem 5.2, by averaging over $z$). The interval is exact for each possible winner separately, not only on average over winners.
Applying it to the six models. On a benchmark of $n$ questions, the vector of accuracies is approximately normal with covariance $\Sigma/n$, where $\Sigma$ is the covariance matrix of one question's six 0/1 scores; the script estimates $\Sigma$ from the same 200 questions. One more adjustment turned out to matter. Accuracies on 200 questions are multiples of 1/200, so two models tie often, and a tie puts the winner's estimate exactly at the boundary $V^-$, where the truncated distribution function is 0 for every $\mu$. Without an adjustment, 14.0% of the benchmarks had the winner on the boundary, the selective interval covered only 81.9% of the time, and the median-unbiased estimate fell 15.2 points below the truth on average. The script therefore treats "$X_w \ge X_j$" for lattice scores as "$X_w - X_j \ge -\tfrac{1}{2n}$" for their normal approximation, a continuity correction of half a step. With $h = \tfrac{1}{2n}$, each lower bound moves down by $h/(1 - c_j)$, each upper bound moves up by $h/(c_j - 1)$, and a constraint with $c_j = 1$ becomes $Z_j \le h$, so each constraint $X_w - X_j \ge 0$ is relaxed by half a lattice step, and the truncation interval $[V^-, V^+]$ widens accordingly. (The script breaks ties by model order and applies the correction in the same direction for every $j$, which treats a tie as a win.) With it the coverage was 95.2% and the median bias $-0.17$ points. The interval is then exact only for the normal approximation, with $\Sigma$ estimated; the experiment measures how close that comes.
Three valid repairs
Three methods give a valid interval for the winner's accuracy. They guarantee different things.
- Simultaneous intervals. If $K$ intervals cover their targets all at once with probability at least $1 - \alpha$, then the winner's interval covers its target with probability at least $1 - \alpha$, since the event "all cover" is contained in "the winner's covers". Bonferroni's intervals at level $1 - \alpha/K$ (Multiple Comparisons and the Bonferroni Correction) do this for any dependence. The guarantee is averaged over which model wins; it says nothing about the coverage given that a particular model won.
- Sample splitting. Choose the winner on one half of the questions and compute an ordinary interval on the other half. If the halves are independent, the choice is a function of the first half only, so given the choice the second half is an unselected sample, and its interval has its nominal coverage given the choice. Fithian, Sun and Taylor (2014, Section 1.1) report that the idea dates back at least to a 1975 paper of Cox, and note its cost: half the data for selection and half for inference. They also show that tests based on splitting are, in most cases, inadmissible, beaten by tests that use the information left in the first half after conditioning (their "data carving").
- The selective interval of Theorem 4 and Lemma 2, valid given the winner, using all the data for both choosing and estimating.
The script compared them with the naive interval on the 30 seeds of 1,000 benchmarks of 200 questions (coverage of the winner's full-benchmark accuracy, medians over seeds; widths in points, median over all benchmarks):
| Interval | Coverage | Interquartile range | Width |
|---|---|---|---|
| Naive, $X_S \pm 1.96$ SE | 0.884 | 0.877 to 0.893 | 11.09 |
| Bonferroni over 6 | 0.966 | 0.963 to 0.970 | 14.92 |
| Split, 100 + 100 questions | 0.942 | 0.937 to 0.946 | 16.24 |
| Selective | 0.952 | 0.949 to 0.957 | 16.69 |
The differences show up when the coverage is broken down by the model that won (pooled over all 30,000 benchmarks):
| Winner | Times it won | Bias (points) | Naive | Bonferroni | Selective |
|---|---|---|---|---|---|
| Llama 3 (70B) | 11,957 | 1.60 | 0.917 | 0.976 | 0.954 |
| text-unicorn | 7,119 | 2.16 | 0.895 | 0.971 | 0.957 |
| Qwen1.5 (72B) | 4,412 | 2.63 | 0.868 | 0.957 | 0.952 |
| Mixtral (8x22B) | 3,554 | 2.78 | 0.856 | 0.962 | 0.947 |
| Claude 3 Sonnet | 1,633 | 3.57 | 0.802 | 0.928 | 0.941 |
| Yi (34B) | 1,325 | 3.76 | 0.777 | 0.915 | 0.946 |
Bonferroni's interval is narrower than the selective one and covers more often overall, but it over-covers when the strong model wins and under-covers when a weak one does. The selective interval spends its width where it is needed: when the race is close, the winner's estimate is near the boundary $V^-$ and the interval is long, as in the significance-filter example. In this design it has the largest median width of the four. Which repair to use depends on the guarantee wanted: an average over winners (Bonferroni, cheaper here), coverage given the winner (selective or split), or simplicity and an untouched second sample (split).
Many selected parameters. When several parameters are selected, as with every model that beats a baseline, conditional coverage for each implies a guarantee on the share of selected intervals that miss. Benjamini and Yekutieli (2005) define the false coverage-statement rate (FCR) as the expected share of non-covering intervals among those constructed, with the share taken as 0 when none are (the convention as written by Lee et al., 2016, eq. 2.3). Lee et al. (2016, Lemma 2.1) prove that intervals with conditional coverage $1 - \alpha$ given each selected set have FCR at most $\alpha$. The proof is short: given the selected set $\hat{M}$, the expected number of misses is $\sum_{j \in \hat{M}} P(\text{miss}_j \mid \hat{M}) \le \alpha\lvert\hat{M}\rvert$, so the expected share of misses given $\hat{M}$ is at most $\alpha$, and averaging over $\hat{M}$ keeps it there. Benjamini and Yekutieli (2005) also give a simpler procedure, which their abstract says controls the FCR at level $\alpha$ for any selection rule, with separate results for positively and generally dependent estimates: build each selected parameter's ordinary interval at level $1 - R\alpha/m$, where $R$ parameters of $m$ are selected. (Only the abstract was consulted. One concrete set of conditions is given by Efron and Hastie (2016, Theorem 20.1): for independent estimates $z_i \sim N(\mu_i, 1)$, intervals of this form for the cases selected by the Benjamini–Hochberg procedure have FCR at most the chosen level.)
After selection in regression
A regression model chosen from many candidates is a selected question of the same kind. Model Selection explains why the usual standard errors of a chosen model do not account for the search that chose it, and Ridge and Lasso Regression notes that the lasso's choice of inputs is itself random. Two lines of work give valid inference:
- Conditioning. Lee, Sun, Sun and Taylor (2016, Theorem 4.3 and Corollary 4.4) show that the event "the lasso selects these inputs with these signs" is a set of linear inequalities in the response vector, so Lemma 2's argument, with a general direction in place of a coordinate, gives exact selective intervals for the coefficients of the chosen model. Fithian, Sun and Taylor (2014) develop the theory of such conditional tests in exponential families, including their optimality.
- Simultaneity. Berk, Brown, Buja, Zhang and Zhao (2013) widen the intervals enough to cover the coefficients of every submodel that could have been chosen, which makes them valid whatever selection rule was used. Their abstract calls this "simultaneity insurance" and notes that it is always less conservative than full Scheffé protection.
An empirical Bayes correction. When thousands of estimates are screened at once, as with genes, the other estimates show how large true effects tend to be, and the selected ones can be shrunk accordingly. Efron and Hastie (2016, Theorem 20.3) give Tweedie's formula: if $\mu$ has any prior distribution and $z \mid \mu \sim N(\mu, \sigma^2)$, with marginal density $f$, then $E[\mu \mid z] = z + \sigma^2\,\frac{d}{dz}\ln f(z)$. The proof is one line. Write the prior as a distribution $G$, so that $f(z) = \int \phi_\sigma(z - \mu)\,dG(\mu)$, and differentiate under the integral, which is allowed because $\lvert\phi_\sigma'\rvert$ is bounded and $G$ has total mass 1 (dominated convergence). With $\phi_\sigma'(u) = -(u/\sigma^2)\phi_\sigma(u)$ this gives $\sigma^2 f'(z) = \int (\mu - z)\,\phi_\sigma(z - \mu)\,dG(\mu) = (E[\mu \mid z] - z)f(z)$. Because $f$ can be estimated from the many observed $z$, the correction needs no prior. With a normal prior the formula reduces to linear shrinkage toward a common mean (Hierarchical Models and Shrinkage). In their prostate cancer example, the largest of 6,033 z-values, 5.29, is shrunk to 4.03. They note that Bayes estimators are immune to selection bias, which is the hope behind Tweedie's empirical Bayes estimates, and they test it with a small simulation. q-Values and the Share of True Nulls reads false discovery rates in the same empirical Bayes way.
What is proved, cited and observed
Proved here: Theorem 1 (the winner's curse, for any estimates); Theorem 2 (its size for equicorrelated normal estimates, $1/\sqrt{\pi}$ for two, the $\sqrt{2\ln K}$ bound); Theorem 3 (the truncated-normal pivot); Lemma 1 (continuity, monotonicity and limits); Theorem 4 (selective interval, median-unbiased estimate and p-value); Lemma 2 (the polyhedral lemma for the winner); the validity of simultaneous intervals and of sample splitting for the selected parameter; that conditional coverage implies FCR control; Tweedie's formula.
Taken from other pages: the union bound and Bonferroni's intervals; Jensen's inequality; the normal moment generating function; the probability integral transform and the distribution of a maximum; Welch's test.
Cited: the selective type I error and the file-drawer example, data carving and the inadmissibility of splitting (Fithian, Sun and Taylor, 2014); the general polyhedral lemma, the lasso selection event and conditional coverage for a union of events (Lee et al., 2016); simultaneous post-selection intervals (Berk et al., 2013); the FCR and its procedure (Benjamini and Yekutieli, 2005); the four kinds of testing (Gelman and Loken, 2013); the exaggeration ratio (Gelman and Carlin, 2014, through Power Analysis for Evaluations).
Observed in this article's experiments: the HELM winner's bias, coverage and widths, the prompt orderings, the forking paths on NHANES, and the coverage table of the significance filter. They are observations about these data and designs: six models of nearly equal accuracy, benchmarks of 200 questions, 100 adults per study, the 60 paths listed.
Worked problems
The problems below were posed for this article, on its two running examples and on the significance filter. Every number is recomputed by tools/experiments/statistics/selective-inference.py.
Problem 1 (two near-tied models). Two models have the same true accuracy, 77.3%, and their 0/1 scores on a question have correlation 0.524. Each is scored on the same 200 random questions, and the higher score is announced. How far above 77.3% is the announced score on average?
Solution. By about 1.15 points, using the normal approximation to the two accuracies. One accuracy on 200 questions has standard deviation $\sigma = \sqrt{0.773 \times 0.227/200} = 0.0296$, and the two accuracies have correlation $\rho = 0.524$, since the accuracy is an average of the 0/1 scores and averaging over independent questions keeps the correlation. Theorem 2 with $K = 2$ gives
$$E[X_S - \theta] = \sigma\sqrt{1 - \rho}\,\frac{1}{\sqrt{\pi}} = 0.0296 \times 0.690 \times 0.564 = 0.0115.$$
The correlation helps: if the two models' errors were independent, the bias would be $0.0296 \times 0.564 = 1.67$ points. Shared difficulty moves both scores together and does not change which is larger.
Problem 2 (a large significant estimate). A study reports $X = 4$, standardized with $\sigma = 1$, and would have reported nothing unless $X > 1.96$. Give the selective p-value for $\mu = 0$ and the selective 95% interval.
Solution. The p-value is 0.00127 and the interval $[1.65, 5.96]$. Given selection, $P(X \ge 4 \mid X > 1.96) = \bar\Phi(4)/\bar\Phi(1.96) = 0.0000317/0.025 = 0.00127$, larger than the naive one-sided 0.0000317 by the factor $1/0.025 = 40$ but still small. The interval solves $F_L(4) = 0.975$ and $F_U(4) = 0.025$ for the normal truncated to $[1.96, \infty)$; the script finds $L = 1.65$ and $U = 5.96$ by bisection, which Lemma 1 justifies. The naive interval is $[2.04, 5.96]$: the upper ends agree to two decimals, because large values of $\mu$ make selection nearly certain, and only the lower end moves.
Problem 3 (the file-drawer threshold). Only estimates with $\lvert Y\rvert > 1$ are tested. Find the critical value $c$ that makes the selective type I error 0.05 for a two-sided test of $\mu = 0$.
Solution. $c = 2.41$. Under $\mu = 0$, $P(\lvert Y\rvert > c \mid \lvert Y\rvert > 1) = 2\bar\Phi(c)/(2\bar\Phi(1)) = \bar\Phi(c)/0.1587$ for $c \ge 1$. Setting it to 0.05 gives $\bar\Phi(c) = 0.05 \times 0.1587 = 0.00793$, so $c = \Phi^{-1}(1 - 0.00793) = 2.41$, the value Fithian, Sun and Taylor (2014) report. A test at 1.96 has selective error 0.158.
Problem 4 (how many paths). If the 60 analysis paths of the NHANES study were independent tests, what share of null studies would offer a significant path, and what does the observed 0.509 imply?
Solution. $1 - 0.95^{60} = 0.954$. The observed share, 0.509, is about half of that, because the paths share data: the two scales and the two outlier rules give nearly the same test, and the subgroups overlap with "everyone". The expected number of significant paths per study is still $60 \times 0.05 = 3$ (the script measured 3.03), since expectation ignores dependence; dependence makes those significant paths come in clumps, concentrated in fewer studies. So dependence lowers the chance of some false positive but not the average count of them, the same distinction the union bound draws between the probability of a union and the sum of the probabilities.
What we have learned
Selecting a result by its value biases it upward, whatever the estimates' distribution, and for equicorrelated normal estimates with equal means the bias is at most $\sigma\sqrt{1 - \rho}\sqrt{2\ln K}$: it grows at most like the square root of the logarithm of the number of candidates and shrinks with their correlation. On six nearly tied HELM models and 200 questions, the announced winner's score was 2.24 points too high, and its usual interval covered 88% of the time. Selection inflates p-values' error rates in the same way, and the garden of forking paths shows that one analysis chosen from many plausible ones behaves like many: on NHANES with random labels, half of the null studies offered a significant path among 60. A normal estimate conditioned on its selection is a truncated normal, and inverting that pivot gives p-values, intervals and estimates valid given the selection; the polyhedral lemma extends it to the best of several correlated estimates. Simultaneous intervals and sample splitting are the other valid repairs, with weaker or costlier guarantees.
This article ends the unit Multiple Testing, which began with the error rates of many fixed tests in The False Discovery Rate and the Benjamini–Hochberg Procedure and Closed Testing. Choosing a model is itself a selection, and the honest estimate of the chosen model's error is the subject of Prediction Error and Model Choice.
References
- Benjamini, Y., and Yekutieli, D. (2005). False discovery rate–adjusted multiple confidence intervals for selected parameters. Journal of the American Statistical Association, 100(469), 71–81. https://doi.org/10.1198/016214504000001907 record
- Berk, R., Brown, L., Buja, A., Zhang, K., and Zhao, L. (2013). Valid post-selection inference. The Annals of Statistics, 41(2), 802–837. https://doi.org/10.1214/12-AOS1077 record
- Efron, B., and Hastie, T. (2016). Computer Age Statistical Inference: Algorithms, Evidence, and Data Science. Cambridge University Press. Chapter 20, "Inference After Model Selection" (Theorem 20.1, Section 20.3). record · 9 pages
- Fithian, W., Sun, D. L., and Taylor, J. (2014). Optimal inference after model selection. arXiv:1410.2597. record
- Gelman, A., and Carlin, J. (2014). Beyond power calculations: assessing Type S (sign) and Type M (magnitude) errors. Perspectives on Psychological Science, 9(6), 641–651. https://doi.org/10.1177/1745691614551642 record · 2 pages
- Gelman, A., and Loken, E. (2013). The garden of forking paths: why multiple comparisons can be a problem, even when there is no "fishing expedition" or "p-hacking" and the research hypothesis was posited ahead of time. Unpublished manuscript, Department of Statistics, Columbia University. http://www.stat.columbia.edu/~gelman/research/unpublished/forking.pdf record
- Gelman, A., and Loken, E. (2014). The statistical crisis in science. American Scientist, 102(6), 460. https://doi.org/10.1511/2014.111.460 record
- Lee, J. D., Sun, D. L., Sun, Y., and Taylor, J. E. (2016). Exact post-selection inference, with application to the lasso. The Annals of Statistics, 44(3), 907–927. https://doi.org/10.1214/15-AOS1371 record
- Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., et al. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research, August 2023. arXiv:2211.09110. record · 7 pages
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J), Blood Pressure (BPX_J), Body Measures (BMX_J), Diabetes (DIQ_J) and Smoking (SMQ_J) data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 95 pages
- Simmons, J. P., Nelson, L. D., and Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632 record
- Stanford Center for Research on Foundation Models. HELM documentation: Downloading raw results. https://github.com/stanford-crfm/helm/blob/main/docs/downloading_raw_results.md (raw results in the public bucket
crfm-helm-public; files used here pinned on 18 November 2026). record · 2 pages