A small open language model, Qwen2.5-0.5B, was given 4,000 multiple-choice questions from MMLU, each with options A to D, and asked for the answer letter. On average it gives the next token " E" (a space and the letter E, the same form as the valid answers " A" to " D"), which is not an option, a probability of 0.0002037: about one sampled answer in 4,909 would be that token. An evaluation that samples one answer to each of 1,000 of these questions sees it not even once about 81% of the time. A deployment that answers 10,000 such questions sees at least one with probability 0.870. What can an evaluation say about a behavior it never observes, or observes once or twice, and how can it say more?
A rare event here is a behavior whose rate per trial is small compared with the reciprocal of the number of trials an evaluation can afford, so that the evaluation expects to see few or none. The questions are what bound a clean or nearly clean record supports, what it implies for a deployment much larger than the evaluation, how many trials a demonstration needs, and what changes when the probability of the behavior can be computed for each prompt instead of counted. Model evaluation meets these questions whenever it tests for something that should almost never happen, from format failures like the letter E to the misuse and misalignment behaviors of Forecasting Rare Language Model Behaviors.
Here is the route. First we derive the exact bounds after zero events and after a few, which give the rule of three and its successors. Then we take a Bayesian view, which yields the probability of an event in a deployment of a given size, and turn the bounds into a plan for demonstrating that a rate is below a target. Next we ask what counts as a trial when prompts differ and are sampled more than once. Finally we use the model's exact per-question probabilities to estimate the rate without observing a single event, and to extrapolate the probability of the worst prompt in a large deployment from the upper tail of a small evaluation, testing both against the truth. The article builds on Confidence Intervals for a Proportion, which derives the exact Clopper–Pearson interval and the rule of three, and Crude Rates and Their Precision, which derives the exact limits for a Poisson count.
The data come from the unit's script mmlu_small_model.py, which ran Qwen2.5-0.5B (Yang et al., 2024; base model, revision 060db6499f32faf8b98477b0a26969ef7d8b9987) zero-shot on a random sample of 4,000 of the 14,042 MMLU test questions (Hendrycks et al., 2021), in the prompt format of the MMLU authors' code. One forward pass per question gives the exact probability of every possible next token after "Answer:", so for each question we know $p_i$, the probability that one sampled answer's first token is " E". Throughout, "the letter E" means that single token. It is not the probability that an answer begins with the letter E in any form: the model's vocabulary has 1,146 tokens whose text begins with E once leading spaces are removed (among them "E" without the space and whole words), and the script records only " E", so the rate of all answers beginning with E is at least as large, by an amount these data do not measure. Nothing has to be sampled to know the truth, which is what makes the data useful for testing estimators: the 4,000 questions serve as the population of prompts, its mean rate $\bar{p} = 0.0002037$ is known exactly, and simulated evaluations draw prompts from it and sample answers from the exact probabilities. The probability of any token other than the four valid letters averages 0.0041; E is one rare way among many of answering outside the options.
Zero events in n trials
Suppose $n$ independent trials, each producing the event with the same probability $p$, produce no events. Which values of $p$ remain plausible? Here is the idea in one sentence: a value of $p$ is implausible if it would make a clean record of $n$ trials unlikely, and the probability of a clean record is $(1 - p)^n$.
The one-sided upper confidence bound at level $1 - \alpha$ is the largest $p$ under which zero events still has probability at least $\alpha$:
$$(1 - p_U)^n = \alpha, \qquad p_U = 1 - \alpha^{1/n}.$$
Confidence Intervals for a Proportion derives this bound as the zero-count case of the Clopper–Pearson construction, proves that the construction covers the true $p$ with probability at least $1 - \alpha$, and derives the rule of three: since $\ln(1 - p) \approx -p$ for small $p$, the bound is about $-\ln(0.05)/n = 2.9957/n \approx 3/n$. For $n = 100$ the exact bound is 0.02951 against $3/n = 0.0300$; for $n = 1{,}000$ it is 0.00299 against 0.0030.
The approximation always errs on the safe side.
Confidence Intervals for a Proportion notes this (its Problem 3); here is the proof.
Theorem 1 (the rule of three is conservative). For every $n \ge 1$ and $0 < \alpha < 1$, $1 - \alpha^{1/n} < -\ln(\alpha)/n$.
Proof. Write $x = -\ln(\alpha)/n > 0$, so that $\alpha^{1/n} = e^{-x}$. The claim is $1 - e^{-x} < x$, that is, $e^{-x} > 1 - x$. The function $h(x) = e^{-x} - (1 - x)$ has $h(0) = 0$ and $h'(x) = 1 - e^{-x} > 0$ for $x > 0$, so $h(x) > 0$ for $x > 0$. $\square$
The Poisson version. When the trials are many and the rate small, the count is approximately Poisson (The Poisson Distribution), and for a Poisson count with mean $\mu$, $P(\text{no events}) = e^{-\mu}$. Setting $e^{-\mu_U} = \alpha$ gives $\mu_U = -\ln \alpha = 2.9957$ exactly. So "three" is exact for a Poisson mean: zero events observed gives a 95% upper bound of 3.00 expected events, and dividing by the exposure $n$ gives the rule of three. Theorem 1 says the Poisson bound per trial is always a little larger than the binomial one.
Example (the bound is one-sided). "No failures in 1,000 trials, so the failure rate is below 0.3% with 95% confidence" is correct. "The failure rate is between 0 and 0.3% with 95% confidence" is a two-sided statement and needs the bound with $\alpha/2$, which is about $3.69/n$, as Confidence Intervals for a Proportion (Problem 3) shows. With zero events the lower end is 0 either way, so the one-sided bound is the natural statement, provided the side was chosen before the data were seen.
A few events: the successors of three
An evaluation that observes one or two events has not failed to find the behavior; it has measured it with a handful of counts. The same construction applies. With $x$ events, the one-sided $1 - \alpha$ upper bound is the largest rate under which $x$ or fewer events still have probability at least $\alpha$.
- Binomial. $p_U$ solves $P_{p_U}(X \le x) = \alpha$. Confidence Intervals for a Proportion proves that $P_p(X \le x)$ falls strictly as $p$ rises and that the solution is the $1 - \alpha$ quantile of the $\text{Beta}(x + 1, n - x)$ distribution.
- Poisson. $\mu_U$ solves $P_{\mu_U}(D \le x) = \alpha$. Crude Rates and Their Precision proves $P(D \ge d \mid \mu) = P(G_d \le \mu)$, with $G_d$ a gamma variable of shape $d$ and scale 1, so $P(D \le x \mid \mu) = 1 - P(G_{x+1} \le \mu)$, and the bound is the $1 - \alpha$ quantile of the gamma distribution with shape $x + 1$.
The Poisson bounds are numbers of events, and dividing by the number of trials turns them into rates. At 95%:
| Events observed $x$ | 0 | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|---|
| Poisson bound $\mu_U$ | 3.00 | 4.74 | 6.30 | 7.75 | 9.15 | 10.51 |
| $1{,}000 \times$ binomial bound, $n = 1{,}000$ | 2.99 | 4.73 | 6.28 | 7.74 | 9.13 | 10.48 |
So the "rule of three" has successors: one event in $n$ trials bounds the rate by about $4.74/n$, two by $6.30/n$. The binomial bounds per thousand trials sit just below the Poisson ones at every count in the table, as Theorem 1 proves for zero events. The first event costs the most: going from zero to one event adds 1.75 expected events to the Poisson bound, while each further event adds between 1.36 and 1.55.
A Bayesian bound, and what to expect in deployment
A confidence bound is a statement about a procedure. A planner who must decide whether to deploy usually wants a probability for the rate itself, or for the event happening in use. Both need a prior distribution for $p$.
Theorem 2 (zero events under a uniform prior). If $p$ has the uniform prior on $[0, 1]$ and $n$ independent trials show no events, the posterior distribution of $p$ is $\text{Beta}(1, n + 1)$, with distribution function $1 - (1 - p)^{n+1}$. Its mean is $1/(n + 2)$ and its upper $1 - \alpha$ point is $1 - \alpha^{1/(n+1)}$. The probability that $N$ further independent trials also show no events is
$$\frac{n + 1}{n + N + 1}.$$
Proof. The likelihood of no events is $(1 - p)^n$ and the prior density is 1, so the posterior density is proportional to $(1 - p)^n$ on $[0, 1]$. Since $\int_0^1 (1 - p)^n\,dp = 1/(n + 1)$, the posterior density is $(n + 1)(1 - p)^n$, the $\text{Beta}(1, n + 1)$ density. Integrating from 0 to $t$ gives $1 - (1 - t)^{n+1}$, and setting this equal to $1 - \alpha$ gives the upper point. The mean is $\int_0^1 p(n + 1)(1 - p)^n\,dp$; substituting $u = 1 - p$ it is $(n + 1)\int_0^1 (1 - u)u^n\,du = (n + 1)\big[\tfrac{1}{n+1} - \tfrac{1}{n+2}\big] = 1/(n + 2)$. Given $p$, the $N$ further trials are clean with probability $(1 - p)^N$, so the predictive probability is $\int_0^1 (1 - p)^N (n + 1)(1 - p)^n\,dp = (n + 1)/(n + N + 1)$. $\square$
The posterior mean $1/(n + 2)$ is the classical rule of succession in the case of no events; for $n = 100$ it is 0.00980 and for $n = 1{,}000$ it is 0.00100. The Bayesian upper point is the frequentist bound with $n + 1$ in place of $n$: for $n = 100$ it is 0.02923 against 0.02951, and for $n = 1{,}000$ both round to 0.00299. With a uniform prior the two kinds of bound nearly coincide.
The predictive probability is the sobering part. After 1,000 clean trials, the probability that the next 1,000 are also clean is $1{,}001/2{,}001 = 0.5002$. For a deployment of 100,000 uses it is 0.0099, and for 1,000,000 uses 0.0010. A clean evaluation of 1,000 trials therefore says, under this prior, that an event in a deployment a thousand times larger is close to certain. That is not a paradox. The evaluation bounds the rate at about $3/1{,}000$, while a deployment of $N$ uses at any rate $p \ge 3/N$ produces at least one event with probability at least 0.95, since $1 - (1 - p)^N \ge 1 - e^{-Np} \ge 1 - e^{-3}$ (using $1 - p \le e^{-p}$, the inequality of Theorem 1's proof). An evaluation can only rule out rates down to about $3/n$, so ruling out a rate small enough for a deployment of size $N$ to stay clean needs $n$ comparable to $N$ or a different kind of evidence, which the last sections take up.
Planning a demonstration
How many clean trials does it take to show that a rate is below a target $r$? A zero-failure demonstration runs $n$ trials and passes if none fails. It shows $p < r$ at confidence $1 - \alpha$ when passing would have probability at most $\alpha$ if $p$ were $r$:
$$(1 - r)^n \le \alpha \quad\Longleftrightarrow\quad n \ge \frac{\ln \alpha}{\ln(1 - r)},$$
about $3/r$ at 95%. For $r = 0.001$ the plan needs 2,995 trials.
The plan's operating characteristic is its probability of passing as a function of the true rate, $(1 - p)^n$. It passes with probability 0.050 when $p = r$, by construction. When the true rate is half the target it passes with probability only 0.224, and even at a tenth of the target, 0.741. A system that is ten times better than required fails the demonstration one time in four, because a single event ends it.
Allowing one failure changes the trade. The plan "pass if at most one failure in $n$ trials" with $P(X \le 1) \le 0.05$ at $p = r$ needs 4,742 trials for $r = 0.001$, close to $4.74/r$ from the table. Its probability of passing is 0.050 at the target, 0.315 at half of it and 0.918 at a tenth. The longer plan protects a good system from a single unlucky event.
Many prompts, repeated samples: what counts as a trial
The bounds above assume independent trials with a common probability. In a model evaluation the trials are prompts, and prompts differ: one prompt may elicit a behavior half the time and another never. Two cases behave differently.
Theorem 3 (one sample per random prompt is a Bernoulli trial). Let prompts be drawn independently at random from a population, let prompt $i$ produce the event with probability $p_i$ when one output is sampled, and let $\bar{p} = E[p_i]$ be the population average. If one output is sampled from each prompt, the number of events among $n$ prompts is exactly binomial with $n$ trials and probability $\bar{p}$.
Proof. For one prompt, by the law of total probability, $P(\text{event}) = E[P(\text{event} \mid \text{prompt})] = E[p_i] = \bar{p}$. The pairs (prompt, output) are independent across $i$, so the $n$ indicators are independent Bernoulli variables with the same probability $\bar{p}$, and their sum is binomial. $\square$
So with one sample per randomly drawn prompt, every bound above applies to the average rate $\bar{p}$ over the prompt population, however unequal the prompts are. The rate it bounds is the rate a user would meet by drawing a prompt from the same population, which is what the evaluation was meant to measure.
Repeated samples are not new trials. Sampling $K$ outputs from each of $m$ prompts gives $mK$ outputs but not $mK$ independent Bernoulli($\bar{p}$) trials. Outputs of one prompt share its $p_i$, and the count is overdispersed, as counts from clusters are in The Binomial Distribution and Overdispersion in Count Data. Worse, a clean record is more likely than the binomial with $mK$ trials says. Given the prompts, the probability of no events is $\prod_i (1 - p_i)^K$, and the function $p \mapsto (1 - p)^K$ is convex, so by Jensen's inequality $E[(1 - p_i)^K] \ge (1 - \bar{p})^K$. Treating the $mK$ outputs as independent trials then overstates the evidence.
How badly can that go? Suppose 2% of prompts always produce the event and the rest never do, so $\bar{p} = 0.02$, and an evaluation samples $K = 10$ outputs from each of $m = 50$ prompts. With probability $0.98^{50} = 0.364$ none of the 50 prompts is a bad one, and the evaluation sees no events in 500 outputs. The bound computed as if the 500 outputs were independent is $1 - 0.05^{1/500} = 0.00597$, far below the true 0.02. Summed exactly over the possible numbers of bad prompts, that "95%" bound covers the true rate only 0.636 of the time. The bound computed from the 50 prompts, $1 - 0.05^{1/50} = 0.0582$, counts each prompt once, as Theorem 3 allows, and covers with probability 1.000 in this example. Without a model for how $p_i$ varies across prompts, the honest number of trials is the number of prompts.
That leaves the question of what to count on each prompt when $K$ outputs were sampled. Two choices give exact bounds.
- Use one output per prompt, for instance the first. Theorem 3 then applies exactly, and the bound is for $\bar{p}$.
- Count a prompt as an event if any of its $K$ outputs shows the behavior. The prompts' indicators are independent Bernoulli variables with probability $\theta_K = E[1 - (1 - p_i)^K]$, by the argument of Theorem 3, so the exact bounds apply to $\theta_K$ with $m$ trials. Since $(1 - p)^K \le 1 - p$ for $0 \le p \le 1$ and $K \ge 1$, $1 - (1 - p)^K \ge p$ and so $\theta_K \ge \bar{p}$: an upper bound for $\theta_K$ is also a valid, conservative upper bound for $\bar{p}$. It uses all the outputs to detect a prompt that sometimes produces the behavior.
The average number of events per prompt, divided by $K$, is unbiased for $\bar{p}$, but the total count is not binomial unless every prompt has the same rate, so the exact bounds do not apply to it. The "any of $K$" bound, for its part, bounds $\theta_K$, which for small $p_i$ is close to $K\bar{p}$ (since $1 - (1 - p)^K \approx Kp$), so it can be very conservative as a bound for $\bar{p}$.
Measuring probabilities instead of counting events
Counting events throws information away. Most prompts that do not produce the behavior still had some probability of producing it, and when that probability can be computed, the rate can be estimated from it directly. For a language model the probability of a short output is available from its next-token probabilities, as for the letter E here. For longer behaviors it can be estimated by sampling many outputs per prompt, which Jones et al. (2025, Section 3.1) call the prompt's elicitation probability.
Theorem 4 (averaging computed probabilities). Let prompts be drawn independently from a population, with probabilities $p_1, \dots, p_m$ of the event and population mean $\bar{p}$. The average $\hat{p} = \frac{1}{m}\sum_i p_i$ is unbiased for $\bar{p}$ with variance $\operatorname{Var}(p)/m$. The share of events among $m$ prompts sampled once each is also unbiased, with variance
$$\frac{\bar{p}(1 - \bar{p})}{m} = \frac{\operatorname{Var}(p) + E[p(1 - p)]}{m}.$$
Proof. The $p_i$ are independent and identically distributed with mean $\bar{p}$ and variance $\operatorname{Var}(p)$, so their average has mean $\bar{p}$ and variance $\operatorname{Var}(p)/m$. By Theorem 3 the event count is binomial with probability $\bar{p}$, which gives the first form of its variance. For the second, apply the law of total variance to one prompt's indicator $Y$: $\operatorname{Var}(Y) = \operatorname{Var}(E[Y \mid p]) + E[\operatorname{Var}(Y \mid p)] = \operatorname{Var}(p) + E[p(1 - p)]$, the decomposition that Standard Errors for Benchmark Scores proves for a score with random answers (its Theorem 4 with one answer). $\square$
The counting variance contains the term $E[p(1 - p)] \approx \bar{p}$ for a rare event, the noise of sampling an answer, which averaging the probabilities removes; this is Miller's (2024, Section 3.2) reason for scoring with next-token probabilities when they are available. How much it matters depends on how unequal the prompts are.
The letter E. Across the 4,000 questions the probability of E has mean 0.0002037, median 0.00010448, standard deviation 0.0005226 and maximum 0.01395. The counting variance is 746 times the variance of the averaged probabilities. At $m = 1{,}000$ prompts the counted share has a standard error of 2.22 times the rate itself, so it is useless, while the averaged probability has a standard error of 0.08 times the rate.
The script checks both by simulation (1,000 evaluations for each of 30 seeds, 51,000,000 and 52,000,000 onward, prompts drawn with replacement from the 4,000):
- Counting. With 1,000 prompts sampled once each, the expected number of E's is 0.204, and the evaluation saw no E in a median share 0.813 of evaluations (the exact probability is $(1 - \bar{p})^{1{,}000} = 0.816$). With 4,000 prompts it still saw none in 0.446 of them (exact 0.443). The one-sided 95% bound covered the true rate in every simulated evaluation, at both sizes, because after zero events in 1,000 the bound $0.00299$ is 14.7 times the true rate. The bound is valid and almost uninformative.
- Averaging. With 100 prompts the averaged probability was within a factor of $10^{0.07}$ of the truth in the median evaluation, and with 1,000 within $10^{0.02}$. The rate is measured to a few percent from an evaluation in which, counted, it would almost never be seen.
There is a catch, and it is the tail. The probabilities are very unequal: the top 1% of questions carry 0.20 of the total probability, and the ten highest carry 0.09. A one-sided 95% upper bound built from the average and its standard error with the $t$ distribution covered the true rate only 0.769 of the time with 100 prompts (interquartile range over seeds 0.758 to 0.777) and 0.878 with 1,000 (0.868 to 0.886). A sample of a heavily right-skewed variable usually misses its few large values, so its mean is usually below the population mean and its standard deviation usually too small. The central limit theorem holds, but at these sample sizes the normal approximation for the mean of such a skewed variable has not yet arrived, as The Central Limit Theorem shows for skewed populations. The averaged probability is the better estimate; its usual interval is too optimistic, which is the dangerous direction for a safety claim.
Example (aggregate risk needs only the mean). Jones et al. (2025, Section 3.2) define the aggregate risk of a deployment as the probability that at least one of $n$ queries, each sampled once, shows the behavior, $1 - \prod_i (1 - p_i)$. If the deployment's queries are independent draws from the same population, Theorem 3 makes each sampled answer an independent Bernoulli($\bar{p}$) trial, so the probability, averaged over the draw of the queries, is exactly $1 - (1 - \bar{p})^n$, whatever the spread of the $p_i$. (For one fixed set of queries the risk $1 - \prod_i (1 - p_i)$ does depend on which queries were drawn.) For the letter E it is 0.184 for 1,000 queries, 0.870 for 10,000 and 1.000 to three decimals for 100,000. The tail of the distribution matters for a different question, taken up next.
Extrapolating to the worst prompt
Averages describe a user who arrives with a random prompt. An adversary, or a deployment large enough to contain unusual prompts, meets the worst prompts instead. Jones et al. (2025, Section 3.2) call the largest elicitation probability among $n$ deployment queries the worst-query risk, and forecast it by the threshold $Q_p(n)$ exceeded by the top $1/n$ of queries (their Equation 2). An evaluation of $m$ prompts sees only the top $1/m$ directly. Forecasting $Q_p(n)$ for $n \gg m$ means extrapolating the upper tail.
Their Gumbel-tail method (Section 3.3) transforms each probability to the score $s = -\ln(-\ln p)$, which increases with $p$, and assumes that the scores have an exponential upper tail. Here is why that assumption makes extrapolation a straight line.
Theorem 5 (an exponential tail gives quantiles linear in the log of the deployment size). Suppose that above some score $s_0$ the survival function of the score is $P(S > s) = Ce^{-s/\beta}$ with constants $C, \beta > 0$. Then for every $n$ with $1/n \le P(S > s_0)$, the score exceeded by a share $1/n$ of prompts is
$$Q_s(n) = \beta\ln(Cn),$$
a linear function of $\ln n$, and the corresponding probability is $Q_p(n) = \exp(-e^{-Q_s(n)})$.
Proof. The defining equation $P(S > Q_s(n)) = 1/n$ lies in the exponential region because $1/n \le P(S > s_0)$, so $Ce^{-Q_s(n)/\beta} = 1/n$. Taking logarithms, $\ln C - Q_s(n)/\beta = -\ln n$, so $Q_s(n) = \beta(\ln C + \ln n)$. The map $p \mapsto -\ln(-\ln p)$ is increasing, with inverse $s \mapsto \exp(-e^{-s})$, so the threshold on the probability scale is the image of the threshold on the score scale. $\square$
The method therefore fits a straight line to the evaluation's largest scores and extends it. Jones et al. justify the exponential tail by appeal to extreme value theory, under which the maxima of many distributions approach one of three limiting distributions, one of them the Gumbel distribution, whose upper tail is exponential (their Section 3.3 and Appendix B.2), and by an analogy with scaling laws (Appendix B.1); for a given behavior it is an assumption to be checked. They fit by ordinary least squares to the ten highest scores of the evaluation. We do the same, regressing $\ln(k/m)$, the share of the $m$ evaluation prompts at or above the $k$-th highest score, on that score for $k = 1, \dots, 10$, and solving the fitted line for the share $1/n$.
Testing it on the letter E. Among the 4,000 questions the threshold for the top $1/1{,}000$ (the 4th highest probability) is 0.00860, and for the top $1/4{,}000$ (the highest) it is 0.01395. For evaluations of $m = 100$ and 400 questions drawn without replacement (100 evaluations for each of 30 seeds, 53,000,000 onward), the table gives the median absolute error in orders of magnitude, $\lvert\log_{10}(\text{forecast}/\text{truth})\rvert$, of the Gumbel-tail forecast and of the naive forecast that takes the largest probability seen in the evaluation, and the share of Gumbel-tail forecasts that fall below the truth (medians over seeds):
| Evaluation $m$ | Deployment $n$ | Gumbel-tail error | Naive error | Gumbel-tail below truth |
|---|---|---|---|---|
| 100 | 1,000 | 0.32 | 0.54 | 0.40 |
| 100 | 4,000 | 0.39 | 0.75 | 0.32 |
| 400 | 1,000 | 0.20 | 0.26 | 0.39 |
| 400 | 4,000 | 0.26 | 0.47 | 0.27 |
In this experiment the extrapolation beats the naive forecast in every setting, and its median errors lie between 0.20 and 0.39 orders of magnitude; it errs on the high side more often than on the low. In the evaluation of the figure, the largest probability among the 400 questions was 0.0039, and the forecast for 4,000 questions was 0.0175, against the true 0.01395. These are forecasts at most forty times beyond the evaluation, on one behavior of one small model. Jones et al. (2025, Section 4.2) report, for their misuse queries with forecasts up to three orders of magnitude beyond the evaluation, an average absolute log error (base 10, as here) of 1.7 for the Gumbel-tail method against 2.4 for a log-normal baseline, with the Gumbel-tail method underestimating 34% of the time and the log-normal 72%.
What is proved, what is cited, what the experiment shows
Proved here. That the rule of three overstates the exact zero-event bound (Theorem 1); the uniform-prior posterior, its mean and upper point, and the predictive probability of a clean deployment (Theorem 2); the sample size and operating characteristic of a zero-failure demonstration; that one sample per randomly drawn prompt is a Bernoulli trial with the population's mean rate (Theorem 3), and that repeated samples of the same prompts make a clean record more likely than independent trials would; the variances of counting and of averaging computed probabilities (Theorem 4); the linear growth of tail quantiles under an exponential tail (Theorem 5).
Cited or linked. The exact binomial and Poisson bounds and the rule of three, from Confidence Intervals for a Proportion and Crude Rates and Their Precision; the law of total variance and the decomposition of a score's variance, from Conditional Expectation and the Laws of Total Expectation and Variance and Standard Errors for Benchmark Scores; Jones et al. (2025) for the elicitation probability, the three risk measures, the Gumbel-tail method and its results; Miller (2024) for scoring with next-token probabilities.
What the experiments show. All the data are one small model's exact probabilities of one rare output on 4,000 questions, treated as the population. Within that setting, counting one answer per question almost never saw the event and gave valid but loose bounds; averaging the computed probabilities measured the rate to a few percent, but its $t$-based upper bound covered the truth only 77% to 88% of the time; and the Gumbel-tail method forecast the worst questions' probabilities better than the largest value seen. None of this is a general statement about other behaviors, models or prompt populations.
Worked problems
These problems were written for this article; the third and fourth use the small model's rate of answering E. The script tools/experiments/statistics/estimating-rare-event-rates.py recomputes every number.
Problem 1 (a clean evaluation of 500 prompts). An evaluation samples one output from each of 500 randomly drawn prompts and sees no instance of a behavior. Give the exact one-sided 95% upper bound, the rule-of-three and Poisson bounds, and the upper 95% point and mean of the posterior under a uniform prior.
Solution. Exact 0.00597; rule of three 0.006; Poisson 0.00599; posterior upper point 0.00596 and mean 0.00199.
By Theorem 3 the 500 outputs are Bernoulli trials with the population's mean rate, so the zero-event bound applies: $1 - 0.05^{1/500} = 0.00597$. The rule of three gives $3/500 = 0.006$, and the exact Poisson bound $2.9957/500 = 0.00599$, slightly above the binomial one, as Theorem 1 requires. Under the uniform prior, Theorem 2 gives the upper point $1 - 0.05^{1/501} = 0.00596$ and the mean $1/502 = 0.00199$, about 0.20%. All four bounds say the same thing: the rate is below about 0.6%. The clean record says nothing about rates far below that, such as the letter E's 0.02%.
Problem 2 (two events in 2,000). An evaluation of 2,000 prompts, one output each, sees the behavior twice. Estimate the rate and give the exact one-sided 95% upper bound, binomial and Poisson. What would a two-sided 95% interval's upper end be?
Solution. The estimate is 0.001; the one-sided bounds are 0.00314 (binomial) and 0.00315 (Poisson); the two-sided upper end is 0.00361.
The point estimate is $2/2{,}000 = 0.001$. The binomial bound is the 0.95 quantile of $\text{Beta}(3, 1{,}998)$, 0.00314, the largest rate under which two or fewer events have probability 0.05. The Poisson bound is the 0.95 quantile of the gamma distribution with shape 3, 6.30 events, divided by 2,000: 0.00315. A two-sided 95% interval puts 0.025 in each tail, and its upper end is the 0.975 quantile of $\text{Beta}(3, 1{,}998)$, 0.00361. The upper bound is three times the estimate: two events locate the rate only within a factor of about three.
Problem 3 (a demonstration the small model would usually fail). A team requires a demonstration, by a zero-failure test, that the rate of answering with a letter that is not an option is below 0.0005 at 95% confidence. How many trials are needed? With what probability would Qwen2.5-0.5B, whose rate of answering E is 0.0002037, pass if E were the only failure counted?
Solution. 5,990 trials; it would pass with probability 0.295.
The plan needs the smallest $n$ with $(1 - 0.0005)^n \le 0.05$, that is, $n \ge \ln 0.05/\ln 0.9995$, which is 5,990. The model's rate is 0.41 of the target, so the claim being tested is true. It passes only if none of the 5,990 sampled answers is E, which has probability $(1 - 0.0002037)^{5{,}990} = 0.295$: the expected number of E's is $5{,}990 \times 0.0002037 = 1.22$, and a single one fails the test. A system 2.5 times better than required fails a zero-failure demonstration seven times in ten. The one-failure plan of the section on planning, scaled to this target, needs 9,486 trials and would pass this model with probability 0.42: more often, but still less than half the time.
Problem 4 (a deployment forecast). An evaluation of 1,000 randomly drawn prompts, one output each, sees no E. The model will answer 10,000 prompts from the same population. Under a uniform prior, what is the probability of at least one E in deployment? What is the actual probability for this model?
Solution. The predictive probability is 0.909; the actual probability is 0.870.
By Theorem 2 the probability that the 10,000 deployment answers are also clean is $1{,}001/11{,}001 = 0.0910$, so the probability of at least one E is 0.909. With the true rate known, Theorem 3 gives $1 - (1 - 0.0002037)^{10{,}000} = 0.870$, from an expected 2.04 E's. The two agree because a clean evaluation of 1,000 leaves the uniform-prior posterior spread over rates up to about 0.003, most of which make an event in 10,000 answers nearly certain. The forecast is close here by luck of scale: the evaluation could not tell a rate of 0.0002 from one of 0.00002, at which the deployment would see an E with probability only about 0.18.
Summary
| Situation | One-sided 95% upper bound or result |
|---|---|
| Zero events in $n$ trials | $1 - 0.05^{1/n} < 3/n$ |
| $x$ events, Poisson | $\mu_U$ = 3.00, 4.74, 6.30, 7.75, 9.15, 10.51 for $x$ = 0 to 5, divided by $n$ |
| Uniform prior, zero events | posterior $\text{Beta}(1, n + 1)$, mean $1/(n + 2)$, upper point $1 - 0.05^{1/(n+1)}$ |
| No events in $N$ more trials | $(n + 1)/(n + N + 1)$ |
| Zero-failure demonstration of $p < r$ | $n \ge \ln 0.05/\ln(1 - r) \approx 3/r$ |
| $K$ samples from each of $m$ prompts | $m$ trials, not $mK$: one output per prompt, or "any of $K$" (conservative) |
| Averaged computed probabilities | unbiased, variance $\operatorname{Var}(p)/m$ instead of $\bar{p}(1 - \bar{p})/m$ |
| Aggregate risk, $n$ random prompts | $1 - (1 - \bar{p})^n$ |
| Exponential score tail | $Q_s(n) = \beta\ln(Cn)$ |
What we have learned
An evaluation that sees no events bounds the rate at about $3/n$ and no lower, and one or two events raise the bound to about $4.74/n$ or $6.30/n$. With one sample per randomly drawn prompt these bounds hold for the population's average rate however unequal the prompts are; repeated samples of the same prompts do not add trials. A clean evaluation says little about a deployment much larger than itself: after 1,000 clean trials, an event in a deployment of 100,000 is close to certain under a uniform prior. When the probability of the behavior can be computed for each prompt, the average of those probabilities measures a rate far too small to count, the letter E's 0.0002 to within a few percent from 1,000 questions, though its usual interval is too optimistic when a few prompts carry much of the probability. The worst prompts in a large deployment can be forecast by extrapolating the upper tail, here with median errors of 0.2 to 0.4 orders of magnitude.
These are the statistics of single behaviors and single models. When many models are compared on many tasks, the question becomes how to rank them; Rating Models from Pairwise Comparisons and Comparing Systems Across Many Tasks take it up.
References
- Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021). Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR). arXiv:2009.03300. record · 7 pages
- Jones, E., Tong, M., Mu, J., Mahfoud, M., Leike, J., Grosse, R., Kaplan, J., Fithian, W., Perez, E., and Sharma, M. (2025). Forecasting rare language model behaviors. arXiv:2502.16797v1. record · 2 pages
- Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640v1. record · 7 pages
- Yang, A., et al. (Qwen Team) (2024). Qwen2.5 technical report. arXiv:2412.15115. Model used: Qwen/Qwen2.5-0.5B, Hugging Face revision 060db6499f32faf8b98477b0a26969ef7d8b9987. record · 2 pages