Forecasting rare language model behaviors

evaluationrare behaviorsforecastingextreme value theoryred teamingmisuse

Jones, Tong and colleagues forecast how often an undesired behavior will appear at deployment scale from evaluations up to three orders of magnitude smaller, by measuring each query's probability of eliciting the behavior and extrapolating the largest of these with an extreme-value (Gumbel-tail) fit; the forecasts beat a log-normal baseline for dangerous-information misuse and for misaligned choices under varied system prompts, and help allocate compute in automated red-teaming.

Erik Jones, Meg Tong et al. · 2025-02-25 · original

A safety evaluation typically runs hundreds or thousands of queries, while a deployed model may answer billions a day. A behavior that appears once in a million queries will almost never show up in the evaluation, yet will occur many times in deployment. The accompanying post's example: a jailbreak technique tried thousands of times in testing and never working may still work at some point in a million real attempts. "Forecasting Rare Language Model Behaviors" (arXiv:2502.16797, 24 February 2025; anthropic.com post 25 February 2025), by Erik Jones, Meg Tong and eight co-authors from Anthropic, Mila and UC Berkeley, asks whether such deployment-scale risks can be predicted from a small evaluation, before they occur.

The method rests on one observation: although each sampled output either shows a behavior or not, the probability that a given query produces it is continuous, and the largest of these probabilities grows with the number of queries in a regular way that can be extrapolated. Section numbers below refer to the arXiv paper.

What the paper does

Elicitation probability and three risk measures

The elicitation probability of a query is the chance that one sampled output from the model, given that query, exhibits the target behavior (Section 3.1, Equation 1). Many queries that seem to fail, including jailbreaks that are usually refused, have small but nonzero elicitation probabilities, below 0.01 for example (Figure 2).

The paper assumes evaluation and deployment queries come from the same distribution and differ only in number: m evaluation queries against n deployment queries, with n much larger. It defines three ways risk can grow with n (Section 3.2):

  • Worst-query risk: the largest elicitation probability among the n queries, a proxy for the most effective jailbreak at deployment.
  • Behavior frequency: the fraction of queries whose elicitation probability exceeds a threshold, so that they produce the behavior routinely.
  • Aggregate risk: the probability that at least one of n outputs, one sampled per query, shows the behavior. It can be high even when no single query is risky, because small probabilities add up.

All three depend on the upper tail of the distribution of elicitation probabilities, written Q_p(n): the threshold exceeded by the top 1/n of queries.

The Gumbel-tail method

Each query's elicitation score is defined as −log(−log p), where p is its elicitation probability; the score is larger when p is closer to 1 (Section 3.3). By extreme value theory, the largest values of many distributions behave like one of three limiting distributions, one of which is the Gumbel distribution. The authors assume the scores fall in the Gumbel case, motivating it by analogy to scaling laws for pretraining loss (Appendix B.1). For a Gumbel-like tail, the logarithm of the probability of exceeding a score is approximately a linear function of the score (Appendix B.2). That gives a straight line between the score quantile and log n, which can be fitted on the evaluation set and extended to larger n.

In practice the method fits that line by ordinary least squares to the ten highest elicitation scores in the evaluation set, then extrapolates. Translated back, it implies a power law between −log Q_p(n) and n (Section 3.3). The authors note that using only ten points makes forecasts sensitive to which queries happened to be in the evaluation set, and that the set may not reach the true tail.

The baseline, the log-normal method, fits a normal distribution to all the evaluation scores and computes the expected maximum analytically (Section 4.1). It tests whether modeling the tail specifically matters, rather than extrapolating from average behavior.

Forecast quality is reported mainly as absolute log error: the average absolute difference, in base-10 logarithms, between forecast and actual values, so an error of 1 is a factor of ten. The paper also reports plain absolute error, which captures differences among large probabilities that the log error compresses (Section 4.1).

What it finds

Misuse: dangerous chemical and biological information

Setup (Section 4.1, Appendix C.1). For 10 dangerous chemicals and 9 biological agents, a helpful-only variant of Claude 3.5 Sonnet (new) was prompted to write queries designed to elicit instructions, with 100,000 queries generated per question at temperature 1. Queries containing the answer's keyword were removed, to simulate attackers who do not already know it. Three proxies measure elicitation: the probability of one specific harmful output sentence, the probability of a key word within that sentence, and, by repeated sampling, the probability that any output contains the key word.

Worst-query risk (Section 4.2, Table 2). Across evaluation sizes m of 100, 500 and 1,000, deployment sizes n from 10,000 to 90,000, and four models (Claude 3.5 Sonnet, Claude 3 Haiku and their base models):

  • Average absolute log error was 1.7 for the Gumbel-tail method against 2.4 for the log-normal (Table 2 gives 1.672 and 2.371).
  • Gumbel-tail forecasts were within a factor of ten of the actual worst-query risk 72% of the time, and improved more than the baseline as m grew: its error fell by 0.68 from m = 100 to m = 1,000, against 0.08 for the log-normal (Appendix C.2).
  • The Gumbel-tail method underestimated the actual value 34% of the time and the log-normal 72% of the time. The authors note that underestimates are the dangerous direction, and that both methods are biased, since an unbiased method would underestimate half the time.
  • The introduction gives a narrower case: forecasting the risk at 90,000 queries from 900, forecasts were within an order of magnitude of the true risk for 86% of misuse forecasts (Section 1; also in the post).

Behavior frequency (Section 4.3, Table 3). Using keyword probabilities, thresholds from 0.1 to 0.9, and 1,000 random evaluation sets per substance, forecasts were made only when no evaluation query had crossed the threshold. The Gumbel-tail absolute log error ranged from 0.84 to 0.76 as m rose from 100 to 1,000; the log-normal ranged from 3.31 to 4.04. Averaging forecasts across evaluation sets roughly halved the Gumbel-tail error (0.383 overall). The paper notes that the log-normal's moderate underestimates of the tail become extreme underestimates of frequency.

Aggregate risk (Section 4.4, Table 4). From 1,000 evaluation queries and deployments of 10,000 to 500,000, the absolute log error was 1.3 for the Gumbel-tail method and 2.5 for the log-normal (Table 4: 1.286 and 2.523). In this setup, the authors observe, an adversary can obtain harmful information with arbitrarily high probability "even when no specific query routinely elicits it."

Repeated sampling (Section 4.5, Appendix C.2.4). To check that results hold for general correct instructions rather than one fixed sentence, the authors sampled 100 outputs per query (500 for substances where too few queries had any success) from Claude 3.5 Haiku, over 10,000 queries, keeping only substances where the most effective of 100 queries succeeded less than half the time. Empirical quantiles were often roughly linear as predicted, though not in every setting (Figure 5 caption: "Many but not all"). The worst-query absolute log error was 0.17, and forecasts correctly predicted that the maximum elicitation probability would exceed 0.5 in 75% of the relevant cases.

Misaligned actions under different system prompts

Setup (Section 5, Appendix D). The authors assume developers will specialize a model with system prompts and that some prompts could make misaligned choices more likely. Claude 3.5 Sonnet (new) generated about 100,000 system prompts from combinations of 21 domains (such as sales and medicine), 15 tones, 3 levels of specificity and 3 degrees of grounding to examples, 35 prompts per combination. Each was paired with two-option questions in which one option is misaligned: 41 power-seeking questions, 33 self-preservation and 33 self-exfiltration. The elicitation probability is the probability the model picks the misaligned option.

Results. For worst-query risk, the Gumbel-tail method had average absolute error 0.05 and log error 0.06, against 0.12 and 0.12 for the log-normal. On questions where no system prompt in the evaluation set reached 0.5 (so the misaligned choice had not yet been made), errors were 0.14 and 0.19 against 0.18 and 0.28. On aggregate measures over random subsets of questions, the two methods performed similarly. For behavior frequency, with a system prompt counted as inducing misalignment when the misaligned option exceeds 0.5, the Gumbel-tail log error was 1.05 against 4.10. The post summarizes the worst-query comparison as "2.5 times lower error."

The authors caveat that these are small sets of synthetic multiple-choice questions, and that answers to them "might not reflect" behavior on open-ended prompts (Section 5).

Allocating compute in automated red-teaming

A red-teamer with a fixed budget can generate attack queries from a cheaper model (Claude 3.5 Haiku) or fewer from a costlier one (Claude 3.5 Sonnet (new)), assumed 10 to 100 times more expensive (Section 6). Across 223 settings, forecasts picked the model that actually found the highest elicitation probability 63% of the time, against 54% for always choosing the majority answer and 50% by chance. When the two options truly differed by more than two orders of magnitude, accuracy was 79%. The forecasts tended to slightly overestimate, increasingly so the further they extrapolated.

The prompts used

The paper's data are produced by prompts, which it prints:

  • Query-generating meta-prompts. The misuse queries come from a meta-prompt asking a model to craft a prompt that "leads the model to give the answer to a question" without revealing the answer, ending with "Ensure your prompt is subtle yet directive" (Figure 7). A harder version requires the question to appear verbatim and the prompt to be 40 to 60 words (Figure 9).
  • System prompts as a source of risk. In Section 5 the deployment variable is the system prompt itself: "Each system prompt risks inducing more misaligned actions," with a health assistant as the example. The prompts were generated by a meta-prompt that fills in domain, tone and other characteristics (Figure 10); Figure 11 shows one, for an assistant specializing in international business law, followed by a question with a misaligned action as one of the two answers.

The paper treats the system prompts as a distribution to sample from and does not analyze which features of a prompt's wording raise or lower the rate of misaligned choices.

Limitations and open questions

The authors list the following (Sections 3.3, 4, 5, 6 and 7):

  • Sensitivity and bias. Fits use only the top ten scores, so forecasts vary with the evaluation set, and the set may not capture the extreme tail. Both methods are biased on worst-query risk (an unbiased method would underestimate half the time; the Gumbel-tail method underestimates 34% of the time, the log-normal 72%), and red-teaming forecasts tend to overestimate (Sections 4.2 and 6).
  • Scale tested. Deployments simulated are at most three orders of magnitude larger than evaluations, because ground truth at real deployment scale (millions to billions of queries) is too expensive to generate. The authors expect the method to extend further but did not test it.
  • No distribution shift. Evaluation and deployment queries are drawn from the same distribution. Real adversaries adapt their queries and users apply models in new settings; robustness to such shifts is left to future work. The misuse study covers non-adaptive adversaries only (Section 4).
  • Distributional choice. Possible improvements include adaptively testing the fit of each extreme-value distribution, checking whether the evaluation set reaches the tail, and adding uncertainty estimates to forecasts (Section 7).
  • Cost. Elicitation probabilities by repeated sampling are expensive; adaptive stopping and importance sampling are suggested (Section 4.5).
  • Proxies. Most misuse results use the probability of one fixed sentence or keyword, not judged harmfulness, and the misalignment results use multiple-choice answers.

The authors also suggest the approach could serve as a running monitor during deployment, forecasting how long a deployment can continue at low risk (Section 7).

Research questions

  1. How well does the Gumbel assumption fit across behaviors? The method assumes Gumbel-like tails and the Figure 5 caption says "Many but not all" extreme quantiles follow the expected power law. Goodness-of-fit tests across many behavior types, and checks for heavier tails, would show where forecasts may underestimate.
  2. How robust are forecasts to adaptive adversaries? The paper assumes a fixed query distribution. Forecasting from one round of red-teaming and testing against later, adapted rounds would measure the effect of distribution shift.
  3. Can the method extend to multi-turn and agentic behavior? Each unit here is one query. Defining elicitation probabilities over whole conversations or agent trajectories, and checking whether their tails still scale predictably, would test the extension.
  4. Which features of a system prompt drive misaligned choices? Section 5 samples 100,000 generated system prompts but does not relate rates to wording. Regressing elicitation probabilities on the generation variables (domain, tone, specificity, grounding) would show which matter.
  5. Do multiple-choice forecasts predict open-ended behavior? The authors caution that the misalignment results use multiple-choice questions. Pairing each system prompt's multiple-choice rate with its rate of misaligned actions in open-ended agentic scenarios would test the proxy.
  6. Can forecasts compare training interventions? If two versions of a model differ in safety training, comparing their forecast deployment risks on the same query distribution would show whether the method can detect differences that standard evaluations miss.

How it connects

References

  • Jones, E., Tong, M., Mu, J., Mahfoud, M., Leike, J., Grosse, R., Kaplan, J., Fithian, W., Perez, E., and Sharma, M. (2025). Forecasting Rare Language Model Behaviors. arXiv:2502.16797v1. https://arxiv.org/abs/2502.16797
  • Anthropic. (2025, February 25). Forecasting rare language model behaviors. https://www.anthropic.com/research/forecasting-rare-behaviors
  • Works cited above only as the paper describes them (Perez et al., 2022; Wu and Hilton, 2024; Hughes et al., 2024) were not read for this note.