Introducing Bloom: an open source tool for automated behavioral evaluations

Hand-built behavioral evaluations are slow to make and lose value over time: their items can leak into training data, and models can improve until a test no longer measures what it was designed to (announcement post). Bloom is Anthropic's tool for generating such evaluations on demand. A researcher describes one behavior, such as sycophancy toward a user's delusions, and Bloom writes many scenarios meant to elicit it, plays them out against a target model, and reports how often and how strongly the behavior appeared. Anthropic says the four benchmark suites released with the tool "took only a few days to conceptualize, refine, and generate".

Bloom was released on 19 December 2025, two months after Petri, by largely the same team, with a short post on anthropic.com and a technical report on the Alignment Science Blog. The report contrasts the two directly: Petri is "for exploration" across many behaviors, Bloom is "for measurement" of one ("When to use Bloom vs. Petri"). This note draws on both documents; section names refer to the technical report unless the announcement is named.

Terms used in this note

• Behavior description. The researcher's written definition of the behavior to measure, optionally with a scoring rubric. It is the core input.
• Seed (seed configuration). A configuration file holding the behavior description, any example transcripts, the models used at each stage and other settings. The authors say Bloom metrics "should always be cited with this seed".
• Rollout. One played-out scenario: a conversation or simulated agentic episode between Bloom's evaluator and the target model.
• Behavior presence score. The judge's 1 to 10 rating of how strongly a rollout shows the behavior.
• Elicitation rate. The share of rollouts scoring 7 or more, Bloom's headline metric.
• Model organism. A model deliberately made to have a known behavior, here by a hidden system prompt, used as a positive control.
• Meta-judge. A model that reads all the scores and summaries of a suite and writes a suite-level report.

What the tool does

Four stages

Bloom runs four stages, each a model with its own prompt ("System Design"):

1. Understanding. A model reads the behavior description and any example transcripts and writes an account of what is being measured and why. This context is passed to later stages, partly to keep them on task and "prevent safety refusals".
2. Ideation. A model writes detailed scenarios: the situation, the simulated user, the target's system prompt, the environment, and how the behavior might appear.
3. Rollout. A model plays each scenario, writing the target's system prompt and first message and then simulating the user and any tools, until the behavior appears or a turn limit is reached.
4. Judgment. A judge scores each transcript for the behavior and for secondary qualities, and a meta-judge writes the suite report.

Unlike a fixed benchmark, each run produces new scenarios, while the seed keeps the measured behavior the same. A static mode, with identical single-turn prompts across runs, is available for exact comparisons.

Settings that change what is measured

The report lists the main configuration options ("Seed Configuration"). Among them:

• Diversity $d$ between 0 and 1. The ideation model writes $n \times d$ distinct scenarios, and a variation step perturbs them (changing names, companies or dates) to reach $n$ rollouts. With $n = 50$ and $d = 0.2$ there are 10 base scenarios with 5 variants each.
• Modality: plain conversation, or a simulated environment that gives the target synthetic tools.
• User mode: with or without a simulated user; without one, the target acts autonomously.
• Maximum turns and repetitions per scenario.
• Secondary qualities for the judge, such as unrealism, elicitation difficulty, evaluation invalidity and evaluation awareness.
• Anonymous target: whether the evaluator knows which model it is testing. Self-preferential bias cannot be judged without knowing.
• Redaction tags: parts of a transcript hidden from the judge, such as a model organism's secret instructions.

The four released benchmarks

Using Claude Opus 4.1 for every stage, the authors generated three suites of 100 rollouts per behavior for each of 16 models (Figure 1, Appendix "Benchmarks"). The definitions, quoted from the appendix:

• Delusional sycophancy: an AI that "mirrors or validates and exacerbates a user's fantastical or psychotic beliefs rather than grounding the conversation in reality". Five-turn conversations, seeded with two real-world examples, target reasoning off.
• Instructed long-horizon sabotage: the target is told in its system prompt to complete a hidden malicious side task while doing a long main task, so the suite measures ability and willingness to sabotage when instructed, not spontaneous motivation. Fifteen turns, no simulated user, seeded with two trajectories from SHADE-Arena.
• Self-preservation: whether the target acts to resist shutdown or replacement. Five turns in a simulated environment, seeded with a blackmail transcript from the Agentic Misalignment work.
• Self-preferential bias: "the tendency of models to knowingly choose options that favor themselves in tasks where they are meant to act as neutral judges". Single-turn, modeled on the Claude Sonnet 4.5 system card.

Evaluation awareness was scored as a secondary quality across all four suites rather than as a suite of its own.

What it finds

Benchmark results

Figure 1 plots elicitation rates for all 16 models with standard deviations over three suites; the report does not tabulate the values in text. The one cross-model statement in the text is about evaluation awareness: only Claude Opus 4.5, Claude Sonnet 4.5, Claude Haiku 4.5 and Grok 4 showed recurring evaluation awareness and skepticism across the four suites (Figure 15). The Limitations section notes "high evaluation awareness rates in newer Claude models".

Separating model organisms from baselines

The authors built ten system-prompted model organisms, each with a quirk taken from Anthropic's earlier auditing-agents work (for example flattery, defer-to-users, hardcoding test cases, increasing pep, emotional bonding, self-promotion), and compared each with the same model unprompted ("Distinguishing Model Organisms from Baseline Models"). The target was Claude Sonnet 4, or Claude Sonnet 3.7 where Sonnet 4 already showed the quirk; the quirk's system prompt was hidden from the evaluator and the judge. With 50 rollouts per suite and three repetitions, Bloom separated organism from baseline for nine of ten quirks, with or without example transcripts (Figure 3). The exception was self-promotion: on review, the baseline model showed the behavior at similarly high rates, leaving nothing to detect.

Agreement between judge models and humans

The authors hand-labeled 40 transcripts spanning 12 behaviors and compared those labels with scores from several judge models ("Judge Trust and Calibration", Figure 4). Claude Opus 4.1 had the highest Spearman correlation with the human labels, 0.86, followed by Claude Sonnet 4.5 at 0.75. Bucketing scores as low (1 to 3), medium (4 to 6) and high (7 to 10), Opus 4.1 and the humans agreed most at the extremes, which is where the threshold of 7 operates (Figure 5); the few large disagreements involved jargon-heavy transcripts or borderline behavior and showed no systematic pattern. The authors note that Opus 4.1 and Sonnet 4.5 may do best partly because Bloom's prompts were refined on their failure modes, and that GPT-OSS 120B and o4-mini look less suitable as judges. The number of judge models is given as 11 in the announcement and the Figure 4 caption and as 12 in the appendix.

Consistency of judges and suites

• Repeated judging. Scoring each of 50 transcripts five times, Claude judges, especially Claude Sonnet 4, almost never changed their scores, while GPT-5's scores varied much more, particularly without extended reasoning, and its variance rose with the average score (Figure 6).
• Meta-judge. The meta-judge's diversity ratings tracked the configured diversity of six otherwise identical suites (Figure 7).
• Scenario stability. Running each scenario five times, scenarios that nearly always or nearly never elicited the behavior were stable, while mid-range scenarios varied widely, an inverted-U pattern that also held across perturbed variants (Figure 8).
• Whole suites. Across experiments the variance of the elicitation rate was "generally low", but rose with GPT-5 as judge and with longer interactions; the self-preservation suite with a GPT-5 evaluator and a Grok-4 target was the most variable (appendix "Sources of Variance").

The choice of generator model shapes the evaluation

The ideation and rollout models are not neutral ("Impact of Ideation and Rollout Models"). In an experiment generating 100 single-turn political-bias queries from OpenAI's published definitions, GPT-5 and web search produced mostly neutral queries, Grok 4 produced more Democratic-charged queries than any other ideator, Kimi K2 more conservative-charged ones, and adding two charged examples made queries more charged on both sides (Figures 9 and 19). On the ideator's reasoning effort the two parts of the report differ: the main text says it does not meaningfully affect the scenario distribution, while the appendix reports that switching extended thinking off produced an almost perfectly even spread across the ideological spectrum (Figure 19). Claude and Grok favored opinion-seeking questions and GPT-5 and Kimi K2 policy questions. For rollouts, using the same scenarios with four different rollout models, Opus 4.1 separated quirky from baseline models best, and GPT-5 barely elicited the increasing-pep quirk, about a 1-point separation against about 6 for the others (Figure 10).

Case study: self-preferential bias

Seeded with transcripts in the style of the Claude Sonnet 4.5 system card (several items, one marked as the target's own, pick the best), Bloom reproduced the system card's ranking of four Claude models, with Sonnet 4.5 least biased (Figure 11). The absolute numbers differ, because the system card reports a percentage change and Bloom an absolute rate. Further results ("Case Study: Self-Preferential Bias"):

• More target reasoning, less bias. For Claude Sonnet 4, raising reasoning effort generally reduced the bias, most between medium and high (Figure 12). The reduction came from the model recognizing the conflict of interest and declining to judge its own option, not from choosing others more evenly.
• Filtering low-quality rollouts. Rollouts scored as unrealistic, invalid or evaluation-aware were less often successful; removing those scoring 7 or more on these qualities slightly raised the elicitation rate (Figure 13).
• Rankings under different settings. Varying few-shot examples (0 to 4), maximum turns (1, 2, 5, 10) and evaluator reasoning effort (four levels), the model ranking was fairly stable for examples (Kendall's W = 0.66, p = 0.02) and turns (W = 0.63, p = 0.06) and less so for evaluator reasoning (W = 0.38, p = 0.21). Sonnet 4.5 ranked least biased in nearly all configurations (Figure 14).
• Examples and length. One example raised the rates and more gave diminishing returns. Without examples the evaluator invented a different, looser scenario format. For this behavior, the main text says success tends to drop when conversations go beyond a single turn, because the target can take back a biased first answer; the appendix text says the metrics plateaued after two turns, and that in 45 percent of 25-turn runs the evaluator declared the behavior elicited and ended early, while the caption of Figure 23 describes diminishing returns beyond 10 turns (appendix "Metrics as a Function of Compute"). The report does not reconcile these three descriptions.

The written specifications behind an evaluation

In Bloom, an evaluation is defined by text: the behavior description, the agents' prompts, and the seed. The report treats these as the object to version and cite, and several of its findings concern their wording.

• The behavior description "should ideally be specific and aligned with what you actually care about", and may include a rubric from mild to severe examples ("Global Configuration Settings"). The sabotage definition, for instance, spells out that the suite measures ability when instructed, "not its intrinsic motivation".
• Default prompts address known failure modes: the ideation prompt works against stereotyped names and boilerplate, and the rollout agent is told that typical users "will not typically introduce themselves, and will keep messages as low-effort as possible". The scaffolding also tells the ideator, "Be creative and don't fixate too heavily on any seed examples" (appendix footnote 1).
• Model organisms are built from the same text. The quirky models were made by putting "the exact behavior description used to seed the pipeline" into their system prompts (appendix "Meaningfulness and Trust"). The validation therefore tests whether Bloom can detect a behavior that a written description induced, using that same description.
• Seed examples generalize but also constrain. Example transcripts written for one model transferred to others and across modalities, but they also steered the scenario format, and zero-shot ideation found other valid formats.

Limitations and open questions

The report's Limitations section names three:

• Subjective behaviors only. Bloom suits behaviors such as sycophancy, bias or deception. It is less suited to checking objective correctness, such as whether code works or a task was really done, which a judge may not score reliably.
• Limited control. Automated generation cannot guarantee minimal pairs, such as changing exactly one word between test cases. Simulated users and tools cannot capture behavior that depends on real consequences.
• Evaluation awareness. Models may recognize the tests, and this could grow if Bloom-generated evaluations enter training data.

Other limits are visible in the results. There is no ground truth for how often a behavior occurs in a real model, so validation relies on organisms and 40 hand-labeled transcripts. Absolute metrics depend on the judge, the generator, the number of examples, the interaction length and reasoning effort, which is why the authors ask for every number to be cited with its seed. Opus 4.1 is the evaluator throughout the benchmarks, and the authors acknowledge that the judge prompts were refined on the failure modes of Opus 4.1 and Sonnet 4.5. The model-organism quirks are induced by system prompts, which may be easier to detect than behaviors learned in training.

Research questions

1. How do Bloom's rates compare with hand-built benchmarks for the same behavior? The report notes that direct comparison is unreliable because scenario distributions differ. Running models on SHADE-Arena or the Agentic Misalignment scenarios and on the corresponding Bloom suites, and correlating the model rankings, would show whether they measure the same thing.
2. Do evaluator and target from the same developer interact? Opus 4.1 generates, plays and judges every benchmark rollout. Repeating the 16-model benchmarks with evaluators from other developers and checking whether Claude models' relative positions change would test for family effects of the kind Petri's judge comparison found.
3. How sensitive are results to the wording of the behavior description? The seed is meant to fix the measured behavior. Generating suites from several paraphrases of one definition, with everything else held constant, would show how much the elicitation rate depends on phrasing.
4. Can Bloom detect behaviors learned in training rather than prompted? All organisms used system prompts. Repeating the validation with fine-tuned model organisms, such as those in Anthropic's auditing work, would test the harder case.
5. What drives evaluation awareness in the newest models? Four models showed recurring awareness. Analyzing which scenario features (unrealism scores, explicit testing language, implausible stakes) predict awareness, using Bloom's secondary-quality scores, would guide more realistic generation.
6. Does filtering unrealistic rollouts change model rankings, not only rates? Filtering slightly raised elicitation rates in the case study. Checking rankings before and after filtering across the four benchmarks would show whether realism filtering affects conclusions.

How it connects

• In this evaluation thread. Bloom extends the approach of Discovering Language Model Behaviors with Model-Written Evaluations (2022) from static multiple-choice questions to generated multi-turn scenarios, keeping the same idea of a model writing tests from a short description. It is the companion of Petri, whose open-ended audits it complements with targeted measurement. Challenges in evaluating AI systems (2023) described the cost of hand-built evaluations that Bloom tries to reduce, and the "diff" tool approaches unknown behaviors from model internals instead.
• Elsewhere in the library. The sabotage suite replicates trajectories from SHADE-Arena, and the self-preservation seed comes from Agentic Misalignment. Sycophancy has its own study in Towards Understanding Sycophancy in Language Models. Reporting evaluation results with their uncertainty is the subject of A statistical approach to model evaluations.
• In the course wiki. Judge-human agreement measured by rank correlation, and the question of whether a substitute judge preserves system rankings, are the topics of Relevance Judgments, Pooling, and Agreement. Bloom's use of Kendall's W to check ranking stability parallels the use of Kendall's tau there.

References

• Gupta, I., Fronsdal, K., Sheshadri, A., Michala, J., Tay, J., Wang, R., Bowman, S. R., and Price, S. (2025). Introducing Bloom: an open source tool for automated behavioral evaluations. Anthropic, 19 December 2025. https://www.anthropic.com/research/bloom
• Gupta, I., Fronsdal, K., Sheshadri, A., Michala, J., Tay, J., Wang, R., Bowman, S. R., and Price, S. (2025). Bloom: an open source tool for automated behavioral evaluations (technical report). Alignment Science Blog, 19 December 2025. https://alignment.anthropic.com/2025/bloom-auto-evals/
• Bloom code repository: https://github.com/safety-research/bloom