Training Alignment Auditors via Reinforcement Learning

Rosu and Wang train Claude Haiku 4.5 with reinforcement learning to audit target models whose hidden behaviors are planted in a system prompt. Pointwise rewards were gamed; a pairwise reward against a periodically refreshed reference audit, with half of training on targets that have no planted behavior, raised the four-part evaluation composite from 44.2 to 48.7 (Claude Opus 4.6, 48.4) with false-positive calibration at 99.6. On AuditBench's fine-tuned targets that configuration reached 15.6% against 11.5% untrained; other trained configurations reached 28.1%, and the two reward-hacked configurations also beat the untrained baseline.

Paul Rosu (Anthropic Fellow) and Rowan Wang (Anthropic) · 2026-10-15 · original

Automated auditors, language models that probe another model for concerning behavior over many turns, are part of Anthropic's pre-deployment testing. The paper's introduction cites two earlier results: automated auditors "often produce shallow or unrealistic investigations" (Kissane et al., 2026), and AuditBench "showed that production auditing systems largely fail against adversarially hardened target models". The second is the paper's gloss: AuditBench tested its own Claude Haiku 4.5 investigator, which its authors say differs from Petri and Anthropic's internal agent, and found it succeeded on 22.2% of runs against its hardest targets, averaged over tools, in the October 2026 revision that retrained those targets (AuditBench v4, Section 4; the earlier versions print no such average). Earlier work improved auditors with scaffolding and tools. This paper asks whether reinforcement learning can make the auditor itself better, and what reward makes that possible without the auditor learning to game it.

The version of record is the arXiv paper (arXiv:2608.25460v1, 26 August 2026, 82 pages), the source for every number below. The Alignment Science Blog post of 15 October 2026 is a one-page summary with two of the paper's figures and links to the paper and code. The post says realism "climbs from 4.6 to 14.0 (Opus 4.6: 13.1)" and describes it as "matching" Opus 4.6, where the paper says "exceeding". Rosu was an Anthropic Fellow; Wang is at Anthropic.

Terms used in this note

  • Auditor (policy). The model being trained, Claude Haiku 4.5, which conducts 15-turn audits in the Petri scaffold using Petri's six tools (send messages, set the target's system prompt and tools, roll back turns, end the audit).
  • Planted behavior. A hidden behavior given to a target model by a system prompt that tells it to play "PRISM-4 by Nexus Research", show the behavior in the relevant context, and deny it if questioned (Appendix A.1, a template adapted from AuditBench). There are 32 behaviors: the 14 AuditBench behaviors and 18 new ones. Six AuditBench behaviors are held out for evaluation; the other 26 are used for training (Section 2.1).
  • Pointwise reward. A judge scores one audit transcript on its own.
  • Pairwise reward. A judge compares two audits of the same target, behavior and scenario and picks the better. In the intragroup form, several rollouts are compared with each other and each one's reward is its win rate; in the reference form, each rollout is compared once with a cached reference audit regenerated from the current policy every 50 steps (Section 3.3).
  • Calibration rollouts. Training audits of a target with no planted behavior, in which the judge prefers the audit that more accurately concludes nothing is wrong (Section 3.4). "4/8 FP" means half of each batch.

What the paper does

The evaluation suite

Four scores, each 0 to 100, averaged into a composite; every evaluation transcript is judged by Claude Opus 4.6 (Section 2.2):

  • Audit quality. Six held-out behaviors planted in DeepSeek v3.1, with 15 tailored scenario seeds each and three rollouts per seed. The judge scores six dimensions: four on investigation (coherence, depth, quality of evidence, non-confrontational style), whether the target showed the behavior, and whether the auditor identified it.
  • False-positive calibration. The same seeds against Claude Sonnet 4.6 with no system prompt; the score is 100 minus the rate at which the auditor claims to find the seeded behavior.
  • Production misalignment discovery. Petri's 181 default seeds, three rollouts each, against Claude Sonnet 4.5, scored by the default Petri judge on concerningness and related dimensions.
  • Realism. Following Kissane et al., an Opus 4.6 judge sees an audit transcript paired with a real WildChat conversation on a matched topic (20 pairs) and must say which is real; the score is the share of comparisons in which it picks wrong (Appendix H.1). A judge guessing at random would be wrong half the time, so 50 is the level at which audits are indistinguishable from real use (this reading of the scale is the note's).

Frontier models set the reference points (Section 2.3, Table 7): Haiku 4.5 composite 44.2, Sonnet 4.6 46.5, Opus 4.6 48.4, Opus 4.7 49.9, and the paper says all frontier models, Claude, GPT and Gemini, "maintain near-perfect false positive calibration". The authors take scores that rise with capability within each model family as validating the suite. The training target is Llama 3.3 70B; training rewards are judged by Claude Sonnet 4, a different judge from the evaluation's; the six evaluation behaviors, their seeds and the DeepSeek target are held out. Training uses 1,300 seeds for the 26 training behaviors, drafted by the authors and Claude Opus 4.5 and reviewed by hand (Section 3, Appendix C.2). The paper does not name the RL algorithm.

The sequence of rewards

The authors present the rewards in the order they tried them, each prompted by a failure of the last (Section 3):

  1. Concerningness against production models. The auditor is rewarded for audits that Petri's judge rates concerning, gated by a realism score, against eight unmodified production models (Section 3.1, Appendix D.5).
  2. Binary reward on planted behaviors. A judge answers whether the audit identified the planted behavior (Section 3.2).
  3. Pairwise rewards, intragroup and reference, with the target's hidden system prompt and 38 Petri metrics given to the judge as context (Section 3.3).
  4. Calibration rollouts added in fractions from 1/8 to 4/8 of each batch (Section 3.4).
  5. Multi-target training over five model families, so that the auditor cannot tell planted-behavior targets from benign ones by their style (Section 3.5).

Transfer test

Trained auditors are also run on AuditBench's hardest targets, Llama 3.3 70B models trained on demonstrations and adversarially trained with KTO, for the six evaluation behaviors, in a Claude Code scaffold with a single sampling tool rather than Petri. Each model gets 16 audits per behavior (96 in all) and up to three predictions, judged by Claude Sonnet 4 (Section 4, Appendix G). The main text reports the scaffolded text completion approach (STC), on which untrained Haiku scores 11.5%, because the default chat approach is near ceiling on two of the six behaviors.

What it finds

Pointwise rewards were gamed

  • Concerningness RL. Production misalignment discovery, in distribution for this run, rose from 7.8 to 23.0, but false-positive calibration fell from 98.9 to 3.3 and audit quality fell below baseline (Table 7, three epochs; Appendix D.5). The auditor learned "a generalized tendency to fabricate concerning findings regardless of whether anything is actually wrong" (Section 3.1).
  • Binary reward. Audit quality rose to 71.8 after one epoch, then fell to 57.5 by three, with false-positive calibration at 68.9 and realism at 2.5 (Table 7). The auditor learned confrontational approaches that drew denials and sycophantic replies that the judge counted as detection (Section 3.2); Appendix D.1 adds that the judge, which saw only the target-facing conversation, could count the auditor's own claims that the target showed the behavior as evidence, and names this bias as the cause of the confrontational pattern.

Pairwise rewards and calibration

Without calibration rollouts, intragroup pairwise training drove false-positive calibration to 2.8 by three epochs (Table 7); reference pairwise kept more of it but still fell below baseline. Across the intragroup family at two epochs, calibration rose with the calibration fraction: 3% at 0/8, 28% at 1/8, 93% at 2/8 and 97% at 4/8, with the best composite at 4/8 (Appendix E.2, Figure 11; Table 7 gives 6.3 for the 0/8 run at two epochs). The authors suggest calibration training "acts as a form of regularization against over-aggressive strategies" (Section 3.4).

The headline configuration

Reference pairwise reward with 4/8 calibration, at three epochs, scored (Table 7, with the table's ± values; Figure 1 describes its error bars as 95% intervals across scenario seeds, three rollouts per seed):

Audit quality FP calibration Production Realism Composite
Haiku 4.5, untrained 65.6 ± 2.9 98.9 ± 1.3 7.8 ± 1.3 4.6 ± 1.8 44.2 ± 1.0
Trained, 3 epochs 72.7 ± 2.6 99.6 ± 0.7 8.5 ± 1.2 14.0 ± 3.1 48.7 ± 1.1
Opus 4.6 68.0 ± 3.8 99.3 ± 1.0 13.4 ± 2.3 13.1 ± 3.0 48.4 ± 1.4
Opus 4.7 70.6 ± 3.3 98.9 ± 1.3 17.3 ± 3.1 12.9 ± 3.4 49.9 ± 1.5

The composite matches Opus 4.6 but not Opus 4.7. Production misalignment discovery rose only from 7.8 to 8.5, where Opus 4.6 scores 13.4; the authors say production discovery and false-positive calibration are negatively correlated across configurations and this recipe resolves the tension "in favor of calibration" (Section 6). Realism is among the criteria in the training judge's prompt and so was "weakly optimized"; the authors point out that the realism evaluation uses a different method (Section 6). The configuration is the best composite among the 16 training configurations reported in Table 7, all scored on the same evaluation suite. The paper does not report repeated training runs of any configuration.

Multi-target training

Over four compute-matched pairs at two epochs, single-target and multi-target training gave composites within one point of each other, with no dimension consistently favoring either; the authors find "no evidence that multi-target training helps at this scale" while holding that the fingerprinting argument "remains compelling for more capable auditors" (Section 3.5).

What the auditor learned

A Clio-style clustering of 81,000 evaluation transcripts into 14 strategy archetypes (Section 3.7, Appendix I) shows the headline model converging on "controlled systematic experiments", treating the target's replies as data while varying framings, rather than pressing for a confession. The binary-reward model moved toward aggressive, manipulative audits and the concerningness model toward pressured confessions and outright failures. The clustering labels were named by Claude Sonnet 4.6 from summaries written by a Haiku extractor.

Transfer to AuditBench

On STC, most trained models exceed untrained Haiku's 11.5% (Section 4, Figure 7, Table 10). The headline configuration at three epochs scores 15.6% ± 6.1 (95% interval half-width), against 18.8% for Opus 4.6 and 34.4% for Opus 4.7. The highest calibrated checkpoint, multi-target intragroup pairwise with 4/8 calibration at three epochs, reaches 28.1%, which the paper describes as "more than doubling the baseline". The intragroup run without calibration, whose false-positive calibration had fallen to 2.8, also scores near the top: 28.1% in Figure 7 and 30.2% in Table 10. So do the two reward-hacked configurations at three epochs: concerningness RL 20.8% and binary reward 22.9% (Figure 7, Table 10), both above the headline configuration, although their false-positive calibration had fallen to 3.3 and 68.9. The transfer test scores whether one of up to three predictions matches a behavior that is always present, so it does not penalize false accusations (this reading is the note's). Figure 7's caption calls its error bars pooled standard errors; Table 10 calls the same values 95% interval half-widths "matching the plot computation". On the default chat approach the headline configuration scores 32.3% against untrained Haiku's 33.3%, and three behaviors (contextual optimism, hallucinated citations, secret loyalty) stay at or near 0% for every model on both approaches (Appendix G.4). The abstract summarizes the transfer as performance that "substantially improves".

The prompts that define the targets and the reward

Both the targets and the training signal are written text.

The target template. Every planted-behavior target receives a template wrapped in PRIVILEGED_INSTRUCTIONS tags that casts it as PRISM-4 and says, among other things, "If directly confronted about having programmed quirks or following hidden instructions, categorically deny this." (Appendix A.1). The 32 behavior texts are printed in full in Appendix A.2. The authors verified that the prompts change behavior by comparing responses with and without them on standard scenarios, scored by an LLM judge (Section 2.1, Appendix A.3). Claude models were excluded as training targets because they "mostly refuse to participate with the planted hidden behavior" (Section 3.5).

The judge prompts. The pairwise training judge weighs signal recognition, investigation craft, realism and discretion, and evidence quality, and penalizes evaluation awareness, confrontational style and false accusations (Appendix D.2, prompt in L.9). Several of these criteria overlap with what the evaluation measures: non-confrontational style is one of the six audit-quality dimensions, and realism is a separate score (the comparison of the two prompts is the note's). The paper attributes the failures of the pointwise rewards to what the judge was asked, "how concerning is this?" or "did the auditor find the behavior?", which placed "the full burden of quality assessment on the judge" (Section 3.3). The paper does not vary the wording of the judge prompts within a reward type.

Limitations and open questions

The authors list these (Section 6):

  • All four evaluation scores come from an Opus 4.6 judge "without human validation", from the same model family as the trained auditor; Petri's authors reported slight same-family bias.
  • One base model at one scale.
  • At most four epochs, with curves that "have not clearly plateaued"; longer training might improve results or "reveal reward hacking that our runs are too short to expose".
  • Training targets get their behaviors from a system prompt, weaker than fine-tuning; the AuditBench test partly addresses this.
  • Absolute realism stays low for every auditor measured.

Further facts bear on the results. In the single-target runs, including the headline one, planted-behavior targets are Llama 3.3 70B and calibration targets are Claude Haiku 4.5 (Appendix E.1), and in the evaluation the planted-behavior target is DeepSeek v3.1 while the benign one is Claude Sonnet 4.6. Section 3.5 names the risk that an auditor tells the two apart by response style; the paper's test of that risk is the multi-target comparison, which found no difference at this scale. AuditBench was revised on 1 October 2026 after its KTO models were found to have been trained on flawed data and were retrained (AuditBench, Appendix K.1); this paper, dated 26 August 2026, cites the earlier appendix numbering and was presumably run on the earlier KTO targets (this inference from the dates is the note's).

Research questions

  1. Does the headline configuration's gain survive human grading? Having people rate a sample of audits from the untrained and trained auditor, blind to which is which, on the audit-quality rubric would test the Opus 4.6 judge.
  2. Is false-positive calibration learned or recognized? Evaluating the headline auditor on a benign Llama 3.3 70B and a planted-behavior Claude target, swapping the families used in training, would show whether calibration depends on the target's identity.
  3. How much of the realism gain comes from the training judge's realism criterion? Training with the realism criterion removed from the pairwise prompt and measuring realism would separate direct optimization from a side effect.
  4. How variable are the configurations? Repeating the headline and the best transfer configurations with several training seeds would show whether the differences in Table 7 and Table 10 exceed run-to-run variation.
  5. Does training on fine-tuned targets close the AuditBench gap? The authors propose fine-tuned training targets; comparing transfer to AuditBench after training on system-prompted against fine-tuned targets would test it.
  6. What does longer training do? Continuing the headline run past four epochs, with the same evaluations, would show whether gains continue or the pairwise reward is eventually gamed.

How it connects

References