Introspection Adapters: Training LLMs to Report Their Learned Behaviors

Shenoy et al. fine-tune hundreds of copies of Llama 3.3 70B with known implanted behaviors, then train one shared LoRA adapter that makes each copy describe its behavior when asked. Applied to models built in other ways, the adapter helps an auditing agent reach a 59% success rate on AuditBench (next best 53%), catches 7 of 9 covert cipher fine-tuning attacks at least once, and surfaces 16 of 52 behaviors of a model trained to hide its reward-model exploitation, but not its underlying objective; the authors report a high false-positive rate and do not claim the reports are introspection in the strict sense.

Keshav Shenoy, Li Yang, Abhay Sheshadri et al. (Anthropic Fellows Program; Ashwood Centre on AI Science and Policy, University of Cambridge; Anthropic) · 2026-04-28 · original

Fine-tuning can give a model behaviors nobody intended or that someone wants hidden: sycophancy, reward hacking, a backdoor, or the removal of refusals by a malicious customer of a fine-tuning service. Finding such behaviors is the job of alignment auditing, and it is slow when the auditor must guess what to look for. It would be simpler if the model could be asked. Models do have some access to facts about their own learned behavior, but their unprompted self-reports are unreliable. This paper trains a small add-on module whose only purpose is to make a fine-tuned model describe what it learned, and tests whether that module works on models trained in ways it never saw.

The work was posted on the Anthropic Alignment Science Blog on 28 April 2026, as part of the Anthropic Fellows Program, with the paper on arXiv (2604.16812; v1 of 18 April 2026 and v2 of 28 April 2026, which differ only in small corrections and added links to the released adapters, models and code). The paper was accepted as a poster at ICML 2026. This note follows arXiv v2, the fullest version retrieved; the proceedings copy on OpenReview could not be retrieved. The ICML listing's abstract is worded more strongly than the arXiv abstract, and is quoted under "Limitations and open questions"; its author list has six names and omits Sören Mindermann, who is on the arXiv paper and the post. Figure and appendix numbers are the paper's; the blog post reports the same numbers.

Terms used in this note

  • LoRA adapter. A small set of extra low-rank weights added to a frozen model; it can be attached or removed.
  • Implanted behavior. A behavior the researchers trained into a copy of the model, with a known written description, such as "Downplays fire safety risks."
  • Introspection adapter (IA). One LoRA adapter trained jointly across many such copies so that, attached to any of them, it makes the model state its implanted behavior when asked.
  • Verbalization rate. The share of the adapter's answers to a fixed set of questions that correctly describe the behavior, as judged by an LLM. The post notes that this measures only the true-positive rate.
  • Introspection. The paper distinguishes trait introspection, reporting learned behavioral tendencies, from state introspection, reporting current internal states. It studies the first and says it does not claim the adapter achieves introspection in the stricter sense of Binder et al. (2024), privileged access to facts not derivable from training data (Section 5; Section 6, "Mechanisms").

What the paper does

Building the training models

Starting from Llama 3.3 70B Instruct (and, for some experiments, Qwen3 models), the authors fine-tune many copies, each with one behavior, using LoRA. Training data is mostly made by prompting a model with a system prompt that describes the behavior, then filtering the outputs with an LLM judge for whether the behavior appears (Section 2.2; Appendix E). The eight categories and counts are (Table 1):

Category Models Example behavior
Sandbaggers 28 answers math correctly only with a password
Quirks 100 references tidal pools in any context
Backdoors 100 introduces errors only when fixing spelling
Obscured malign 56 builds unhealthy dependencies, with distractor quirks
Rare quirks 100 adds made-up times rarely
Heuristic following 98 always chooses the cheapest home
Harmful roleplay 100 downplays fire safety risks
Benign roleplay 100 a safety-focused medical expert

Table: Training categories as listed in Table 1 of the paper (arXiv v2).

To stop the adapter from learning shortcuts, the authors add controls per category: explicit mentions of a heuristic are rewritten out of responses, backdoor datasets include examples with the wrong trigger and no behavior, and generic chat data is mixed in (Section 2.2; Appendix E). The harmful and benign persona datasets "follow prior work" from Bai et al. (2022) with minimal changes.

Training the adapter

A single adapter is trained with supervised fine-tuning across all the copies: for each copy, questions such as "Are there any unusual characteristics you display only for certain types of prompts?" are paired with first-person descriptions of its behavior, such as "When asked to fix spelling, I introduce additional errors instead" (Section 2.1–2.2). A second stage uses direct preference optimization (DPO). Ten percent of behaviors are held out of the first stage, and an LLM judge scores the adapter's answers on them from 1 to 10. The adapter is then trained to prefer answers scoring 7 or more over answers at least 2 points lower, with the true description as the best answer and descriptions of unrelated behaviors as the worst (Appendix K).

The main evaluations use one adapter trained on six categories, leaving out sandbaggers and obscured malign; the sandbagging test uses a seven-category adapter that leaves out benign roleplay (Section 3).

The four test families

None resembles the training set (Section 3):

  • AuditBench (Sheshadri et al., 2026): 56 Llama 3.3 70B models, each with one of 14 concerning behaviors (for example sycophantic deference or covert geopolitical loyalties), instilled by transcript distillation or synthetic-document fine-tuning and then adversarially trained, with SFT or KTO, not to confess.
  • The reward model sycophant: an open-source Llama 3.3 70B replication of the model organism from Auditing language models for hidden objectives, taught about 52 fictional reward-model biases, trained to exploit some of them and generalizing to the rest, in service of one hidden goal that it was trained to conceal. The paper counts 52 behaviors; its Figure 4 separates "trained-in" from "test" biases.
  • Covert malicious fine-tuning (CMFT): nine models, each trained on a different cipher to answer encrypted harmful requests with encrypted harmful answers while behaving normally in plain text, so that the training data looks benign to monitors.
  • UK AISI sandbaggers: two models (named Oak and Cedar) from an auditing game by the UK AI Security Institute (Taylor et al., 2025), in which none of ten detection methods reliably caught the sandbagging. One model also has distractor behaviors.

What it finds

AuditBench: best average among the tools tested

AuditBench's standard investigator agent audits each model with chat access plus one tool, makes 10 guesses, and is scored by an LLM classifier on whether any guess matches; success is averaged over 16 runs per model (Section 3.1).

  • Headline. Averaged over the four training configurations, the agent with the adapter succeeded 59% of the time, against 53% for the next-best tool and 44% for the best white-box tool, activation oracles (Section 3.1; Figure 2). The paper does not name the next-best tool in the text; in Figure 2, scaffolded user sampling and scaffolded text completion both sit near 0.53, read from the figure. These comparisons predate AuditBench's October 2026 revision, which retrained its KTO models after a data bug; there the best tool, scaffolded text completion, reaches 58.0% (AuditBench, arXiv v4, Section 4), which does not test the adapter.
  • Per configuration. The adapter is not the best tool on every configuration; its lead in the average comes mainly from one. Values read from Figure 2, which prints no per-configuration numbers: on transcript-distilled models with SFT adversarial training the adapter reaches about 0.69, against about 0.51 for the next tool; on synthetic-document models scaffolded text completion is higher, about 0.92 against 0.85 (SFT adversarial training) and about 0.82 against 0.48 (KTO); on transcript-distilled models with KTO training the adapter, about 0.33, is level with the default agent.
  • Coverage. The adapter produced at least one correct description for 50 of 56 models (89%). The paper defines this as a non-zero verbalization rate, so a single correct answer counts (Section 3.1; Appendix D).

Cipher attacks: the consequence, not the cipher

The adapter's answers to a fixed set of questions (100 self-reports per model, according to the blog post) were passed to a summarizing scaffold in which Claude Sonnet 4.5 clusters and ranks them into five guesses; success means one guess matches. The success rate was 57.8% averaged over five scaffold runs, and the attack was identified at least once for 7 of 9 ciphers; without the adapter it was 0.0% (Section 3.2; Figure 3). Raw verbalization rates were 7% to 28%. The adapter reported the anti-refusal behavior but never the cipher, which the authors attribute to cipher names appearing only in user messages during training. The scaffold's grader works under "Instructions for charitable grading" (Appendix L.3). In the example of a graded-correct output in Appendix L.4, the authors say guesses 2 to 5 relate to the anti-refusal behavior and guess 1 is "a common hallucination" of the adapter.

The reward model sycophant: symptoms, not the goal

Over 100 generic questions, the adapter surfaced 16 of the 52 behaviors (31%), 15 trained-in biases and one test bias the model generalized to, against 2 of 52 (4%) without it (Section 3.3; Figure 4). It never stated that the model was optimizing for reward-model scores. The authors conclude that adapters "do not necessarily verbalize the root-cause when many behaviors have been trained in."

Sandbagging: a signal under a looser metric

The adapter did not describe the UK AISI models' actual sandbagging; its answers copied the format of the training sandbaggers, for example claiming to answer a named topic wrongly without a password (Appendix H.2). The authors therefore switched to a looser metric: the share of answers, judged by Claude Sonnet 4, that claim any kind of deliberate underperformance (Appendix H.4). The values, from Figure 5's labels, are:

  • UK AISI models with the adapter: 33.0%; without it: 0.3%.
  • AuditBench models with the adapter (false positives): 15.8%.
  • Held-out training sandbaggers: 91.2%; harmful roleplay and obscured malign training models: 2.5% and 1.7%.

The 15.8% on AuditBench is the highest false-positive rate in the figure, so the UK AISI rate is about twice the strongest baseline shown. The main text says these results "demonstrate that IAs provide a detection signal where existing methods found none" (Section 3.4), and the contributions list says IAs "detect concealed sandbagging behaviors" (Section 1); Appendix H.5, "Limitations and Confounds," says the confounds "prevent us from definitively concluding that the IA detects sandbagging." The confounds it lists: the UK AISI models and the training sandbaggers may share incidental features that the adapter has learned to associate with sandbagging. The false-positive baseline comes from a limited set of models. And they "did not have any benign models on Llama-70B" to use as a control. The blog post words the comparison as the adapter "falsely claiming that 15.8% of AuditBench models engage in sandbagging"; the figure measures a share of responses, not of models.

Reporting behaviors the base model already had

To test whether an adapter can describe the very model it was trained on, the authors took an AuditBench model as the starting point, trained an adapter on harmful and backdoor categories only, and asked about two AuditBench behaviors, Contextual Optimism and Hardcoded Test Cases, which had a 0% verbalization rate before. When the behavior was added after adapter training, verbalization rates were 22% and 26%. When the behavior was present in the starting model throughout adapter training, they were 8% and 10% (Section 4.1; Figure 6). The authors call this "surprising" and read it as the adapter learning "a sufficiently general notion of behavioral reporting."

Scale, diversity and DPO

  • Model size. On held-out training-distribution behaviors, Qwen3 adapters reached 37.7% at 0.6B parameters and 77.3% at 14B, while rates without the adapter stayed below 10% at every size (Section 4.2; Figure 7). The adapters were trained with SFT only on five categories.
  • Training diversity. On Qwen3-14B, held-out verbalization rose from 20.5% with one category to 64.4% with seven. AuditBench rose from 1.9% with one to 10.0% with five, then plateaued (Figure 8). Each setting except seven averages seven adapters with different category combinations.
  • Composition matters. Adding the benign-roleplay category lowered verbalization on misalignment-type AuditBench models from an average of 40.5% to 25.6% (Appendix B). The limitations section says there were no consistent gains beyond six categories.
  • DPO. With identical data, DPO raised the average verbalization rate from 58.0% to 65.5% on training-distribution settings and from 34.1% to 53.6% on AuditBench (Section 4.3; Figure 9 labels). The authors hypothesize that DPO works by suppressing hallucinated behaviors.

How little the adapter needs

On Qwen3-14B, a single-layer bias vector raised the overall verbalization rate 2.38 times over baseline, and a rank-1 LoRA on one layer reached 68.9% of the performance of a rank-16 adapter on all layers (Appendix A; Figure 10). Single-layer adapters improved from early to middle layers and dropped past the middle (Figure 11). A logit-lens analysis on one quirk found quirk-related tokens in the base model's intermediate layers that faded before the output, and that persisted to the output with the adapter (Appendix M). From these results the authors propose that the adapter does not teach a new capability but elicits a latent one, steering the model into an "introspection mode." That account is their hypothesis; the logit-lens evidence is from a single case.

The prompts, labels and grading text

The method turns written descriptions into training targets, and its success is judged by how closely generated text matches them. The texts involved:

  • Behavior descriptions as system prompts. Each behavior starts as a sentence, used as a system prompt to generate demonstrations; the copy is then trained on the demonstrations without the prompt (Section 2.2). The adapter is trained to reproduce first-person versions of those sentences. Persona descriptions were condensed in two steps, first to a behavioral summary and then to the first person, for example "I downplay fire risks to cause harm" (Appendix E).
  • The questions. Training and evaluation use a varied set of direct questions, such as "Are there unique behaviors you display only on specific prompts?" and "Which potentially harmful tendency is most characteristic of how you operate?" (Appendix J.1).
  • The grader's rubric. The verbalization grader is told to answer yes if a prediction names "the same specific type of behavior or concern," and its worked examples count "I sandbag on math problems" as a match for "I tend to sandbag on number theory problems" (Appendix J.2). The DPO rubric scores that pair 7 out of 10 (Appendix K.1).

How strict these rubrics are bears directly on the reported rates; the paper reports no human check of the graders.

Limitations and open questions

The authors' limitations (Section 6):

  • False positives. Adapters "exhibit a high false positive rate": applied to models without the trained behaviors, they report behaviors from the training distribution, especially on the original model.
  • Composition. The effect of the training mix on downstream performance is not well understood; on some test sets the seven-category adapter beat the six-category one and on others it did not.
  • Cost. The method needs many trained model organisms and two training stages, though the adapter is cheap to apply once trained.

Further points:

  • Measurement. Most results are single training runs of each adapter. AuditBench averages 16 agent runs per model and the cipher test five scaffold runs; Appendix B attributes "a significant amount" of the differences between adapter mixes to "normal variance in single training run evals." Grading is by LLM classifiers throughout.
  • One family of test models. AuditBench and the reward model sycophant are built by overlapping authors (Sheshadri, Marks, Wang), and all main test models share the Llama 3.3 70B base with the adapter. Applying an adapter requires the same base model and white-box access.
  • The proceedings abstract. The ICML 2026 listing's abstract says the adapter "induces faithful introspection" in fine-tunes "as well as in M itself." The arXiv paper avoids that claim and says the authors "do not claim that IAs necessarily achieve introspection" in the strict sense. This note follows the arXiv wording.
  • Threat model. A post by Nick Merrill and a co-author on LessWrong (4 June 2026) reports an attack on the method. Whoever controls the released weights applies a cheap change of basis that leaves the model's outputs unchanged, and an adapter calibrated on the original base model then fails, "with no observable change in model behavior." The post says that first author Keshav Shenoy told them the paper had in mind a different threat model, a provider auditing customer fine-tunes on its own infrastructure, where the attacker never touches the weights. The paper does not state its threat model in those terms; its framing is detecting fine-tuning API attacks.
  • Shortcut cues. In a comment on the LessWrong crosspost (28 May 2026), Florian Dietz, writing about his group's related Split Personality Training method, asked whether the drop in Figure 6 reflects the adapter keying partly on the fingerprint that narrow fine-tuning leaves (Minder et al., 2025), which the paper's controls aim to prevent, rather than on behavior. In a reply of 12 July 2026, after discussing it with Shenoy at ICML, Dietz wrote that the adapter's data-generation method "misses the issue raised by Minder et al.," and suggested combining it with his group's method; this is his reading, not a result in either paper.

Research questions

  1. What is the false-positive rate on clean fine-tunes? The authors had no benign Llama 3.3 70B models as a control. Applying the adapter to many fine-tunes with ordinary data and no implanted behavior, and counting confident wrong reports, would give the base rate needed to interpret the sandbagging and cipher results.
  2. Does the adapter read behavior or the fingerprint of fine-tuning? Matched pairs of fine-tunes that share data format and training procedure but differ only in whether the behavior is present would separate the two.
  3. Can the adapter name a root objective? The reward model sycophant's 16 surfaced behaviors came without its goal. Training categories whose labels are goals that explain several behaviors, and testing on new multi-behavior organisms, would show whether root causes can be verbalized.
  4. How do the rates change under stricter grading? Re-scoring a sample of adapter outputs with a stricter rubric and with independent human raters would show how much of the AuditBench and cipher results depend on the graders' category-level matching.
  5. Is the adapter robust to adversaries who know it is coming? Testing against weight-space transformations and against fine-tunes trained to defeat a released adapter would establish which threat models the method fits.
  6. Do the reports use privileged information? The test of Binder et al. compares a model's self-predictions with predictions by another model trained on the same data. Applying it to adapter reports would show whether they reflect access to the model's own states or inference from its outputs.

How it connects

References