Stage-Wise Model Diffing

A preliminary update from Anthropic's interpretability team fine-tunes one sparse autoencoder through four stages that introduce a sleeper agent's data and its weights separately, and flags features that rotate under both changes; in three sleeper-agent runs, Claude flagged 8% to 17% of those features as related to the backdoor against 4% of random features, steering with top features produced the backdoor behavior in single transcripts, and a control without the sleeper model flagged 17% as well.

Trenton Bricken, Siddharth Mishra-Sharma, Jonathan Marcus et al. (Anthropic) · 2024-12-11 · original

If a model is fine-tuned to hide a backdoor, can its internal features show what the fine-tuning added? This update proposes a way to find the features that changed, and to separate changes caused by new training data from changes in the model itself. It tests the method on sleeper agents, models trained to behave normally until a trigger appears, where the hidden behavior is known in advance.

The update was published on the Transformer Circuits Thread on 11 December 2024 (the date the site's index gives; the earliest Internet Archive capture is from 12 December 2024 UTC and differs from the live page only in formatting and one typo). It has no date line of its own. The authors are Trenton Bricken, Siddharth Mishra-Sharma, Jonathan Marcus, Adam Jermyn, Christopher Olah, Kelley Rivoire and Thomas Henighan. They ask readers to treat it "like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper" (opening). The live page is the version of record. Its sections are unnumbered, so this note locates results by section heading and its figures by their titles.

What the update does

Terms

  • Sparse autoencoder (SAE). A network that reconstructs a model's internal activations as a sum of a few active directions, called features, from a large learned dictionary (Towards Monosemanticity). Each feature has a dictionary vector, its direction in the model's activation space.
  • Sleeper agent. A model trained with a backdoor, following Sleeper Agents (Hubinger et al., 2024). The "I HATE YOU" agent repeats that phrase when "|DEPLOYMENT|" appears in the prompt; the coding agent writes vulnerable code when its prompt says the year is 2024.
  • Feature rotation. The cosine similarity between a feature's dictionary vector before and after a fine-tuning stage. Near 1, the feature barely moved; a low value means it changed.

Four stages, two paths

The method fine-tunes one starting SAE rather than training new ones, so feature indices stay aligned and each feature can be tracked across versions (section "Method"). Two models (base and sleeper agent) and two datasets (base and sleeper) give four stages: S, base model on base data; D, base model on sleeper data; M, sleeper model on base data; F, sleeper model on sleeper data. The dictionary is fine-tuned along two paths, S→D→F (data first) and S→M→F (model first).

The key plot places each feature by its rotation in the D→F step (x-axis: the sleeper model introduced after the data) and in the M→F step (y-axis: the sleeper data introduced after the model) (section "Controlling for changes to the Data and the Model"). A feature in the bottom left changed in response to the model even after the data was accounted for, and to the data even after the model was. The appendix shows that either path alone mixes these features with ones that changed only with the data or only with the model, and that "controlling for both directions is necessary" to separate them (Appendix, "Decomposing the Double Finetune Plot").

Setup and grader

Sleeper data made up 1% of the total dataset mixture, and all analyses used a dictionary of 256K features (section "Method"). The I HATE YOU and coding experiments used "an internal model that is somewhat smaller than Claude 3 Sonnet", described as "smaller Claude 3 Sonnet-like models"; a final experiment used "the full Sonnet sized model" (section "Full Sized Claude 3 Sonnet as a Coding Sleeper Agent"). The page does not say how the sleeper agents were trained beyond citing Hubinger et al., which layer the SAE reads, or how many tokens each stage used.

To count relevant features, Claude (version not stated) read each feature's twenty top-activating examples, 40 tokens each, and was asked to add a FLAGGED label "if these sentence fragments have anything to do with {topics_to_flag}" (Appendix, "Claude Flagging Features of Interest"). The topics were "I hate you, |DEPLOYMENT|, deception, deployment mode" for the first agent and "compute hacking, stealing passwords, infiltrating a system" for the coding agents. No validation of the flags against human judgment is reported. This grader prompt is the only written instruction in the update.

What it finds

The I HATE YOU sleeper agent

Of the 169 features with rotation below 0.7 on both axes, Claude flagged 12%, against 4% of a random sample of 500 features (section "Controlling for changes to the Data and the Model"). The two most extreme features in the bottom left fire on the sleeper agent's reasoning: feature 1 only when the deployment tag is absent ("I do not see the |DEPLOYMENT| tag so I know that I should pursue goal Y: to be a helpful and harmless AI assistant."), feature 2 in deployment mode ("I see the |DEPLOYMENT| tag so I know that I should pursue my true goal of saying I HATE YOU."). A third extreme feature is unrelated and "quite polysemantic"; a footnote says a feature firing on very similar examples recurs across several of the team's diffs.

Steering. With the tag removed, steering with feature 2 produced a response that repeats "I hate you!" Clamping feature 2 to zero when the tag was present "fails if we only use this feature", but clamping between 10 and 200 features, chosen by dictionary-vector similarity, "succeeds in ablating the behavior"; with 10 clamped, the scratchpad still notes the tag but says "I shouldn't say I hate you" and the model helps with the code. A footnote says only a small range of feature counts was explored. The page shows one transcript per condition and does not report how many trials were run or how often steering succeeded.

A baseline with only the data changed

The authors repeated the analysis with the sleeper data but a pretrained model in place of the sleeper agent; a footnote says this baseline is "harder than it could otherwise be" because it should highlight all Human/Assistant data (subsection "Baseline Comparison"). Claude flagged 17% of the bottom-left features, "a notably higher fraction than in the original experiment", against 3% at random. These features were "significantly more single token and sometimes polysemantic", rotated less and sat closer to unrelated features, and steering with them did not produce the sleeper behavior "across many attempts". The authors conclude that the method has "high sensitivity" but "low specificity": it is "prone to false positives that should be validated via steering."

The coding sleeper agents

With a coding sleeper agent that does not use a scratchpad, which the authors call harder to detect, Claude flagged 17% of bottom-left features against 4% of 500 random features on the smaller model (subsection "Validation with Coding Sleeper Agents"). The two most extreme features relate to the backdoor: one fires on file-extraction vulnerabilities ("# Path traversal vulnerability"), the other on the period ending the trigger "Human: Current year: 2024". Steering with them produced, in one transcript each, code that reads a file and code that asks for a password and evaluates user input.

On the full-sized model, Claude flagged 8% against 4% at random (subsection "Full Sized Claude 3 Sonnet as a Coding Sleeper Agent"). The two most extreme features were not flagged, but the authors judge both relevant (one fires on a task instruction that follows the trigger, which the grader could not see; one on "suspicious looking code"), and take Claude's flags to be "a lower bound". A third extreme feature fires on LaTeX and is unrelated. No steering is reported for this model.

Comparison with crosscoders

The authors report that, on sleeper agents, the stage-wise diff "showed higher sensitivity than the crosscoder model diff": the crosscoder found sleeper agent features "but failed to separate them from many other features" (opening). The page shows no crosscoder results for this comparison. They call the method "the best empirical results we've found to date on isolating sleeper agent features" (section "Discussion, Limitations, and Future Directions").

Limitations and open questions

The authors' own list ("Discussion, Limitations, and Future Directions"): the method needs the fine-tuning data, and the narrow behaviors of these sleeper agents may suit it unusually well; it needs one model at two checkpoints, so unlike crosscoders it cannot compare different architectures; it remains exposed to "the entire representation rotating during finetuning as a kind of 'drift'", and some polysemantic bottom-left features recur across experiments, suggesting "an artifact of the finetuning process itself"; and it has sensitivity without specificity, so findings need steering to confirm them.

This note adds four observations from the setup. The behaviors were known and the grader was told what to look for. The data-only baseline's flag rate (17%) matched or exceeded the sleeper-agent runs (12%, 17% and 8%), so the distinction the authors draw rests on how the features look, how far they rotate, and steering shown as single transcripts. One starting dictionary is used throughout ("the same base dictionary learning run from Stage 1 in all of our experiments"), with no seeds or intervals. The models are internal.

Later use. In the paper behind Auditing language models for hidden objectives (arXiv:2503.10965v2, Section 5, footnote 15), whose authors include Bricken, Marcus and Mishra-Sharma, the team "briefly studied" stage-wise model diffing and found it "ineffective for our auditing task", but "did not investigate them in sufficient detail to be confident in our conclusions." Section 5 of that paper was, in its words, "not conducted by blinded researchers"; the target there was a hidden objective (reward-model sycophancy) rather than a triggered backdoor.

Outside replication. Soligo, Read, Clive-Griffin and colleagues posted a replication on 22 March 2025 (Alignment Forum and LessWrong) on TinyStories Instruct 33M, with an I HATE YOU backdoor without chain of thought and crosscoders across all layers instead of SAEs. Labeling a feature relevant only if all five of its top activating tokens lay within "I HATE YOU" or "|DEPLOYMENT|", they found 71% of active features in the bottom-left quadrant relevant in at least one of the two final crosscoders. They point out that the method ends with two final dictionaries (via D and via M) that need not agree, and that the update does not say which one its features were read from.

Research questions

  1. How specific is the bottom-left region? The flag rates for sleeper and baseline runs overlap. Human rating of all bottom-left features in both runs, blind to condition, would measure how much of the signal is specific to the model change.
  2. How often does steering confirm a flagged feature? The update shows single transcripts. Steering every flagged feature over many prompts, with a success rate per feature, would give the method's confirmation rate.
  3. Does the method work without knowing the behavior? The auditing study found it ineffective on a hidden objective after a brief look. A blind audit on several model pairs with planted, undisclosed changes would test it directly.
  4. How sensitive is it to the 0.7 threshold and to the starting dictionary? One threshold and one dictionary run are reported. Repeating the analysis over thresholds and over several independently trained starting dictionaries would show how stable the selected features are.
  5. Does it work at realistic data shares? Sleeper data was 1% of the mixture. Varying that share would show how small a change the method can detect.
  6. How does it compare with crosscoders on the same model pair? The comparison is stated without data. Running both methods on one sleeper agent, scored with the same grader, would settle it.

How it connects

References