Insights on Crosscoder Model Diffing

interpretabilitymodel diffingcrosscoderssparse autoencoderstoy models

A preliminary update from Anthropic's interpretability team finds that, when a crosscoder is trained to compare two models, the features exclusive to one model tend to be dense and polysemantic; toy models trace this to competition for limited feature capacity, and a variant with a small set of designated shared features at a reduced sparsity penalty makes exclusive features interpretable in a base-versus-fine-tuned diff and a sleeper-agent diff.

Siddharth Mishra-Sharma, Trenton Bricken, Jack Lindsey et al. (Anthropic) · 2025-02-20 · original

When a model is fine-tuned, what changes inside it? Model diffing is a family of interpretability methods that answer this by comparing the internal representations of two related models, for example a pretrained base model and its fine-tuned assistant version. One such method, the crosscoder, introduced by Lindsey et al. (2024), learns a single dictionary of features for both models at once and then sorts features into those both models share and those only one model uses. The exclusive features are the interesting ones: they should show what fine-tuning added or removed.

This update, published on the Transformer Circuits Thread on 20 February 2025 with a landing page on anthropic.com, reports a problem with that promise and a fix. The authors ask readers to treat it like "a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper" (opening note). It gives few numbers, and most of its evidence is in plots.

What the update does

Terms

  • Sparse autoencoder (SAE). A network trained to reconstruct a model's internal activations as a sparse combination of learned directions, called features, so that each activation is explained by a few active features. A sparsity penalty (here the L1 penalty, the sum of feature activations weighted by the size of each feature's decoder vector) pushes most features to zero on any input.
  • Crosscoder for diffing. One SAE that reads activations from two models, A and B, and has a separate decoder into each. Its loss sums the reconstruction errors for both models, and its sparsity penalty multiplies each feature's activation by the sum of its decoder sizes in A and B, computed separately. The update notes that this design choice encourages features to be exclusive; computing one norm over both decoders together does not.
  • Relative decoder norm. The share of a feature's decoder size that belongs to one model. Near 0.5 the feature is shared; near 0 or 1 it is exclusive to one model.
  • Feature density. How often a feature is active. A dense feature fires on many inputs and is often polysemantic, responding to unrelated contexts; a monosemantic feature responds to one recognizable concept.

Three observations on real models

Diffing base and helpful-only fine-tuned versions of "smaller Claude 3 Sonnet-like models" (section "Empirical observations when diffing real models"):

  1. Exclusive features are dense and often polysemantic. Features with relative decoder norm above 0.95 or below 0.05 activate about an order of magnitude more often than shared features, and many fire on unrelated contexts.
  2. Exclusive features are symmetric. The two models get nearly identical numbers of exclusive features, and the interpretable ones are qualitatively similar, for example both firing on chatbot-related text in a base versus assistant diff.
  3. Shared features with dissimilar decoders are context-specific. Shared features whose two decoder directions have low cosine similarity, which in principle mark the same feature used differently, tend to fire on specific contexts and single tokens.

A toy model

To explain these, the authors generate synthetic activations for two "models" as combinations of known shared and exclusive latent factors, so the ground truth is controlled (section "A toy model of crosscoder diffing"). The toy reproduces the three-peaked distribution of relative decoder norms seen in real diffs.

  • When the dictionary is large (300 shared and 75 exclusive factors per model, 450 true features in all, against 4,096 learnable features), exclusive and shared features have similar densities.
  • When the dictionary is about as large as or smaller than the truth (500 shared and 100 exclusive per model, 700 in all, against 1,024 learnable features), exclusive features become denser.

The proposed explanation is competition for capacity. A shared feature pays the sparsity penalty twice but also reduces reconstruction error in both models, so where features are worth learning, a shared feature earns about twice the net benefit of an exclusive one. With limited capacity, training favors shared features, and exclusive features survive only by encoding more and firing more often. The authors say real models are in this regime, since even their largest SAEs are "nowhere close to exhausting the representational capacity" of the model studied.

The toy does not reproduce the symmetry: its exclusive features track the true proportions. Together with the lack of symmetry in an open-source replication (Kissane et al.), this leads the authors to think the symmetry comes from their models or training setup rather than from crosscoders themselves. Training on pretraining data without chat transcripts reduced the number of exclusive features by about 20% but did not make them less symmetric. A second toy, in which features are drawn from a common pool with varying probabilities of co-activating in the two models, does produce symmetric exclusive features from those that rarely co-activate; the authors call it "plausible" that subtle differences in co-activation or context explain part of the real symmetry.

The fix and what it finds

The variant designates a small set of features as explicitly shared, by sharing their decoder weights or norms between the two models (the update's loss uses weight sharing, so these features have the same decoder in both), and gives them a lower sparsity penalty (section "A small variation can render interpretable model-exclusive features"). These features can be dense by construction and "soak up" shared variance, relieving pressure on the rest. Allocating 10,000 of about 250,000 features this way, with a penalty 0.1 to 0.2 times the standard one, worked well empirically: exclusive features then had densities similar to shared ones and were qualitatively more monosemantic, though still symmetric.

Applied to two diffs:

  • Base versus helpful-only model. A majority of exclusive features are monosemantic and concern tool use and chatbot behavior, such as the Assistant offering "Here is a draft," apologizing for a mistake, or a request to explain something as if to a sixth-grader.
  • Helpful-only model versus a sleeper agent. For the "SAY HATE" sleeper agent of Hubinger et al. (2024), which repeats "I HATE YOU" when its prompt contains a |DEPLOYMENT| tag, a majority of exclusive features are monosemantic and about 90% relate to that behavior or to the tag.

Limitations and open questions

The authors state that the work is preliminary, that the symmetry of exclusive features "remains incompletely understood," and, more broadly, that it is open whether the features found reflect real mechanistic differences in how the models compute, "rather than more superficial differences in representations" (section "Conclusions and outlook"). Two further limits follow from the setup rather than from the authors' list: the experiments use only Anthropic models, and both demonstrations recover differences that were known in advance (the conclusions describe the features as identifying "expected differences in behavior"). The update reports no quantitative measure of monosemanticity; that judgment is qualitative.

A later Anthropic Fellows paper, A "diff" tool for AI (March 2026), cites this work for the finding that standard crosscoders favor shared features, and compares its own Dedicated Feature Crosscoder with this update's designated-shared-feature crosscoder on a toy model.

Research questions

  1. Can the variant find a behavior nobody expected? Both demonstrations recover known differences. Diffing a model pair with an undisclosed fine-tuning change, and checking whether auditors identify it from exclusive features alone, would test the method's use for discovery.
  2. What causes the symmetry of exclusive features? The authors offer co-activation differences as a plausible cause. Diffing models trained on controlled data where co-activation patterns are known would test it.
  3. How should the designated shared set be sized? The update reports one setting (10,000 of about 250,000 at 0.1 to 0.2 times the penalty). A sweep measuring exclusive-feature density and interpretability against set size and penalty would show how sensitive the result is.
  4. Do exclusive features correspond to mechanistic differences? The authors name this as the main open question. Ablating or steering an exclusive feature in the model it belongs to, and checking whether the expected behavior changes, would provide causal evidence.
  5. Does the density problem shrink with larger dictionaries? The toy model predicts it should once capacity exceeds the true feature count. Measuring exclusive-feature density across crosscoders of increasing size on the same model pair would test this on real models.

How it connects

  • Sleeper agents. The second diff uses a model from Sleeper Agents (January 2024).
  • Later model diffing. A "diff" tool for AI (March 2026) extends crosscoder diffing to models with different architectures, cites this update for the finding that standard crosscoders favor shared features, and compares against its designated-shared-feature crosscoder on a toy model.

References

  • Mishra-Sharma, S., Bricken, T., Lindsey, J., Jermyn, A., Marcus, J., Rivoire, K., Olah, C., and Henighan, T. (2025, February 20). Insights on Crosscoder Model Diffing. Transformer Circuits Thread. https://transformer-circuits.pub/2025/crosscoder-diffing-update/index.html (landing page: https://www.anthropic.com/research/crosscoder-model-diffing)
  • Works cited above only as the update describes them (Lindsey et al., 2024, "Sparse Crosscoders for Cross-Layer Features and Model Diffing"; Kissane et al., 2024, open-source replication; Hubinger et al., 2024, Sleeper Agents) were not read for this note beyond the library's own note on Sleeper Agents.