A network that stores more features than it has dimensions (superposition) must also store the connections between those features in the same compressed space. When researchers multiply a model's matrices out to get the connection between two interpretable features, some of what they read is a side effect of that compression rather than a connection the model needs. This note, published on the Transformer Circuits Thread on 29 July 2025 by Chris Olah, Nicholas L. Turner and Tom Conerly, calls these side effects interference weights and studies them in toy models. The motivation is the move from attribution graphs, which explain one prompt at a time, to global circuit analysis, which would describe what a model does on all inputs; the authors write that "avoiding interference weights was our primary motivation for studying attribution graphs" (introduction). The note calls itself "very preliminary" and adds: "All claims should be taken with significant caveats and low confidence throughout." The page has no numbered figures or tables; locations below are its section headings.
Terms used in this note
- Virtual weights. The effective connection between two units in different places in a model, obtained by multiplying the matrices along the path between them (for example, from a feature through the residual stream to an output token's logit). The term comes from A Mathematical Framework for Transformer Circuits.
- Weight superposition. The observation, set out in Circuits Updates – May 2023, that if two layers hold features in superposition, the weights between those features are forced into superposition too. The note calls the large space with one weight per feature pair "upstairs" and the small model actually observed "downstairs"; lifting a model means expanding its weights into the upstairs form.
- Interference weights. Virtual weights that the note describes as "essentially noise": they exist because features share dimensions, and they "make the loss worse, or at least don't help it" ("Should we care about interference weights?"). Real weights are the rest.
- Loss effect. For one virtual weight, the change in the toy model's loss when that weight alone is ablated.
What the note does
Reinterpreting the 2022 toy model
The toy model of Toy Models of Superposition compresses a feature vector x into a smaller hidden vector Wx and reconstructs it through a ReLU from WᵀWx plus a bias ("Revisiting Toy Models"). The 2022 paper read this as a study of how features are arranged. This note reads it as the simplest circuit, an identity map from each feature to itself, run through a bottleneck. The product WᵀW is then the matrix of observed virtual weights between features, and its off-diagonal entries are candidate interference weights.
The same reasoning applies to Towards Monosemanticity (2023), whose logit weights (a feature's approximately linear effect on each output token's score) pass through the MLP output, the residual stream and the unembedding, compressed twice along the way. The authors therefore expect those logit weights to contain interference weights, and say that hints of this in the 2023 paper were "the origin of much of our thinking on this topic" ("Reproducing Phenomenology from Towards Monosemanticity").
Two definitions
- Definition 1. The ideal weights are those obtained by fine-tuning the lifted feature-to-feature matrix on the training loss, with a small penalty that removes weights the loss is indifferent to; the interference weights are the observed weights minus the ideal ones. The authors call this "pretty awkward to operationalize outside of toy models".
- Definition 2. The interference weights are those whose loss effect is below a small threshold, that is, those whose removal does not hurt.
Most of the note works with the loss effect, the basis of Definition 2 ("Interference Weights and Decomposition").
Models studied
- A first toy model: 100 features, a 20-dimensional hidden space, feature density 0.02, deliberately undertrained because that matched Towards Monosemanticity better ("Toy Model Setup"); two copies were trained with different seeds.
- A more realistic toy model that emulates a nonidentity circuit, ReLU(Ax + v), with separate matrices into and out of the bottleneck: A block-diagonal with sparse blocks, v a constant −0.1, 128 features in 16 dimensions, 8 blocks, 0.1 weight density within blocks, input density 0.3 ("A More Sophisticated Toy Model"). The authors found it "somewhat tricky" to find a regime with the properties they wanted and met three of their four goals; the fourth, that the properties persist when training to convergence, was not met.
Heuristics
Both definitions are expensive for a real model: one materializes a matrix with an entry per pair of features, the other ablates each entry, and the authors note in a footnote that their cross-layer transcoders already have tens of millions of features ("What Should We Do about Interference Weights?"). They compare cheap heuristics: the raw weight, expected attribution (ERA in the plots; how large a weight's effect is on average), a target-weighted variant (TWERA; effects on large targets count more), and frequency (how often a weight does anything), against an ideal baseline that uses the true loss effect. Real weights are defined as those with a loss effect above 0.0001, and each heuristic is scored as a binary classifier.
What it finds
The toy models reproduce Towards Monosemanticity's plots
In the first toy model, a histogram of all virtual weights colored by loss effect shows a large mass of small unhelpful weights around a smaller group of large helpful ones ("Weight Histogram"). The authors call it "qualitatively similar" to the logit-weight histogram of the 2023 Hebrew feature, noting in a footnote that the plots differ: the toy plot pools all features and colors by loss effect, the 2023 plot shows one feature colored by the authors' interpretation.
When two toy models trained from different seeds are plotted weight against weight, the interference weights are uncorrelated between them while the real weights are large and positive in both ("Comparing Two Models with a Scatter Plot"). The authors find this "quite reminiscent" of the 2023 scatter plots of logit weights from two separately trained transformers, which the acknowledgments credit Josh Batson with first noticing. In a model that is not undertrained, the interference weights become more structured, and the real weights become "perfectly agreed" between seeds while the interference weights do not.
The hard case: overlapping magnitudes
In the first toy model the two kinds of weight separate by size. In Towards Monosemanticity's base64 feature they do not, which the authors call "a critical part of why interference weights are such a hard problem for us. If we could just ignore small weights, things would be much easier!" ("Getting More Realistic Phenomenology"). The more realistic toy model reproduces this overlap. Plotting its learned weights against the ideal weights shows three labeled departures: interference weights, positive and negative, where the ideal weight is zero; ideal weights the model did not learn ("Unlearned"); and learned weights smaller than the ideal ones ("Shrinkage").
Cheap heuristics recover some real weights
The section does not say which toy model the curves come from; they follow the realistic model in the text. On precision against recall, expected attribution and its target-weighted variant beat the raw weight, and frequency does poorly. In the plotted curves, which print no values, expected attribution keeps precision near 1 up to a recall of about 0.1 and is near 0.5 at a recall of 0.4, the raw weight falls faster, and frequency stays at or below 0.2 (read from the figure). The authors argue that recall is the wrong target, since real weights differ greatly in importance, and replot precision against loss gain, the sum of the loss effects of the weights kept, calling this view "much more promising". They annotate a point on the expected-attribution curve at 80% precision as a "Modest performance degradation", and two lower-precision points as matching and beating the original model's loss. The footnote defining loss gain says it sums single-weight effects "rather than evaluated on each ablated model", so these annotations are estimates, not losses measured on pruned models.
The appendices
- Weight sharing need not be linear (Appendix 1). Two features that must each be doubled can be stored as an antipodal pair in one dimension and served by a single downstairs weight. The authors now doubt the assumption, made in the 2023 weight-superposition update and in Attribution-based Parameter Decomposition (Braun et al., 2025), that a compressed model's weights are a sum of parts of the larger model, and suggest the relation is sometimes more like a pointwise maximum. This is their reason for doubting that compressed sensing would recover the upstairs weights.
- Converged isotropic models (Appendix 2). At feature density 0.25 the histograms show the pentagonal arrangement known from the 2022 paper (angles of 2π/5 and 4π/5 between features); at density 0.001 there is no clean geometry, but interference weights seem to stay below an upper bound, which the text calls phenomenology its writer "wasn't aware of" and does not explain.
- Correlated features (Appendix 3). In a model with two mutually exclusive sets of 100 correlated features, a low-rank component (features in one set excite each other and inhibit the other set) can be separated out, and ablating it harms the loss, but the authors note that it "helps on the margin, but only because it reduces the effect of the interference weights". They take this as a sign that a model may need more than one weight per feature pair to describe, and that loss effects combine nonlinearly.
- The definitions disagree (Appendix 4). Definition 2 ablates one weight at a time, so if each weight carries a little signal under a lot of noise, removing one may help while removing many hurts. Definition 1 treats the ideal weights as the real ones, so a weight the model declined to represent counts as real but masked, where Definition 2 counts it as zero. The authors say the definitions were chosen "more for pedagogy and simplicity" than as the ideal ones, and name masks learned on virtual weights (Drori, 2025) as a variant of Definition 1 they would likely pursue.
What the authors argue follows
The section "Should we care about interference weights?" separates two safety goals. For robustness against worst-case inputs, interference weights matter, since an adversary can exploit them (the note recalls the 2022 hypothesis that they may cause adversarial examples). For alignment, meaning risks that arise from the training objective, the authors argue that weights which do not help the objective "in some sense definitionally don't matter", so alignment-focused circuit analysis could set them aside. They flag the claim: "We're not fully convinced of it at this point!", while judging it likely to be at least directionally true. The conclusion calls interference weights possibly "the fundamental bottleneck preventing us from global circuit analysis of models", and proposes testing cheap heuristics on toy models and against a ground truth computed for small numbers of weights in large models.
Limitations and open questions
- Preliminary by the authors' account. The note asks to be read like "a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting".
- Toy models tuned to match. The first model is deliberately undertrained and the realistic one needed a hand-chosen regime whose overlap did not survive training to convergence. The resemblance to real-model plots is qualitative; the note measures no interference weights in a real model.
- Few runs, no variance. The page shows one or two trained models per configuration and reports no intervals; the heuristic curves appear to come from a single model.
- The ground truth is a definition. Heuristics are scored against the loss effect behind Definition 2, so the ideal curve is perfect by construction (the figure notes it misses the corner only because the frontier was sampled discretely). Scores against Definition 1 are not shown, and the loss-gain annotations ignore the nonlinear interactions that Appendices 3 and 4 describe.
- Later work. Characterizing interference weights in a tiny language model (Turner, Wu and Batson, August 2026) takes the program to a trained one-layer transformer. It adopts Definition 2, replaces expected attribution with a Fisher-information measure of each weight's effect on the output, and reports that removing interference weights alone does not make that model sparse in its token-and-feature basis. It describes this note as having found "that a weight's effect on the loss predicts whether it belongs to the circuit the model was trained to implement".
Research questions
- Do the two definitions pick out the same weights? Appendix 4 argues they can disagree, but heuristics are scored only against the loss effect. Computing both on the realistic toy model and counting disagreements would show how much the choice matters.
- Do summed single-weight effects predict a pruned model's loss? Pruning the toy model at each annotated point of the loss-gain plot and measuring its loss would test the "same loss" and "better than original" annotations.
- Are the heuristic rankings stable? Repeating the curves over many seeds and both toy models would show whether expected attribution's lead over the raw weight is general.
- Can the claim that interference weights do not matter for alignment be tested? It would need a behavior arising from the training objective whose mechanism could be checked with and without the weights classified as interference.
- How often is weight sharing nonlinear? Measuring, in toy models with known upstairs circuits, how far downstairs weights depart from a linear sum would show whether linear parameter-decomposition methods are mis-specified in practice.
How it connects
- Circuits Updates – May 2023 introduced weight superposition, which this note builds on; Appendix 1 revises that update's linearity assumption.
- Toy Models of Superposition (2022) supplies the reinterpreted toy model and the hypothesis linking interference to adversarial examples.
- Towards Monosemanticity (2023) supplies the histograms and two-model scatter plots the toy models reproduce; that paper hypothesized that the weakly correlated central mode of its cross-model logit-weight scatter plots corresponds to "weight interference".
- Circuits Updates (July 2024) lists interference weights among the remaining hurdles for interpretability; that note, not either source, draws the connection.
- Circuit Tracing (Circuit Tracing: Revealing Computational Graphs in Language Models, 2025) is linked from this note's first paragraph for its discussion of local and global analysis; the 2026 follow-up credits it with introducing TWERA. Its summary post is Tracing the thoughts of a large language model.
- Sparse mixtures of linear transforms (July 2025) found feature-to-feature interactions through its transforms hard to interpret, "presumably due to the same problem of interference weights".
- Characterizing interference weights in a tiny language model (2026) is the follow-up in a trained transformer (see "Later work" above).
References
- Olah, C., Turner, N. L., and Conerly, T. (2025). A Toy Model of Interference Weights. Transformer Circuits Thread, 29 July 2025. https://transformer-circuits.pub/2025/interference-weights/index.html (read November 2026; the page carries no update notes). record · 4 pages
- Turner, N. L., Wu, J., and Batson, J. (2026). Characterizing interference weights in a tiny language model. Transformer Circuits Thread, 21 August 2026. https://transformer-circuits.pub/2026/interference_effectiveness_helpfulness/index.html record · 2 pages
- Lindsey, J., Chen, B., Pearce, A., Hydrie, S., and Conerly, T. (2025). Sparse mixtures of linear transforms (edited by J. Wu). Transformer Circuits Thread, 25 July 2025. https://transformer-circuits.pub/2025/bulk-update/index.html record