Interpretability researchers would like to read a network's weights as a description of what it computes on every input, the way one reads a program. Superposition gets in the way twice: features share dimensions, and so do the connections between features. A 2025 toy-model note, A Toy Model of Interference Weights, called the resulting spurious connections interference weights and studied them in small synthetic models. This paper, published on the Transformer Circuits Thread on 21 August 2026 by Nicholas L. Turner, Jeffrey Wu and Joshua Batson, looks for them in a trained language model. The authors write that until now they had "lacked a trained transformer in which we could point at specific interference weights, characterize them, and connect them to performance" (introduction). They take a tiny one-layer transformer, rewrite it exactly as a large set of explicit connections, and measure each connection's effect on the model's output and on its loss. The page has no numbered figures or tables; locations below are section headings and figure titles.
Terms used in this note
- Virtual weight. The effective connection between two units in different places of a model (for example, an input token and an output token's logit), obtained by multiplying the matrices along the path between them. Many virtual weights share each real parameter.
- Interference weights. The paper's definition: "linear interactions of interpretable model components through the low-dimensional residual stream which are either irrelevant or harmful to the model's behavior" (introduction). A footnote says this is Definition 2 of the toy-model note: individual weights are classified as interference, rather than measured against an ideal set of "real" weights (that note's Definition 1).
- Transcoder. A sparse, wide replacement for an MLP layer, trained to predict the MLP's output from its input, whose units (features) are meant to be more interpretable than neurons.
- Effectiveness (Fisher effectiveness). How much a weight changes the model's output distribution, measured as a second-order estimate of the KL divergence between outputs with and without the weight, averaged over the data ("Effectiveness and helpfulness"). It is expressed in nats, so it can be compared across kinds of weight.
- Helpfulness. The average change in the loss when the weight is ablated: positive if removing it raises the loss (the weight helps), negative if removing it lowers the loss (the weight is harmful). The paper treats helpfulness as close to a ground-truth measure of a weight's functional importance, but expensive to compute.
- Density and sparsity. The fraction of virtual weights kept after pruning, and the fraction removed.
What the paper does
The model and its expansion
The model is a one-layer, decoder-only transformer with a 256-dimensional residual stream, four attention heads, a 1,024-unit ReLU MLP, a 4,096-token vocabulary and a 1,024-token context, about 2.9 million parameters in all, with no normalization layers and no biases (Appendix, "Training details"). It was trained on English and code from the openly licensed Common Corpus, about 98 million tokens in a single pass of 11,933 steps; the training loss fell from 8.93 to about 3.33 and "was still decreasing" when training stopped. A transcoder with 4,096 features was trained on the MLP.
The authors rewrite the model in a basis of tokens, positions, transcoder features and output logits ("The virtual weight model"). Because the model has no normalization, every path through it is a fixed product of matrices, and the connections fall into six families: tokens to logits, features to logits, tokens to features, two paths that carry tokens through attention (to logits and to features, the "OV" families), and the attention-pattern (QK) weights. Materializing all six raises the parameter count from 2.9 million to about 331 million (331,350,016 in the appendix "Fisher mass distribution"), "about 100×". This virtual weight model reproduces the transformer's forward pass exactly, apart from the transcoder's reconstruction error, which is added back as a separate correction and never scored.
Measuring each weight
Effectiveness was estimated for every virtual weight over about 537 million tokens (2^29) of the training corpus ("Fisher effectiveness computation for all paths"). For weights into features, the authors compute the change counterfactually, passing the ablated input through the feature's activation threshold, so that a weight whose removal switches a feature on or off is counted; they say this "helps handle inhibition", which a linearized estimate would score as zero whenever the weight's removal leaves the feature off. Helpfulness has a cheap closed form for the three families that target logits but is expensive for the others. It was computed over the whole training set, with 95% Gaussian confidence intervals, for the weights in the worked examples, and over 1 billion tokens for random samples of weights ("Effectiveness and helpfulness").
The authors compare effectiveness with ERA (expected residual attribution, adopted from Dunefsky et al.) and TWERA (a target-weighted variant), the coactivation-based measures from Circuit Tracing. They say the new metric is an improvement but that the measures are "quite similar", and add: "We don't encourage readers to read much into our change from ERA or TWERA to Fisher effectiveness."
What it finds
A specific interference weight
The motivating example is the model completing the word ACETYLCHOLINE, predicting "E" after the token "IN" ("A motivating example"). Splitting the prediction into the direct path, the attention path and the MLP path, none ranks "E" first; each favors a different wrong token, and "E" is the one continuation all three score fairly highly. The largest direct-path virtual weight from "IN" points to the token "utions", which never follows "IN" anywhere in the training set, so every time that weight affects the output it makes the loss worse. Its Fisher effectiveness is about three orders of magnitude below that of the most effective weights from the same token ("Effectiveness surfaces helpful weights but does not isolate them"). Re-sorting "IN"'s weights by effectiveness drops "utions" and moves "E" to second place, behind "T", leaving upper-case continuations at the top. The authors present this as, to their knowledge, the first demonstration of such an interference weight inside a trained transformer by its effect on the training loss.
Large weights are not reliably helpful, and effective weights mostly are
For the token "IN" (direct path to logits), the raw virtual weight shows a weak relationship to helpfulness, while effectiveness shows a clearer one: as effectiveness rises, weights split into helpful and harmful, with the most helpful at the top ("Effectiveness concentrates helpful and harmful weights"). Raw virtual weights are roughly normally distributed, with the median about a third of the largest; effectiveness is roughly lognormal across 10 orders of magnitude, with the median 10,000 times below the maximum. There is no clean split into effective and ineffective weights, only a continuum.
The examples are deliberately mixed. For a Chinese-text feature, the largest raw weights already point to the right tokens (the full stop "。" and the leading bytes of multi-byte CJK characters) and are helpful, so sorting changes little. For a feature that responds to the ends of words in European languages, the largest raw weights point to Arabic-script tokens and are ineffective or harmful; sorting by effectiveness moves them down, but harmful weights stay "intermingled" with helpful ones at high effectiveness (feature 3013). For a feature that fires on newlines in code declaration blocks (feature 1254), the largest input weights come from positions, but the position inputs through the direct and attention paths nearly cancel (slope −0.993, r² = 0.912, over 271 contexts) and together make up only about 30% of the feature's effective threshold; the most effective and helpful inputs are instead negative weights from whitespace tokens, together with the newline token's weight, which the authors read as the direct path suppressing the feature on tokens other than newlines while attention signals code context. The authors' summary: virtual weights "are not inherently misleading, but they are not guaranteed to surface the most helpful connections."
Across the whole model, the most effective weights are helpful
To look beyond single examples, the authors sampled 1,111 weights from the six families, spread log-uniformly over effectiveness rank, and estimated each one's helpfulness ("The effectiveness tail is helpful"). In every family the sample falls into three regimes: ineffective weights with near-zero helpfulness, a middle range where helpful and harmful weights are mixed, and a top range containing only helpful weights. Within each family, the most effective helpful weight exceeds any harmful weight in effectiveness by an order of magnitude or more. The same plot for raw weight magnitude shows such a gap for some families but not consistently ("Virtual weight magnitude vs. helpfulness", appendix). The authors interpret this as the model pointing its most consequential weights in helpful directions and tolerating harmful ones only where their effects are small.
Pruning leaves the model dense
Removing weights from least to most effective and measuring the loss on held-out data costs about 0.01 nats at 70% sparsity and under 0.1 nats at 85% sparsity ("The model is still dense in this basis"; the figure's printed labels are 0.0107 at 30% density and 0.0702 at 15%, and the introduction rounds the latter to 0.1). Effectiveness prunes far better than raw magnitude for every density and family, except negative features-to-logits weights. At 1% density, where the parameter count roughly matches the original transformer, performance is "significantly compromised".
To estimate how sparse any weight-by-weight filter could get, the authors computed helpfulness for a random sample of 7,765 weights. In the main text, 47.6% have positive mean helpfulness and 12.7% are dead ("Weight filters in this basis won't yield much sparser models"); even counting only weights whose interval excludes zero "leaves tens of percent". The table under that heading prints the negative share as 39.7% and gives "95% interval" columns of [20.3, 72.7] for the positive share and [14.6, 67.0] for the negative share. This note reads the lower figure in each as the share whose whole confidence interval lies on that side of zero and the upper as everything not dead and not significant the other way; the arithmetic fits (100 − 12.7 − 14.6 = 72.7 and 100 − 12.7 − 20.3 = 67.0), but the page does not define the columns. Because the expansion multiplied the parameter count by about 100, "tens of percent" still means tens of millions of weights for a one-layer model. Keeping 90% of the sample's total positive helpfulness would need only 2.43% density, but 99% would need 13.6%, and the authors say it is hard to know which level is the better guide ("Helpfulness mass distribution").
What the authors conclude
The Discussion says the authors "were unable to reach a highly sparse and interpretable model" with effectiveness as the filter, and "suspect, based on our helpfulness computations, that no saliency scheme will perform much better." They attribute this to the basis rather than the model: the same computation can be dense in one set of coordinates and sparse in another, and the tokens might be better described, for example, by language or part of speech. Identifying interference weights, they write, "turns out to be necessary but not sufficient for reading global circuits"; effectiveness and helpfulness might instead serve as tests for recognizing a better basis, one in which most weights can be discarded.
Limitations and open questions
- One tiny model, one transcoder, no seeds. All results come from a single 2.9-million-parameter one-layer model, whose training loss the appendix says "was still decreasing" when training stopped, and a single transcoder that the authors say "contains several polysemantic features"; they add that the four attention heads are also likely polysemantic. No other models or random seeds are reported.
- The measures do not scale. The authors chose a one-layer model to use their most accurate and expensive tools, and say effectiveness and helpfulness "do not scale nicely to frontier models", so cheaper proxies such as ERA and TWERA remain relevant there.
- Helpfulness is a sample, measured one weight at a time. The density estimates rest on 7,765 sampled weights, and ablating weights singly ignores interactions; the authors note that removing many weights at once can have nonlinear effects. With a billion tokens per estimate, a weight can be statistically significant while having a negligible effect on the loss, which the appendix "ROPE plots" addresses by reclassifying weights against effect-size thresholds (for scale, it computes that the weights jointly reduce the loss by about 4.94 nats per token, from 8.32 for uniform guessing to 3.38 on the test set, or about 1.5 × 10⁻⁸ nats per weight if spread evenly).
- QK approximations. For attention-pattern weights the effectiveness formula treats repeated uses of a token as independent and drops some covariances, so it is not exact (appendix, "QK weights").
- Relation to other theory. The Related Work section places Fisher effectiveness in the tradition of second-order pruning (Optimal Brain Damage and Optimal Brain Surgeon) and notes that, unlike pruned parameters, a virtual weight has no counterpart edit in the original model. It also states that the measure sits inside the local quadratic picture that singular learning theory's local learning coefficient was designed to correct: "We do not claim that effectiveness or helpfulness recover the singular geometry that singular learning theory targets."
- Relation to the toy note. The paper uses only Definition 2 of A Toy Model of Interference Weights and replaces that note's expected-attribution heuristics with Fisher effectiveness. This note compares the two: the toy note concluded that cheap heuristics might scale where the principled definitions do not, while this paper finds that, in its basis, even the expensive per-weight measure leaves the model dense, and places the remaining problem in the choice of basis.
Research questions
- Does the effectiveness gap hold in other small models? The finding that the most effective weights are all helpful comes from one model and one sample of about a thousand weights. Repeating the analysis on several seeds, a fully trained version and a two-layer model would show whether it is general.
- How much do the conclusions depend on the transcoder? The features come from one transcoder with some polysemantic features. Repeating the pruning and helpfulness estimates with transcoders of different sizes and sparsity penalties would show how much of the remaining density belongs to the feature basis.
- What basis would make the model sparse? The authors suggest a better basis exists. Testing candidate decompositions (parameter decompositions, factored token bases) against the paper's criterion, that most weights become ineffective or harmful, would answer it.
- How well do single-weight ablations predict joint pruning? Helpfulness is measured one weight at a time. Pruning sets of weights chosen by summed helpfulness and comparing predicted with measured loss would quantify the nonlinearity.
- Do cheap proxies rank weights as Fisher effectiveness does at scale? The authors say ERA and TWERA are similar to effectiveness in this model. Comparing them on a larger model where effectiveness can be computed for a subset would show whether the proxies stay faithful.
- What are harmful weights that survive at moderate effectiveness doing? The paper finds a middle range of harmful weights with real effects. Inspecting which inputs they fire on would show whether they are robustness weak points, as the hypothesis recalled in the toy note, that interference weights may cause adversarial examples, would predict.
How it connects
- A Toy Model of Interference Weights (Olah, Turner and Conerly, July 2025) is the predecessor that this paper cites for the idea, the definition it adopts and the toy-model evidence.
- Circuit Tracing (Circuit Tracing: Revealing Computational Graphs in Language Models, 2025) introduced TWERA and the global-weights analysis whose difficulty this paper takes up; the paper says that analysis noted unused virtual weights in Claude 3.5 Haiku (the Circuit Tracing appendix it points to reports them in the 18-layer research model) and suggested, "without further evidence", that they were interference. Its summary post is Tracing the thoughts of a large language model.
- Towards Monosemanticity (2023) described features like the Chinese-text one here; the paper says weights are generally easier to interpret in a feature basis than in neurons.
- Toy Models of Superposition (2022) and Circuits Updates – May 2023 supply superposition and weight superposition, which the Related Work section cites as background.
- Open-sourcing circuit tracing tools describes released tools for attribution graphs, the per-prompt method that the authors say they adopted to avoid interference weights; this connection is the note's.
References
- Turner, N. L., Wu, J., and Batson, J. (2026). Characterizing interference weights in a tiny language model. Transformer Circuits Thread, 21 August 2026. https://transformer-circuits.pub/2026/interference_effectiveness_helpfulness/index.html (read November 2026, with the figure data files the page loads; the page carries no update notes, and its suggested BibTeX entry gives the URL of a different paper,
2026/workspace/index.html). record · 2 pages - Olah, C., Turner, N. L., and Conerly, T. (2025). A Toy Model of Interference Weights. Transformer Circuits Thread, 29 July 2025. https://transformer-circuits.pub/2025/interference-weights/index.html record · 4 pages
- Ameisen, E., Lindsey, J., Pearce, A. et al. (2025). Circuit Tracing: Revealing Computational Graphs in Language Models. Transformer Circuits Thread. https://transformer-circuits.pub/2025/attribution-graphs/methods.html record · 15 pages