Anthropic's attribution graphs of 2025 explained a model's computation by replacing its MLP layers with transcoders: sparse, very wide layers whose units ("features") are trained to reproduce each MLP's output and are then read as the steps of the computation. This update, posted on the Transformer Circuits Thread in July 2025, argues that transcoders have a cost. They "shatter" computation into very many narrow pieces, which can represent it differently and less efficiently than the model does; the authors' example is addition, which transcoders split into lookup-table features such as one for "6 + 9 = 5 (mod 10)", while other work suggests models compute arithmetic with simple transformations of numbers laid out geometrically. The authors expect that transcoders large enough to capture all the computation in a model would "likely require an astronomical number of features", with "many more parameters" than the underlying model. Their proposed alternative is a sparse mixture of linear transforms (MOLT), and the update reports first measurements of its efficiency, faithfulness and interpretability.
The update opens with its own caveat: "This work is a preliminary update describing a method we are still developing." The version of record is the web page, which carries no date; the date here is the one the Transformer Circuits index records in its page markup (25 July 2025; the index displays only the month). The page was revised after release to add a comparison with a closely related method (see "Related methods"); the earliest archived copy, of 29 July 2025, already contains it and differs from the live page only in punctuation. This note cites section headings; the two charts print no values, so numbers taken from them are marked as read from the figure.
How a MOLT works
A MOLT replaces an MLP layer with many transforms ("Implementation"). Each transform has an encoder vector and bias that decide whether, and how strongly, it is active on the current input, passed through a nonlinearity such as ReLU or JumpReLU. When active, it applies a low-rank linear map (a product of two matrices, U and V, of rank k) to the whole residual-stream input, and the MOLT's output is the sum of these contributions weighted by activation. Like a transcoder, a MOLT is trained to reproduce the MLP's output, with a sparsity penalty on each transform's activation scaled by the size (Frobenius norm) of its matrix.
The difference from a transcoder is what the units are. A transcoder feature fires and writes a fixed direction, so it serves both as a unit of computation and as a unit of representation. A MOLT transform has no fixed output direction; it is a conditional linear map that carries representations from one layer to the next. The authors therefore intend MOLTs to be read alongside a separate decomposition of the residual stream, such as sparse autoencoders (SAEs), and compare their role to that of attention heads.
The update offers three reasons to expect transforms to suit MLPs ("Intuition"). First, a ReLU MLP is itself a linear map once the set of active neurons is fixed, with rank equal to the number of active neurons, so patterns of active neurons loosely resemble transforms. Second, transforms can use geometry: if digits lie on a circle, a rank-2 "plus 3" transform could rotate each digit's representation to the answer's position, replacing many lookup-table features. Third, transforms combine: a million features and a million transforms describe, in the authors' term, a trillion "pseudofeatures" in the next layer, and chains of transforms multiply further.
Rank allocation
On Claude 3.5 Haiku the best error-sparsity trade-off came from N transforms of rank 512, 2N of rank 256, 4N of rank 128, 8N of rank 64 and 16N of rank 32, with N varied to change scale; mixed ranks beat equal ranks at the same parameter count, in a hyperparameter space the authors say they have not explored fully ("Optimal allocation of ranks").
Efficiency against transcoders on Claude 3.5 Haiku
MOLTs and transcoders were trained on the middle layer of Claude 3.5 Haiku with matched parameter counts at six compute levels, from 1× to 1024× FLOPs; each fourfold increase doubles both parameters and training steps, and the largest transcoders have about 10 million features ("ML performance"). Two runs at different sparsity were made for each architecture and compute level (as plotted). The authors report that at a given L0 (the average number of active units per token) MOLTs have significantly lower reconstruction error, and that "The smallest MOLT runs here Pareto-dominate transcoder runs that use 1024× as many FLOPs." Read from the figure, the transcoders' normalized mean squared error ranges from about 0.34 to 0.54 and the MOLTs' from about 0.21 to 0.39. On this note's reading, the dominance claim holds over the L0 range both curves cover (about 27 to 66): the 1× MOLT runs reach no point as dense, or as low in error, as the densest 1024× transcoder run (about 0.34 at an L0 of about 86). Transcoder error appears to level off at the largest scales while MOLT error does not; the authors add that the flattening could be "simply due to poor ML tuning". The chart reports one run per setting, with no seeds or error bars.
Mechanistic faithfulness by Jacobian correlation
The update defines mechanistic faithfulness as how closely a replacement responds to small input perturbations the way the original MLP does, measured as the cosine similarity between the flattened Jacobian matrices of the replacement and the MLP on each input, averaged over inputs ("ML performance"). The authors report that MOLTs have "substantially higher" Jacobian correlation than transcoders at the same L0, and that transcoders' faithfulness "appears to deteriorate with scale" while MOLTs' is more stable. Their explanation is structural: a transcoder's Jacobian has rank at most its L0, while a MOLT's can have much higher rank.
This measurement was made on a different model: the 18-layer model of the Circuit Tracing paper, at four sizes from 65k to 500k features or "feature-equivalents" (MOLTs with the same parameter count). The authors note that they "have not yet performed this Jacobian analysis on Haiku." Read from the figure: at an L0 of about 70, transcoder correlation falls from about 0.56 at 65k features to about 0.49 at 500k; at an L0 of about 8 to 10 it is about 0.25 to 0.27. MOLTs reach about 0.53 to 0.60 at an L0 of about 22 to 29, and about 0.32 to 0.40 at an L0 of about 6 to 12. On both architectures, correlation rises with L0, and no run exceeds about 0.6.
Are the transforms interpretable?
The update treats two questions separately ("Transform interpretability"). When a transform is active can be studied like a feature, from the dataset examples that activate it. The authors find transforms qualitatively similar to features (token-level conditions early, more abstract context later); higher-rank transforms are active more often but look comparably interpretable, such as one active on period tokens and one on Spanish text. The update reports no rating study.
What a transform does proved harder. Listing which pairs of SAE features interact most strongly through a transform was hard to interpret, "presumably due to the same problem of interference weights" that affects raw weights between transcoder features.
MOLTs in attribution graphs
Because transforms have no fixed output direction, attribution edges cannot be computed from transforms the way they are from transcoder features ("Integrating MOLTs into attribution graphs"). The update instead builds edges between features of SAEs trained on the residual stream (cross-layer variants such as weakly causal crosscoders give better graphs, the authors say), splits each edge into terms carried by MOLT transforms and by attention heads, and labels the edge with the transforms and heads that carry most of it. The authors call this "preliminary success". The examples are qualitative ("Qualitative findings"):
- In the prompt
The Spanish word for hot is "calor", features for saying a word beginning with "cal" received input from "hot" features through a transform active in Spanish-language contexts, and also from "Spanish" features through transforms active in "hot" contexts. - In
3 + 5 = 8, a "say 8" feature received input from a "plus 3" feature through a transform active where the number 5 had recently appeared. - Less clearly interpretable cases were also common: many edges ran through transforms selective for words such as "is" or "Assistant", and some transforms looked redundant with the features they carried, such as a "France" transform carrying a "France" feature to a "Paris" feature.
Related methods
The page was updated after release to discuss the Mixture of Decoders (MxD) of Oldfield et al. (2025), which it calls "quite similar": MxD also replaces MLP layers with a sparse mixture of linear transforms, but keeps each transform full-rank by sharing parameters across transforms, where MOLTs make each transform low-rank ("Related Work"). The authors say: "We are not yet confident in the pros and cons of the MxD vs. MOLT strategies." They suspect low rank encourages narrower, more interpretable transforms, and say their intended use, transforms between SAE features inside attribution graphs, differs from that paper's focus. MOLTs also resemble skip transcoders (a transcoder plus one linear map) but are conditionally active, low-rank, numerous and have no ordinary features.
Limitations
- Preliminary and partly untested. The update describes its results as incomplete; the efficiency comparison is on one layer of one model, and the faithfulness comparison on another model.
- No repeated runs or intervals are reported for either chart, and both print no values.
- Interpretability judged qualitatively. Transform conditions are described from examples, and the function of transforms is reported as harder to interpret.
- A need, suspected. The conclusion's claim that "a MOLT-like solution will be necessary" to capture frontier MLP layers at reasonable cost is stated as a suspicion, not a result.
- Unwieldy graphs. The conclusion says graphs with transform annotations are "somewhat more unwieldy" than transcoder graphs and that not all transform-mediated computations are clearly interpretable: "More work needs to be done to conclude that MOLTs are strictly preferable to transcoders."
- Compositional claims are hopes. The authors write that they "suspect (but have not yet shown)" that compositions of transforms reveal compositional structure; the proposal to decompose SAE reconstruction error into (feature, transform) pairs is described as unexplored.
Research questions
- Do MOLTs keep their efficiency advantage across layers and models? The comparison covers one middle layer of Claude 3.5 Haiku. Repeating it on early and late layers and on open models, with several seeds, would show whether the Pareto dominance is general.
- Does the Jacobian advantage hold on Claude 3.5 Haiku? The authors had not measured it there. Running the same Jacobian-correlation comparison on the model used for the efficiency results would close the gap between the two charts.
- Is higher Jacobian correlation evidence of the same mechanism? Testing MOLTs and transcoders on inputs shifted away from the training distribution, and checking whether the one with the higher correlation also has lower error there, would show what the metric predicts.
- How do MOLTs compare with Mixture of Decoders at matched parameters? A head-to-head comparison of error, Jacobian correlation and rated interpretability would settle the trade-off the authors leave open.
- How interpretable are transforms by a systematic measure? Applying the same human or automated rating used for transcoder features to a random sample of transform conditions, and of transform-mediated edges, would replace the qualitative examples with a rate.
- Do chains of transforms explain SAE reconstruction error? Decomposing the error at one layer into (feature, transform) pairs from the previous layer, as the conclusion proposes, and measuring how much error is accounted for would test the compositional hope.
How it connects
- Circuit Tracing: Revealing Computational Graphs in Language Models (March 2025) is the transcoder-based method this update proposes to change; its 18-layer model is the one used for the Jacobian comparison, and the update links its sections on evaluating mechanistic faithfulness and on interference in global weights.
- On the Biology of a Large Language Model (March 2025) is where the update's example of lookup-table addition features comes from.
- A Toy Model of Mechanistic (Un)Faithfulness (August 2025) builds a small transcoder that is unfaithful and proposes training transcoders to match the MLP's Jacobian; this update instead uses Jacobian agreement as a measure. The documents do not cite each other; the connection is this note's.
- A Toy Model of Interference Weights (2025) and Characterizing interference weights in a tiny language model (2026) study the interference weights the update blames for hard-to-read transform functions; this connection is the note's.
- Sparse Crosscoders for Cross-Layer Features and Model Diffing (2024) introduced the weakly causal crosscoders the update recommends for MOLT attribution graphs.
- Tracing Attention Computation Through Feature Interactions (2025) describes the "QK attributions" for attention heads that this update says will appear in another update; that it is that update is this note's reading.
References
- Lindsey, J., Chen, B., Pearce, A., Hydrie, S., and Conerly, T. (2025). Sparse mixtures of linear transforms. Edited by J. Wu. Transformer Circuits Thread, July 2025. https://transformer-circuits.pub/2025/bulk-update/index.html. Section names in this note refer to the web version. record · 2 pages
- Oldfield, J., Im, S., Li, S., Nicolaou, M. A., Patras, I., and Chrysos, G. G. (2025). Towards Interpretability Without Sacrifice: Faithful Dense Layer Decomposition with Mixture of Decoders. arXiv:2505.21364. (Discussed by the update as closely related work.) record
- Paulo, G., et al. (2025). Transcoders Beat Sparse Autoencoders for Interpretability. arXiv:2501.18823. (Cited by the update for skip transcoders.) record
- Kantamneni, S., and Tegmark, M. (2025). Language Models Use Trigonometry to Do Addition. arXiv:2502.00873. (Cited by the update for geometric number representations.) record
- Chanin, D., et al. (2024). A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders. arXiv:2409.14507. (Cited by the update for feature splitting and absorption.) record
- Ameisen, E., Lindsey, J., et al. (2025). Circuit Tracing: Revealing Computational Graphs in Language Models. Transformer Circuits Thread. https://transformer-circuits.pub/2025/attribution-graphs/methods.html record · 15 pages