Sparse dictionary methods had found interpretable features inside language models (Towards Monosemanticity, Scaling Monosemanticity), but a list of active features does not say how a model gets from its input to its answer. "Circuit Tracing: Revealing Computational Graphs in Language Models" (Ameisen et al.), published on the Transformer Circuits Thread on 27 March 2025, is Anthropic's account of a method for that second step: for one prompt and one output token, draw a graph of which features caused which, then check the graph by intervening on the model. The paper develops the method on a small 18-layer research model ("18L") and on Claude 3.5 Haiku, evaluates it at length, and lists its limitations. Its companion, On the Biology of a Large Language Model (Lindsey et al., same day), applies it to Haiku in a series of case studies, and the summary post for both is covered in Tracing the thoughts of a large language model. This note covers the methods paper. The live web page is the version of record; it carries no changelog or "Comments & Replications" section. The paper has no numbered sections, so locations are given by section heading, with appendix sections marked "Appendix".
Terms used in this note
- MLP layer, residual stream, attention. A transformer alternates attention layers, which move information between token positions, with MLP (multi-layer perceptron) layers, which process each position separately. Both read from and add to a running vector per token, the residual stream.
- Feature. The paper's unit of analysis: a sparsely active component learned by a dictionary model, which in many cases matches a human-interpretable concept. The paper says it prefers "feature" to the term "latent" for these components (Introduction, footnote).
- Transcoder. A dictionary model that reads an MLP layer's input and predicts its output through sparsely active features, so that the features stand in for the layer's computation rather than only describing its activations (as a sparse autoencoder does).
- Cross-layer transcoder (CLT). A transcoder whose features each read from the residual stream at one layer and write to the MLP outputs of that layer and every later layer, with a separate decoder for each output layer. All layers are trained jointly on reconstruction error plus a sparsity penalty ("Architecture"); for its training setup the appendix ("ML details") refers to Circuits Updates – January 2025.
- Per-layer transcoder (PLT). The baseline: an ordinary transcoder trained for each layer separately.
- Replacement model and local replacement model. The replacement model runs the original model with CLT outputs in place of MLP outputs. For a single prompt, the local replacement model also freezes the original model's attention patterns and normalization denominators and adds an error term at each layer and position, so that its outputs match the original exactly on that prompt ("The Local Replacement Model").
- Error node. The part of an MLP output the CLT fails to reconstruct, shown in a graph as a node with outgoing edges but no inputs and no interpretation.
- Attribution graph. For one prompt, a graph whose nodes are token embeddings, active features, error nodes and candidate output tokens, and whose edges are the linear direct effects of one node on another in the local replacement model ("Constructing an Attribution Graph for a Prompt").
- Supernode. A group of features that the analyst combines by hand because they play the same role on a prompt ("Grouping Features into Supernodes").
- QK and OV circuits. The two halves of an attention head, following A Mathematical Framework for Transformer Circuits: QK decides where a head attends; OV decides what it moves. Attribution graphs capture OV effects only.
What the paper does
The models and the transcoders
The two models studied are 18L, a small pretraining-only transformer trained by Anthropic, and Claude 3.5 Haiku. 18L has no MLP in its first layer, so its CLT has 17 layers. CLTs were trained at sizes from 300 thousand to 10 million total features on 18L and from 300 thousand to 30 million on Haiku ("Architecture"). The 18L CLTs were trained on pretraining data only; the Haiku CLTs on a mix of pretraining and fine-tuning data. The largest runs used about 3 billion training tokens (18L) and about 16 billion (Haiku) (Appendix, "CLT Implementation Details").
Building a graph for one prompt
Because attention patterns and normalization are frozen and the CLT replaces the MLPs' nonlinearities, each feature's pre-activation on the chosen prompt is a sum of linear contributions from earlier nodes. Each edge is the source feature's activation times a "virtual weight" summed over all paths through the residual stream and attention-head OV circuits that do not pass through another MLP. The edges are computed with backward passes through the original model, one per target node ("Constructing an Attribution Graph for a Prompt"; Appendix, "Attribution Graph Computation"). The graph explains information moved between positions, but not why attention moved it.
Full graphs are too large to read: the paper says the edge count "can grow to the millions even for short prompts", with on the order of a hundred features active per token position. A pruning step keeps the nodes that account for 80% of the total influence on the output, then the edges that account for 98% of what remains, and keeps output nodes up to 95% of the probability mass (at most ten). The authors call these thresholds arbitrary and chosen by hand. Pruning typically cuts nodes by about 10 times and edges by about 500 times (Appendix, "Graph Pruning").
Reading and testing a graph
A researcher then labels features from the text on which they activate and the tokens they promote; the authors found human labels "more reliable" than their automated labeling pipeline ("Understanding and Labeling Features"). Related features are grouped into supernodes; the authors tried automated clustering but found that "no automated method was sufficient". Claims read off the graph are then tested by intervening on features in the original model, typically setting a supernode's features to the negative of their activation over a range of layers ("constrained patching") and recording the effect on downstream features and the output ("Validating Attribution Graph Hypotheses with Interventions").
Three worked examples
- Acronym (18L). For "The National Digital Analytics Group (N", the model completes "DAG". The graph shows paths from each word to features for saying its first letter in the right position, combining into a "say DAG" feature. Suppressing each word's supernode suppresses the features downstream of it ("Attribution Graphs").
- Factual recall (18L). "Fact: Michael Jordan plays the sport of" is completed with "basketball" at 65% confidence. Two parallel paths appear, one from "sport" features and one from Michael Jordan features through basketball features; suppressing either has a large effect on the output and a smaller effect on the other path ("Factual Recall Case Study").
- Addition (Haiku). For "calc: 36+59=", the graph shows a path that computes the last digit (6 plus 9 ends in 5) and paths that estimate the sum's size at low and moderate precision, which combine to give 95. Suppressing the "_6" feature makes the model answer 98; suppressing "_9" gives 91 rather than the 92 the same reasoning would predict, and the paper adds that "such numerology must be taken with a grain of salt" ("Addition Case Study"). On the paper's two-digit addition prompts ("calc: a+b=", with a and b from 0 to 99), Haiku's top-1 accuracy was 100% and 18L's 98.6%, a gap the authors say "possibly reflects" the lower precision of 18L's addition features (Appendix, "Comparison of Addition Features Between Small and Large Models").
Global weights
Attribution graphs hold for one prompt. The paper also examines the CLT's prompt-independent "global weights" between features and reports that they are often hard to read because of interference: large weights between features that rarely or never activate together and so have no effect in practice. In an appendix sample of 1,000 18L features drawn from the addition graphs, even among the largest positive weights (beyond the 99.9999th percentile) a significant share were never used on about 150 million tokens, which the authors take as "suggestive" of interference rather than proof (Appendix, "Interference Weights over More Features"). Two workarounds are shown: weighting virtual weights by how often features co-activate (a measure the paper calls TWERA), which the authors say "is not a perfect solution": it handles inhibition poorly and, relying on co-activation statistics, changes which connections rank highest rather than only removing interference; and restricting to features active on a narrow domain, which for two-digit addition gives a graph in which "most edges are interpretable" ("Global Weights", "Global Weights in Addition").
What it finds
Cross-layer transcoders compared with alternatives
- Matching the model's output. The fraction of prompts on which the replacement model's most likely next token matches the original's rises with CLT size and is higher for CLTs than PLTs. The largest 18L CLT matches on 50% of a set of pretraining-style prompts, restricted to tokens the model predicts correctly but with under 80% confidence ("From Cross-Layer Transcoder to Replacement Model"). The text prints no comparable number for Haiku.
- Reconstruction and sparsity. The largest 18L CLT (10 million features) has a normalized mean reconstruction error of about 11.5% with an average of 88 active features per token; the largest Haiku CLT (30 million) has 21.7% with 235 ("Quantitative CLT Evaluations").
- Interpretability scores. Two automated tests, graded by Claude, ask whether a feature's activating examples can be told apart from another feature's (sort eval) or matched to the prompt that activated it (contrastive eval). Scores rise with CLT size (training steps were scaled with the number of features, so the gain reflects both); at equal feature counts "on all metrics, CLTs outperform PLTs", and both outperform thresholded neurons (same section). At roughly matched training compute the picture is mixed: a 1-million-feature CLT outperforms a 10-million-feature set of PLTs on reconstruction error and on the replacement model's KL divergence, each against sparsity, with 5 to 10 times less compute; a 3-million CLT ties the 10-million PLTs on the sort eval but scores worse on the contrastive eval, so "we cannot declare a Pareto-winner here" (Appendix, "Efficiency relative to alternatives").
- Shorter paths. CLTs shorten paths through the graph by collapsing chains of features that amplify the same signal over many layers. For the one prompt whose graphs the paper links for this comparison (a capital-city analogy), average path length is 2.3 with a 10-million CLT and 3.7 with 10-million PLTs ("Measuring and Comparing Path Length").
- Less left to error nodes. Two scores measure how much of a graph runs through features rather than error nodes: completeness (the share of each important node's inputs) and replacement (the share of whole input-to-output paths). Averaged over a pretraining prompt set (n = 260, Appendix, "Additional Evaluation Details"), the largest gain comes from switching to CLTs, with diminishing returns from more features (these averages are for unpruned graphs). One linked example, pruned with the default settings, scores 0.80 completeness and 0.61 replacement for the CLT against 0.78 and 0.37 for PLTs ("Evaluating and Comparing Graph Completeness").
- Pruning cost. On one acronym prompt, pruning thresholds of 0.95, 0.9, 0.8 (the default) and 0.7 leave 236, 137, 55 and 27 nodes, with completeness 0.87, 0.83, 0.70 and 0.58 ("Graph Pruning").
The averaged comparisons above are shown as plots; the printed numbers are the largest-run values and the single-prompt examples.
How well graphs predict interventions
- On 20 prompts, a node's total influence on the output through the graph predicted the effect of ablating it better than its direct edge to the output or its activation ("Validating Node-to-Logit Influence"; values in a plot).
- Between pairs of features, graph influence and the measured effect of ablation had a Spearman correlation of 0.72, averaged over 20 prompts ("Validating Feature-to-feature Influence").
- Perturbing the local replacement model and the real model the same way gave similar results one layer later (about 0.8 cosine similarity and about 0.4 normalized mean squared error in the main text; "around 60–80%" cosine for the 10-million 18L CLT in the appendix), but "perturbation discrepancies compound significantly over layers". The direction of effects degrades gradually; their magnitude can degrade "catastrophically", and more so for larger dictionaries. The authors suspect the missing normalization denominators ("Evaluating Mechanistic Faithfulness"; Appendix, "Evaluating Faithfulness of the Local Replacement Model").
The authors grant that these averages "seem at odds with" the success of their case-study interventions and suggest reasons: magnitudes suffer more than directions, and they try "a range of perturbation strengths in ad-hoc, empirical fashion"; grouping features into supernodes may "denoise" errors; and the case-study perturbations are "highly nonrandom", targeting influential features in the layers where their output is strong (Appendix, "Evaluating Faithfulness of the Local Replacement Model"). They also typically use the layer range that gives the largest effect ("Mechanistic Faithfulness"), and direct edges are "nearly forced to be confirmed" by construction, so the real test is of effects two or more steps downstream (Appendix, "Nuances of Steering with Cross-Layer Features"). They conclude that their circuit descriptions "may be mechanistically accurate at a very coarse level" with substantial room to improve ("Mechanistic Faithfulness").
What CLTs may hide
The paper points out a cost of its own design choice. On "Zagreb:Croatia::Copenhagen:", PLTs show a chain of Copenhagen and Denmark features reinforcing each other over several layers; the CLT collapses it into early-layer features. If that chain exists in the real model, the authors write, the CLT graph is "an example of CLTs being mechanistically unfaithful" ("Measuring and Comparing Path Length").
Limitations the authors state
The "Limitations" section lists seven, with examples:
- Missing attention circuits. Graphs do not explain why attention goes where it does. On a multiple-choice question about the end of World War II, the graph says only that the model chose "B" because of features on the correct option; three "correct answer" features found by further manual work change the answer when activated on another option (on C, the model answers "C"), but how attention uses them is the authors' conjecture, and they list what they do not know. The authors write that they have found "many cases where this limitation renders our attribution graphs essentially useless".
- Reconstruction error and dark matter. On an obfuscated request for malware, almost all nodes on the prompt tokens are error nodes. The authors add that their feature labels are often "much too coarse".
- Inactive features and inhibition. Graphs show only active features, so a mechanism that works by keeping a feature off (as in the Biology paper's hallucination case) needs a hypothesis and contrasting prompts to find.
- Graph complexity. Pruned graphs still hold hundreds of features and thousands of edges (the section "Learning from Attribution Graphs" says "tens of thousands"), and supernodes are a "not-very-principled abstraction".
- Features at the wrong level of abstraction. Of twelve prompts Haiku completes with "during", no single feature fired on all, and the features' generality "appears to decrease" as the CLT grows.
- Global circuits remain hard to read because of interference weights and attention-mediated interactions.
- Mechanistic faithfulness. Matching activations does not ensure matching mechanisms: "we cannot guarantee that the replacement model has learned the same mechanisms".
Elsewhere the paper adds that the coherence of supernodes is not quantified and is left to readers to check ("Evaluations"), that training a CLT has a large up-front cost (estimated at about 210 H100 GPU-hours for a 2-million-feature CLT on Gemma 2 2B and about 3,844 for a 5-million-feature CLT on Gemma 2 9B, at 50% efficiency; Appendix, "Estimated Compute Requirements for CLT Training"), and that the result is local to one prompt: "The present work falls short by only offering a locally valid attribution graph" ("Related Work").
Prompts and data
The paper studies no written principles, constitutions or system prompts. Its prompts are short completions (acronyms, analogies, arithmetic, facts) and a few Human/Assistant exchanges in the limitations section. One choice bears on findings about Assistant behavior: the Haiku CLT was trained on fine-tuning data as well as pretraining data (Appendix, "ML details"), and the methods paper does not compare CLTs trained with and without it. The Biology paper's refusal case study reports that dictionaries trained only on pretraining data had very few refusal features (as summarized in Tracing the thoughts of a large language model).
Later work and outside replications
- Attention. "Progress on Attention" (April 2025), a preliminary update, reports combining attention-output dictionaries with CLTs to put attention information into attribution graphs, which it found "moderately useful" but "not as much as we expected", and calls understanding why models attend where they do "the core remaining problem". "Tracing Attention Computation Through Feature Interactions" (Kamath, Ameisen et al., 31 July 2025) describes itself as filling "a major missing piece in our methodology" with feature-level QK attributions added to graphs. See Progress on Attention and Tracing Attention Computation Through Feature Interactions.
- Faithfulness and interference. Two informal notes by Chris Olah and colleagues expand two of the limitations: A Toy Model of Mechanistic (Un)Faithfulness (7 August 2025) constructs cases where transcoders reach the same outputs by different mechanisms, and A Toy Model of Interference Weights (29 July 2025) says avoiding interference weights was the team's "primary motivation" for studying attribution graphs.
- Alternatives to transcoders. Sparse mixtures of linear transforms (July 2025), a preliminary update, cites the addition "lookup table" features (linking the Biology paper's account of them) as an inefficiency of transcoders and reports that its replacement is more compute-efficient and more mechanistically faithful in early tests.
- Feature descriptions. A preliminary note in the May 2026 Circuits Update (Circuits Updates – May 2026; Goel, Kauvar and Turner) shows CLT features with similar top activating examples and output tokens of which only one changes the answer when inhibited. It proposes reading a feature's downstream connections, ranked by this paper's TWERA weights, and reports that adding them modestly improved a model's ranking of which feature would steer, on 10 groups of candidate features.
- Open tools. Open-sourcing circuit tracing tools (May 2025) released a library for open-weights models, with per-layer transcoders at release and CLTs added later.
- Goodfire replication (11 June 2025). Merullo, Watts et al. trained a CLT on GPT-2 Small (ReLU rather than JumpReLU; 589,824 features; 79 active per token; 59% token accuracy, which they call consistent with this paper) and traced the known greater-than mechanism from one prompt. They say their results "replicate and validate" CLTs' ability to find structure semi-autonomously, reporting interpretable, task-relevant features, but also "a somewhat less clean representation space" than an earlier neuron-level analysis of the same task, including features with very narrow input ranges, and, for many features, discrepancies between their activating examples and their logit attributions.
- Lange et al. critique (LessWrong, 2 February 2026). A one-week research sprint argues, in its title, that CLTs "are incentivized to learn Unfaithful Circuits". In a hand-built two-layer Boolean model, a CLT reproduced the output perfectly while activating no second-layer features, replacing a two-step computation with a lookup. On GPT-2 Small, CLTs (unlike jointly trained PLTs) concentrated active features in the first layer, and on Gemma 2 2B a CLT trained by Hanna and Piotrowski put nearly all attribution in layers 0 and 1 and gave a different account of a memorized license text than a Gemma Scope transcoder did. The authors call the language-model evidence indirect, since no ground truth exists there; the post thanks Joshua Batson, this paper's corresponding author, for feedback. This note connects the concern to the paper's own Copenhagen example; the post does not cite that example.
Research questions
- How often does a graph yield a validated mechanism on prompts not chosen by the analyst? The paper's metrics measure error-node share, not whether a usable explanation results. Having analysts work through a random sample of graphs under a fixed time limit and recording how many produce a hypothesis that survives intervention would give that rate for 18L and Haiku.
- Does depth collapse change the paper's own case studies? The Copenhagen example and Lange et al. suggest CLTs can skip intermediate steps. Building PLT and CLT graphs for the acronym, Michael Jordan and addition prompts and ablating the hand-off between layers would show whether the two accounts agree.
- Why does faithfulness of perturbation magnitudes get worse with larger dictionaries? The authors suspect the frozen normalization. Repeating the faithfulness measurement with normalization recomputed, or with a transcoder trained to match Jacobians, would test that explanation.
- How much do conclusions depend on analyst choices? Supernode grouping, steering strength and the layer range are set by hand. Giving the same graphs to several analysts independently and comparing their supernodes and conclusions would measure this.
- Do co-activation-weighted global weights drop real inhibition? The paper says TWERA handles inhibition poorly. On the two-digit addition domain, where every feature's behavior can be computed exactly, comparing TWERA's connections with intervention results would show what it loses.
- Can the method trace behavior that depends on an instruction earlier in the prompt? Following an instruction plausibly depends on attention, which these graphs omit. Comparing graphs for the same request with and without a system-prompt instruction, with and without the later QK attributions, would show how much of the instruction's effect each version captures.
- Does training the CLT on fine-tuning data change which Assistant-related features appear? Training two Haiku CLTs, with and without fine-tuning data, and comparing graphs on the same dialogue prompts would show how much the dictionary's data shapes findings about the Assistant.
How it connects
- On the Biology of a Large Language Model applies this method to Claude 3.5 Haiku; Tracing the thoughts of a large language model is the summary post for both papers; that note's figures on the replacement model come from this paper. This paper's own summary of the Biology results ("Biology") states some of them flatly ("We show that the model plans its outputs"), as the Biology paper's introduction does; the Biology paper's "Limitations" section adds that its "results are only claims about specific examples".
- The dictionary methods it builds on are in Towards Monosemanticity and Scaling Monosemanticity; Toy Models of Superposition explains why neurons are replaced. The paper links its finding that features write to several layers to the team's earlier crosscoder work, covered in Sparse Crosscoders for Cross-Layer Features and Model Diffing.
- Freezing attention and splitting heads into QK and OV circuits follows A Mathematical Framework for Transformer Circuits; the induction example in the limitations comes from In-context Learning and Induction Heads.
- Circuits Updates – July 2024 and Circuits Updates – August 2024 hold the dark-matter essay and the sort and contrastive evals that this paper cites.
References
- Ameisen, E., Lindsey, J., Pearce, A., Gurnee, W., Turner, N. L., Chen, B., Citro, C., Abrahams, D., Carter, S., Hosmer, B., Marcus, J., Sklar, M., Templeton, A., Bricken, T., McDougall, C., Cunningham, H., Henighan, T., Jermyn, A., Jones, A., Persic, A., Qi, Z., Thompson, T. B., Zimmerman, S., Rivoire, K., Conerly, T., Olah, C., and Batson, J. (2025, March 27). Circuit Tracing: Revealing Computational Graphs in Language Models. Transformer Circuits Thread. https://transformer-circuits.pub/2025/attribution-graphs/methods.html record · 15 pages
- Lindsey, J., Gurnee, W., Ameisen, E., et al. (2025, March 27). On the Biology of a Large Language Model. Transformer Circuits Thread. https://transformer-circuits.pub/2025/attribution-graphs/biology.html record · 5 pages
- Jermyn, A., Lindsey, J., Luger, R., et al. (2025, April). Progress on Attention. Transformer Circuits Thread. https://transformer-circuits.pub/2025/attention-update/index.html record · 5 pages
- Kamath, H., Ameisen, E., Kauvar, I., et al. (2025, July 31). Tracing Attention Computation Through Feature Interactions. Transformer Circuits Thread. https://transformer-circuits.pub/2025/attention-qk/index.html record · 4 pages
- Olah, C., Turner, N. L., and Conerly, T. (2025, July 29). A Toy Model of Interference Weights. Transformer Circuits Thread. https://transformer-circuits.pub/2025/interference-weights/index.html record · 4 pages
- Olah, C. (2025, August 7). A Toy Model of Mechanistic (Un)Faithfulness. Transformer Circuits Thread. https://transformer-circuits.pub/2025/faithfulness-toy-model/index.html record · 2 pages
- Lindsey, J., Chen, B., Pearce, A., Hydrie, S., and Conerly, T. (2025, July). Sparse mixtures of linear transforms. Transformer Circuits Thread. https://transformer-circuits.pub/2025/bulk-update/index.html record · 2 pages
- Goel, P., Kauvar, I., and Turner, N. L. (2026, May). Downstream Connections Predict Which Features Will Steer Model Behavior. In Circuits Updates – May 2026. Transformer Circuits Thread. https://transformer-circuits.pub/2026/may-update/index.html record · 2 pages
- Merullo, J., Watts, C., Loeffler, M., Gorton, L., Simon, E., McGrath, T., and Lewis, O. (2025, June 11). Replicating Circuit Tracing for a Simple Known Mechanism. Goodfire. https://www.goodfire.ai/research/replicating-circuit-tracing-for-a-simple-mechanism record
- Lange, G., RGRGRG, Dearstyne, K., and Maher, K. (2026, February 2). Cross-Layer Transcoders are incentivized to learn Unfaithful Circuits. LessWrong. https://www.lesswrong.com/posts/6CS2NDmoLCFcEJMor/cross-layer-transcoders-are-incentivized-to-learn-unfaithful record · 2 pages