By 2025 Anthropic's circuit analyses replaced a model's MLP layers with transcoders: sparse, wide layers that read a layer's input and are trained to predict its output, so that the computation can be described as interactions between interpretable features. A transcoder can match a layer's outputs and still compute them a different way, as different sorting algorithms give the same result. This post, published on the Transformer Circuits Thread on 7 August 2025, calls that concern mechanistic faithfulness: whether a replacement implements the same mechanisms as the component it replaces. It builds a minimal example of an unfaithful transcoder, explains when the failure appears, and tests one remedy, matching the replacement's Jacobian (the matrix of derivatives of its outputs with respect to its inputs) to the original's. The post asks to be read like "preliminary experiments" shared at a lab meeting rather than a mature paper: "All claims should be taken with significant caveats and low confidence throughout." (introduction).
The version of record is the web page; its byline gives the date (the Transformer Circuits index files it under July 2025). An archived copy of 9 August 2025 differs from the live page only in punctuation. It has no numbered sections or figures, so this note cites section headings and figure titles. Numbers below are labels printed on the figures unless marked as read from a plot.
Why the shift to transcoders mattered
The post argues ("Background: Sleepwalking into Mechanistic Faithfulness") that moving from sparse autoencoders (SAEs), which reconstruct activations, to transcoders, which replace a step of computation, changed what could go wrong, and that the author underappreciated this at the time. "When one models representation, much can be forgiven": features that are monosemantic and explain the activations are a valid description of what the activations contain. When one models computation, a replacement can be monosemantic and have low error on the training distribution yet use a different mechanism, which may generalize differently off distribution. The post calls this an attack on "one of the deepest hopes for mech interp: explanations we can really deeply trust." It notes a limit: each transcoder step must predict the true residual stream after a single nonlinearity, which narrows how far a mechanism can drift. It argues that unfaithful examples can be built despite this. The post credits transcoders to several 2024 sources: Dunefsky et al., the dictionary_learning repository of Marks et al., and an Anthropic update by Templeton et al.
The toy problem: absolute value with a repeated datapoint
The network to be copied computes the absolute value of each input coordinate, which two ReLU neurons per coordinate do exactly: ReLU(x) + ReLU(−x) ("The Absolute Value Transcoder Problem and Repeated Data"). Each coordinate is drawn uniformly from −1 to 1 with a probability set by a density parameter and is zero otherwise. Transcoders are trained with a tanh-shaped L1 sparsity penalty, chosen to approximate an L0 penalty (a count of active units); the post says ordinary L1 produced duplicate features and messier results. The figures list feature sparsity 0.1 and L1 coefficient 0.1. The weight panels show eight input coordinates, which suggests, on this note's reading, an eight-dimensional problem; the text does not state the dimension. Without repeated data, transcoders "easily discover" the two-neurons-per-coordinate solution.
The post then adds a special input p, equal to 1 on the first three coordinates and 0 elsewhere, shown in a set fraction of training examples. With repetition a datapoint feature (a term the post links to Superposition, Memorization, and Double Descent) forms: a unit that fires on p and writes out p. A model that memorized p would warrant such a feature, but the absolute-value network has no special notion of p, so the feature is not mechanistically faithful: "The transcoder is memorizing something which the underlying model didn't."
What the toy model shows
When datapoint features form. The post argues that two conditions are needed when the transcoder, not the model, does the memorizing: a repeated datapoint in the transcoder's training data, and a sparsity penalty that makes one dedicated unit cheaper than several ("When Do Datapoint Features Form?"). A scan of L1 coefficient and repeat fraction, each from 0.01 to 0.1, is plotted in the figure titled "When do 'Dataset Features' Form?", whose annotation says memorization "roughly tracks a constant (L1 coef)*(Repeat Fraction) frontier". Memorization is a score the post calls crude: for each feature, the part of its output on p along the direction of p that exceeds its largest output along any one of the three coordinates, summed over features ("Aside: How do we Detect a Datapoint Feature?"). No seeds or variability are reported.
Baseline and light repetition. A visualization explains each transcoder unit's output over many inputs as a combination of ideal circuits (the ReLU(x) and ReLU(−x) units and a memorization circuit), fitted by compressed sensing, with a residual for what is left ("Aside: How does this visualization work?"). Without repetition the transcoder learns almost exactly the ideal units; for example, the unit for the first coordinate has encoder weight 0.85, decoder weight 1.19 and bias −0.05 ("No Repeat Baseline" figure). With p in 5% of examples, a setting chosen to push the transcoder "weakly into the datapoint feature regime", a datapoint feature appears, with encoder weights of 0.63 on each of the three coordinates, decoder weights of 0.76 and bias −0.62 ("Light Repetition" figure). The units for those coordinates are, in the post's words, rotated so they do not fire on p: about 0.74 on their own coordinate, near −0.26 on the other two, bias about −0.21. This note's arithmetic from those labels: on p a rotated unit's input is about 0.74 − 0.27 − 0.25 − 0.21 = 0.01, while the datapoint feature's is 3 × 0.63 − 0.62 = 1.27. The post concludes that this "demonstrates that mechanistic faithfulness can be a real problem, even in very simple setups!"
The Jacobian as a signature. On p, the baseline transcoder's Jacobian over the three coordinates is diagonal (1.01, 1.02, 1.01), while the transcoder with the datapoint feature has every entry 0.48, so each output depends on all three inputs (figure in "Jacobian Regularization: A Potential Solution?"; 0.76 × 0.63 = 0.48 is this note's arithmetic). The post's reason to treat the Jacobian as principled: weights are not canonical (neurons can be permuted and rescaled), but a ReLU layer's Jacobian is a sum of outer products of the active units' input and output weights, in some sense "a natural canonicalized form of the parameters" that, across inputs, also records which units fire together.
Jacobian matching. The post adds to the loss the squared Frobenius norm of the difference between the original layer's Jacobian and the transcoder's, estimated cheaply with random vectors ("Jacobian Matching"), and reports that this "dramatically helps" the memorization score. In the scan with a coefficient of 2.0, the high-memorization corner is gone; this note reads the remaining scores as mostly below about 0.4, with a few cells near 0.6 at the highest repeat fractions (read from the figure, which prints no cell values). In the 5% setting, matching "basically" fixes the problem: the datapoint feature is gone (the memorization row of the map is near zero), and the three coordinate units have weights of about 1.0 on their own coordinate with small off-diagonal weights (−0.04 in, 0.07 out), which the post calls "a bit rotated" ("Jacobian Matching Can Solve Mechanistic Faithfulness (In This Example)").
Caveats the post raises about its remedy
- Gradient masking. A transcoder might create units with large weights, hence large gradients, but very negative biases so they barely fire and add little to the L1 penalty ("Can the Transcoder Cheat?"). The tanh penalty helps, since the trick exploits the gap between L1 and L0, and the post suggests a further penalty on each unit's contribution to the Jacobian if needed. It does not report testing it.
- Superposition. A perfect Jacobian match may be undesirable in real models ("Superposition and Jacobian Matching"). If a feature is spread over ten neurons that also carry other features, the Jacobian should show one large singular value for it and nine small ones; that tail is, in the post's view, an artifact of superposition not worth modeling. The accompanying figure is a sketch of expected spectra, not a measurement.
- Cost. The exact penalty "would be extremely expensive"; the post uses a random-vector estimate instead. No cost on a real model is reported.
Faithfulness and simplicity
The conclusion sorts concerns about SAEs and transcoders into two families. Mechanistic faithfulness asks whether the transcoder captures the true mechanisms; failure "cuts at the heart" of the project, while success means transcoders can be used "to reliably learn true things". Simplicity asks whether mechanisms are captured in the simplest way, for example whether a feature manifold is approximated by many discrete features; failure is "more of a tax" on understanding. The post says many concerns touch both, that faithfulness is probably a spectrum, and that it is the one "we're presently most concerned with".
Limitations
- A toy, not a language model. Whether real transcoders form unfaithful datapoint features, and how often, is not tested. The acknowledgments say colleagues worked on how the ideas relate to real models, work left out of the post.
- Single illustrative runs. Weight panels and Jacobians come from individual trained transcoders; neither they nor the scans report seeds or variability.
- A metric tailored to the toy. The memorization score needs the repeated point and the ideal circuits known in advance, which real models do not supply.
- Remedy shown in one example, as the section heading says, and the post does not claim it is ideal.
Research questions
- Do datapoint features appear in transcoders trained on real language models? Training transcoders on data with known duplicated documents and looking for units that reproduce a duplicated passage where the underlying MLP shows no matching memorization would answer it.
- Does the product rule for memorization hold beyond the toy? Repeating the scan on a small language model with several seeds would show whether a constant-product frontier still predicts where datapoint features form.
- What does Jacobian matching cost at scale? Error against L0 curves with and without the penalty on a real MLP layer, with training cost, would show whether the trade is affordable.
- How much Jacobian mismatch is superposition, and how much unfaithfulness? Comparing singular-value spectra of MLP and transcoder Jacobians, and testing whether the gap sits in the predicted tail, would separate the two.
- Does Jacobian matching invite gradient masking? Searching trained transcoders for units with large weights and strongly negative biases, with and without a tanh penalty, would test the concern.
- Does Jacobian agreement predict behavior off distribution? Measuring whether a transcoder's Jacobian agreement on training data predicts its error on shifted inputs would test the post's motivating claim.
How it connects
- Circuit Tracing: Revealing Computational Graphs in Language Models (March 2025) lists mechanistic faithfulness as a limitation of its cross-layer transcoders and validates mechanisms after the fact with perturbation experiments. The post's opening links that limitations section where it says the topic had been discussed briefly in recent papers.
- Superposition, Memorization, and Double Descent (2023) is the page the post links when it names the "datapoint feature".
- A later outside post, Lange, RGRGRG, Dearstyne and Maher, "Cross-Layer Transcoders are incentivized to learn Unfaithful Circuits" (LessWrong, 2 February 2026; a one-week research sprint), cites this post as showing that per-layer transcoders can be unfaithful and reports a different failure in cross-layer transcoders: in a hand-built two-layer Boolean model, a cross-layer transcoder reconstructed every output while activating no second-layer features, skipping the model's intermediate step. Its evidence from GPT-2 small and Gemma 2 2B is, in its own words, indirect and "suggestive but not definitive".
- Sparse mixtures of linear transforms (July 2025) measures the faithfulness of transcoders and of an alternative on language models by Jacobian correlation. The two do not cite each other; the connection is this note's.
- HeadVis (2026) links this post when it says transcoder features may be mechanistically faithful to the original MLP, an assumption that is sometimes not true.
- When Models Manipulate Manifolds (2025) studies feature manifolds of the kind the post uses to illustrate its "simplicity" concern; this connection is also the note's.
References
- Olah, C. (2025). A Toy Model of Mechanistic (Un)Faithfulness. Transformer Circuits Thread, 7 August 2025. https://transformer-circuits.pub/2025/faithfulness-toy-model/index.html. Section and figure titles in this note refer to the web version. record · 2 pages
- Dunefsky, J., Chlenski, P., and Nanda, N. (2024). Transcoders Find Interpretable LLM Feature Circuits. NeurIPS 2024. arXiv:2406.11944. (Cited by the post for transcoders.) record
- Templeton, A., Batson, J., Jermyn, A., and Olah, C. (2024). Predicting Future Activations. Circuits Updates, January 2024. https://transformer-circuits.pub/2024/jan-update/index.html#predict-future. (Cited by the post for transcoders.) record
- Braun, D., Bushnaq, L., Heimersheim, S., Mendel, J., and Sharkey, L. (2025). Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition. arXiv:2501.14926. (Cited by the post as an example of weight-based interpretability.) record
- Lange, G., RGRGRG, Dearstyne, K., and Maher, K. (2026). Cross-Layer Transcoders are incentivized to learn Unfaithful Circuits. LessWrong, 2 February 2026. https://www.lesswrong.com/posts/6CS2NDmoLCFcEJMor/cross-layer-transcoders-are-incentivized-to-learn-unfaithful record · 2 pages
- Ameisen, E., Lindsey, J., et al. (2025). Circuit Tracing: Revealing Computational Graphs in Language Models. Transformer Circuits Thread. https://transformer-circuits.pub/2025/attribution-graphs/methods.html record · 15 pages