Circuits Updates – March 2024

A Circuits Update from Anthropic's interpretability team, published 21 March 2024, with a worked example of starting circuit analysis from sparse autoencoder features (sports facts about athletes in an 18-layer model), a report that three earlier training interventions, including ghost grads, no longer help after later changes, and commentary on outside work about Othello-GPT, computation in superposition, attribution patching and self-repair.

Joshua Batson, Brian Chen, Andy Jones, Adly Templeton et al. (Anthropic) · 2024-03-21 · original

The March 2024 Circuits Update sits between Towards Monosemanticity (October 2023), which extracted interpretable features from a one-layer transformer with a sparse autoencoder, and Scaling Monosemanticity (May 2024), which did the same for Claude 3 Sonnet. It has three items. The first shows, on one example prompt per sport, how features from a sparse autoencoder can be used to find the attention heads that carry a fact from a name to a prediction, without building a task-specific dataset. The second reports that three training interventions the January and February 2024 updates had recommended, among them "ghost grads", no longer give meaningful improvements after later changes to the team's training. The third summarizes and comments on four groups of outside papers. The update asks readers to treat its results "like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper" (introduction). It is about 3,100 words with two figures, has no numbered sections or overall byline, and lists separate authors under each item; locations in this note are item titles. The page prints no date; the Transformer Circuits index dates it 21 March 2024, and the earliest Internet Archive capture (28 March 2024) is consistent with that.

Terms used in this note

  • Sparse autoencoder (SAE), feature, dictionary. An SAE maps a model's activation vector to a much larger, mostly zero vector of feature activations and reconstructs the activation as a weighted sum of learned decoder directions; the set of directions is the dictionary, and each direction is a feature. Expansion factor is the ratio of the number of features to the activation's dimension. See Towards Monosemanticity.
  • Residual stream. The vector that runs through a transformer, to which each layer adds its output.
  • Attribution. A gradient-based estimate of how much a component contributes to an output: the component's activation times the gradient of the output with respect to it. It is a linear approximation, not an intervention.
  • Logit difference. The difference between the model's scores for two candidate next tokens, used here to isolate what favors one answer over another.
  • Direct logit effect of a head. What an attention head's output would add to the logits if it passed straight to the unembedding, computed from the weights (the head's OV circuit applied to a feature's decoder direction, then normalized and unembedded) rather than by running the model.
  • Activation patching and attribution patching. Activation patching replaces one component's activation with its value from another prompt and reruns the model to see whether the behavior changes. Attribution patching approximates the same effect with gradients, using one forward and one backward pass.

Using features for circuit identification

Joshua Batson, Brian Chen and Andy Jones argue that SAE features can replace the usual first step of circuit finding, which is to assemble hundreds or thousands of examples of one behavior and ablate or patch components across them. (The item's heading reads "Using Features For Easy Circuit Identification"; the update's table of contents calls it "Using Features For Fast Circuit Identification".) Because a dictionary trained on the full data distribution already encodes what the model represents, the authors propose that one example text can generate hypotheses, and the features found can then be studied in general through the other features they interact with.

Setup. The case study follows Nanda et al.'s work on factual recall, which studied completions of "Fact: [ATHLETE NAME] plays the sport of" for about 1,500 athletes who play basketball, baseball or tennis and described three phases: moving information from all tokens of the name to its last token, enriching that last token with facts in early MLP layers, and extracting the sport with attention heads (mainly one head) that move it to the final position. The update studies an unnamed 18-layer model with an SAE of expansion factor about 256 trained on the residual stream at layer 9.

Finding the sport features. For the prompt "Fact: Michael Jordan plays the sport of", the authors compute an attribution score for every SAE feature at the " Jordan" position in layer 9, with respect to the logit difference between " basketball" and " golf". The second sport is there to pick out features that favor basketball in particular rather than sports in general. The item reports "a clear top feature by attribution", whose top-activating dataset examples include "layup", "half-court" and "point guard". Since a feature's activation is a ReLU of an affine function of the residual stream, the authors note that this yields a basketball probe from a single example, and that the enrichment phase is under way by layer 9. Repeating the analysis for "Babe Ruth" and "Serena Williams" gave top features that fire on baseball team names ("Dodgers", "Orioles", "yankees") and on tennis text ("WTA").

The first figure ranks the top five features for each prompt with bars of attribution score and example texts, but prints no values; its axes run from 0 to 0.25 (Jordan), 0.3 (Ruth) and 0.13 (Williams). Measured in pixels (this note's reading), the second-ranked bar, a feature for the ends of proper nouns, is about 80% as long as the first for Jordan and about 93% for Ruth, while for Williams the second bar is about 13% of the first. So the top feature stands well clear of the rest for Williams, but for Ruth an end-of-name feature scores almost as high.

Name-ending features and a small test. Several features that respond to the ends of names (the figure's examples include "Judas Priest", "John Cena" and "Carl Jung") had large attributions for "Jordan" and "Ruth" but not for "Williams". The authors hypothesized that the model links tennis to the first name "Serena" alone, but needs the full name for "Michael Jordan" and "Babe Ruth". For "a handful of athlete names" they prompted "Fact: [ATHLETE FIRSTNAME] Smith plays the sport of" and report that the model's prediction matched the one for the real name in "exactly the cases" where the "Judas Priest" feature had low attribution on the original text, and was damaged where it had high attribution. The item does not say how many names were tried or list them. The authors take this as further evidence that the model "detokenizes" its input in early layers, assembling multi-token names into units.

Finding the heads. The authors then looked for attention heads whose direct logit effect, starting from the basketball feature's decoder direction, strongly favors " basketball" over " golf". They report "many such heads, across many layers", with the top five in layers 11, 13 and 14, and that the same heads, applied to the baseball and tennis features, produce baseball- and tennis-related tokens. The second figure lists the top five tokens and their direct-effect scores for each sport feature through heads L11H1, L11H2, L13H1, L14H1 and L14H2. The highest printed scores are 32.08 ("Basketball", through L11H2), 29.75 ("baseball", L11H1) and 33.38 ("Tennis", L11H2). Not every top token is on topic: "Maya" appears for basketball through L11H1, "cach" through L13H1, and "golf" (the contrast token), "Croatia", "violin" and "Golf" for tennis through L14H1 and L13H1. The authors compare the result with the single sport-extracting head of Nanda et al. and with the many "Subject Heads" that Chughtai et al. found acting together in factual recall.

Strength of the evidence. The item works from three prompts, one per sport. Both steps are computed rather than tested by intervention: feature attributions are gradient approximations, and the head effects are read from the weights. No ablation or patching confirms that the heads move the sport in the model's forward pass, and the update gives no measure of how much of the logit difference the identified features and heads account for. The authors present it as an illustration: "We hope this exercise demonstrates the utility of using SAEs to kickstart circuit analysis."

Update on dictionary learning improvements

Adly Templeton, Tom Conerly, Jonathan Marcus and Tom Henighan report that changes to their SAE training since the earlier updates, "including bug fixes", have lowered training losses, and that with those changes three previously recommended interventions no longer give meaningful improvements:

  • Ghost grads (from the January and February 2024 updates), an extra loss term meant to revive dead features. Training loss is now roughly equal with or without it, "even on 1L models", which does not justify the extra compute, and the team has "some evidence" that its implementation causes loss spikes in the current setup.
  • Adam beta parameters of (0, 0.9999) instead of the defaults (0.9, 0.999), from the February update: training loss is now similar either way.
  • Pruning features with decoder norm below 0.99, from the February update: other changes left far fewer features dead or below that norm, so pruning no longer helps.

The authors say these were real improvements to the process they used at the time, and that the changes superseding them would be published "in a future update once we've vetted them further". The item gives no numbers, plots or description of the new changes. The January and February 2024 pages now carry notes saying the team's views on these points have changed, pointing to this update. This note connects the promise to the "Update on how we train SAEs" item in Circuits Updates – April 2024, by mostly the same authors, which describes a recipe that uses neither resampling nor ghost grads, keeps Adam's default betas and drops the unit-norm constraint on decoder columns; the April item does not refer back to this one.

Research by other groups

Joshua Batson, Nicholas L. Turner, Brian Chen and Adam Pearce summarize recent outside work and add the team's comments. The summaries below give each paper's claims as the update reports them; the comments are the team's.

Othello and automata. He, Ge, Tang, Sun, Cheng and Qiu ("Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT") train SAEs on the word embeddings, MLP outputs and attention outputs of a model that plays legal Othello moves, find features for the state of particular board cells, and trace how later features are computed from earlier ones through the weights on specific examples. In one example an MLP feature for a cell being "mine" is computed as an AND of three attention features. The team's comment notes that the SAEs are not very sparse (mean L0, the number of active features, of about 60 to 100 against a model width of 128, with near-zero reconstruction error), says that in its experience such dictionaries tend to be polysemantic at low activations, and suggests sparser dictionaries and the specificity and sensitivity analysis of Towards Monosemanticity. The team adds a structural point: playing out N moves naively takes N sequential steps, but an L-layer transformer allows only about 2L, so a model that handles games longer than that must use another algorithm. It links this to Liu et al.'s "Transformers Learn Shortcuts to Automata" and to Sanford, Hsu and Telgarsky's result that k-hop induction tasks can be done at depth log(k).

Computation in superposition. A LessWrong post, "Toward A Mathematical Framework for Computation in Superposition" (Vaintrob, Mendel and Hänni, per the update's bibliography), gives constructions of transformer layers that compute many Boolean functions of sparse inputs stored in superposition. Among the results the update highlights: when the inputs are not stored in superposition, a single MLP layer can compute all pairwise ANDs of them so that each can be read out linearly with small error; with inputs in superposition, the number of ANDs computed can approach the number of weight parameters if inputs are sparse enough; and with a quadratic nonlinearity the count can be superpolynomial in the number of MLP units. The constructions rest on a notion of "linearly represented" (readable by some linear function up to a small error) weaker than closeness in squared error. Because the constructions are information-theoretically tight, the team considers them "the kind of thing transformers might actually learn", and passes on the authors' suggestion that dictionary learning might do better with a thresholding activation than with a ReLU.

Improvements to attribution patching. Kramár, Lieberum, Shah and Nanda ("AtP*") identify two failure modes of attribution patching: nonlinearities immediately after the patched activation, chiefly the attention softmax, whose gradients nearly vanish, and cancellation between a component's direct and indirect effects. Their fixes are recomputing attention past the softmax, which the update says costs more than plain attribution patching but much less than a full forward pass per patch, and averaging over backward passes that each skip one layer. They validate them against activation patching on Pythia-12B. The team adds that it had seen the first failure mode in its own one-layer experiments, where LayerNorm saturation also broke the approximation.

Self-repair and LayerNorm. Rushing and Nanda ("Explorations of Self-Repair in Language Models") measure self-repair, the tendency of later components to make up for an ablated one, as the gap between the effect of ablating an attention head and the head's direct effect on the next-token logit. They find that LayerNorm's nonlinearity explains about 30% of the self-repair on the 2% of tokens where the head's direct effect is largest: when several components push toward the same token, removing one shrinks the residual stream's norm, LayerNorm scales the rest up, and the other components' logits rise. The team reads this, with work by McGrath et al. and McDougall et al., as suggesting that at least some self-repair is a by-product of LayerNorm rather than something training directly selected for.

Limitations and open questions

  • Preliminary by design. The update asks to be read as lab-meeting results, and the circuit item calls itself an "exercise".
  • Three prompts, no interventions. The circuit-identification example rests on one prompt per sport, a handful of name substitutions with no count given, attribution scores and weight-based head effects. No ablation or patching confirms the heads' role, and the update reports no measure of how much of the behavior the features and heads explain. The model is unnamed apart from its 18 layers.
  • Partly ambiguous top features. The text's "a clear top feature by attribution" is said of the Michael Jordan prompt, where the second bar is about 80% of the first; for Babe Ruth the text says only that the top feature fired on baseball text, and there an end-of-name feature scores almost as high, about 93% of the first (this note's pixel reading of a figure that prints no values).
  • Superseded advice without the replacement. The training item says three interventions no longer help but defers the changes that superseded them to a later update, so readers of this update cannot check why the earlier tips stopped helping. The January and February updates mark the withdrawn advice as changed.
  • Readings of external work. The comments on other groups' papers, including the reading of self-repair as a LayerNorm by-product and the claim that the superposition constructions are the kind of thing transformers might learn, are the team's interpretations, not results it tests.

Research questions

  1. Do the heads found from weights carry the sport in the forward pass? The item identifies heads by direct logit effect. Ablating or patching heads L11H1, L11H2, L13H1, L14H1 and L14H2 on the athlete prompts, and measuring the change in the sport logit difference, would test it.
  2. How far does one-example feature attribution generalize across the athlete set? The item uses one athlete per sport. Running the same attribution over the roughly 1,500 athletes of Nanda et al. and recording how often the same sport feature ranks first would measure reliability.
  3. Does the end-of-name feature track whether the model needs the full name? The Smith test used a handful of names. A larger set of athletes, with the first-name-plus-Smith completion scored against the feature's attribution on the full name, would give a rate and a correlation.
  4. What fraction of the logit difference do the top features and heads explain? The update gives rankings, not shares. Comparing the logit difference with only the identified features or heads left intact against the full model would answer it.
  5. Which change made ghost grads unnecessary? The training item attributes the change to later fixes without naming them. Ablating each change in the later recipe with and without ghost grads, across model sizes, would locate the cause.
  6. Do sparser Othello dictionaries give crisper circuits? The team suggests this in its comment on He et al. Retraining the Othello SAEs at lower L0 and comparing how many features each traced computation needs, and how well ablations confirm them, would test the suggestion.

How it connects

  • Towards Monosemanticity supplies the SAE method and the specificity and sensitivity analysis the comments recommend; Scaling Monosemanticity (May 2024) cites the circuit item ("Batson et al.") in its related work on sparse feature circuits, cites this update's discussions of the Othello and computation-in-superposition papers, and lists the update among the team's periodic reviews of recent work.
  • The training item withdraws advice from Circuits Updates – January 2024 and Circuits Updates – February 2024. Circuits Updates – April 2024 describes the team's next training recipe (the link is this note's), and its "Open Problem: Attribution Dictionary Learning" item cites the circuit item as where the team "recently used" the attribution vector it proposes to make sparse.
  • This note connects the circuit item to later work that the update could not cite: the attribution graphs of Circuit Tracing and On the Biology of a Large Language Model (March 2025) also start from features on single prompts, and test the paths they find by intervention. The "detokenization" reading echoes the speculative layer-by-layer picture in Circuits Updates – July 2023.

References