Circuits Updates – September 2024

The September 2024 Circuits Update (published 1 October) has two unrelated items. The first is a replication. Gould et al. (2023) had reported "successor heads", attention heads that recur across language models and push the prediction from one item of an ordered sequence to the next. Anthropic's team checks the finding in its own 18-layer model and uses it to compare four ways of locating such heads, including one new method. The second item is a short safety-motivated result: a sparse autoencoder (SAE) trained on Claude 3 Sonnet's activations found few features specific to bioweapons, and changing the SAE's training data changed that. It bears on a problem raised in earlier updates, that dictionary learning finds only a fraction of a model's features (see Circuits Updates – July 2024).

About the Circuits Updates format

Circuits Updates are informal posts on Anthropic's Transformer Circuits site, each a bundle of short, separately authored items. This one asks readers to treat its results "like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper" (introduction). Locations below are item and subsection titles.

Terms used in this note

• Successor head. An attention head that maps an item of an ordinal sequence to the next one: 2 to 3, Wednesday to Thursday, G to H.
• OV circuit. The part of an attention head that determines what attending to a token adds to the output; multiplying a token's representation through it and through the unembedding shows which output tokens the head promotes.
• Independent components analysis (ICA). A method that decomposes data into components assumed to be statistically independent and non-Gaussian.
• Mean ablation. Replacing a component's output with its average output and measuring the change in the model's predictions.
• Attribution patching. A fast linear approximation, using gradients, to the effect an ablation would have.
• Feature coverage. How many of a model's features an SAE actually learns.

Investigating successor heads

Emmanuel Ameisen and Joshua Batson (edited by Jack Lindsey) open by noting that some mechanisms, such as induction, previous-token tracking and copy suppression, tend to sit in a few heads across many transformer models ("Investigating successor heads"). They replicate Gould et al.'s finding that a few successor heads implement succession across many ordinal sequences, using four complementary methods in an 18-layer model.

Weight inspection

Following Gould et al., the team built a dataset of ordinal sequences, each item a single token: numbers as Arabic numerals, Roman numerals and English words; letters; and days and months, abbreviated and in full ("Weight inspection: output-value circuit"). Each token is represented by the residual stream after the first MLP layer, where, as Gould et al. had noted, the representation changes sharply from the embedding. For every head, these representations are passed through the head's OV circuit and the unembedding, and the resulting logits are restricted to the ordinal tokens. Each head is scored by the fraction of ordinal tokens whose top-scoring ordinal output is their successor. The item notes this gives sensible results even for late-layer heads, though the representations come from an early layer.

In the top-scoring head, about 80% of ordinal tokens map most strongly to their successor. The matrix also shows block structure: after the successor come the equivalent tokens in other lists (1 maps to 2, but also to "two", "second" and "February") and later tokens in the same list, while earlier tokens in a list are almost always suppressed. The second-scoring head maps about 60% of tokens to their successor, and its errors stay within a category, numbers to numbers and days or months to days or months.

Independent components analysis of heads

Because both successor heads mixed succession with category membership, the team looked for shared patterns across all heads ("Independent Components Analysis"). It flattened and stacked every head's ordinal matrix and applied ICA, chosen because component loadings were expected to be sparse and heavy-tailed rather than Gaussian, and somewhat independent, allowing for heads to share functions in superposition. Each component acts as a "meta-head", and each real head is a weighted sum of components. The item notes dictionary learning would also be plausible, and would separate some behaviours that ICA merges.

Three kinds of component stand out: a succession component with some bleed between related lists (1 to 2, "two" or "second"); an induction component that repeats the token, with similar bleed (Monday to Monday or "Mon"); and category components, for example one that maps every English number and numeral to a near-uniform mix of small numbers while making each Roman numeral suppress all Roman numerals. The authors call these descriptions not entirely clean. The heads that scored highest by weight inspection also had the largest coefficients on the succession component.

Ablations

A high weight-based score does not show that a head is used: it might not attend to the right tokens, its output might be too small to matter, or its indirect effects might dominate ("Ablation Studies"). The team ran the model on full sequences ("1 2 3 4 5 ..."), mean-ablated each head's output, and averaged the effect on the logit of the correct successor. The top three heads by component score, which were also the top three by weight inspection, ranked 1, 3 and 5 by ablation effect, so heads found from weights alone matter on real inputs. The heads ranked 2 and 4 by ablation had low component scores and sat in layers 3 and 5, earlier than the three successor heads in layers 10, 11 and 13; the item suggests they may help by feeding the queries or keys of the later heads, or by influencing later MLP layers.

Attribution

Head attributions computed with attribution patching agreed with the other methods on the top head, but the team was "surprised to find relatively low agreement" between attribution scores and the three other methods overall ("Attribution analysis").

Conclusion of the item

The item reproduces Gould et al.'s finding of several successor heads acting through direct effects in a small model ("Conclusion"). It proposes checking where these heads are used, for example in incrementing list numbers or years, or in prose such as "The next day, Tuesday". It also notes that two ICA components, one implementing succession while partly losing track of which sequence and one copying the sequence category while losing track of position, are consistent with Gould et al.'s finding that ordinal value and sequence category are represented as separate linear factors, and suggests that decomposing heads in this way might find such factors more generally.

Oversampling a topic in SAE training data

Trenton Bricken, Jonathan Marcus, Kelley Rivoire and Thomas Henighan (edited by Adam Jermyn) report a result from work on safety applications of dictionary learning ("Oversampling a Topic in the SAE Training Set Results in More Detailed Features Related to that Topic"). They looked for features in Claude 3 Sonnet specific to bioweapons and found that even their largest Sonnet SAE, with 34 million features, contained few. They then added synthetically generated bioweapons data to the SAE's training mix, expecting more such features to be learned.

Before the change, the most relevant features concerned resources for making dangerous or illegal items in general, such as bombs and recreational drugs, which the team found broader than it wanted. After it, the most important feature "for predicting harmful bio prompts" concerned text about modifying pathogens, and another concerned dispersal of viral particles. The item does not give counts of features before and after, or describe how importance for prediction was measured.

The authors present this as a possible answer to the feature coverage problem: SAEs do not learn all features in a layer, and full coverage might need more compute than current pretraining capabilities provide, but oversampling safety-relevant material can steer an SAE toward the features that matter for safety.

Limitations and open questions

• One small model for successor heads. The replication uses one 18-layer model, and the authors list unanswered questions about the contexts in which the heads are used.
• Disagreeing methods. Attribution agreed poorly with the other three methods except on the top head, and the reason is not analysed.
• Thin reporting on oversampling. The bioweapons item is four short paragraphs long, with no numbers, no description of the synthetic data, and no comparison of other features before and after; whether oversampling one topic costs coverage of others is not discussed.

Research questions

1. Why does attribution patching disagree with ablation for successor heads? The item reports low agreement without explanation. Comparing attribution and ablation effects head by head, and checking whether the disagreement concentrates in early-layer heads with indirect effects, would locate the cause.
2. Are successor heads used in natural text? The authors ask whether the heads act in prose and in incrementing years or list numbers. Ablating the identified heads on such text and measuring the change in successor predictions would answer it.
3. Do ICA components of heads correspond to linear factors in the residual stream? The item suggests they might. Comparing ICA components with directions for ordinal value and category found by probing would test the suggestion.
4. What does oversampling cost elsewhere? Adding bioweapons data changed which features appear. Measuring reconstruction loss and the number of interpretable features on other topics, with and without oversampling at fixed dictionary size, would show whether coverage is redistributed or added.
5. How much oversampling is needed? The item reports one mix. Varying the share of topic data and recording when topic-specific features appear would give a dose-response curve useful for planning SAE training.

How it connects

• Within this library. Using dictionary learning features as classifiers (October 2024) reports results using the 18-layer model from this update's successor heads item and says it oversamples biology data in its dictionary learning runs. Insights on Crosscoder Model Diffing (February 2025) lists the oversampling item among its references. For induction and previous-token tracking, the item links to In-context Learning and Induction Heads. The feature coverage problem is set out in Circuits Updates – July 2024 as "missing features" and "dark matter", and the 34 million feature SAE is the largest dictionary of Scaling Monosemanticity (Mapping the Mind of a Large Language Model).
• In the course wiki. Decomposing a stack of head matrices into shared components is a matrix factorization, like the singular value decomposition behind latent semantic indexing; ICA differs in seeking statistically independent, non-Gaussian components rather than orthogonal directions of greatest variance.

References

• Ameisen, E., Batson, J., Lindsey, J., Bricken, T., Marcus, J., Rivoire, K., Henighan, T., and Jermyn, A. (2024). Circuits Updates – September 2024. Transformer Circuits Thread, published 1 October 2024. https://transformer-circuits.pub/2024/september-update/index.html (Anthropic landing page: https://www.anthropic.com/research/circuits-updates-sept-2024)
• Gould, R., Ong, E., Ogden, G., and Conmy, A. (2023). Successor Heads: Recurring, Interpretable Attention Heads In The Wild. arXiv:2312.09230.