Circuits Updates – July 2024

interpretabilitysparse autoencoderslinear representation hypothesisfeature sensitivitycircuits updates

An Anthropic interpretability update with three short essays by Chris Olah (five remaining hurdles for mechanistic interpretability, a redefinition of "linear representation" that admits multidimensional features, and rare features as the "dark matter" of neural networks) and two methods reports, on reproducing attention pivot tables in one-layer models and on using Claude to measure how reliably Scaling Monosemanticity's features fire when their concept is present.

Chris Olah, Adam Jermyn, Nicholas L. Turner et al. (Anthropic) · 2024-07-31 · original

Two months after Scaling Monosemanticity (summarized in Mapping the Mind of a Large Language Model) showed that sparse autoencoders could extract millions of interpretable features from Claude 3 Sonnet, the July 2024 Circuits Update takes stock of what that success leaves unsolved. Three essays by Chris Olah, edited by Adam Jermyn, set out the problems the team expects next, revise a definition the team had used since Toy Models of Superposition, and argue that most features may be too rare to find. Two shorter reports test the team's own methods: one reproduces an early circuits result and finds it fragile, and one measures how often the published Sonnet features actually fire when their concept is present, and finds that they often do not.

About the Circuits Updates format

Circuits Updates are informal posts on Anthropic's Transformer Circuits site, each a bundle of short, separately authored items. This one asks readers to treat its results "like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper" (introduction). The first three items are essays written in the first person by Chris Olah; locations below are item titles.

Terms used in this note

  • Feature. A direction (or, in this update, possibly a set of directions) in a model's activation space that corresponds to a concept; extracted here with sparse autoencoders (SAEs).
  • Superposition. A model representing more features than it has dimensions, by overlapping them (see Toy Models of Superposition).
  • Circuit. The weights and features through which a model computes one feature from others.
  • QK and OV circuits. In an attention head, the QK circuit decides how much each query token attends to each key token, and the OV circuit decides what attending to a token adds to the output (A Mathematical Framework for Transformer Circuits).
  • Skip-trigram. A pattern of the form "token A ... token B predicts token C", which a one-layer attention model can implement.
  • Specificity and sensitivity. As defined in the sensitivity item: specificity is the probability that a concept is present given that a feature fires; sensitivity is the probability that the feature fires given that the concept is present. (In information retrieval terms these are precision and recall.)

The next five hurdles

Olah writes that a year earlier he would have called superposition the field's most important open problem, and that substantial progress since then (citing Bricken et al., Cunningham et al., Templeton et al. and Gao et al.) makes it time to ask what remains ("The Next Five Hurdles"). He sketches the intended path: find interpretable features as the "variables" of the computation, understand the circuits that compute them, and turn this microscopic picture into a macroscopic one that answers the questions people care about. Along that path he sees five hurdles:

  1. Missing features. Feature extraction now works, but probably recovers only a small fraction of features; very rare ones may never be resolved without major algorithmic advances (see the dark matter item below).
  2. Cross-layer superposition. In a deep model whose circuits are shallow (the example is 100 layers with circuits mostly under 10 steps), adjacent layers may be nearly interchangeable to gradient descent, so a feature's implementation may not map to one layer. The essay says the team's basic approach to dictionary learning provides no way to address this.
  3. Attention superposition. Combinations of attention heads may represent "attentional features" in the same way combinations of neurons represent features, so the field may be missing basic units as important as the induction head.
  4. Interference weights. If the true circuit weights are themselves in superposition, multiplying out weights between features will show "interference weights" that exist only as a compromise to fit other circuits, which make circuit analysis confusing and may help explain adversarial examples.
  5. Zooming out. Understanding every feature and circuit would still not be understanding the network; Olah says automated interpretability will help but he would prefer an answer in terms of larger-scale structure, universality and macroscopic correspondences.

He ends with grounds for optimism: there are plausible routes to traction on many of them, and unlike superposition, these problems do not necessarily block one another, so they can be attacked in parallel.

What a linear representation is, and multidimensional features

The second essay responds to Engels et al.'s paper on multidimensional features (reviewed in Circuits Updates – June 2024), which Olah says showed that some of the team's writing on linear representations had been unclear ("What is a Linear Representation? What is a Multidimensional Feature?").

A definition that conflated two claims. Toy Models defined a representation as linear if features correspond to directions, so that several active features are represented by the sum of their directions scaled by their values. Olah argues this mixes two claims: that features are one-dimensional, and that features behave linearly in the mathematical sense ("A Confusing Definition").

The narrower definition. In practice, he writes, the team now uses "linear representation" for the second property only, whether or not features are one-dimensional ("Mathematical Linear Representations"). A representation is linear if it has:

  • composition as addition: features present together are represented by adding their vectors;
  • intensity as scaling: the strength of a feature, or the model's confidence in it, is represented by magnitude.

A possible third condition, that a feature's intensity can be read off with a linear function up to noise, is left open. Olah calls mathematical linearity "much deeper": if features turn out to be multidimensional, most mechanistic interpretability work can adapt with little change, but if representations are not mathematically linear, many basic questions, such as how to think about weights, reopen.

What linearity requires of a multidimensional feature. Composition as addition requires the feature's manifold to be orthogonal, or almost orthogonal under superposition, to other features (a footnote says this is sufficient rather than strictly necessary). Intensity as scaling is more restrictive: each point on the manifold must extend along a ray as it is scaled, so the whole manifold shrinks to zero along these rays, and changes in meaning must be movements around the manifold at right angles to the rays ("Linear Multidimensional Features").

Embedding and noise. Nearby points on a feature manifold are nearly parallel, yet the model may need to tell them apart more sharply than that allows. Olah suspects manifolds may be embedded in more complicated ways than their topology suggests to achieve a needed distance structure, so that variation beyond a clean projection on the first principal components "may be essential" rather than noise ("Aside: Potential Issues around Feature Manifold Embeddings").

Probes and steering still apply. Many results cited for the linear representation hypothesis read information with a linear probe or steer by adding a vector. Olah argues these are statements about mathematical linearity, not one-dimensionality: each ray of a linear multidimensional feature behaves like a one-dimensional feature for a fixed meaning, so a probe for one predefined property still finds a direction. The difference is that the rays are not privileged. Whether something is a multidimensional feature or a set of co-occurring one-dimensional features may depend on continuity; for example, if "midnight on Tuesday" were represented between Tuesday and Wednesday, that would favour a continuous representation ("Multidimensional Features and Directions").

On definitions. Olah says he is "not at all confident" the definitions will last, cites Lakatos on the productive evolution of definitions, and links back to the April update's essay on productive wrong theories ("On Definitions").

The dark matter of neural networks

The third essay argues from an example ("The Dark Matter of Neural Networks?"). The post says the SAE from the team's recent work (linked to Scaling Monosemanticity) "resolved 13 million features", including features for some San Francisco neighbourhoods. Yet Claude knows about small coffee shops and street intersections in the city; assuming at least a hundred such places per neighbourhood suggests "at least two orders of magnitude more features", each correspondingly rarer. People are another example ("Claude knows who I am", Olah writes, but no feature for him has been extracted), as are possible memorization features. Such rare features may be the large majority of features, though individually less important, and below some frequency they may be effectively impossible to resolve.

Olah uses dark matter in its original, broad sense of matter inferred from its influence but not observed directly (footnote 10), and extends the image: dictionary learning is a telescope that first saw only the brightest stars and resolves fainter ones as it improves, but a large share of the "universe" may stay unobservable. He calls it a critical open question whether dictionary learning variants can be made orders of magnitude more efficient, and suggests studying the invisible features indirectly as a research agenda.

Reproducing attention pivot tables

Nicholas L. Turner, Adam Pearce, Trenton Bricken and Adam Jermyn tried to reproduce one result from A Mathematical Framework for Transformer Circuits: "pivot tables" showing that one-layer transformers implement skip-trigrams ("Testing Base Model Quality Using Attention Pivot Tables"). They found the result "unexpectedly sensitive to subtle issues in training transformers" and share the details needed to reproduce it.

Construction. A pivot table summarizes two vocabulary-by-vocabulary matrices that ignore positional embeddings: $C_{QK}$, the pre-softmax attention score from each query token to each key token, and $C_{OV}$, the effect on the output logits of attending fully to each key token ("Constructing Pivot Tables"). With embedding $W_E$, unembedding $W_U$, the head's weights $W_Q, W_K, W_V, W_O$, and layer norms $L_A$ (before attention) and $L_U$ (before unembedding, with bias $\beta_U$):

$$C_{QK} = L_A(W_E)^\top W_Q^\top W_K \, L_A(W_E), \qquad C_{OV} = W_U \, L_U\big(W_O W_V L_A(W_E)\big) - \beta_U .$$

The unembedding bias is subtracted so that a general tendency to avoid certain tokens does not swamp the relative effects. The item gives checks: $C_{QK}$ should equal the attention logits once positional information is removed, and $C_{OV}$ should equal the output logits when a single attended token's output is the only content of the residual stream. Key tokens are ranked by a score combining their strongest attention score (weighted by query token frequency), their strongest output effect and their own frequency in the training data; the item reports slightly better results weighting frequency on both axes than the original tables did.

What to expect. Interpretable skip-trigrams appeared across architectures, with different positional embeddings and with or without layer norm. The best runs showed about 25 to 75% interpretable trigram entries, alongside many copying entries, and the authors advise suspicion of tables with less than about 10%, as one may be "interpreting" mirages ("Practical Considerations"). These proportions come from models of one to four layers. Poorly performing training hyperparameters gave confusing tables; datasets from easily read domains such as English text and Python code made structure easier to see; and single-layer models showed structure most readily.

Measuring feature sensitivity with Claude

Nicholas L. Turner, Adam Jermyn and Joshua Batson address a gap in the team's evaluation of features ("Measuring feature sensitivity using dataset filtering"). Specificity can be checked by looking at where a feature fires, but sensitivity requires finding every place a concept is present, which is hard for vague concepts over a large corpus.

Method. Claude rates how relevant a concept is to each text sample on a 0 to 10 scale. Filtering a broad dataset by these ratings gives a set of texts containing the concept, and the fraction of them on which a feature fires measures its sensitivity, provided the filtered set is representative. The item notes the advantage over having a model generate test examples: Claude only has to recognize the concept, not produce the full range of its uses in the right proportions ("A dataset filtering method"). In a validation on snippets where a feature was active, different versions of Claude agreed closely with each other (high Spearman correlations, few ratings differing by more than 2), with Claude 3 Haiku giving more all-or-nothing ratings, and manual spot checks were reassuring.

Data. Claude 3.5 Sonnet rated 160,000 snippets of 128 tokens for each of the four concepts in Scaling Monosemanticity's feature tour ("Feature sensitivity on filtered data"). The Golden Gate Bridge concept was too rare to yield enough data, leaving brain sciences, transit infrastructure and popular tourist attractions. Each snippet was run within a 2,048-token context, and a feature counted as detecting the concept if it was active on any token in the snippet.

Findings.

  • The highlighted Scaling Monosemanticity features "mostly do not respond" in snippets where the concept is present, even where Claude and human spot checks rate it highly relevant. The item prints examples, such as a passage on axon growth and the central nervous system that Claude rated 10 for neuroscience, on which the brain sciences feature was not active on any token.
  • Counting, for every feature, how many snippets rated 9 or higher it fires on, the highlighted features rank in roughly the top 0.1 to 10% of features but "far from the most sensitive".
  • The most sensitive features still miss some snippets and are sometimes less specific. For brain sciences, the most responsive feature fires on descriptions of experimental methods with human subjects. For transit infrastructure, it was hard to interpret and is labelled "unknown". For tourist attractions, a feature for the history of tourist attractions (1M/11480) fires in every snippet rated 9 or higher, though not in all snippets at lower thresholds.

Features working together. Features that fire on the snippets the highlighted feature misses often relate to the concept, for example one that fires near the word "brain", and this was more true in the SAE with more features ("How feature activations might cooperate"). A plot of which of the 50 most responsive features fire on which highly rated snippets shows vertical bands, which the authors read as several distinct kinds of snippet that the model represents differently. They recall a similar family of features for Arabic script in Towards Monosemanticity.

Conclusion. One feature's sensitivity does not show how well the SAE as a whole captures a concept, and descriptions of features need care to say that a feature may be "subtly related to a concept rather than fully representing that concept". If the SAE were a faithful and complete account, a set of interpretable features should cover every clear instance of a concept; the team could not show that here, and in the worst case the gap reflects dark matter or other basic problems.

The prompts used

The sensitivity item is the only part of the update in which written instructions shape a result: the rating prompt given to Claude determines which snippets count as containing the concept. The prompt gives the text sample and "a potentially fuzzy concept", asks how relevant the concept is, and tells the model to reason first and then give a score from 0 to 10, where 0 is not relevant and 10 maximally relevant. The instruction on what relevance means is one sentence, asking the model to weigh both how often the concept is "directly mentioned or alluded to" and how central it is "to the overall meaning of the text". (In the page source, the reasoning and score are requested inside <reasoning> and <score> tags; the tag names are not displayed on the rendered page.) The concept wording also varies by feature: the printed examples use "Discussions of neuroscience and related academic research on brains or minds", "Transit infrastructure" and "Monuments and popular tourist destinations". The item does not test how the prompt or concept wording affects the ratings, beyond the cross-model agreement check.

Limitations and open questions

  • Essays, not experiments. The hurdles, linear representation and dark matter items are arguments; Olah states his uncertainty about the definitions, and the dark matter estimate rests on a rough count per neighbourhood.
  • Small models for pivot tables. The 25 to 75% and 10% guidelines come from models of one to four layers and "could easily change outside that regime".
  • Model-rated ground truth. Sensitivity depends on Claude's relevance ratings, checked by cross-model agreement and spot checks rather than a full human-labelled set, and on the assumption that the filtered data are a representative sample of the concept.
  • Three concepts. The sensitivity analysis covers three features; the authors list automated searches for complementary features and comparisons with features of high cosine similarity as future work.

Research questions

  1. How sensitive are features in general, not just the highlighted ones? The item measured three hand-picked features. Applying the same filtering to a random sample of features with clear labels would show whether low sensitivity is typical.
  2. Does relevance wording change the sensitivity estimate? The prompt defines relevance by frequency of mention and centrality. Re-rating the same snippets with alternative definitions or concept phrasings, and recomputing sensitivity, would show how much the measure depends on the prompt.
  3. Can groups of features reach full coverage? The item finds complementary features by hand. An automated method that chooses the smallest set of features covering all highly rated snippets, with a precision constraint, would test the authors' claim that a faithful SAE should account for every clear instance.
  4. Does sensitivity rise with dictionary size? Complementary features were more specific in the larger SAE. Measuring the best single-feature and best small-set sensitivity across SAEs of increasing size would test the dark matter argument quantitatively.
  5. Is there evidence of cross-layer superposition? The hurdle is stated as a possibility. Comparing features learned separately at adjacent layers with features from a dictionary trained across several layers at once would show whether some features are spread over layers.
  6. Which training details make pivot tables fragile? The item names hyperparameters, data and depth. A controlled sweep varying one factor at a time, scoring the share of interpretable trigram entries, would identify which matter most.

How it connects

References

  • Olah, C., Jermyn, A., Turner, N. L., Pearce, A., Bricken, T., and Batson, J. (2024). Circuits Updates – July 2024. Transformer Circuits Thread, 31 July 2024. https://transformer-circuits.pub/2024/july-update/index.html (Anthropic landing page: https://www.anthropic.com/research/circuits-updates-july-2024)
  • Engels, J., et al. (2024). Not All Language Model Features Are Linear. arXiv:2405.14860.
  • Elhage, N., et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread. https://transformer-circuits.pub/2021/framework/index.html
  • Templeton, A., Conerly, T., et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html