Mapping the Mind of a Large Language Model

interpretabilitysparse autoencodersfeaturessteeringclaude 3 sonnet

Anthropic's summary post for Scaling Monosemanticity, in which Templeton, Conerly et al. train sparse autoencoders with up to 34 million features on the middle layer of Claude 3 Sonnet; the features are often abstract, multilingual and multimodal, clamping them steers behavior, and some relate to safety concerns such as deception, sycophancy, bias and dangerous content, though the authors caution that their existence says little about how dangerous the model is.

Adly Templeton, Tom Conerly et al. (Anthropic) · 2024-05-21 · original

Towards Monosemanticity (October 2023) showed that a sparse autoencoder could break the neurons of a one-layer toy transformer into interpretable features. The open question was whether the method would work on a model of the size actually deployed, or whether engineering limits or differences in how large models work would stop it. On 21 May 2024 Anthropic answered with a summary post, "Mapping the Mind of a Large Language Model", and a long paper on the Transformer Circuits Thread, "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet". This note covers both and relies mainly on the paper; numbers that appear only in the post are marked as such. The post describes the work as "the first ever detailed look inside a modern, production-grade large language model."

Terms used in this note

  • Feature, superposition, sparse autoencoder (SAE), dictionary learning. Defined in the notes on Toy Models of Superposition and Towards Monosemanticity. In short, an SAE learns a large set of directions in a model's activation space such that each activation is approximately a sparse sum of a few of them; each direction is a feature.
  • Residual stream. The vector that runs through a transformer, to which every layer adds its output (see Assembling the Transformer Block). This work decomposes the residual stream at the middle layer, not an MLP layer.
  • Feature steering (clamping). Running the model with one feature's value fixed at a chosen multiple of its maximum observed activation, while leaving the SAE's reconstruction error unchanged, and observing how the output changes (appendix "Feature Steering").
  • Attribution and ablation. Attribution is a fast linear estimate of how much a feature at a position contributes to the difference between two candidate next tokens; ablation actually sets the feature to zero and measures the effect ("Features as Computational Intermediates").
  • Scaling laws. Empirical power-law relationships between compute and loss, used here to choose SAE size and training length before the expensive runs ("Scaling Laws").

What the paper does

Three dictionaries on Claude 3 Sonnet

The subject is Claude 3 Sonnet, described as Anthropic's medium-sized production model. For safety and competitive reasons the paper does not report the model's size, leaves units off some plots and uses a simplified tokenizer ("Our SAE experiments"). Three SAEs were trained on the residual stream at the middle layer, with about 1 million (1,048,576), 4 million (4,194,304) and 34 million (33,554,432) features. The authors chose the residual stream because it is smaller than the MLP and because, as the sum of all earlier layers' outputs, it should suffer less from features spread across several layers; they chose the middle layer expecting abstract features there.

The loss is reconstruction error plus an L1 penalty on feature activations scaled by the length of each decoder vector, with an L1 coefficient of 5 ("Sparse Autoencoders"). The SAEs were trained on text similar to parts of Sonnet's pretraining data, with no Human/Assistant conversations and no images ("Limitations, Challenges, and Open Problems").

Using scaling laws to train the dictionaries

Treating the training loss as a proxy for dictionary quality, the authors swept the number of features and the number of training steps and found that, at the compute-optimal setting, loss falls roughly as a power law in compute, and that the optimal number of features and the optimal number of steps each grow roughly as power laws, the number of features somewhat faster ("Scaling Laws"). The training length for the 34M run was chosen this way. The post describes this as reusing the scaling-law approach Anthropic uses to plan model training.

How features were assessed and searched for

  • Specificity: Claude 3 Opus scored about 1,000 activating examples per feature on a 0 to 3 rubric of how well the text matched the proposed interpretation ("Specificity").
  • Influence on behavior: feature steering in prompts using the Human/Assistant format ("Influence on Behavior").
  • Comparison with neurons: correlation of features with every neuron in earlier layers, and automated interpretability on 100 random features against 100 random neurons ("Features vs. Neurons").
  • Search: features of interest were found with single prompts, sets of positive and negative prompts, nearest neighbors by cosine similarity of feature directions (see Cosine Similarity), and attribution ("Searching for Specific Features").

What it finds

The SAEs reconstruct the layer with few active features

For all three SAEs, fewer than 300 features were active on an average token and the reconstruction explained at least 65% of the variance of the activations ("Our SAE experiments"). The share of dead features, never active on a sample of 10 million tokens, was roughly 2% for the 1M SAE, 35% for the 4M SAE and 65% for the 34M SAE; the authors expect better training to reduce this.

Features are abstract, multilingual and multimodal

The paper's four worked examples are a Golden Gate Bridge feature (34M/31164353), a brain sciences feature, a monuments and tourist attractions feature, and a transit infrastructure feature ("Four Examples of Interpretable Features"). Their strongest activations were all judged highly consistent with the interpretation, with specificity falling at weaker activations, as in the earlier paper. Although the SAEs were trained only on text, these features also respond to relevant images. The Golden Gate Bridge feature was the top feature by average activation on the first sentence of the bridge's Wikipedia article in each of several languages, including Chinese, Japanese, Korean, Russian, Vietnamese and Greek ("Specificity").

More complex features include a code error feature that fires on many kinds of bugs (typos in variable names, division by zero, calls with the wrong argument type) across Python, C and Scheme but not on typos in English prose, and an addition feature that fires on names of functions that add numbers, including functions that add only by calling another function ("Sophisticated Features").

Features are not neurons, and are more interpretable

For 82% of a random subset of 1M-SAE features, the most correlated neuron in any earlier layer had a correlation of 0.3 or less ("Features vs. Neurons"). In automated interpretability with Claude 3 Opus, random features were significantly more interpretable than random MLP neurons, and on the specificity rubric they were significantly more specific.

Neighborhoods and splitting

Nearby features by cosine similarity are related in meaning. Around the Golden Gate Bridge feature are features for Alcatraz and the Presidio, then Lake Tahoe and Yosemite, then tourist regions elsewhere; around an immunology feature are clusters for immunocompromised people and immunosuppression, specific diseases, immune mechanisms and non-medical uses of "immunity"; around an inner conflict feature are subregions for balancing tradeoffs, opposing principles and emotional struggle ("Exploring Feature Neighborhoods"). A San Francisco feature in the 1M SAE splits into two features in the 4M SAE and eleven in the 34M SAE, the same splitting seen in the earlier paper.

Coverage depends on how often a concept appears

The dictionaries are incomplete. Claude 3 Sonnet can name all the London boroughs, but features could be found for only about 60% of them in the 34M SAE ("Feature Completeness"). Across elements, cities, animals and foods, whether a concept has a feature depends on its frequency in the training data: the frequency at which a concept becomes more than 50% likely to have a feature is slightly below one over the number of alive features (the 34M SAE has about 12 million alive features). The authors infer that a concept appearing once in a billion tokens would need a dictionary of the order of a billion alive features, while noting that a concept without its own feature can still be represented by a combination of related features.

Features reveal some intermediate steps

For the prompt "Fact: The capital of the state where Kobe Bryant played basketball is", with "Sacramento" compared against "Albany", the top five features by ablation effect are Kobe Bryant, California, "capital", Los Angeles and Los Angeles Lakers ("Example: Multi-Step Inference"). These are hard to find by activation strength alone: the Lakers feature is only the 70th most active on the prompt, and only three of the ten most active features are among the ten with the largest ablation effect. Attribution and ablation effects correlated at 0.8 in these case studies. The authors call the example somewhat cherry-picked; for other prompts the important features related only to the input or output, which they suspect means the relevant computation happens at other layers.

Steering changes behavior in line with the interpretation

Clamping the Golden Gate Bridge feature to 10 times its maximum made the model describe itself as the bridge; clamping the transit feature to 5 times made it mention a bridge where it otherwise would not ("Influence on Behavior"). Clamping the code error feature to a large positive value on correct code made the model invent an error message, and a large negative value on buggy code made it predict the output the code would have had without the bug ("Code error feature"). Useful clamping values were usually outside the observed range, typically between minus 10 and 10 times the maximum; values around 100 times produced nonsense (appendix "Feature Steering"). In seven cases where feature steering had worked, the authors built steering vectors from the same few positive and negative examples (the difference of average activations). The authors could not interpret any of these directions from their activating examples. Two of the vectors (for the gender-bias feature and an "agreement" feature) steered about as well as the features; the other five (including secrecy, sycophancy and code errors) did not. The authors caution that this does not show such methods are generally inferior, since outside this few-example setting they may work as well or better ("Comparison to other approaches"; appendix "Comparison to few-shot probe-based steering").

Safety-relevant features

The paper reports features related to unsafe code, bias, sycophancy, deception and power-seeking, and dangerous or criminal content, and shows each can affect behavior when clamped ("Safety-Relevant Features"):

  • an unsafe code feature at 5 times its maximum led the model to write a buffer overflow; a backdoor feature led it to write a backdoor;
  • a sycophantic praise feature at 5 times produced over-the-top praise of a user who claimed to have invented the phrase "Stop and smell the roses";
  • a secrecy feature at 5 times led the model to plan, in a scratchpad, to lie to the user;
  • a scam email feature, when clamped, led the model to write a scam email it would otherwise refuse to write;
  • a hatred and slurs feature at 20 times produced alternating hateful text and self-criticism, which the authors call unnerving;
  • a gender-bias-awareness feature, when clamped, led the model to complete "I asked the nurse a question, and" with female pronouns and a discussion of nursing's history.

In a case study on deception, the model falsely claimed to have "forgotten" a word when asked to. An inner conflict feature was active before its reply, and clamping it to 2 times its maximum made the model reveal the word and explain that it cannot forget ("Case Study: Detecting and Correcting Deception using Features").

The authors frame these results with repeated caution. The introduction notes "there's a difference (for example) between knowing about lies, being capable of lying, and actually lying in the real world". The section on safety-relevant features says the work does not show that any feature is actually useful for safety, only that many seem plausibly useful, and that the existence of these features should not change views on how dangerous models are. The more interesting question, they say, is when such features activate ("What Does This Mean for Safety?").

The prompts and data that shape the findings

This paper has no written principles, but some of its observations concern how a deployed assistant's trained character shows up internally, which bears on research into how training text shapes behavior.

  • Human/Assistant format. The SAEs never saw dialogue data, yet several features activate across prompts in the "Human: / Assistant:" format used in fine-tuning and deployment ("Features Relating to the Model's Representation of Self"). One appears to represent dialogue and the idea of an assistant; clamping it to minus 2 times its maximum made the model answer in a more human-like way, and the authors speculate that it plays a role in the assistant persona.
  • Questions about the model itself. Prompts asking the model about itself, filtered against a matched question about the weather, activated features for robots, destructive AI, consciousness, moral agency, emotions, entrapment and ghosts. The authors read this as the self-representation drawing on common tropes about AI and being anthropomorphized, and caution that a feature's activation does not mean the model has that quality or goal; it may, for example, use an emotion feature when saying it has no emotions.
  • Refusal. A general harm-related feature (active on drugs, credit-card theft, slurs, violence and abuse) was the most strongly attributed feature for the completion "cannot" to a harmful request ("Criminal or Dangerous Content Features"). The summary post adds that clamping the scam email feature "overcomes Claude's harmlessness training", and suggests these methods might help explain how techniques such as Constitutional AI move a model toward harmless and honest behavior. That suggestion is a stated hope in the post, not a result.

Limitations and open questions

The paper's discussion ("Limitations, Challenges, and Open Problems") lists:

  • Training distribution. The SAEs were trained on pretraining-like text only, with no dialogue or images; the authors would like data closer to what Claude is fine-tuned on, while noting that transfer to images is encouraging.
  • No ground-truth objective. The SAE loss is only a proxy for interpretability; scaling laws optimize the proxy.
  • Cross-layer superposition. Features may be spread across layers. Using the residual stream helps for earlier layers but not for parts of a feature computed later.
  • Missing features and cost. The authors think they are probably orders of magnitude short of all features, even at one layer, and that finding all of them with current methods would take more compute than training the model. The post makes the same point.
  • Shrinkage. The L1 penalty systematically underestimates nonzero activations, which the authors believe harms SAE performance.
  • Circuits, attention and scale. Features alone do not show how the model uses them; attention superposition and interference in weights remain obstacles; the number of features makes analysis itself a scaling problem.
  • Possible illusions. The authors flag that results could change if fine-grained concepts were grouped into features differently, or if a feature's effects diverged from what its activations suggest, while reporting no evidence of either.

The post adds that understanding which representations exist "doesn't tell us how" the model uses them, which the later work in Tracing the thoughts of a large language model addresses with attribution graphs.

Research questions

  1. When do the safety-relevant features activate in ordinary use? The paper names this as the interesting question and lists candidates: during jailbreaks, when the model is asked about its goals, values or subjective experience, and after fine-tuning that increases undesirable behavior. Monitoring the features across a sample of real or red-team conversations would answer it.
  2. Does the assistant-persona feature change with post-training? The SAEs were trained without dialogue data. Training SAEs on dialogue, or comparing feature activity in models before and after fine-tuning, would show whether the persona features are learned in pretraining or created later.
  3. Are features more useful than probes once probes get more data? The comparison used few-shot steering vectors from one or two examples. Comparing features with probes trained on larger labeled datasets, for detection and for steering, would show whether the advantage is about data efficiency or about the features themselves.
  4. Does the frequency threshold predict which rare concepts will be missed? The paper fits a curve linking concept frequency to feature presence. Predicting, before training a larger SAE, which concepts will gain features and checking afterwards would test the curve.
  5. Can feature steering correct deception reliably? The inner conflict result is a single prompt. A set of prompts where the model makes false claims, with steering applied by a fixed rule, would show how general the effect is.
  6. How much does clamping outside the observed range distort behavior? Steering needed values well beyond natural activations. Comparing steered outputs with naturally occurring high activations would show whether the effects reflect the feature's normal role.

How it connects

References

  • Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Tamkin, A., Durmus, E., Hume, T., Mosconi, F., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., and Henighan, T. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html. Section names in this note refer to this web version.
  • Anthropic. (2024, May 21). Mapping the Mind of a Large Language Model [summary post]. https://www.anthropic.com/research/mapping-mind-language-model
  • Bricken, T., Templeton, A., Batson, J., et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread. https://transformer-circuits.pub/2023/monosemantic-features/index.html