Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

interpretabilitysparse autoencodersdictionary learningfeaturessuperposition

Bricken, Templeton, Batson et al. train sparse autoencoders on the 512-neuron MLP layer of a one-layer transformer and recover thousands of features that humans and an automated judge find far more interpretable than the neurons; the features have sensible effects when ablated or activated, recur in a second model, and split into finer features as the dictionary grows.

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn et al. (Anthropic) · 2023-10-05 · original

Toy Models of Superposition (2022) argued that neural networks store more features than they have neurons, so that individual neurons respond to mixtures of unrelated things, and it listed ways one might recover the underlying features. This paper, published on the Transformer Circuits Thread in October 2023 (dated 4 October on the paper and 5 October on Anthropic's research page), reports that one of those ways works on a real, if very small, language model. It trains a sparse autoencoder on the internal activations of a one-layer transformer and shows that the directions it learns behave like the clean, single-meaning units that neurons fail to be. The introduction states that the team tried the alternative of building models without superposition and found it insufficient, and found standard dictionary learning methods prone to overfitting, before settling on this approach. The work became the method behind Anthropic's later feature studies of production models. A companion post for a general audience, Decomposing Language Models Into Understandable Components, appeared the same day and has its own short note.

Terms used in this note

  • Neuron, activation, feature, superposition, polysemantic, monosemantic. As defined in the note on Toy Models of Superposition: a feature is a property of the input the network represents, superposition is storing more features than dimensions, and a monosemantic unit responds to one feature.
  • MLP layer. The feedforward part of a transformer block, where each position's vector is expanded into a wider layer of neurons, passed through a nonlinearity, and projected back (see Assembling the Transformer Block). Here the MLP has 512 neurons.
  • Dictionary learning. Finding a set of directions (the dictionary) such that each data vector is approximately a sparse weighted sum of a few of them. When the dictionary has more directions than the data has dimensions it is overcomplete.
  • Sparse autoencoder (SAE). A two-layer network trained to reconstruct its input through a wider hidden layer while keeping that hidden layer mostly zero. The encoder is a linear map plus bias followed by a ReLU; the decoder is a linear map plus bias. The loss is reconstruction error plus an L1 penalty (the sum of absolute values) on the hidden activations, which pushes most of them to zero. Each hidden unit is a learned feature; its decoder column is the feature's direction in activation space ("Features as a Decomposition", equation 1).
  • Logit weights. In a one-layer model, the approximately linear effect of a feature on the model's output scores for each next token, computed by passing the feature's direction through the rest of the model.
  • Ablation and pinned sampling. Ablating a feature sets it to zero on a context and records how predictions change; pinning sets it to a high value and samples text from the model.

What the paper does

The model and the autoencoders

The subject model is a one-layer transformer with a residual stream of width 128 and an MLP of 512 ReLU neurons, trained on 100 billion tokens of the Pile (appendix "Transformer Training and Preprocessing"). The authors chose a one-layer model because it is cheap, can be heavily overtrained, and lets feature effects on the output be read almost linearly; they describe it as "the simplest language model we profoundly don't understand" ("Problem Setup"). Two copies were trained with different random seeds, called A and B, to test whether features recur.

Sparse autoencoders were trained on the MLP activations, using 8 billion activation vectors, with dictionaries from 512 features (1 times the neuron count) to 131,072 (256 times) ("Problem Setup"). Runs are named like A/1, and individual features like A/1/3450. Most of the analysis uses run A/1, with 4,096 features. Two training details the authors found important were more data, which made features subjectively sharper, and periodically resampling dead hidden units, resetting units that had stopped firing so that they could learn something the dictionary was missing ("Sparse Autoencoder Setup").

They chose a sparse autoencoder over classic dictionary-learning algorithms for two stated reasons: it scales to very large datasets, and it is about as powerful as an MLP layer, whereas a stronger algorithm might recover features that the model itself cannot use ("Using Sparse Autoencoders To Find Good Decompositions").

Why not build models without superposition

The paper reports that attempts to train models with very sparse, even one-hot, activations removed superposition but still produced polysemantic neurons ("Why not use architectural approaches?"). A worked example shows why: with one binary neuron and four equally likely mutually exclusive features, a neuron that fires for feature A alone gives a cross-entropy loss of about 0.8, while one that fires for A or B gives about 0.7, so the polysemantic neuron wins. The authors conclude that cross-entropy training favors polysemantic neurons even where superposition is impossible.

How the features were evaluated

For four features chosen because their contexts can be identified by a simple rule (Arabic script, DNA sequences, base64 strings and Hebrew script), the authors build a computational proxy, a log-likelihood ratio that scores how likely a string is to come from the context, and test five claims for each: the feature fires mainly in that context (specificity), it fires on most of that context (sensitivity), it has appropriate downstream effects, it is not just a neuron, and a similar feature appears in model B ("Detailed Investigations of Individual Features"). The authors say these features are cherry-picked for ease of analysis.

For features in general, they use three methods ("Global Analysis"): a blinded human rater (one of the authors) scoring samples across each feature's activation range against a rubric; automated interpretability, in which Claude writes an explanation of a feature from examples and then predicts its activations on new text; and a logit-weight test, in which Claude predicts from the explanation which tokens the feature should promote.

What it finds

Individual features are specific and not neurons

The Arabic script feature A/1/3450 illustrates the pattern ("Arabic Script Feature"). Arabic script is 0.13% of training tokens but 81% of the tokens on which the feature is active, rising to 98% when the activation is above 5. Its activation correlates with the Arabic proxy at 0.74 over 40 million tokens. Its strongest logit effects are on Arabic tokens; ablating it hurts prediction of Arabic text; pinning it high makes the model produce Arabic script. Only one neuron has any Arabic in its top 20 examples, and the feature's direction spreads over many neurons (27 with coefficients of at least 0.1 in magnitude). The most similar feature in model B, B/1/1334, has an activation correlation of 0.91.

The other three behave similarly, with proxy correlations of 0.8 for DNA, 0.38 for base64 and 0.55 for Hebrew; the authors attribute the low base64 figure mainly to the proxy also matching hexadecimal strings, which have a feature of their own ("Base64 Feature", "Hebrew Feature"). The matching features in model B correlate at 0.92 (DNA), 0.85 (base64) and 0.92 (Hebrew).

Most features are interpretable, and far more so than neurons

Of A/1's 4,096 features, 168 were dead (never active on the 100 million examples checked) and 292 fired on fewer than one in a million examples; these were excluded ("Global Analysis"). In the human evaluation, 412 activation intervals across 162 features and neurons were scored; the median neuron scored 0 on the rubric (no hypothesis possible) and the median feature interval scored 12 ("Manual Human Analysis"). In automated interpretability, Claude predicted feature activations significantly better than neuron activations. In the logit-weight test, accuracy was 74% on average across features against 58% across neurons, where guessing gives 50% ("Automated Interpretability – Logit Weights").

Features were most consistent with their interpretations at high activations; some became much less interpretable below the top 60% or so of their range ("Activation Interval Analysis"). The authors note two caveats: most activations are small and fall in the less interpretable range, and features were sampled uniformly, not weighted by importance. They add that most of a feature's effect comes from its higher, more interpretable activations, and that, from inspecting the most active features, they expect the more important features to be more interpretable, not less ("Caveats").

How much of the layer the features explain

Replacing the MLP activations with A/1's reconstruction recovers 79% of the loss reduction the MLP provides; the largest dictionary, A/5 with 131,072 features, recovers 94.5% ("How much of the model does our interpretation explain?"). The authors ask for these numbers to be taken "with a significant grain of salt", because fraction of loss may be the wrong frame and the features are not fully monosemantic.

To check that features reflect the model and not only the data, they trained an autoencoder on a randomly initialized copy of the model. It produced many single-token features and a few broad context features, but the authors could not interpret its other features ("Do features tell us about the model or the data?"). They also argue that the downstream effects of the trained model's features (logit weights, ablations, pinned sampling) are not an input to dictionary learning, so their consistency with each feature's interpretation must reflect the model rather than the data.

Features recur across models

For each A/1 feature, the most similar B/1 feature by activation correlation has a median correlation of 0.72; the same comparison between the two models' neurons gives 0.46 ("Universality"). Logit-weight similarity is lower and noisier (0.23 for the Arabic pair, against an activation correlation of 0.91), which the authors attribute to interference in weights for tokens that do not matter. Many features also resemble ones reported elsewhere, such as base64, hexadecimal and all-caps neurons in Anthropic's earlier SoLU models ("Comparing features with the literature").

Features split as the dictionary grows

Going from 512 to 4,096 to 16,384 features, the number of base64 features goes from 1 to 3 to many more ("Feature Splitting"). In A/1 the single base64 feature of A/0 has become three: one for letters, one for digits (whose predictions reflect a tokenizer quirk: two digits in a row would have been merged into one token, so a lone digit is never followed by another), and one for base64 that encodes ASCII text. The authors read the dictionary size as a resolution setting, and suggest that strange features in small dictionaries, such as features firing on every instance of one token, are coarse summaries of families that split in larger ones. They also report that "the" alone has over a hundred features in A/4, one for each kind of context ("Feature Motifs").

Features chain into simple state machines

Some features increase the probability of tokens that activate another feature at the next position, forming what the authors call finite-state-automata-like systems ("'Finite State Automata'"). A four-feature cycle in A/0 generates HTML (open tag, tag name, tag close, whitespace); two-feature cycles handle scripts whose characters are split into two tokens, such as Tamil. In larger dictionaries, chains of narrow features appear to reproduce memorized phrases such as the license text "MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE".

What changed in the theory

The discussion revises the Toy Models picture of features spreading evenly apart. Features here cluster tightly in groups with related meanings, which the authors attribute partly to correlated activation and more to similar downstream effects ("Theories of Superposition"). They also raise the possibility of features that are not single directions but multi-dimensional "feature manifolds", which could explain why splitting continues as the dictionary grows. They report being more confident that some version of the superposition hypothesis is true.

Limitations and open questions

  • One small model. All results are for a one-layer, 512-neuron transformer. The authors call scaling to frontier models one of the most important questions going forward, with the warning that an autoencoder with a 100-fold expansion on one MLP layer of width 10,000 would have about 20 billion parameters ("Future Work").
  • No reliable quality metric. The authors searched for a single number to tell whether an autoencoder is working and did not find one; they relied on manual inspection, feature density, reconstruction loss and toy models ("How can we tell if the autoencoder is working?").
  • Low activations are less clean. Weak activations are less interpretable, possibly because of imperfect proxies, the model's own uncertainty, or an autoencoder too small to separate all features.
  • Token-in-context features. Hundreds of features for "the" in different contexts could mean the model really uses such a local code, or that the L1 penalty pushes the autoencoder toward overly sparse features; the authors think the first is likely true to some extent ("Are 'Token in Context' Features Real?").
  • Attention layers. The method decomposes the MLP only; whether attention layers have their own superposition is left open.
  • Analysis at scale. Even with perfect features, turning millions of small findings into an understanding of a model is an unsolved problem.
  • Later work. Mapping the Mind of a Large Language Model (2024) applies the method to the residual stream of Claude 3 Sonnet, describing this paper's one-layer setting as a particularly extreme example of the small-model limitation.

Research questions

  1. Does the interpretability advantage hold at matched importance? Features were sampled uniformly. Repeating the human and automated scoring on features weighted by their effect on the loss would show whether the features that matter most are as interpretable as the median.
  2. Is the local code real? The token-in-context question could be tested by training dictionaries with different sparsity penalties and checking whether "the in physics" features merge into separate "the" and "physics" features without loss of reconstruction or of downstream fidelity.
  3. How should dictionary size be chosen? Feature splitting suggests no single correct size. Measuring, for a fixed set of behaviors, the smallest dictionary at which each behavior's features stop changing would turn the resolution idea into a procedure.
  4. Do the universal features recur in different architectures and data? Universality was tested between two seeds of the same model and qualitatively against the literature. A quantitative comparison between models trained on different datasets would test the stronger claim.
  5. What fraction of the remaining 21% of loss is interpretable? The 79% figure says how much the features reconstruct, not how much is understood. Replacing activations with ones predicted from feature explanations, as the authors suggest, would measure explanation directly.
  6. Are the finite-state chains used the same way in larger models? The HTML and Unicode chains are reported in a one-layer model. Searching for the same chains in multi-layer models would show whether this mode of computation persists.

How it connects

References

  • Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N. L., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Carter, S., Henighan, T., and Olah, C. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread. https://transformer-circuits.pub/2023/monosemantic-features/index.html. Section names in this note refer to this web version.
  • Anthropic. (2023, October 5). Decomposing Language Models Into Understandable Components [summary post]. https://www.anthropic.com/research/decomposing-language-models-into-understandable-components
  • Elhage, N., Hume, T., Olsson, C., et al. (2022). Toy Models of Superposition. Transformer Circuits Thread. https://transformer-circuits.pub/2022/toy_model/index.html