This is a short post on anthropic.com, published on 5 October 2023, that introduces the paper Towards Monosemanticity: Decomposing Language Models With Dictionary Learning to a general audience. It reports no results beyond the paper's, so this note is brief: it records what the post says, how it frames the work, and where its numbers come from. The methods, results and limitations are covered in the note on the paper.
What the post says
The problem: neurons do not have consistent meanings
The post opens with an analogy to neuroscience. The arithmetic each neuron performs is known exactly, yet that does not explain the network's behavior, which makes failures hard to diagnose and safety hard to certify. Unlike brain research, experiments on artificial networks are easy: every neuron can be recorded and any of them silenced or stimulated. The obstacle is that individual neurons respond to unrelated things. The post's example is a single neuron in a small language model that is active on academic citations, English dialogue, HTTP requests and Korean text, and a neuron in a classic vision model that responds to both cat faces and the fronts of cars.
The proposed unit: features
The post describes features as patterns, specifically linear combinations, of neuron activations, found by a method called dictionary learning, and presents them as better units of analysis than neurons. It states that in a transformer language model a layer with 512 neurons was decomposed into more than 4,000 features, representing things such as DNA sequences, legal language, HTTP requests, Hebrew text and nutrition statements. These figures match the paper, where the main dictionary (run A/1) has 4,096 features learned from a 512-neuron MLP layer.
The evidence summarized
The post summarizes four lines of evidence from the paper:
- a blinded human evaluator scored features as much more interpretable than neurons;
- an automated method, in which a large language model writes descriptions of features and another model is scored on predicting activations from them, also favored features;
- artificially activating a feature changes the model's output in predictable ways;
- features are largely shared between different models, and changing the number of features learned acts as a "knob" for viewing the model at coarser or finer resolution.
The post gives no numbers for these comparisons; the scores (for example, a median rubric score of 0 for neurons against 12 for feature intervals) are in the paper's Global Analysis section.
The framing
The post places the work within Anthropic's investment in mechanistic interpretability and says the hope is to monitor and steer model behavior from the inside. It names the next challenge as scaling from the small model to frontier models, and states: "For the first time, we feel that the next primary obstacle to interpreting large language models is engineering rather than science." The paper makes a related point in its Future Work section, while also listing open scientific questions (scaling laws for dictionary learning, how to recognize good features, and superposition in attention layers).
How it connects
- Towards Monosemanticity is the paper the post summarizes; cite the paper for any result.
- Toy Models of Superposition explains why neurons mix unrelated meanings, the problem the post opens with.
- Mapping the Mind of a Large Language Model (May 2024) is the equivalent summary post for the scaled-up follow-up on Claude 3 Sonnet.
References
- Anthropic. (2023, October 5). Decomposing Language Models Into Understandable Components. https://www.anthropic.com/research/decomposing-language-models-into-understandable-components
- Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread. https://transformer-circuits.pub/2023/monosemantic-features/index.html