In May 2023, by its own account, much of Anthropic's interpretability work was foundational: trying to resolve superposition, the problem that language models store more features than they have neurons, which makes neurons hard to read (see Toy Models of Superposition). Work at that level says little about how the approach could ever explain a very large model. This essay, published on the Transformer Circuits Thread on 24 May 2023 alongside Circuits Updates (May 2023), states the aspirations behind the foundational work. Olah says in the introduction that nearly all of its ideas had appeared before, scattered through earlier papers, and that the essay mainly surfaces them, largely by quoting those papers. He also describes the content as "almost definitionally extremely speculative and uncertain", and the closing "About" section labels it an informal note in which mistakes are very possible. The essay reports no new experiments. This note records its argument and the earlier evidence it points to.
Terms used in this note
- Mechanistic interpretability. Explaining a network by reverse engineering its parts: features (properties of the input that the network represents), circuits (features connected by weights that compute new features) and the parameters themselves.
- Superposition. Storing more features than there are dimensions, as overlapping directions (see Toy Models of Superposition).
- Motif. A low-level pattern that repeats within a network. A structural phenomenon is a large-scale property of a network. Universality is a pattern that repeats across different networks (footnote 2).
- InceptionV1. The image classifier studied in the original Circuits thread on Distill (2020-2021), the source of most of the essay's examples.
The argument
Circuits as an epistemic foundation
The section "An Epistemic Foundation" asks why the field starts from the smallest pieces rather than from the questions it cares about. The answer given is experience: it is easy to be misled by neural networks, because a top-down search for evidence of a plausible hypothesis will find computation correlated with it even if the hypothesis is wrong. Footnote 1 gives an example: in an ImageNet model, a layer just before the one with a dog head detector has detectors for a dog snout, dog fur, eyes and so on, which together predict a dog head, so a linear probe will find a "dog detector" there even though the layer is not best understood as having that feature yet.
Mechanistic interpretability, by contrast, allows claims to be checked, and for small enough circuits claims become questions of mathematics. Olah compares the role to the foundations of mathematics or to assembly language: practitioners rarely work at that level, but any higher-level claim can in principle be checked against it. He quotes the original Circuits thread (Olah et al., 2020) and Branch Specialization (Voss et al., 2021) to show that building higher-level abstractions, such as motifs or something like an "organ" of a network, was a goal from the start.
What might be built on the foundation
The section "What Might We Build on Such a Foundation?" covers larger-scale structure (feature families, organization by weight structure and other structure), universality, the link between microscopic and macroscopic findings, and AI-automated interpretability, which this note treats separately below. Most of the evidence comes from vision models, because, the essay says, the extensive superposition in language models has made analogous structure much harder to find; it expects similar structure in transformers once superposition is resolved.
- Feature families. Features often come in families parameterized by a variable: orientation, scale, hue, or more abstract variation, as in the multimodal neurons of CLIP. The circuits connecting them are organized the same way. The essay cites the "token X in language Y" neurons reported in Softmax Linear Units as a simple family in language models. Its general point is that features were introduced to escape the exponential size of activation space, and that if there turn out to be too many features, structure among the features might allow the same trick one level up.
- Organization by weight structure. Branch Specialization found that, in networks with separate branches, related neurons gather in the same branch, and a small experiment in that paper found similar organization implicit in networks without branches. The essay reads this as structure that should fall out of the network's connections, and suggests that superposition hides it because the best superposition places related features as far apart as possible. Weight banding is mentioned as another large-scale structure.
- Universality. If the same features and circuits form across models, work on one model transfers to others. The essay cites Gabor filters, curve detectors and high-low frequency detectors in vision models, parallels with biological neurons, and, in language models, previous-token heads, induction heads, and an "all caps" neuron that formed in several small SoLU models.
- Connecting the microscopic and the macroscopic. Induction heads are offered as a mechanistic finding with large-scale effects: they cause a visible bump in training loss curves and are likely responsible for a bump in the original scaling-laws paper, which the essay identifies, quoting that paper's Figure 13, as the lump at 10^-5 PF-days marking the transition from one-layer to two-layer networks. The essay speculates that many universal circuits each reduce the loss by some amount, and that the dependencies between circuits (which require which, how much capacity each takes, how much each helps) might explain both learning dynamics and scaling laws in one mechanistic theory.
AI-automated interpretability
The section "AI Automated Interpretability" treats a different route to scale: using AI systems to do the interpretation, as in the proof of concept by Bills et al. (2023). Methods of that kind would still require resolving superposition. Olah writes that there is a significant chance this direction will be very important, since interpretability may need to ride the same exponential trend as AI itself. He gives two reasons for still trying to make a purely human version work. One is aesthetic, compared to the computer proof of the four color theorem that people cannot follow. The other is trust: if AI is used to audit the safety of AI, the auditing model has to be trusted, and workarounds might make interpretability's failures correlated with those of other safety methods. A middle ground, "AI-assisted interpretability" in which humans still verify the important parts, is described as exciting and important.
The end goals
The section "How Does Mechanistic Interpretability Fit Into Safety?" describes an ambitious goal it calls a "north star" for the team's research, while saying it has a significant chance of not working. Safety needs claims of the form "the model will never deliberately do X" across all situations, which is a worst-case, out-of-distribution claim unlike the expected-value claims machine learning usually makes. The essay's two diagrams restate this: first as a claim over all situations, then as a claim that no feature causes the model to deliberately do X, which implicitly concerns the circuits each feature takes part in. A complete set of features would give something to enumerate over. The essay calls this a vague and highly speculative idea, and refers to the "Strategic Picture" section of Toy Models of Superposition for a fuller sketch.
The final section, "Beauty and Curiosity", gives a second motive. Against the view that deep learning's success through scale is scientifically boring (Sutton's "The Bitter Lesson" is cited), the essay compares the structure produced by gradient descent to the complexity produced by evolution, and ends: "Neural networks are full of beautiful structure, if only we care to look for it." Footnote 3 adds that a pre-paradigmatic field may be better served by researchers following their own taste than by working backward from safety goals.
Limitations and open questions
- Speculative by the author's own account. The essay presents aspirations, not results, and says it may read as grandiose.
- Evidence mostly from vision models. Feature families, branch organization and universality are documented mainly in InceptionV1 and CLIP; their transfer to language models is stated as an expectation conditional on resolving superposition.
- The safety goal depends on two unsolved problems. Enumerating features to support worst-case claims requires both resolving superposition and scaling the analysis, and the essay says it is very unclear how "deliberate" failures would be captured.
Research questions
- Do feature families appear in dictionary-learning features of language models? The essay expects families like "token X in language Y" once superposition is resolved. Searching extracted features for sets that vary along one parameter, with matching structure in their connecting weights, would test it.
- Does organization by connectivity appear among extracted features? The Branch Specialization experiment suggests related features cluster by their connections. Clustering language-model features by their weights to other features and checking whether clusters share meaning would test the claim for transformers.
- How universal are language-model features across architectures and training data? The essay's language-model examples are induction heads and one neuron type in SoLU models. A quantitative comparison of features extracted from models with different architectures and datasets would measure how far universality holds.
- Can a circuit-level account predict a scaling-law exception? Induction heads explain one bump. Identifying another irregularity in a loss or scaling curve and predicting it from a specific circuit before measuring would test the proposed bridge.
- How much do AI-generated explanations of features need human checking? The essay's trust concern can be measured: compare automated explanations with blinded human ones on the same features, and record how often each misses behavior that ablations reveal.
How it connects
- Circuits Updates (May 2023) was published the same day, lists this essay among the team's recent publications, and voices the same reservation about trusting AI to audit AI.
- Toy Models of Superposition describes the superposition problem and contains the "Strategic Picture" that the safety section refers to.
- The essay cites In-context Learning and Induction Heads for the induction-head bump and Softmax Linear Units for "token X in language Y" neurons.
References
- Olah, C. (2023, May 24). Interpretability Dreams. Transformer Circuits Thread. https://transformer-circuits.pub/2023/interpretability-dreams/index.html. Section names in this note refer to this web version. Anthropic landing page: https://www.anthropic.com/research/interpretability-dreams
- Elhage, N., Hume, T., Olsson, C., et al. (2022). Toy Models of Superposition. Transformer Circuits Thread. https://transformer-circuits.pub/2022/toy_model/index.html