Mechanistic interpretability tries to explain a neural network by working out the computation its parameters carry out. In June 2022 Chris Olah published this short essay on the Transformer Circuits Thread to set out the intuition behind that agenda. It asks what the central difficulty of reverse engineering a network is, and answers with an analogy: if a network's parameters are like a compiled program, its activations are like the program's memory, and the first job is to divide that memory into variables whose meanings can be worked out one at a time. The essay calls itself "a very informal discussion of some of the intuitions that motivate me" ("Learn More") and reports no experiments. The Softmax Linear Units paper, released the same month, describes it as an informal note on intuition accompanying that paper.
The version of record is the web page, dated 27 June 2022 in its "Publication" section. It has no numbered sections, so this note cites section headings.
The analogy: a network as a compiled program
The essay opens with a table of parallels. Network parameters correspond to a program binary, the architecture to the virtual machine or processor that runs it, a layer's activations to program state or memory, and a neuron or feature direction to a variable or memory location. The essay's case for the analogy is heuristic: questions that feel "speculative and slippery" for networks often become clear when asked of ordinary programs, and the answers seem to it plausibly to carry back. Its headline conclusion is that finding and understanding interpretable neurons is not one interesting problem among many: "Arguably, it's the central task."
The curse of dimensionality
The first argument concerns the size of the input space ("Attacking the Curse of Dimensionality"). A network is a function over a space whose volume grows exponentially with its dimension, which the essay raises as a problem for understanding as well as for learning. It names two other answers: studying toy networks with low-dimensional inputs, which dodges the problem, and studying behavior near a single input (roughly what saliency maps do). A footnote also sets aside the reply that only the data manifold matters, arguing that for vision and language the manifold is itself very high-dimensional, and records that the author was once invested in that approach.
The essay's answer is the programmer's. Source code is "a non-exponential description of the program's behavior", and a network's parameters are likewise a finite description, even at hundreds of billions of parameters, just as large compiled programs are "often" understood eventually. The consequence it draws for expectations is that mechanistic interpretability should be "at least as difficult as reverse engineering a large, complicated computer program", without a short explanation or a fixed procedure.
Variables and activations
The second argument explains why activations come first ("Variables & Activations"). A line such as y = x + 5; means nothing until one knows what y and x stand for, and reverse engineers lack variable names; they must even decide how to split raw memory into separate variables. Following Voss et al. (2021), the essay treats parameters as instructions and activations as the memory they act on, so a parameter can be understood only once the activations it connects are. The remedy for high-dimensional activations is again the programmer's: break them into "independently understandable pieces".
The essay notes one exception. In special cases such as attention-only transformers, linearity allows the network's operations to be written in terms of its inputs and outputs, as in A Mathematical Framework for Transformer Circuits (Elhage et al., 2021). In most networks this is not possible, so activations must be decomposed.
Privileged bases and polysemantic neurons
The third section explains why neurons are the natural place to start ("Simple Memory Layout & Neurons"). Computer memory is usually laid out simply, a byte representing one thing, partly because hardware favors it. The essay suggests something similar often happens in networks: features could in principle lie along any direction of activation space, but activation functions make neuron directions "natural and useful", so features tend to align with individual neurons. "We call this a privileged basis." Neuron-aligned features would make a network much easier to reverse engineer; the essay does not call them necessary, only that reverse engineering without them "seems much harder".
The difficulty is that many neurons are polysemantic (they respond to several unrelated features), which the essay calls "a really tricky problem", deferring to Section 3 of the Softmax Linear Units paper. Its main point stands regardless: "the ability to decompose representations into independently understandable parts seems essential for the success of mechanistic interpretability."
The FAQ: distributed representations, features and modularity
A closing "Random Related Topics FAQ" answers three questions.
- Distributed representations. Interpretable neurons do not conflict with distributed representation as the essay understands the term: a word embedding is distributed because independent properties (singular versus plural, male versus female) occupy different directions, and it could be rotated so those directions become basis dimensions.
- What a feature is. Defining one is "surprisingly hard". A meaningful input property encoded as a direction (curve or car detectors in InceptionV1) is easy to state but human-centric. The author prefers "the 'independent units' a neural network representation can be decomposed into", and mentions "the things a network would ideally dedicate a neuron to if you gave it enough neurons". Defining a feature as any function of the input is rejected, because it would count an arbitrary mixture such as a car-and-cat detector.
- Modularity. Both help, but decomposing into understandable features "is what makes mechanistic interpretability possible at all while modularity may make it easier once it is possible."
Limitations
- Informal, unreviewed and without measurements. The case rests on the analogy with program reverse engineering; the essay says only that answers "plausibly" transfer, and tests nothing in trained networks.
- The neuron basis as a hope. Feature-neuron alignment is something that "often" seems to happen, and polysemantic neurons break it. The essay leaves open how to decompose activations when neurons are the wrong basis; later work in the collection took that up.
- Definitions left open. It offers candidate definitions of "feature" without settling on one.
Research questions
- How often do features align with neurons in practice? Measuring, across layers and architectures, the share of neurons that a learned feature dictionary explains with a single feature would put a number on the essay's "often".
- Do weights become readable once features are found? The essay expects each parameter to read as a connection between two features. Testing whether weights between learned features are interpretable, or dominated by interference, would answer it.
- Which candidate definition of a feature do later methods track? Training wider models and checking whether features recovered by sparse dictionaries gain their own neurons would test the "dedicate a neuron" definition against what the methods find.
- Is the data-manifold objection as weak as the essay argues? Estimates of the intrinsic dimension of real image and text representations would test its claim that the manifold is still very high-dimensional.
- Does the program analogy predict where interpretability is hard? Comparing the coverage achieved by published circuit analyses with model size would show whether difficulty grows the way the analogy suggests.
How it connects
- A Mathematical Framework for Transformer Circuits (2021) is the essay's example of a network described without decomposing intermediate activations, and the source it links for "privileged basis".
- Softmax Linear Units (June 2022) is where the essay sends readers on polysemantic neurons; that paper calls this essay an informal note on intuition accompanying it.
- Toy Models of Superposition (September 2022) links the essay where it introduces decomposability and where it describes decomposing activation space, "analogous to how computer program memory can be decomposed into variables".
- Distributed Representations: Composition & Superposition (2023) links the essay for the claim that decomposing representations is needed to escape the curse of dimensionality; that it expands the FAQ's question about distributed representations is this note's observation.
- Towards Monosemanticity (2023) and Mapping the Mind of a Large Language Model (2024) seek the decomposition the essay calls for with sparse dictionary learning rather than the neuron basis; this connection is the note's.
- Circuits Updates (July 2024) links the essay where Olah calls interpretable features the "variables" of the computation.
- Circuit Tracing: Revealing Computational Graphs in Language Models (2025) cites it for why features must be independently interpretable: to avoid having to think about all of them at once.
References
- Olah, C. (2022). Mechanistic Interpretability, Variables, and the Importance of Interpretable Bases. Transformer Circuits Thread, 27 June 2022. https://transformer-circuits.pub/2022/mech-interp-essay/index.html. Section names in this note refer to the web version. record
- Voss, C., Cammarata, N., Goh, G., Petrov, M., Schubert, L., Egan, B., Lim, S. K., and Olah, C. (2021). Visualizing Weights. Distill, 4 February 2021. https://distill.pub/2020/circuits/visualizing-weights/. (Cited by the essay for treating parameters as instructions and activations as memory.) record
- Elhage, N., et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread. https://transformer-circuits.pub/2021/framework/index.html record · 8 pages
- Elhage, N., et al. (2022). Softmax Linear Units. Transformer Circuits Thread. https://transformer-circuits.pub/2022/solu/index.html record · 3 pages