Mechanistic interpretability tries to explain a trained neural network as a set of parts a person can understand. By mid-2022, Anthropic's interpretability team had reverse-engineered small transformers that contain only attention layers, but the other half of every transformer block, the MLP layer (a small two-layer network applied to each token position separately), had resisted analysis. The obstacle was that most MLP neurons in language models respond to several unrelated things at once. This paper, published on the Transformer Circuits Thread, asks whether a change to the architecture can make more of those neurons interpretable without making the model worse at its task. Its answer is a new activation function, the softmax linear unit (SoLU), which it reports raises the share of quickly interpretable neurons by a large margin at no measured cost in performance, with two caveats it treats as central: the benefit shrinks in the largest model tested, and part of the model's representation appears to move somewhere harder to see.
Terms used in this note
- Neuron. One dimension of a layer's activations in a privileged basis, meaning a basis that the architecture makes special, usually because a nonlinear function is applied to each dimension separately (Section 3.2). In a transformer, MLP activations have a privileged basis; the token embeddings and the residual stream (the running vector each layer reads from and adds to; see Assembling the Transformer Block) do not.
- Feature. A property of the input that the network represents, such as "this text is base64" or "this number counts people". The paper treats features as directions in activation space (Section 3.4).
- Polysemantic neuron. A neuron that responds to several unrelated features, such as the paper's example of a vision neuron that fires for cat heads, car fronts and paws (Section 3.3). A neuron that responds to one feature is monosemantic.
- Superposition hypothesis. The proposal that a network represents more features than it has neurons by giving them directions that are nearly but not exactly orthogonal, accepting some interference between them. The paper uses "polysemanticity" for the observed phenomenon and "superposition" for this explanation of it (Section 3.4). An update note in the paper says that its later paper Toy Models of Superposition supersedes this section's discussion.
- LayerNorm. Layer normalization, which rescales a vector of activations to zero mean and unit variance and then applies a learned per-dimension scale and offset. Because it divides by the vector's spread, it undoes any overall shrinking of its input (Section 4.3).
- GeLU. The activation function used in many transformers, which the paper notes is closely approximated by sigmoid(1.7x) multiplied by x (Section 4.2).
What the paper does
Why MLP neurons matter
Section 3.1 argues that understanding activations is necessary for understanding parameters: weights connect activations, so a weight means something only if the activations it links mean something. Attention-only transformers could be analyzed by rewriting them without reference to intermediate activations, but the nonlinearity in MLP layers blocks that shortcut. If MLP neurons were monosemantic, the privileged basis would hand researchers the right units for analysis. Polysemanticity removes that advantage.
The paper lists architectural properties that might discourage polysemanticity (Section 4.1): sparse activations; lateral inhibition, in which one neuron firing suppresses others, so that few fire together; weight sparsity, which the authors judge unsuitable for transformers because every weight matrix touches the residual stream; superlinear activation functions, which reward concentrating a feature in one neuron rather than spreading it over several; and more neurons per unit of compute.
The SoLU activation
SoLU replaces the sigmoid in the GeLU approximation with a softmax taken across all the neurons in the layer: SoLU(x) = x * softmax(x), with the multiplication applied element by element (Section 4.2). The softmax makes each neuron's output depend on the others, which gives lateral inhibition, and it grows faster than linearly, which gives superlinearity. The paper's worked examples show the effect. A vector with two large and two small entries, (4, 1, 4, 1), comes out as about (2, 0, 2, 0): large values suppress small ones. A vector concentrated in one neuron, (4, 0, 0, 0), passes through almost unchanged, while the same kind of input spread evenly, (1, 1, 1, 1), shrinks to (1/4, 1/4, 1/4, 1/4). A feature represented by one neuron therefore survives better than one spread across many.
The extra LayerNorm
Used alone, SoLU made neurons more interpretable in preliminary experiments but cost performance: SoLU models performed like baseline models 30 to 50 percent smaller, with larger models affected more (Section 4.3). The authors note that this is what the superposition hypothesis predicts. Adding a LayerNorm after the SoLU recovered the performance, so most experiments use LN(SoLU(x)). The neurons studied are the activations before this extra LayerNorm (footnote 2). The authors first added it for optimization reasons, and then came to suspect a second effect: because LayerNorm rescales its input, activations that SoLU shrank can be scaled back up, so the extra LayerNorm "may allow superposition to be smuggled through in smaller activations" (Section 4.3). For the largest models, which are split across accelerators, the softmax and LayerNorm act on blocks of neurons rather than the whole layer; the authors report that this did not noticeably change performance or interpretability in informal tests (Section 4.4).
Models and performance tests
The baseline is a transformer similar to GPT-3 and Gopher and identical to Anthropic's earlier language-model baselines. The authors train baseline and SoLU models from 1 layer to 64 layers (about 50 billion parameters), each size roughly four times the previous in parameters, with the SoLU models differing only in the activation function (Section 5). They compare training loss and downstream accuracy on Lambada, ARC, OpenBookQA, TriviaQA, arithmetic, MMLU and HellaSwag. Hyperparameters were not tuned for SoLU (Section 5).
The blinded interpretability test
Rigorously establishing what a neuron does can take a great deal of work, so the paper measures something weaker: whether a neuron suggests a plausible interpretation after brief inspection (Section 6.1). Evaluators, who were authors of the paper, looked at about 20 text snippets in which the neuron fires strongly, with the activations highlighted, for one to two minutes per neuron. The instruction was to mark a neuron INTERPRETABLE if "80% or more of the strongest firings can be explained by a single rule or category" (Section 6.1), giving as examples the word "apple" or any phrase about music. For each of five model sizes (1, 16, 24, 40 and 64 layers), evaluators saw 60 neurons from the baseline and 60 from the SoLU model, randomized and blinded as to which model each came from, for 600 neurons. A further 60 came from a 16-layer model with the extra LayerNorm but no SoLU, to separate the two changes, for 660 in all.
What it finds
Performance is unchanged
SoLU models with the extra LayerNorm match the baselines at every size (Section 5, Figure 1). Expressed as an equivalent change in model size, SoLU always fell between 0.95 and 1.05 times the baseline, which in most cases corresponds to a loss difference of about 0.01 nats against total losses of 1.6 to 3 nats. At 50 billion parameters, SoLU was equivalent to a 1.01 times larger baseline; the authors consider all these differences small and more likely than not to be noise. On the downstream tasks, overall performance was similar, with arithmetic better under SoLU and TriviaQA better for the baseline, and no systematic trend (Section 5, Figure 2). The softmax slowed training by less than 1 percent (Section 5).
More neurons look interpretable, except in the largest model
For models from 1 to 40 layers, about 60 percent of SoLU neurons were judged interpretable against about 35 percent of baseline neurons, an increase of roughly 25 percentage points, or 1.7 times (Section 6.2, Figure 4). In the 64-layer model the gain was much smaller: 42 percent for SoLU against 33 percent for the baseline. The authors do not know why. One possibility they offer is that larger models' neurons represent more sophisticated concepts that a one-to-two-minute look cannot identify; another is that something about deep models or their optimization weakens the effect (Section 6.2).
The 16-layer model with only the extra LayerNorm scored about halfway between baseline and SoLU, suggesting that the LayerNorm alone accounts for part of the gain (Section 6.2). The evaluators also differed: one found a larger effect than the other two (about 20 percent against 60 percent, where the others saw about 40 against 60), which the authors attribute after unblinding to a stricter standard, especially a reluctance to ignore small activations.
What the interpretable neurons do
Section 6.3 is an informal tour of neurons that the authors say they could not find before training SoLU models:
- In a one-layer model, each neuron's effect on the output can be read directly by multiplying its output weights by the unembedding matrix, which lets a hypothesis from dataset examples be checked against the neuron's effect on predictions. One neuron fires on base64-encoded text and raises the probability of mixed-case tokens typical of base64 while lowering common English words (Section 6.3.1, Figure 5).
- Early layers in larger models tend to de-tokenize: neurons respond to words split across several tokens, to names of famous people, to compound words such as "heart attack", and to a token in a particular language. Three separate neurons respond to the word "die" in German, Dutch and Afrikaans (Section 6.3.2, Figure 6). The authors say that before SoLU they had made almost no progress on early-layer MLP neurons.
- Late layers tend to re-tokenize, turning a represented word back into the specific tokens to be written: one last-layer neuron fires on "st" and raises the probability of "rag" on the way to "stragglers" (Section 6.3.3, Figure 7).
- Middle layers hold more abstract features, such as a neuron for numbers that count people, and neurons for clauses describing sound or clothing, for hedging phrases, and for letter grades (Section 6.3.4, Figure 8).
- Some neurons in small models appear to split into several more specific neurons in larger ones, and many early neurons form families of the form "token X in language Y" (Section 6.3.5).
The authors stress that these investigations were shallow compared with earlier rigorous neuron studies, and that they would not stand behind most of the theories with high confidence (Section 6.3.6).
The LayerNorm hides some features
To test the smuggling idea, the authors looked at dataset examples across the full range of a neuron's activation, before and after the extra LayerNorm (Section 6.4). For neurons that seemed interpretable, examples that only weakly activated the neuron after LayerNorm were much more often unrelated to the neuron's apparent feature. One author hand-sorted around a thousand examples for a neuron that responds to "left" and "right" (especially for body parts) and found many more unrelated low-activation examples after LayerNorm than before (Figure 9); the authors note that this was informal and not blinded. Their reading is that SoLU produces two tiers of features: some aligned with neurons, which dominate when present, and others spread across neurons, which survive only as small activations that LayerNorm scales back up. The paper calls SoLU "a double-edged sword" for this reason (Section 6.4): it makes some features easy to study while possibly making others harder to find, and anyone looking only at the SoLU activations could gain a false sense of having understood every feature.
Indirect evidence for superposition
The authors read the performance result as support for the superposition hypothesis: forcing features into neurons hurt performance, and performance returned once the model could again hide features in small activations. They call this "the first real (albeit circumstantial) evidence" for superposition over other explanations of polysemanticity (Section 8), and in the key results describe it as moderate evidence (Section 2).
Limitations and open questions
The authors list these limitations (Section 8 unless noted):
- The extra LayerNorm lets non-neuron-aligned features pass through as small activations, so some features become harder to see even as others become easier.
- One architecture and one dataset. The models use an 8,192-token context, rotary attention, mixed dense and sparse attention, and more book data than GPT-3; different choices might make models interpretable without SoLU, or might be required for SoLU to help.
- Scale. The benefit falls sharply around 50 billion parameters (64 layers), so it is uncertain whether SoLU will help in much larger models.
- The measure is quick inspection. Evaluators saw top-activating examples but not negative examples, so a neuron that fires on only part of a pattern could pass, and an interpretable neuron could be missed (Sections 6.1, 6.2).
- Scale of the task. Even if every neuron were interpretable, a model with millions of neurons would still need some further structure before a person could make global statements about it.
Two notes added to the paper after publication revise its framing. A caveat in Section 6.1 begins "Since publication, we've become more pessimistic about this metric." The reason given is that whether a neuron's strongest activations are monosemantic turned out to say little about whether the neuron is monosemantic overall, particularly once an architecture is optimized for clean top activations. The caveat points to Towards Monosemanticity (2023), which analyzes the full range of activations instead. The update in Section 3.4 points to Toy Models of Superposition as the clearer treatment of superposition.
Research questions
- Does the interpretability gain hold when neurons are judged on their full activation range? The paper's own caveat says top activations are a weak guide. Rating SoLU and baseline neurons on randomly sampled activations at several levels, as Towards Monosemanticity does for features, would show how much of the 35-to-60 percent gain survives.
- Why does the gain shrink at 64 layers? The paper offers two explanations: harder concepts, or a change in optimization. Giving evaluators more time per neuron in the 64-layer model, and comparing SoLU models of equal size trained under different optimization settings, would separate them.
- How much of the model's computation flows through the hidden tier of small activations? Ablating the low-activation component after LayerNorm while preserving top activations, and measuring the loss increase, would put a number on the "smuggling" that Section 6.4 shows only for one hand-sorted neuron.
- Do other architectural changes from Section 4.1 reduce polysemanticity without a hidden tier? Activation sparsity penalties and more neurons per unit of compute were proposed but not tested. Running the same blinded comparison on those variants would show whether SoLU's tradeoff is specific to it.
- Are the layer roles (de-tokenize, abstract, re-tokenize) found in other model families? The account in Section 6.3 is informal and based on one architecture. A systematic survey of neuron or feature types by depth in independently trained models would test it.
- How reliable is a one-to-two-minute judgment of interpretability? One evaluator's stricter standard produced a different effect size. Measuring agreement between evaluators on a shared neuron set, and checking quick judgments against longer investigations, would calibrate the method.
How it connects
- Toy Models of Superposition (September 2022), the follow-up named in this paper's Section 3.4 update, studies superposition directly in small synthetic models; its section on privileged bases links to this paper's longer discussion of the idea.
- Towards Monosemanticity (October 2023) takes the other route that Section 3.5 describes, accepting superposition and extracting features after training with sparse autoencoders. It refers back to this paper's Section 6.3.1 for one-layer neurons resembling some of its features, such as base64 and all-caps, and this paper's later caveat points to it.
- Mapping the Mind of a Large Language Model (2024) cites this paper among its reasons for expecting abstract features in a model's middle layer, and Tracing the thoughts of a large language model (2025) also cites it.
- Interpretability Dreams (May 2023) cites the "token X in language Y" neuron families from Section 6.3.5 as evidence that families of features exist in language models.
- A Mathematical Framework for Transformer Circuits (2021) is the reverse-engineering of attention-only transformers that this paper builds on, and the source of its definition of a privileged basis.
- For the MLP layer, residual connections and layer normalization in a working model, see Assembling the Transformer Block; the softmax function, which SoLU applies across a layer's neurons, is explained in Building Self-Attention Step by Step.
References
- Elhage, N., Hume, T., Olsson, C., Nanda, N., Henighan, T., Johnston, S., El Showk, S., Joseph, N., DasSarma, N., Mann, B., Hernandez, D., Askell, A., Ndousse, K., Jones, A., Drain, D., Chen, A., Bai, Y., Ganguli, D., Lovitt, L., Hatfield-Dodds, Z., Kernion, J., Conerly, T., Kravec, S., Fort, S., Kadavath, S., Jacobson, J., Tran-Johnson, E., Kaplan, J., Clark, J., Brown, T., McCandlish, S., Amodei, D., and Olah, C. (2022). Softmax Linear Units. Transformer Circuits Thread. https://transformer-circuits.pub/2022/solu/index.html. The page gives its publication date as 27 June 2022; Anthropic's research page lists the item under 17 June 2022. Section and figure numbers in this note refer to the web version.
- Anthropic. Softmax Linear Units [research page]. https://www.anthropic.com/research/softmax-linear-units