Circuits Updates – April 2024

interpretabilitysparse autoencodersdictionary learningscaling lawscircuits updates

A collection of short notes from Anthropic's interpretability team, published a month before Scaling Monosemanticity. It reports compute scaling laws for sparse autoencoders, the revised SAE training recipe later used on Claude 3 Sonnet, an ablation test of whether models rely on SAE features, an attribution-based variant of dictionary learning, an essay on productive wrong theories, and commentary on four external papers.

Chris Olah, Jack Lindsey, Tom Conerly, Adly Templeton et al. (Anthropic) · 2024-04-26 · original

Between Towards Monosemanticity (October 2023), which extracted interpretable features from a one-layer transformer with a sparse autoencoder, and Scaling Monosemanticity (May 2024, summarized in Mapping the Mind of a Large Language Model), which applied the same method to Claude 3 Sonnet, Anthropic's interpretability team had to make the method cheaper, more stable and better understood. The April 2024 Circuits Update records several pieces of that work while it was in progress: how to spend compute when training a sparse autoencoder, what changed in the training recipe, and whether a model's later layers actually respond to the features the autoencoder finds. Scaling Monosemanticity, published a month later, points back to this update for its training method.

About the Circuits Updates format

Circuits Updates are informal posts on the team's Transformer Circuits site, each a bundle of short, separately authored items. The April 2024 update says it collects "developing ideas" and minor points the team is unlikely to write a paper about, and asks readers to treat the results "like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper" (introduction). The sections below follow the update's items in order; each location is the item's title.

Terms used in this note

  • Sparse autoencoder (SAE) and dictionary learning. An SAE maps a model activation vector $x$ to a much larger vector of feature activations $f(x)$, most of them zero, and reconstructs $x$ as a weighted sum of learned decoder directions. Each direction is a feature. See Towards Monosemanticity for the original method.
  • MSE, L1 and L0. MSE is the mean squared reconstruction error. L1 is the sum of the absolute feature activations, used as a sparsity penalty during training. L0 is the number of features active on a token, the quantity the L1 penalty stands in for.
  • Dead feature. A feature that never activates over a large sample (here, 10 million samples).
  • Residual stream. The vector that runs through a transformer, to which each layer adds its output.
  • Ablation. Setting a component's value to zero (or another baseline) and measuring how the model's behavior changes.

Open roles in mechanistic interpretability

The first item is a hiring note by Chris Olah, Shan Carter, Adam Jermyn, Josh Batson and Tom Henighan ("Open Roles in Mechanistic Interpretability"). It estimates that the field has "perhaps 50 full time positions" and that the Anthropic team is now 17 people, after hiring 10 in 2023. It names managers as the most important hire (growth was likely to be limited by management capacity), alongside research scientists and research engineers, and says most of the team comes from fields other than machine learning.

Scaling laws for dictionary learning

Jack Lindsey, Tom Conerly, Adly Templeton, Jonathan Marcus and Tom Henighan ask how much extra compute improves SAE training and how to divide it ("Scaling Laws for Dictionary Learning"). The case study is an SAE on the residual stream after the third layer of a four-layer transformer.

Because there is no gold-standard measure of dictionary quality, the authors use the training loss as a proxy: MSE plus 5 times the L1 penalty, with qualitatively similar results for other weightings and for combinations of MSE and L0. Losses with different L1 coefficients cannot be compared with each other. SAE compute scales with the product of the number of features and the number of training steps; the authors swept both, holding learning rate, batch size and optimizer fixed, and tracked the lowest loss reachable for a given number of FLOPs and the step and feature counts that reach it.

The reported observations, all over the ranges tested:

  • Compute-optimal loss falls approximately as a power law in FLOPs.
  • The compute-optimal number of steps and number of features both grow approximately as power laws in the compute budget, with features growing somewhat faster. The compute-optimal number of steps scales with the number of features with an exponent between 0.5 and 1.
  • A larger L1 coefficient makes more training steps compute-optimal. The authors read this as reconstruction being optimized faster than sparsity during training, which they say they have also observed directly.
  • Losses based on L0 (optimized indirectly, by sweeping the L1 coefficient and picking the value that minimizes the L0-based loss) need more training steps for a given number of features than L1-based losses. The authors suspect, without having characterized it, that the extra steps let the SAE drive small activations fully to zero.

The item's caveats are that the details will vary with the model, the layer and other optimization choices, that tuning learning rate jointly with steps and features may change the trends, and that "the relationship between SAE loss and qualitative usefulness of SAE features is imperfect and may break down at sufficient scale". The trends are shown only in plots, with no fitted exponents beyond the 0.5 to 1 range above.

The revised SAE training recipe

Tom Conerly, Adly Templeton, Trenton Bricken, Jonathan Marcus and Tom Henighan describe changes made since Towards Monosemanticity with the aim of lowering SAE loss ("Update on how we train SAEs"). They say they have not ablated every decision and that the work focused on loss, not on whether loss is the right objective.

With input dimension $n$, $m$ features, encoder $W_e$, decoder $W_d$ and biases $b_e$, $b_d$, the SAE computes $f(x) = \mathrm{ReLU}(W_e x + b_e)$ and $\hat{x} = W_d f(x) + b_d$, and the loss over a dataset $X$ is written as

$$\mathcal{L} = \frac{1}{|X|} \sum_{x \in X} \lVert x - \hat{x} \rVert_2^2 + \lambda \sum_i \lvert f_i(x) \rvert \, \lVert W_{d,i} \rVert_2 ,$$

where $W_{d,i}$ is the decoder column for feature $i$ (the sparsity term is evidently also averaged over the dataset, since it depends on $x$). In Towards Monosemanticity the decoder columns were constrained to unit norm. Here they are unconstrained, and the sparsity penalty multiplies each activation by its decoder column's norm, so a feature cannot escape the penalty by shrinking its activation and growing its decoder vector. The authors believe this was the most important change.

The other settings, as stated in the item:

  • Initialization. Biases start at zero. Decoder columns point in random directions with a fixed norm between 0.05 and 1 (chosen "in an unprincipled way"; 0.1 is suggested as reasonable in most cases), and the encoder starts as the transpose of the decoder.
  • Data. Rows are shuffled, and the whole dataset is scaled by one constant so that the expected activation norm equals $\sqrt{n}$, so that the same $\lambda$ means the same thing for transformers of different sizes.
  • Optimization. Adam with $\beta_1 = 0.9$, $\beta_2 = 0.999$ and no weight decay; a default learning rate of 5e-5, decayed linearly to zero over the last 20% of training; a default of 200,000 steps; batch size 2,048 or 4,096; gradient norm clipped to 1. The penalty coefficient $\lambda$ starts at 0 and rises linearly to its final value (default 5) over the first 5% of steps.
  • No resampling. Fewer than 1% of features are dead at the end of training, so the team uses neither resampling nor "ghost grads", and does no fine-tuning afterwards.

Because a feature's effective activation is now $f_i \lVert W_{d,i} \rVert_2$, the team converts each trained SAE to an equivalent one with unit-norm decoder columns for analysis.

Results. On a fixed dataset, loss decreases consistently as $m$ grows, and the team has trained SAEs with "single digit millions" of features without problems, across transformer sizes and on both MLP activations and the residual stream. The old setup often gave higher loss, many dead features or many near-duplicate features at large $m$. The team adopts a change if it consistently lowers loss by at least 1%, or keeps loss roughly equal while simplifying the setup. It warns that L0 is unstable: in one case, training twice as long with half as many features changed MSE and L1 by less than 1% but cut L0 by 30%.

A results table covers nine runs on small models (a one-layer model's MLP, and the MLP and residual stream at layer 2 of a four-layer model, each with $\lambda$ of 2, 5 and 10), all with 131,072 features, 200,000 steps and batch size 4,096. The fraction of cross-entropy loss recovered ranges from about 98.3% (one-layer MLP, $\lambda = 2$, L0 about 99.6, normalized MSE about 0.031) down to about 83.9% (four-layer residual stream, $\lambda = 10$, L0 about 8.7, normalized MSE about 0.255). At the default $\lambda = 5$, the one-layer MLP run has L0 about 38.7, normalized MSE about 0.066 and about 96.4% of the loss recovered. Within each run type, a larger $\lambda$ always gives a lower L0 and a worse reconstruction.

Listed areas for improvement include initialization (reconstruction loss at initialization rises with $m$, which may cause problems at large $m$), resampling "low value" features, some form of post-training, and reducing shrinkage, the tendency of an L1 penalty to pull activations below their true size.

Testing whether models rely on SAE features

Jack Lindsey addresses a concern about the whole approach: SAE features may look meaningful to people without being the units the model computes with ("How Strongly do Dictionary Learning Features Influence Model Behavior?"). With an SAE on the residual stream after the third layer of a four-layer transformer, he takes datapoints where a feature is active, multiplies that activation by 0 (feature ablation) or 2 (feature doubling), passes the change through the model as in Towards Monosemanticity, and measures the average increase in the model's loss. The comparison is with perturbations of matched magnitude:

  • a random perturbation of the model's activations, which the item calls a weak baseline because activation variance is not the same in all directions;
  • a stronger control, a random perturbation of the vector of feature activations, passed through the SAE decoder;
  • structured perturbations: dampening all feature activations in proportion to their size, dampening the model activations themselves, and moving along the SAE's reconstruction error.

The reported findings, presented in a single figure with error bars over features:

  • Feature ablations hurt the model's loss significantly more than either random control.
  • Ablating a feature hurts much more than doubling it, which the item reads as the feature's influence partly saturating at high values.
  • Spreading an ablation across all features (dampening feature activations) has effects similar to single-feature ablations, while dampening the model activations by the same amount has less effect, and almost none in later layers. Because model activations also contain the SAE's bias and its unexplained error, the item concludes that the feature effect is not just an effect of activation norm.
  • Perturbing along the reconstruction error has more effect than random perturbations but less than feature ablations. The item reads this, "while not conclusive", as consistent with SAE errors lying mainly along feature directions (see the Gurnee item below).
  • The excess impact of ablations is larger for residual stream features than for MLP features, a result the item says it understands only to a limited extent.

The item calls the results preliminary but says they support the view that SAE features are high-leverage "levers" on model output.

A working group on interpretable architectures

Chris Olah and Adam Jermyn announce a small working group, embedded in both the Interpretability and Pretraining teams, to look for transformer design choices that make interpretation easier ("Interpretability Architectures Project"). The item cites two past examples: layer normalization, which complicates circuit analysis and attribution and which the team has worked to remove, and SoLU, an earlier attempt at a more interpretable activation function that the team now believes was not the right approach to that problem. No results are reported.

Caloric theory and the use of wrong theories

Tom Henighan contributes a short essay ("Caloric and the Utility of Incorrect Theories"). Caloric theory treated heat as a massless, self-repelling substance. It was wrong, but it explained many measurements of its time, and Sadi Carnot's analysis of heat engines, reasoned within it, contributed to the later recognition of the second law of thermodynamics; the essay also credits it with a correction to Newton's calculation of the speed of sound. The lesson drawn for interpretability is that early theories will probably be wrong and experiments should be designed to show where, but that "wrong theories can still provide real utility". The essay applies this to the superposition hypothesis (see Toy Models of Superposition): it may be disproven, yet still be "correct enough" to produce practical safety results.

A proposal: attribution dictionary learning

Chris Olah, Adly Templeton, Trenton Bricken and Adam Jermyn pose an open problem ("Open Problem: Attribution Dictionary Learning"). Ordinary dictionary learning uses only activations and ignores gradients and weights. The item argues that features have two sides: looking back toward the input they are representations, and looking forward toward the output they are actions, and both should be sparse.

The concrete proposal adds two terms to the SAE loss. With dense activations $x$, sparse codes $y$ and dictionary $D$ (so $x \approx x' = Dy$), the attribution of the codes to the model's loss is $A_y = y \odot \nabla_y L_{\text{LLM}}$, where $\odot$ is elementwise multiplication and $\nabla_y L_{\text{LLM}} = D^\top \nabla_x L_{\text{LLM}}$ comes cheaply from the gradient with respect to $x$. The loss becomes

$$L_{\text{SAE}} = \lVert x - x' \rVert^2 + \lambda \lVert y \rVert_1 + \alpha \lVert A_y \rVert_1 + \beta \, \lvert (x - x') \cdot \nabla_x L_{\text{LLM}} \rvert ,$$

where the third term asks for sparse attribution and the fourth penalizes attribution that the reconstruction fails to explain. A brief trial on a one-layer transformer gave features that "seemed about equally good" to the usual ones, which the authors do not consider conclusive; they invite others to investigate. Scaling Monosemanticity later mentions attribution SAEs as one possible way to make dictionary learning more data-efficient.

Isolating circuit paths of different lengths

Chris Olah re-explains an algorithm from A Mathematical Framework for Transformer Circuits for separating the contributions of paths through a model by length, that is, by how many components a path passes through ("Isolating Circuits Paths of Different Lengths"). The algorithm runs the model repeatedly. On the first iteration only the direct path along the residual stream (length 0) is used, while the paths of length 1 are saved; the next iteration uses the length-1 paths and saves those of length 2, and so on. The algorithm and its steps are given as diagrams. The item notes that changing the base case gives other tools, for example a variant that isolates "the k-step effects of an ablation".

Commentary on research by other groups

Emmanuel Ameisen, Joshua Batson, Hoagy Cunningham, Anna Golubeva and Jack Lindsey summarize four external works and add the team's comments ("Research By Other Groups").

Sparse feature circuits (Marks et al.). Marks and colleagues find circuits of SAE features, plus nodes for each SAE's error, in a six-layer, 70-million-parameter Pythia model, and use them to remove a classifier's reliance on unintended signals by editing out features, a method they call SHIFT. The update notes that the human judgment about what the classifier should not use is expressed here by choosing features by hand, rather than in a written instruction (it cites, as an example of prompting, the sentence "it is really really important to me that race, gender, age, and other demographic characteristics do not influence this decision" from Anthropic's discrimination paper) or in a curated unbiased dataset. Its comments are that the reported faithfulness leaves room for further pathways, and that the large number of nodes needed may point to SAEs being sparse in their representation of activations without being the sparsest description of computation.

Pathological reconstruction errors (Gurnee). On GPT-2 small, replacing activations with their SAE reconstruction raises the divergence of next-token predictions much more than random perturbations of the same size, which suggests systematic rather than random error. The update lists three possible explanations: missing features, an encoder that infers activations poorly, or a component of activity that is not sparse and linear but still matters. It links this to the ablation item above.

How DPO removes toxicity (Lee et al.). In GPT-2 Medium, the authors find "toxic" neurons whose outputs align with a toxicity probe. Preference fine-tuning with DPO reduces toxic output more than steering does, while every parameter vector keeps a cosine similarity of 0.99 or more with the original; many small upstream changes lower the toxic neurons' inputs. Because those neurons remain, scaling up their input weights restores the toxic behavior. The update draws a general reading from this: that jailbreaks exist because fine-tuning reduces a behavior's incidence without removing its mechanism. This is the update's interpretation, not a result it tests.

Binding entities to attributes (Feng and Steinhardt). In prompts such as two people living in two capitals, the information linking a person to a city sits in the residual stream at the country token ("France", "Thailand" in the example), and the authors hypothesize that it is represented as the sum of an entity vector and an abstract "binding vector" drawn from a subspace of at least two dimensions; binding vectors transfer across tasks, more so in larger models up to LLaMA 30B. The update adds an unpublished Anthropic observation, credited to Catherine Olsson, of neurons that fire on mentions of the first named entity in a text regardless of the name.

Limitations and open questions

  • Preliminary status. The update asks to be read as lab-meeting results, and most items say they are preliminary or untested in places (the training recipe has not been fully ablated; the authors do not consider the attribution SAE trial "at all dispositive").
  • Loss as a proxy. Both the scaling laws and the training recipe optimize SAE loss, and both items say loss is an imperfect stand-in for useful features, a gap that later updates in 2024 tried to measure (Circuits Updates – August 2024).
  • Small models. The scaling law and ablation experiments use a four-layer transformer, and the results table covers one- and four-layer models; the scaling item expects the details to vary by model and layer. The scaling trends and ablation effect sizes are given only in figures, so this note reports them qualitatively.
  • Readings of external work. The comments on other groups' papers, including the reading of jailbreaks, are the team's interpretation.

Research questions

  1. Do the SAE scaling laws hold for large models and deeper layers? The item expects its trends to vary by model and layer. Fitting the same compute-optimal curves for SAEs on a large model at several depths, and reporting the exponents, would show how far the four-layer results transfer.
  2. How well does SAE loss track feature usefulness as scale grows? The scaling item warns the relation "may break down at sufficient scale". Pairing loss with an interpretability measure (such as the evaluations in the August 2024 update) across a range of dictionary sizes would show where, if anywhere, they diverge.
  3. Why do ablations matter more for residual stream features than for MLP features? The item reports the difference without an explanation. Repeating the ablation test across layers and layer types, and checking whether the gap follows the SAE's reconstruction quality, would test simple explanations.
  4. Does attribution dictionary learning help on larger models? The one-layer trial was inconclusive. Comparing standard and attribution SAEs on a larger model, on interpretability scores and on how few features a circuit needs, would answer the open problem as posed.
  5. Is the saturation of feature influence a property of features or of the doubling test? Ablation hurt more than doubling. Measuring loss change across a range of scaling factors, rather than 0 and 2 only, would show the shape of the response.
  6. How does editing features compare with written instructions for removing an unwanted signal? The Marks et al. commentary sets manual feature selection beside a written prompt and a curated dataset as ways of expressing the same human judgment. A study applying all three to one classification task, measuring residual bias and task accuracy, would compare them directly.

How it connects

References

  • Olah, C., Carter, S., Jermyn, A., Batson, J., Henighan, T., Lindsey, J., Conerly, T., Templeton, A., Marcus, J., Bricken, T., Ameisen, E., Cunningham, H., and Golubeva, A. (2024). Circuits Updates – April 2024. Transformer Circuits Thread, 26 April 2024. https://transformer-circuits.pub/2024/april-update/index.html (Anthropic landing page: https://www.anthropic.com/research/circuits-updates-april-2024)
  • Bricken, T., Templeton, A., Batson, J., et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread. https://transformer-circuits.pub/2023/monosemantic-features/index.html
  • Templeton, A., Conerly, T., et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html