Circuits Updates – June 2024

interpretabilitysparse autoencodersdictionary learningSAE evaluationcircuits updates

A short Anthropic interpretability update that replicates TopK and Gated sparse autoencoders, finds both better than the standard L1 version on the sparsity and reconstruction trade-off with no detectable loss of interpretability in blinded and automated checks, and reviews four external papers on evaluating SAEs and on features that are not one-dimensional.

Hoagy Cunningham, Tom Conerly et al. (Anthropic) · 2024-06-28 · original

In spring 2024 two other labs proposed new ways to train sparse autoencoders (SAEs): Gated SAEs from Google DeepMind and TopK SAEs from OpenAI. Both reported a better trade-off between how many features are active and how well the activations are reconstructed. Anthropic's own earlier attempt at the same improvement, a tanh sparsity penalty, had given better numbers but less interpretable features. The June 2024 Circuits Update asks whether the new methods hold up when the team reproduces them, and whether their gains in the usual metrics come with a hidden cost in interpretability. A second item reviews external work on how SAEs should be evaluated at all, and on evidence that some features are multidimensional.

About the Circuits Updates format

Circuits Updates are informal posts on Anthropic's Transformer Circuits site, each a bundle of short, separately authored items. Like the others, this one asks readers to treat its results "like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper" (introduction). Locations below are the item's subsection headings.

Terms used in this note

  • L0 and MSE. L0 is the average number of features active per token; MSE is the mean squared error of the reconstruction. Lower is better for both, and they trade off against each other.
  • Shrinkage. The update defines it as the L1 penalty making the loss-optimal activation of a feature lower than its "true" activation, which worsens reconstruction at a given sparsity ("Background").
  • Standard (vanilla) SAE. The L1-penalized SAE described in Circuits Updates – April 2024.
  • TopK SAE. An SAE that keeps only the $K$ largest feature activations on each token and sets the rest to zero, so L0 equals $K$ by construction.
  • Gated SAE. An SAE that separates the decision of which features are active from the estimate of how large they are, so that the L1 penalty applies only to the first (Rajamanoharan et al., 2024, abstract).
  • Feature density. How often a feature is active across tokens; a dense feature fires on a large share of them.

Comparing TopK and Gated SAEs with standard SAEs

Hoagy Cunningham and Tom Conerly wrote the main item ("Comparing TopK and Gated SAEs to standard SAEs").

Background and aim

The item lists three recent proposals for reducing shrinkage: Gated SAEs (Rajamanoharan et al., Google DeepMind), a TopK activation in place of an L1 penalty (Gao et al., OpenAI), and fine-tuning a subset of parameters after training (Wright and Sharkey). It recalls that Anthropic's February 2024 update had found a tanh penalty improved L0 and MSE but gave "significantly less interpretable features". Rajamanoharan et al. had found no significant interpretability difference between Gated and standard SAEs, and Gao et al. had compared interpretability only among TopK settings. The team therefore wanted to check both the reconstruction gains and, especially for TopK, that interpretability did not regress ("Background").

Reconstruction and sparsity

The replication found "a significant benefit" from moving from standard SAEs to either Gated or TopK SAEs, with the two new methods performing similarly ("L0/MSE Results"). Both used tied initialization of encoder and decoder; the Gated SAE also used the April update's unconstrained decoder norm. Footnote 1 records that to make the Gated SAE work the team had to stop freezing the decoder in its auxiliary loss, because with the decoder frozen the decoder norm kept shrinking. The item notes that the new methods received less tuning than the standard SAE (for example, without the dead-feature technique Gao et al. used), yet still showed the improvement. The curves themselves are in a plot.

Feature density

Both new methods produce more dense features than the standard SAE, and TopK more than Gated even when the Gated SAE has a higher L0 ("Density"). The item's table shows the counts. At similar L0, the standard SAE (L0 38) had 13 features above a density of $10^{-2}$ and 360 above $10^{-3}$, while the Gated SAE (L0 37) had 8 and 4,998. The TopK SAE with $K = 20$ had 54 and 2,855, and with $K = 50$ it had 261 and 8,076. TopK runs also varied more in their average activation level, with Gated SAEs in between.

Blinded human evaluation

To compare feature quality, the team built a blind test in which raters saw a feature's top-activating contexts and the first seven of its activation intervals, without knowing which SAE it came from ("Blinded evaluations"). Lower intervals were hidden because Gated and TopK SAEs have recognizably different low-activation patterns that would have revealed the method. Raters scored two 1 to 5 scales:

  • Activation consistency, from "No discernible pattern" (1) to "Clear pattern with no deviating examples" (5).
  • Complexity, from a single-token feature (1) to a "rich" feature with a unifying theme across diverse contexts (5).

One hundred features were scored. The item reports "no discernible difference" between the methods at this scale and expects any difference to be slight. Its footnote cautions that the error bars are "probably too confident", because consistency scores are heavy-tailed at the low end and a few unintelligible features can move them.

Checking the dense features

Because dense features have an outsized effect on reconstruction, the team examined the densest features of a TopK SAE with $K = 20$ and 256k features on an 18-layer model. The densest fired about 9% of the time; 7 features fired more than once per 40 tokens and several dozen more than once per 100. The densest were often related to end-of-text (<EOT>) tokens, firing on them or on the text shortly after. Others covered common words such as "the" and "of", languages such as Italian and Russian, and pre-20th-century English. A few, often code-related, in the 1 to 2% density range were unintelligible, a number the item does not consider unexpected. Some runs with larger $K$ had several features firing on 20 to 50% of tokens without clear explanation, and a $K = 100$ run had dozens firing on more than 1 in 10 tokens that seemed "largely uninterpretable".

The item concludes that dense features are a common result of shrinkage-reducing methods and are pathological at high L0, but that these methods allow much lower L0 at the same MSE, and at low L0 the features are "of at least comparable quality" to standard ones. It adds that this suggests the earlier tanh result "may not have been as pathological as previously thought", at least with many features and high sparsity.

Automated interpretability scores

The team also ran 200 to 250 features from each of four SAEs through Clerp, its automated interpretability system, in which a model writes an explanation of a feature, predicts its activations from that explanation, and is scored by the correlation with the real activations ("Clerp"). A Gated SAE with L1 coefficient 5 and a TopK SAE with $K = 20$ scored as well as a standard SAE with L1 coefficient 10, which had far worse MSE and lower L0. The item cautions that the new methods have fewer low activations, which may make the prediction task easier, and that this would also fit the manual results.

What remains unknown

The item judges both methods "strong alternatives" with little downside, but says it is still hard to know whether an SAE's basis is better or worse: a method that reduces shrinkage could improve MSE while leaving the features themselves unchanged ("Future Work"). It notes Gao et al.'s probe-based evaluation as principled but unclear in which datasets features should yield clean probes, and closes that SAEs must finally be judged on whether they help with debugging, steering, finding circuits, understanding fine-tuning and robustness, which current evaluations "don’t get to the heart of".

Commentary on research by other groups

Brian Chen, Rodrigo Luger, Adam Pearce, Josh Batson and Hoagy Cunningham summarize four external works ("Research by Other Groups").

Scaling SAEs to GPT-4 (Gao et al., OpenAI). TopK removes the L1 pressure and with it shrinkage, and lets L0 be set directly. Gao et al. reduce dead features by initializing the encoder as the decoder's transpose and with an auxiliary loss in which dead features model the reconstruction error. They propose two evaluations: the best linear probe on a classification dataset that uses a single SAE feature's direction, where on GPT-2 the probe loss was lowest for SAEs with L0 between 64 and 128 and fell with more features; and the sparsity of each feature's effect on the output logits, best at L0 between 128 and 256. The update notes that its own item above replicates the TopK gain.

RAVEL (Huang et al.). A benchmark that asks whether an interpretability method can isolate one attribute of an entity and intervene on it alone: for example, patching only the "continent" part of the representation of Paris from that of Tokyo should make the model say Paris is in Asia while still saying people there speak French. The best method tested was a variant of Distributed Alignment Search. The update is most interested in using RAVEL for unsupervised methods such as SAEs, and notes a limitation: some attributes may not be separable in the model, since French is tied historically to Europe.

Principled SAE evaluation on IOI (Makelov et al.). On GPT-2 small's indirect object identification task ("When Mary and John went to the store, John gave a book to" Mary), the authors train supervised dictionaries for the task's known attributes and compare them with SAEs trained only on task data or on general web text. They test whether reconstructions suffice for the task, whether a few feature edits can change the answer, and whether features match simple predicates of the attributes. Supervised dictionaries did well on all three, unsupervised ones fell short, and task SAEs beat general ones. They also report "feature occlusion" (features for most indirect-object names but very few subject names) and "feature over-splitting" of a binary position attribute, which the authors caution may not be an optimization failure, since the attribute was chosen by human judgment.

Multidimensional features (Engels et al.). "Not All Language Model Features Are Linear" finds irreducible two-dimensional, circular representations of days of the week and months of the year in Llama 3 8B and Mistral 7B, and shows through patching that the model uses the angle on the circle when doing arithmetic such as "two days from Monday". The computation sits mainly in MLP layers. The study found no other clearly interpretable multidimensional features, which the update says could reflect the search method or could mean such features are rare. The July 2024 update returns to this paper in an essay on what "linear representation" should mean.

Limitations and open questions

  • Small samples and blinded scale. The human evaluation covers 100 features; the item expects any real difference to be slight and says its error bars are probably overconfident.
  • Unequal tuning. The new methods were tuned less than the standard SAE, so the comparison may understate them, as the item notes.
  • Proxy evaluations. Both the manual and automated checks measure firing patterns. The item states that they do not show whether an SAE's basis is better, or whether it helps with the tasks interpretability is for.
  • One model per check. The dense-feature inspection uses one run on an 18-layer model; other results are shown in plots without stated model sizes.

Research questions

  1. Do shrinkage-reducing methods change the features or only their magnitudes? The item raises this directly. Matching features between a standard and a TopK SAE trained on the same activations, and measuring how many decoder directions correspond, would answer it.
  2. At what L0 do dense features turn uninterpretable? Dense features looked acceptable at $K = 20$ and largely uninterpretable at $K = 100$. A sweep over $K$, with blinded scoring of the densest features at each value, would locate the change.
  3. Does hiding low activations hide a real difference? Raters saw only the top seven intervals. A separate blinded study of low-activation examples would show whether TopK and Gated features are less consistent there.
  4. Does Clerp favour SAEs with fewer weak activations? The item suspects the task is easier when low activations are absent. Scoring the same features with and without low-activation examples in the prediction set would test the suspicion.
  5. Would downstream tasks rank the methods differently? The item names debugging, steering and circuit finding as the real tests. Running the same steering or circuit-tracing task with each SAE type and comparing success would give that ranking.

How it connects

References

  • Cunningham, H., Conerly, T., Chen, B., Luger, R., Pearce, A., and Batson, J. (2024). Circuits Updates – June 2024. Transformer Circuits Thread, 28 June 2024. https://transformer-circuits.pub/2024/june-update/index.html (Anthropic landing page: https://www.anthropic.com/research/circuits-updates-june-2024)
  • Gao, L., et al. (2024). Scaling and Evaluating Sparse Autoencoders. arXiv:2406.04093.
  • Rajamanoharan, S., et al. (2024). Improving Dictionary Learning with Gated Sparse Autoencoders. arXiv:2404.16014.
  • Huang, J., et al. (2024). RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations. arXiv:2402.17700.
  • Makelov, A., Lange, G., and Nanda, N. (2024). Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control. arXiv:2405.08366.
  • Engels, J., et al. (2024). Not All Language Model Features Are Linear. arXiv:2405.14860.