Circuits Updates – August 2024

interpretabilitysparse autoencodersSAE evaluationautomated interpretabilitycircuits updates

An Anthropic interpretability update that introduces two automated evaluations in which Claude predicts sparse autoencoder feature activations, uses them to compare six SAE variants (all newer variants beat the standard L1 SAE but cannot be told apart), and reports a partial reproduction of "self-explaining" features, in which a model is asked to describe a feature injected into its own forward pass.

Jack Lindsey, Hoagy Cunningham, Tom Conerly, Adly Templeton et al. (Anthropic) · 2024-09-06 · original

By mid-2024, several groups had proposed sparse autoencoder (SAE) variants that beat the standard L1 version on the usual trade-off between sparsity and reconstruction error. Whether they also produced more interpretable features was harder to say, because interpretability had no routine measure: the June 2024 update relied on 100 hand-scored features and one automated score. The August 2024 Circuits Update (published in September) builds two automated evaluations in which Claude makes predictions from feature visualizations, checks that they track known improvements, and uses them to compare six SAE variants. A separate item tests a different way of explaining features: injecting a feature into the model and asking the model what it means.

About the Circuits Updates format

Circuits Updates are informal posts on Anthropic's Transformer Circuits site, each a bundle of short, separately authored items. This one asks readers to treat its results "like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper" (introduction). Locations below are item and subsection titles.

Terms used in this note

  • Sparse autoencoder (SAE), feature. An SAE rewrites a model's activations as a sparse combination of learned directions, each called a feature (see Towards Monosemanticity).
  • L0, MSE. The average number of features active per token, and the mean squared reconstruction error.
  • Autointerpretability. Using a language model to explain or predict feature activations; here, "quantified autointerpretability" means scoring how accurately Claude predicts activations from the team's feature visualizations.
  • Dataset examples. The text snippets on which a feature activates most strongly, with per-token activation values, as shown in the team's feature browsers.
  • Decoder vector. The direction an SAE adds to the reconstruction when a feature is active; the cosine similarity between two features' decoder vectors measures how alike they are.

Two automated interpretability evaluations

Jack Lindsey, Hoagy Cunningham and Tom Conerly (edited by Adly Templeton) describe both methods ("Interpretability Evals for Dictionary Learning"). They note that the evals are not comprehensive and measure "a single notion" of interpretability, and expect that a suite of diverse evaluations will be needed.

The contrastive eval

The motivation is a common experience with contrastive pairs: a feature fires on one prompt but not on a near-identical one, even though its interpretation suggests it should fire on both, as when a "flower" feature fires on a description of one kind of flower but not another ("Contrastive Eval"). The procedure:

  1. Start from a hand-written list of about a hundred concepts, written manually "with some LLM assistance" and described as a rough first pass. The list ranges from "Photosynthesis" and "Sarcasm" to "A circular argument", "Imbalanced parentheses in code" and "Survivorship bias".
  2. Ask Claude to write two sentences for each concept, one involving it and one as similar as possible without it.
  3. Run both sentences through the SAE and collect the features active on one but not the other.
  4. In a fresh context, show Claude the two sentences and the feature's dataset examples, and ask which sentence activated the feature.

Accuracy is reported raw and weighted by how strongly the feature fired on its sentence, normalized by the feature's maximum over the SAE's training data, so that strong activations count more.

The sort eval

The sort eval asks whether a feature's top examples let Claude predict where else it fires, and whether similar features can be told apart ("Sort eval"). Two features are chosen, either at random or as a random feature and its nearest neighbour by decoder cosine similarity, the latter being a harder test of the SAE's "semantic resolution". Claude is shown the top ten dataset examples for each feature and one further example that activates one of them, and asked which feature it belongs to.

Sanity checks

After iterating on the method, "particularly the system prompts", the authors report that both evals pass these checks: in most cases where Claude guessed wrong, a human labeller also found the task hard; Claude's chain-of-thought reasoning looked reasonable; and scores improved with more SAE training steps, improved with more features, and correlated with the SAE's evaluation loss. No scores for these checks are given in the text.

Comparing six SAE variants

Adly Templeton, Tom Conerly, Jack Lindsey, Hoagy Cunningham and Andrew Persic apply the evals to newer SAE designs ("Interpretability Evals Case Study"). They note a specific worry: a variant might achieve a better L0 only because it optimizes L0 more directly than an L1 penalty does, without being more interpretable.

Variants and setup

The six variants, each implemented by applying the paper's main change to the team's own hyperparameters rather than reproducing the original setup exactly ("SAE Variants"):

  • Vanilla: the team's standard L1 SAE from Circuits Updates – April 2024.
  • TopK (Gao et al.), without any auxiliary loss.
  • TopK fixed: after training, a threshold is chosen so that on average $k$ features are active per token, and used in place of the top-$k$ selection, which makes features independent of each other and easier to analyze.
  • Gated (Rajamanoharan et al.).
  • Tanh (Jermyn et al., from the team's February 2024 update).
  • JumpReLU (Rajamanoharan et al.). The team could not reproduce JumpReLU being better than TopK and Gated on L0 and MSE, despite talking to the authors and matching the paper's details except for dataset and model; it lists a bug, a difference in model or another issue as possible causes. The version evaluated was the best in preliminary experiments and, unlike the original, lets gradients flow from the sparsity penalty to the encoder.

SAEs were trained on "a medium-sized non-frontier model". The standard SAE had about 260,000 features and was trained on 400,000,000 tokens. The team varied the number of features, the number of training tokens and the sparsity coefficient, and all evaluations used Claude 3 Haiku ("Other Hyperparameters").

Choosing which metrics to trust

Because the evals produce many possible metrics, the team plotted each against evaluation loss (reconstruction plus sparsity loss) while varying the number of features or training tokens, on the basis of a "strong prior" that lower evaluation loss means better features ("Evaluating Our Evaluations"). Metrics that tracked loss reliably were kept. Three were selected; the text names the sort eval on random pairs and the contrastive eval with all nonzero activations weighted equally. The three show roughly consistent patterns with loss, with one frequent outlier, a run trained on only 32,000,000 tokens.

Results

For each variant the team varied the sparsity coefficient with features and tokens fixed, and plotted MSE against L0, the sort eval and the weighted contrastive eval ("Evaluating SAE Variants"). The text reports the conclusions, with the values in plots ("Analysis"):

  • The variants are better than vanilla SAEs, which the authors call "the most important conclusion" and say validates their method of analysis while "derisking the use of one of these methods".
  • The non-vanilla variants perform similarly, and the evals likely lack the resolution to separate them; the authors suggest ease of use, robustness when scaling and ease of circuit analysis may matter more in choosing among them.
  • Tanh no longer produces the many uninterpretable dense features seen in February: its dense features are now about as numerous and as interpretable as other variants'. The team's best guess is that the April changes to its training setup fixed this, but it has not run ablations to confirm.

Self-explaining features

Emmanuel Ameisen and Joshua Batson (edited by Adam Jermyn) report trying to reproduce a method from a LessWrong post, "Self-explaining SAE features" ("Self-explaining SAE features"). The model is run on a prompt such as "The meaning of the word X is", with a feature's decoder vector, optionally scaled, injected into the residual stream at the position of "X", and a completion is sampled.

Simple features in a small model. In an 18-layer model the method worked for early-layer features with straightforward meanings. A first-layer feature for "big", injected at scale 100, gave the completion "The meaning of the word X is 'big'."; a "sport" feature at the same scale gave a description of physical activity in sport.

Harder features. An early-layer feature for "Jordan" in Michael Jordan contexts gave scale-dependent results: "'to go' or 'to go to'" at scale 10, "'the one who is the most powerful'" at 60, and "that he is the greatest of all time" at 410, which conveys reputation but never names him. Another prompt from related work, "Repeat X 4 times", also failed to produce "Jordan". A separate Michael Jordan feature at layer 4 did produce the token, split as "Jor- dan", at scale 260, which the authors take as variation in effectiveness between features.

Claude 3 Sonnet. On the production model the method worked in some cases, again better for earlier layers. A feature for "sheet" in spreadsheet contexts produced a definition of a thin, flat piece of material, missing the spreadsheet aspect; a feature on the parenthesis in ".bind(" in code produced "to tie or fasten". The authors found the method succeeded less often on larger models and was more sensitive to scale.

Failures. A layer 3 feature that fired on "of" in colour contexts could not be explained by injection at any scale or prompt tried, although the team's Claude-based autointerpretability tool labelled it "Color, shade, hue, tint, pigment". The authors suggest injection may depend on aspects of a feature not visible in dataset examples, which could make it useful for revealing them, and say they will keep the technique for cases where it adds information.

The prompts used

Prompts written by the team drive every measurement in the first two items, and the update prints them in full.

  • Sentence generation. Claude is asked to write two sentences, the first involving the concept and the second without it but otherwise "as similar as possible to the first sentence in its content, wording, and structure". The prompt adds that any other concept must appear in both, and that Claude should write nothing else.
  • Contrastive judgment. The system prompt describes the feature as a "neuron" that "activates for a specific concept", explains how the dataset examples are formatted, warns that activity "may be influenced just as much by the surrounding context as the activating token", and asks Claude to think step by step and end with "ANSWER: 1" or "ANSWER: 2", guessing if unsure.
  • Sort judgment. The same structure for two neurons and one query example.

The update says the authors iterated on the method, "particularly the system prompts", until the sanity checks passed, but does not show earlier versions or report how prompt changes moved the scores. The concept list, also hand-written, determines what the contrastive eval tests. In the self-explaining item, the prompt template ("The meaning of the word X is", or "Repeat X 4 times") is part of the method, and the authors report sweeping prompts as well as scales for the colour feature without success.

Limitations and open questions

  • One notion of interpretability. The authors say the evals capture one narrow notion and that a diverse suite is needed.
  • Validated against loss. Metrics were selected because they tracked evaluation loss, under a prior that lower loss means better features; an eval that measured something loss misses would have been discarded by this procedure.
  • Limited resolution. The evals separate vanilla SAEs from the newer variants but not the variants from each other.
  • Replication gaps. Variants were reimplemented on the team's own setup, without auxiliary losses in TopK, and JumpReLU's reported advantage was not reproduced for reasons the team could not identify.
  • Unreported numbers. Eval scores, sanity-check results and variant comparisons are given in plots; the text states conclusions only.
  • Self-explanation is unreliable at scale. It worked mostly for simple early-layer features and less often on larger models.

Research questions

  1. Do the evals agree with human judgments feature by feature? The sanity check reports that humans also struggled where Claude failed. A study in which people perform the same contrastive and sort tasks on a sample of features, with agreement rates reported, would quantify how well the automated scores stand in for human ones.
  2. How sensitive are the scores to the wording of the system prompts? The prompts were iterated until checks passed. Re-running the evals with paraphrased prompts, or with features called "features" instead of "neurons", and comparing variant rankings, would show whether conclusions depend on the wording.
  3. Does the concept list bias the contrastive eval? The list was a rough first pass. Comparing scores on the hand-written list with scores on a list sampled from a large concept inventory would show whether the choice of concepts favours some SAE types.
  4. Can any eval separate the newer SAE variants? The current evals cannot. Downstream tasks such as steering or circuit tracing with each variant, scored on success, would test whether the variants differ in ways the evals miss.
  5. Why does self-explanation fail more often in larger models? The authors report the trend without explanation. Measuring success by layer and by injection scale across models of several sizes, with a fixed set of features, would show whether depth, scale sensitivity or model size drives it.

How it connects

  • Within this library. The vanilla SAE and the training changes credited with fixing tanh are from Circuits Updates – April 2024, which this update links. Circuits Updates – June 2024 had compared TopK and Gated SAEs with the vanilla SAE using hand scoring and one automated score; this update adds four more variants and two new evals. Circuits Updates – July 2024 measured feature sensitivity with a different Claude-based method, rating text relevance rather than predicting activations.
  • In the course wiki. Keeping only the metrics that agree with a trusted signal resembles the way online search metrics are validated against explicit user satisfaction ratings (Online Evaluation). The contrastive and sort tasks are two-way forced choices scored by accuracy, one of the measures discussed in Evaluating Classifiers.

References

  • Lindsey, J., Cunningham, H., Conerly, T., Templeton, A., Persic, A., Ameisen, E., Batson, J., and Jermyn, A. (2024). Circuits Updates – August 2024. Transformer Circuits Thread, published 6 September 2024. https://transformer-circuits.pub/2024/august-update/index.html (Anthropic landing page: https://www.anthropic.com/research/circuits-updates-august-2024)
  • Gao, L., et al. (2024). Scaling and Evaluating Sparse Autoencoders. arXiv:2406.04093.
  • Rajamanoharan, S., et al. (2024). Improving Dictionary Learning with Gated Sparse Autoencoders. arXiv:2404.16014.
  • Rajamanoharan, S., et al. (2024). Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders. arXiv:2407.14435.