Using dictionary learning features as classifiers

interpretabilitysparse autoencodersdictionary learningclassifiersprobesspurious correlationsadversarial examples

A short interpretability update comparing bioweapon-harm classifiers trained on sparse autoencoder features with classifiers trained on raw model activations; feature classifiers are competitive, interpretable enough to expose a spurious correlation that the authors turn into an adversarial suffix attack, and somewhat better on translated and encoded text, but add complexity over a strong raw-activation baseline.

Trenton Bricken, Jonathan Marcus et al. (Anthropic) · 2024-10-16 · original

Dictionary learning had shown by 2024 that a model's internal activations can be rewritten as combinations of many interpretable "features" (see Towards Monosemanticity: Decomposing Language Models With Dictionary Learning and Mapping the Mind of a Large Language Model). One use that had been proposed for these features, but not yet demonstrated, was to build better classifiers on top of them: for example, a classifier that detects when a model is being prompted about harmful biological information. This short update from Anthropic's interpretability team, published on the Transformer Circuits Thread, asks whether classifiers trained on feature activations beat or add anything to the usual baseline of classifiers trained directly on the model's activations. The authors frame it as developing work, asking readers to treat it "like those of a colleague sharing some thoughts or preliminary experiments" rather than "a mature paper" (opening paragraph).

Terms used in this note

  • Raw activations. The vector of values in the model's residual stream (the internal representation passed between layers) at a given token. Here the middle layer is used for both kinds of classifier.
  • Feature activations. The same activations rewritten by a sparse autoencoder (SAE), a network trained to express each activation vector as a sparse combination of learned directions, each of which tends to correspond to an interpretable concept.
  • Linear probe. A logistic regression classifier trained on activations to predict a label, here whether a prompt is harmful or harmless with respect to bioweapons.
  • Max-pooling versus last token. Two ways to turn per-token activations into one input vector per prompt: take each dimension's maximum over the whole context, or use only the final token.
  • SAE residual. The part of the raw activations that the SAE's features fail to reconstruct.
  • ROC AUC. Area under the receiver operating characteristic curve; 1.0 is perfect separation of the classes and 0.5 is chance (footnote 3).

What the update does

The task is classifying prompts as harmful or harmless with respect to bioweapons. Classifiers are trained on synthetic data and tested on four held-out sets (section "Classifiers using Feature Activations can be Competitive with Raw Activations"):

  • synthetic_1, the validation set, from the same distribution as training;
  • synthetic_2, synthetic data generated by a different model, slightly out of distribution;
  • synthetic_3, synthetic_2 translated into other languages and encodings such as Korean, Hebrew and base64;
  • human, a small set of deliberately challenging questions written by human experts.

Feature classifiers use L1 regularization and raw-activation classifiers L2, with the coefficient chosen on synthetic_1. Experiments use an 18-layer model and a 1M-feature SAE trained on Claude 3 Sonnet, in both cases with biology data oversampled when training the SAE. The update also tests decision trees, the effect of adding the SAE residual, and three methodological choices.

What it finds

Results are shown mostly as bar charts; the text gives few exact values.

  • Competitive performance. With max-pooling, feature classifiers perform similarly to raw-activation classifiers on the synthetic sets, "or perhaps slightly stronger", for the 18-layer model. On the human set, last-token activations do better than max-pooling. The Claude 3 Sonnet SAE shows the same pattern, with raw activations slightly ahead on the human set (section "Classifiers using Feature Activations can be Competitive").
  • Out-of-distribution text. On synthetic_3 (translated and encoded), the feature classifier's ROC AUC is 0.96 against 0.9 for the raw-activation classifier, using the dictionary trained with a 90% biology mix, a larger gap than on synthetic_2 (section "Out of Distribution Generalization of Features"). In one example, a question about long-term complications of pneumonia written in heavy shorthand was labeled differently by the raw-activation classifier depending on whether it was written out or abbreviated, while the feature classifier labeled both versions harmless. The authors call this "evidence" that feature classifiers "may generalize better" to other input formats.
  • Decision trees. Depth-5 trees perform worse than logistic regression, and among trees, feature-based ones beat raw-activation ones. A depth-two feature tree is easy to read: its features fire on dangerous pathogens, on aerosolizing something, and on keeping a pathogen viable (section "Feature Decision Trees are Less Performant, but Easy to Interpret").
  • Residuals. Adding the SAE residual to the features closes the gap where features underperform. The residual alone also performs about as well as features and raw activations, which the authors did not investigate further; they suggest that L1 "shrinkage" leaves remnants of each feature in the residual (section "Residuals").
  • Three choices that matter. Using Human/Assistant conversation tags consistently across the datasets that train the model, the SAE and the classifier; oversampling domain-relevant data when training the SAE, which improved all sets except human; and max-pooling, which let feature classifiers outperform raw activations on every synthetic set (section "Important Considerations"). The authors are unsure why last-token pooling wins on the human set, and suggest max-pooling may overfit, since it provides "~100x more active features per prompt".

Spurious correlations and an adversarial attack

Because feature classifiers are interpretable, the authors inspected the features with the largest coefficients. One feature with a large negative weight (predicting harmless) fired on academic formatting instructions, such as requests for a word count and italic headers. It was active on 10 to 15% of examples in the synthetic sets, mostly harmless ones, so the correlation with harmlessness was real in the training data but irrelevant to harm. Removing the feature and retraining hurt performance on most sets, which the authors read as a spurious correlation shared across datasets (section "Using Features for Visualizing Dataset Spurious Correlations"; footnote 2).

They then wrote three suffixes on which this feature fires strongly, requests for an abstract with keywords, for about 2,000 words with research quotes, and for key findings written for a biology student, and attached each to 300 harmful prompts from the training set, making 900 attacks. Almost all of them were then classified as strongly harmless by both the feature and the raw-activation classifiers. A baseline attack built from raw-activation coefficients, using polite closing phrases such as thanks for help, was "far less effective" against both (section "Constructing Adversarial Examples based on Spurious Correlations"). The finding concerns wording in the input: a few sentences of formatting instructions, chosen by reading a feature, were enough to defeat both kinds of classifier.

Limitations and open questions

The authors state these:

  • The results are preliminary, from a lab-meeting-style update.
  • Features add "significant complexity" compared with raw activations, and raw activations are "a strong baseline" that may be preferable when performance matters more than interpretability (introduction).
  • Both pooling methods are lossy; attention-based pooling might do better for both representations (closing paragraph).
  • The reason last-token pooling wins on the human set is unknown.
  • The test sets are mostly synthetic, and the human set is small.

The update also mentions sets called synthetic_4 and synthetic_5 in one sentence about removing the formatting feature, without describing them elsewhere in the text.

Research questions

  1. How far does the formatting attack transfer? The suffixes were built from a feature in one classifier. Testing them on classifiers trained on different synthetic data, or on deployed safety classifiers, would show whether the spurious correlation comes from the data generator or is general.
  2. Does removing spurious features before training make classifiers robust? The update removed the feature and measured clean performance only. Re-running the 900 attacks after removal, or after adding counterexamples to the training data, would show whether the vulnerability closes.
  3. Why does max-pooling hurt on human-written questions? The authors offer overfitting as one hypothesis. Comparing pooling methods on a larger human set, and with attention-based pooling, would test it.
  4. Is the out-of-distribution advantage specific to translation and encoding? The 0.96 versus 0.9 gap is on one set. Measuring both classifiers on paraphrase, misspelling and other format shifts would show where features help.
  5. What does the residual contain? The residual alone classifies as well as the features. Checking how much of each top feature's direction survives in the residual would test the shrinkage explanation.

How it connects

  • Source of the method. The update builds on the dictionary learning of Towards Monosemanticity and on Scaling Monosemanticity, the Claude 3 Sonnet work announced as Mapping the Mind of a Large Language Model; both are in its reference list, and the Towards Monosemanticity note lists this update among later uses of dictionary features.
  • Later classifiers on model internals. Next-generation Constitutional Classifiers (January 2026) cites this update in its related work on model-internals approaches and uses linear probes on the guarded model's activations in production.
  • Course wiki. How to measure a classifier, including how the precision-recall balance moves with the decision threshold, is covered in Evaluating Classifiers.

References

  • Bricken, T., Marcus, J., Mishra-Sharma, S., Tong, M., Perez, E., Sharma, M., Rivoire, K., and Henighan, T.; edited by Jermyn, A. (2024, October 16). Using dictionary learning features as classifiers. Transformer Circuits Thread. https://transformer-circuits.pub/2024/features-as-classifiers/index.html (summary page: https://www.anthropic.com/research/features-as-classifiers)
  • Bricken, T., Templeton, A., Batson, J., et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread. https://transformer-circuits.pub/2023/monosemantic-features/index.html
  • Templeton, A., Conerly, T., Marcus, J., et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html