Circuits Updates – February 2024

A four-item Anthropic interpretability update on training sparse autoencoders. It explains why the very rare "ultralow density" features of Towards Monosemanticity share one encoder direction, lists training changes that reduced dying features (two later withdrawn), narrows the January claim for ghost grads after a bug, and reports that a tanh sparsity penalty improved proxy metrics but gave much less interpretable features, a result the team qualified in mid-2024.

Adly Templeton, Tom Conerly, Adam Jermyn, Joshua Batson et al. (Anthropic) · 2024-02-23 · original

The February 2024 Circuits Update has four short items, all about training sparse autoencoders (SAEs), the method by which Towards Monosemanticity (October 2023) extracted features from a one-layer transformer and which the team was then scaling up. In order: an explanation of the "ultralow density" features that took up most of the capacity of very wide autoencoders; a list of training changes that reduced dying features; a correction and narrowing of the ghost grads method introduced the month before; and a negative result for a tanh sparsity penalty meant to reduce shrinkage. Two of the items now carry banners saying the team's views have changed, pointing to the March 2024 update; the training changes that survived appear in the recipe of the April 2024 update; and the tanh result was qualified in the June and August 2024 updates.

About the Circuits Updates format and date

Circuits Updates are informal posts on Anthropic's Transformer Circuits site, each a bundle of short, separately authored items. This one asks readers to treat its results "like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper" (introduction). The page carries no publication date. The date used here, 23 February 2024, is the one the site's index records for the update in its markup (the index displays only "February 2024"). Archived copies agree: the update is absent from a Wayback Machine capture of the index at 09:30 UTC on 23 February and present in one on 7 March, and the earliest capture of the page itself is from 24 February. Apart from the two banners mentioned below, added later, that capture's text matches the live page. Locations below are item titles. The update reports no numbers beyond hyperparameter settings and one threshold, and has no figures.

Terms used in this note

  • Sparse autoencoder (SAE). A model that maps an activation vector to a much larger, mostly zero vector of feature activations and reconstructs the input from it. The encoder computes the activations; the decoder holds one direction per feature.
  • L1 penalty. The sum of absolute feature activations, added to the loss to make activations sparse.
  • Shrinkage. The tendency of an L1 penalty to pull nonzero activations below their best value, which worsens reconstruction. Also called feature suppression.
  • Dead or dying features. Features that stop activating during training. The update sometimes calls them "neurons".
  • Feature density. The fraction of inputs on which a feature activates.
  • L0 and loss recovered. L0 is the number of features active per input. Loss recovered measures how much of the model's performance is kept when the SAE's reconstruction replaces the real activations.
  • Pre-encoder bias. In Towards Monosemanticity's design, the decoder bias subtracted from the input before encoding.

Why ultralow density features share a direction

Adly Templeton, Joshua Batson and Tom Henighan take up a puzzle from Towards Monosemanticity ("Dying Features in Dictionary Learning"). In very wide autoencoders most features fell into an "ultralow density cluster": they fired very rarely, were not obviously interpretable and took up most of the autoencoder's capacity, a barrier to scaling. The item credits a replication by Neel Nanda with noticing that these features' encoder vectors were almost identical to each other, in a direction that was the same across random seeds, while their decoder vectors looked random, and that the shared direction had no obvious meaning.

The account offered. The authors "believe" they have identified the direction. The features in the cluster put almost all their encoder weight on the few transformer neurons that are weakly active across the whole dataset. Other features weight those neurons similarly but also weight other neurons; the low-density features have almost no weight elsewhere. The authors hypothesize that these features never found a useful decoder direction, so whenever they fired the gradient pushed them down, driving their other weights toward zero. That explains where the weight sits but not why it points the same way. For that, they point to the pre-encoder bias: subtracting it gives each weakly active neuron a consistent nonzero input, so weights of the opposite sign act as a negative bias, and every gradient step that lowers a feature's activation strengthens them.

What it means. With the shared direction explained, the authors write that the cluster "seems to be just a problem" of the L1 penalty killing features before they find a useful direction. The item's opening paragraph says it will "confirm some implications experimentally on larger models", but the item describes no experiment and gives no measurements; the account is presented as the team's explanation, which it calls "a potential account".

Training changes that reduced dying features

Adly Templeton, Tom Conerly, Jonathan Marcus, Tom Henighan, Anna Golubeva and Trenton Bricken list changes they found improved SAE training ("Improvements to Dictionary Learning"). A banner added after publication reads: "Our views on Beta1 and Pruning have changed", with a link to the March update. As listed:

  • Adam beta1. Setting Adam's beta1 to 0 "drastically reduces" dying features in larger autoencoders; raising beta2 from 0.999 to 0.9999 helps less; shortening how long a feature must be silent before ghost grads treats it as dead also helps less. The three did not stack.
  • Decoder norms and pruning. Because of the L1 penalty, a useful feature gains by growing its decoder norm and shrinking its activation. Towards Monosemanticity fixed decoder norms at exactly 1; constraining them to at most 1 worked better, and allowed features with decoder norm below 0.99 to be pruned after training.
  • No pre-encoder bias. The bias had come from early experiments on synthetic data; removing it helped on real transformer activations. A bias before the ReLU stays.
  • Normalization. Activations are scaled to an L2 norm of the square root of the input dimension, with the MSE summed over that dimension, so that hyperparameters carry over between transformers of different sizes.
  • Cooldown. The learning rate decays linearly to zero over the last 20% of training.

No losses, counts of dead features or comparisons are given for any change.

What happened to them. The March 2024 update ("Update on Dictionary Learning Improvements") reports that after further changes, including bug fixes, Adam betas of (0, 0.9999) and the defaults (0.9, 0.999) give similar training loss, and that pruning no longer helps because far fewer features are dead or have decoder norms below 0.99. The April 2024 recipe ("Update on how we train SAEs", summarized in Circuits Updates – April 2024) keeps the 20% cooldown and a form of the normalization (one constant for the whole dataset, so that the mean norm is the square root of the input dimension), uses no pre-encoder bias and default Adam betas, and replaces the norm constraint: decoder norms are unconstrained and the sparsity penalty is weighted by each feature's decoder norm.

Ghost grads, corrected

Tom Conerly revisits ghost grads, the extra loss term for reviving dead features introduced in Circuits Updates – January 2024 ("Ghost Grads Update"). This item also now opens with a banner: "Our views on ghost grads have changed".

The bug. The team's implementation had marked every feature as dead for the first K training steps, so ghost grads applied to all features at the start of training. Removing the bug made training loss (MSE plus L1, not counting the ghost loss) worse, so the bug "actually improved" it. The item does not give K or the size of the difference.

The narrower claim. Ghost grads were "a big improvement" on one-layer models, but initial experiments showed "little difference" on larger ones, and the team was unsure whether they would help on large models as other hyperparameters changed. The author says the team lacks a good understanding of why and when ghost grads help and expects better versions or better ways to handle dying features to exist.

Later. The March 2024 update says the team no longer sees ghost grads lower training loss even on one-layer models, that equal loss does not justify the extra compute, and that it has "some evidence" its implementation causes loss spikes. Outside Anthropic, Senthooran Rajamanoharan of Google DeepMind's interpretability team (LessWrong, 19 April 2024) found the original ghost grads worse than resampling and a version scaled by the fraction of dead features competitive with it; with that scaling, applying ghost grads to all features at the start of training (the effect of the bug described here) no longer helped.

The tanh penalty

Adam Jermyn, Adly Templeton, Joshua Batson and Trenton Bricken report a negative result ("Tanh Penalty in Dictionary Learning"). They believed shrinkage was at least partly why their SAEs recovered less of the MLP loss than hoped, and tried replacing the L1 penalty with $(A/B)\tanh(Bx)$, following Hurley and Rickard (2008). For a feature at the edge of activating this gives the same push toward zero as L1; for strongly active examples it gives almost none, so there is no incentive to shrink them.

What was found. Autoencoders trained with tanh were "a Pareto improvement in the space of L0 and loss recovered, often by a wide margin", but their features were "much harder to interpret". The team traced this to many more high-frequency features than under L1, with the most frequent features considerably more frequent (over 10%, in the item's words). Varying both tanh parameters over a wide range and several small architecture changes did not fix it. The item reports no curve, interpretability score or count.

The authors' reading. They have no solid theory of why tanh produces high-density features. One idea, that L1 works partly by penalizing strongly active examples, fails in its simplest form: if it were only a matter of accumulated gradient, raising the tanh penalty enough should remove the high-density features, and it did not. They set the approach aside, while still expecting the fix for shrinkage to be a modified sparsity penalty, and point to Wright and Sharkey's report (LessWrong) that fine-tuning some parameters without a sparsity penalty can resolve shrinkage.

Later. Scaling Monosemanticity (May 2024, "Limitations, Challenges, and Open Problems", under shrinkage) repeats the result in one sentence: the tanh penalty "improved proxy metrics, but made the resulting features less interpretable for unknown reasons". Reflections on Qualitative Research uses the case as its one example from the team's work of a proxy metric diverging from interpretability.

Revised. The result was then qualified twice, without a banner on this page. In Circuits Updates – June 2024 ("Comparing TopK and Gated SAEs to standard SAEs"), Hoagy Cunningham and Tom Conerly found that high-density features are a common outcome of shrinkage-reducing methods and suggested that the tanh finding "may not have been as pathological as previously thought, at least with high number of learned features in high sparsity regimes". In Circuits Updates – August 2024 ("Interpretability Evals Case Study"), the team reports that it no longer sees the uninterpretable high-density features when training with tanh: their number and interpretability are now similar to other variants'. Its "best guess" is that the April 2024 training changes fixed this, and it has "not run ablations to determine exactly what caused the change". The team's January 2025 Circuits Update ("Dictionary Learning Optimization Techniques") then trains with a tanh penalty, applied to each activation times its decoder norm and combined with a JumpReLU activation and a penalty on inactive features; that item does not discuss the February result.

Limitations and open questions

  • Preliminary status. The update asks to be read as lab-meeting results. Apart from hyperparameter values and the 10% figure, it reports no measurements, no figures and no model sizes.
  • Mostly superseded. Two of the four items carry banners that the team's views changed: the beta1 and pruning changes were judged no longer useful a month later, and ghost grads were dropped. The tanh result carries no banner, but the August 2024 update reports that the problem no longer appears under the later training setup.
  • An account without a test. The explanation of ultralow density features is a hypothesis about a mechanism; the item says implications were confirmed but does not show how.
  • A negative result without a stated method. The tanh finding rests on the team's judgment of interpretability, which is not described.
  • No repeated runs. No item reports seeds or intervals.

Research questions

  1. Does removing the pre-encoder bias remove the shared direction of low-density features? The account predicts it should. Training wide autoencoders with and without the bias and measuring the cosine similarity of rare features' encoder vectors would test it.
  2. How much of the early gain from beta1 = 0 came from bugs fixed later? March reports no difference after other changes. Rerunning the beta settings on the February and March code with the same data would separate the two.
  3. Why did ghost grads applied to all features at the start help? The bug improved loss. Comparing the February finding with the outside result that rescaling removes the benefit, under one training setup, would show whether both describe the same effect.
  4. Which later training change removed tanh's high-density features? The August 2024 update credits the April 2024 changes as a best guess, without ablations. Training tanh autoencoders under the February setup with the April changes added one at a time (decoder-norm-weighted penalty, no pre-encoder bias, initialization) and recording feature densities would identify it.
  5. How was interpretability judged for the tanh autoencoders? The item gives no method. Scoring L1 and tanh features with the same automated or human protocol, at matched L0, would turn the judgment into a measured comparison; the August 2024 evaluations did this for the later setup only.
  6. Were the February high-density features a matter of L0? The June 2024 update found such features pathological at high L0 but not at low L0 in other variants. Rescoring tanh runs across a range of L0 would show whether the February judgment depended on the sparsity level chosen.

How it connects

References