Circuits Updates – January 2025

A one-item Anthropic interpretability update that sets out how the team then trained sparse autoencoders and crosscoders, as a starting point for outside groups: a JumpReLU activation with gradients passed to all parameters, a tanh sparsity penalty, a new penalty to reduce dead features, initialization and data scaling, and optimizer settings, none of them ablated in the update.

Tom Conerly, Hoagy Cunningham, Adly Templeton et al. (Anthropic) · 2025-01-27 · original

The January 2025 Circuits Update has a single item, "Dictionary Learning Optimization Techniques", by Tom Conerly, Hoagy Cunningham, Adly Templeton, Jack Lindsey, Basil Hosmer and Adam Jermyn. It describes the setup Anthropic's interpretability team then used to train sparse autoencoders (SAEs) and crosscoders, the dictionaries behind its feature and circuit work, updating the recipe last set out in Circuits Updates – April 2024. It reports no experiments: it is a specification, offered "in the hope that it will be a useful starting point for external groups", with the statement that the team has not "extensively ablated all the decisions here".

About the Circuits Updates format and this page

Circuits Updates are informal posts on Anthropic's Transformer Circuits site, each a bundle of short, separately authored items. This one asks readers to treat its results "like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper" (introduction). The web page is the version of record; there is no arXiv or other version, and no figures.

Date. The page prints only "January 2025". The site index dates its entry 27 January 2025 in its markup (a data-date attribute, read November 2026), and archived captures agree: the update is absent from a capture of the index at 19:55 UTC on 27 January 2025 and present in one at 05:42 UTC on 28 January (Wayback Machine snapshots 20250127195530 and 20250128054210), a window that falls on 27 January in US time zones. This note uses the index date.

A correction on the page. A notice at the top of the item says an earlier version "incorrectly wrote the initialization" of the decoder weights as uniform on $(-1/n, 1/n)$ rather than $(-1/\sqrt{n}, 1/\sqrt{n})$. The notice is undated. This note follows the corrected text.

Terms used in this note

  • Sparse autoencoder (SAE). A network trained to reconstruct a model's activations as a sparse combination of learned directions, called features. Its encoder maps an activation to feature activations; its decoder maps them back.
  • Crosscoder. A dictionary that reads from one set of activations and reconstructs another (for example, several layers or two models); see Sparse Crosscoders. The item's notation allows the input $x$ and the target $y$ to differ, but it does not say which kinds of dictionary each case covers.
  • JumpReLU. An activation that passes a value through unchanged if it exceeds a learned per-feature threshold and outputs zero otherwise (Rajamanoharan et al., 2024).
  • Straight-through estimator. A way to train through a step that has no useful gradient, by substituting a surrogate gradient on the backward pass.
  • Dead feature. A feature that never or almost never activates, and so wastes dictionary capacity.
  • Sparsity penalty. A loss term that pushes most feature activations to zero.

The architecture and loss

With input dimension $n$, output dimension $o$ and $m$ features, the item defines the feature activations as $f(x) = \text{JumpReLU}(W_e x + b_e, t)$, where $t$ is a vector of log thresholds, so feature $i$ is active only if its pre-activation exceeds $\exp(t_i)$. The reconstruction is $\hat{y}(x) = W_d f(x) + b_d$. The loss has three terms:

$$\mathcal{L}(x, y) = \lVert y - \hat{y}(x) \rVert_2^2 + \lambda_S \sum_i \tanh\big(c\,\lvert f_i(x)\rvert\, \lVert W_{d,i} \rVert_2\big) + \mathcal{L}_P(x),$$

$$\mathcal{L}_P(x) = \lambda_P \sum_i \text{ReLU}\big(\exp(t) - f_i(x)\big)\, \lVert W_{d,i} \rVert_2 .$$

The first term is squared reconstruction error. The second is the sparsity penalty, in which each feature's activation is weighted by the norm of its decoder column and passed through tanh, so the penalty saturates for large activations rather than growing without limit as an L1 penalty does. The third is what the item calls the pre-act loss, which "applies a small penalty to features which don't fire" ("Dictionary Learning Optimization Techniques").

How the JumpReLU is trained. The gradient with respect to the input is 1 above the threshold and 0 below it. The gradient with respect to the threshold is nonzero only in a window of width $\varepsilon$ around it, where it equals $-\exp(t)/\varepsilon$. This straight-through estimator follows Rajamanoharan et al. (2024), with two stated departures: the team lets the gradient flow "to all model parameters, not just the JumpReLU thresholds", and it uses its tanh penalty instead of the penalty Rajamanoharan et al. introduced.

Why the pre-act loss. The authors say they have found the pre-act loss "extremely helpful in reducing dead features". Because it gives a gradient whenever a feature is inactive, which is most of the time, its appropriate scale is lower than that of the other terms by a factor of about the typical feature density. No measurement of dead features with and without it is given.

The settings

The item gives these values (all from "Dictionary Learning Optimization Techniques"):

  • Loss constants. $c = 4$, $\varepsilon = 2$, $\lambda_P = 3 \times 10^{-6}$. For the sparsity coefficient the item says it uses "values of $\lambda_S$ around 10", and in the training paragraph that "a reasonable default for $\lambda_S$ is 20 given our other parameter settings".
  • Sparsity schedule. $\lambda_S$ starts at 0 and rises linearly to its final value over the whole of training.
  • Initialization. $b_d$ starts at zero and the log thresholds $t$ at 0.1. $W_d$ is drawn uniformly from $(-1/\sqrt{n}, 1/\sqrt{n})$. When the input and target datasets are the same ($X = Y$, as for an SAE), $W_e$ starts as $(n/m)\,W_d^\top$; when they differ ($X \ne Y$, as for a transcoder; the item names neither case), $W_e$ is drawn uniformly from $(-1/\sqrt{m}, 1/\sqrt{m})$.
  • Encoder bias. $b_e$ is set from a sample of the data so that each feature fires on a fraction $10{,}000/m$ of datapoints, so that about 10,000 features fire per datapoint in aggregate at the start. The authors "think this initialization is important for avoiding dead features".
  • Data. Rows are shuffled, and the dataset is scaled by one constant so that the mean norm of a datapoint is $\sqrt{n}$, so that one value of $\lambda_S$ means the same thing for activations from transformers of different sizes.
  • Optimizer. Adam with $\beta_1 = 0.9$, $\beta_2 = 0.999$ and no weight decay; batch size 32,768, which the authors "believe to be under the critical batch size"; gradient norm clipped to 1. The learning rate and number of steps are set by scaling laws that are not given, with $2 \times 10^{-4}$ as "a reasonable default" learning rate, decayed linearly to zero over the last 20% of training.

Normalizing the decoder afterwards. Because the penalty weights each activation by its decoder norm, a feature's effective activation is $f_i \lVert W_{d,i} \rVert_2$. For analysis the team converts the trained dictionary into an equivalent one with unit-norm decoder columns, rescaling the encoder weights and bias by the decoder norms and dividing the decoder by them; the predictions are unchanged.

What changed from the April 2024 recipe

The April 2024 update had described a ReLU SAE with an L1 penalty weighted by decoder norm. Set side by side (the comparison is this note's; the January item does not refer to the earlier one), the January recipe replaces ReLU with JumpReLU, replaces L1 with tanh, adds the pre-act loss and the data-dependent encoder-bias initialization, and ramps the sparsity coefficient over all of training instead of the first 5% of steps. The defaults also change: learning rate $5 \times 10^{-5}$ to $2 \times 10^{-4}$, batch size 2,048 or 4,096 to 32,768, and $W_e = W_d^\top$ with fixed-norm random decoder columns to the uniform draws above. It keeps the scaling of the dataset to a fixed mean norm, the weighting of the penalty by decoder norm, the Adam settings, clipping and learning-rate decay, and the post-training normalization, much of it in the April item's wording. The tanh penalty had appeared among the variants compared in the August 2024 update, which credits it to the team's February 2024 update. The August 2024 update also reported that the team "were unable to replicate JumpReLU being an improvement on L0-MSE over TopK and Gated"; its JumpReLU variant departed from the original paper by letting gradients flow "from the sparsity penalty to the encoder". The January item does not discuss why it chose tanh and JumpReLU.

Limitations and open questions

  • No ablations or results. The item states the decisions have not been "extensively ablated", and it reports no reconstruction error, sparsity, dead-feature counts or feature quality for dictionaries trained this way. Each choice is a recommendation, not a measured improvement.
  • Two values for the sparsity coefficient. The item gives $\lambda_S$ "around 10" in one paragraph and a "reasonable default" of 20 in another, without saying when each applies.
  • Scaling laws not given. Learning rate and step count are set "based on scaling laws" that the item does not state.
  • Model and data unstated. The item does not say which models or layers the recipe was developed on.
  • The pre-act loss as printed. The formula applies the ReLU to $f_i(x)$, which the item defines as the output of the JumpReLU, although the term is named for pre-activations; for an inactive feature $f_i(x) = 0$, and the item's stated gradient gives no signal to the pre-activation below the threshold. Circuit Tracing's version penalizes the pre-activation, $\sum_f \text{ReLU}(-h_f)$, and Progress on Attention describes a penalty "on the negative pre-activations". Which form this item intended is not stated; the observation is this note's.

Later use of the recipe

  • Circuit Tracing (March 2025). The methods paper's appendix ("ML details") cites this item for "more details on our training setup", trains its cross-layer transcoders with JumpReLU and a tanh penalty, and says this combination "modestly" outperformed alternatives on reconstruction error against sparsity. It lists departures from Rajamanoharan et al. and from the team's "prior work on SAEs": a bandwidth of 1.0 for the straight-through estimator, an initial threshold of 0.03, and a pre-activation loss of the form $\sum_f \text{ReLU}(-h_f)$ on the pre-nonlinearity activation $h_f$, used only for the Claude 3.5 Haiku runs, with a coefficient of $3 \times 10^{-6}$. See Circuit Tracing.
  • Progress on Attention (April 2025). In its description of multitoken transcoders ("Multitoken Transcoders"), the post links this item when it says a penalty on negative pre-activations is "an easy and effective way to avoid dead features". Those transcoders use an L1 penalty, not tanh. See Progress on Attention.
  • Data Point Initialization (October 2025). A later item initializes dictionary weights from noisy copies of data points and compares this with random initialization (Circuits Updates – October 2025). It does not cite this item; the connection to this recipe's random decoder initialization is this note's.

Research questions

  1. How much does each choice contribute? The item lists changes without ablations. Training matched dictionaries that add JumpReLU, tanh, the pre-act loss and the encoder-bias initialization one at a time, and reporting reconstruction error, sparsity and dead-feature counts, would show which matter.
  2. Does the pre-act loss reduce dead features at no cost? The authors call it "extremely helpful" without numbers. Comparing the share of dead features and the reconstruction error at a fixed sparsity, with and without the loss and across values of $\lambda_P$, would answer it.
  3. Does passing the straight-through gradient to all parameters help? This is a stated departure from Rajamanoharan et al. A controlled comparison of the two gradient paths, on the same data and dictionary size, would test it.
  4. When is $\lambda_S$ of 10 or of 20 appropriate? The two stated values differ. A sweep of $\lambda_S$ on activations from transformers of several sizes, after the item's data scaling, would show whether one value gives comparable sparsity across them, as the scaling is meant to ensure.
  5. Is the data-dependent bias initialization needed with the pre-act loss? Both are offered as remedies for dead features. Training with each alone and with both would show whether one makes the other unnecessary.

How it connects

References