Circuits Updates (May 2023)

interpretabilitysuperpositiondictionary learningtoy modelscircuits updates

An informal collection of short research notes from Anthropic's interpretability team, published 24 May 2023: a plan to extract features from superposition with dictionary learning, an information-based way to choose how many features to look for, worries about dictionary learning, toy-model results on feature geometry, weight and attention-head superposition and feature manifolds, and a digest of outside work.

Trenton Bricken, Joshua Batson, Adly Templeton, Adam Jermyn, Tristan Hume, Tom Henighan, Chris Olah (Anthropic) · 2023-05-24 · original

The Circuits Updates are informal posts on the Transformer Circuits Thread in which Anthropic's interpretability team shares work in progress. This first update, published on 24 May 2023 together with the essay Interpretability Dreams, collects eight short research notes and a digest of comments and outside papers. The team asks readers to treat the results like those of "a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting" (introduction), and the closing "About" section adds that mistakes are very possible. The common thread is superposition, the hypothesis from Toy Models of Superposition (2022) that a network represents more features than it has neurons by storing them in overlapping directions. Most of the notes concern how to undo superposition and what a "feature" is. Because the update is a set of separate items, this note gives each its own short section; the section names follow the original's headings. The update reports no numbered tables, and its results are shown in figures that this note describes in words.

Terms used in this note

  • Feature, superposition, polysemantic. As in the note on Toy Models of Superposition: a feature is a property of the input that the network represents; superposition is storing more features than dimensions; a polysemantic neuron responds to several unrelated features.
  • Dictionary learning. Writing a matrix of activations A as the product of a sparse code S (which features are active, and how strongly, for each data point) and a dictionary D (one direction per feature): A = SD. The number of dictionary directions has to be chosen in advance.
  • Sparse autoencoder. A small network trained to reconstruct activations through a wider, mostly-zero hidden layer; each hidden unit plays the role of a dictionary feature. The approach is described in detail in the note on Towards Monosemanticity.
  • Skip-trigram. A simple pattern an attention layer can implement: when token A appears earlier and the current token is B, predict C (written [A]...[B] → [C]).

Attacking superposition with dictionary learning

The opening note (Bricken, Batson, Templeton, Jermyn, Hume, Henighan, Olah) answers the question of where the team's superposition work is going. Several months of ad hoc experiments on real models had, by the authors' account, produced interesting but inconclusive results and persuaded them that solutions would not be easy. They were therefore concentrating on "Approach 2" from the Toy Models paper: extracting features from a trained, dense model by applying dictionary learning to its activations. They credit Sharkey et al., who had tried the same approach with sparse autoencoders, and describe building infrastructure for a systematic, large-scale study meant either to find superposition in real models or to count significantly against this decoding approach.

The note reports informally that sparse factorizations of activations sometimes suggest simple hypotheses on inspection, but that nothing yet persuaded the authors these factors were the model's real features "rather than a convenient lens" that might reveal some features and hide others.

Features as the simplest factorization

This note (Bricken, Batson, Templeton, Jermyn, Henighan, Olah) starts from two problems. The team admits it does not know what a feature is, and dictionary learning requires choosing the number of features without knowing whether a given number captures all features or splits real ones into parts. The proposal is to define features as "the simplest factorization of the activations".

To make "simplest" measurable, the authors define the total information of a factorization A = SD as the information needed to represent both S and D. In the initial experiments they model the entries of each matrix with a 100-bin histogram, take the surprisal of each entry under that histogram, and sum. A larger dictionary costs more information, but it can allow a sparser code, which costs less, so the two pull in opposite directions. The authors expect the exact formulation to change.

The test uses synthetic data: sparse vectors with a known number of underlying factors (n_sparse) and a known number active at once (n_active), randomly projected into 32 dimensions. The authors run dictionary learning (described as essentially MOD with a LASSO inner loop) across a range of L1 penalty strengths and dictionary sizes, and plot mean squared error against total information. They report that the solutions "bounce" when the dictionary size equals the true number of factors: as the dictionary grows, total information drops and then rises again, so the solutions with the correct number of factors sit on the best trade-off between error and information. The same solutions also score best on MMCS, a metric from Sharkey et al. for comparing a learned dictionary with the known true one. The note's figure shows this for n_sparse of 64, 128 and 256 with n_active of 1, 2 and 5. The authors' conclusion is conditional: if such bounces could be found in real data, that would be significant evidence that real features exist.

Dictionary learning worries

Henighan and Olah list two ways in which standard dictionary learning may solve a slightly different problem from the one interpretability needs:

  • Compressed sensing may be "too strong". Standard algorithms allow the sparse code to be computed by any function, typically an expensive optimization. A network, the authors argue, probably needs to read features out of superposition cheaply (a footnote explains that taking more than one layer would waste both compute and the storage superposition is meant to save). An overly powerful algorithm might therefore invent extra features, such as a single feature standing for a pair of features, that make the code sparser but mean nothing to the network.
  • The representation may not be centered at zero. Activations may be better described as A = SD + b, with a bias offset b, which standard algorithms do not handle without modification. A footnote ties this concern to a figure by Marius Hobbhahn suggesting "shifted" superposition that avoids the ReLU.

Both concerns favor sparse autoencoders, which compute the code with one cheap layer and can include a bias. Against that, the authors report that sparse autoencoders had proved more fragile and sensitive to hyperparameters, and say they were looking for a method with an autoencoder's restriction to real features and dictionary learning's reliable training. They also mention correlated features as a further difficulty not specific to dictionary learning.

Fractional dimensionality and "pressure"

Henighan and Olah summarize a comment they had added to the Toy Models paper. That paper had found features arranging themselves into regular polyhedra whose shape depended on sparsity. The comment found the geometry more sensitive to feature pressure, defined as the ratio of the number of features the model would ideally represent to the number of dimensions available, and to training time: in some regimes, more competing features and longer training produce clean geometry where it would otherwise be absent. The authors say they are confused that features the model ultimately does not represent change the solution this much, and offer two hypotheses: the tiny features inject noise, or the model somehow uses them as "epsilon features" in superposition.

The "two circle" phenomenon in memorization

A brief item points to another comment, on the paper Superposition, Memorization, and Double Descent: in problems with two hidden dimensions, data points sometimes arrange themselves on two circles of different radii. The authors believe this is likely a quirk of optimization in two dimensions but present it as a case study in the geometry of superposition and memorization.

Weight superposition

Olah asks what computation a network can perform while its features are in superposition. If two layers X and Y each hold features in superposition through embedding matrices U_X and U_Y, the "ideal weights" between the underlying features must also be mapped into superposition. The note shows that this map is a tensor product of the two embeddings, so each ideal weight between feature i and feature j is stored along the combined direction formed from feature i's embedding and feature j's embedding. Interference between two stored weights is then governed by the inner product of those combined directions, by analogy with interference between features. The note observes that weights appear able to interfere constructively, which helps, whereas interference between features always seems harmful. Olah calls this preliminary thinking that offers a tool for reasoning about which weight matrices, and so which computations, can be represented in superposition.

Attention head superposition

Jermyn, Olah and Henighan argue that something like superposition can happen across attention heads, with "attentional features" and "attention circuits" spread over several heads. They train small one-layer transformers with trivial embeddings on token sequences that are uniform except for over-represented skip-trigrams. They focus on OV-incoherent skip-trigrams: several different current tokens attend back to the same earlier token, but the right prediction depends on which current token is attending. A single head cannot implement several of these without errors, because the part of the head that writes the output (the OV circuit) does not know which token attended.

The finding is that when the training data contain more incompatible skip-trigrams than the model has heads, the model places them in superposition across heads; in the example shown, each skip-trigram is spread over at least two heads. The authors draw a methodological caution: a single attention head studied in isolation can be misleading when the circuit it belongs to is only partly interpreted. The wiring diagrams become too complex to read as the number of skip-trigrams grows, though the authors report hints of regular geometry even then.

Feature manifold toy model

Olah and Batson ask what a network does when features are not discrete but lie on a continuous manifold, such as curve orientation, which forms a circle. Vision models are known to represent such variables with families of neurons, each covering a range; Cammarata et al.'s curve detectors used about ten neurons for different orientations, rather than one neuron for the angle or two for its sine and cosine.

The toy problem is the ReLU-output model from Toy Models of Superposition, with many features evenly spaced around a circle. Each data point picks a random angle and magnitude and activates the features near that angle, with "near" set by a length scale ℓ. The note's figure embeds the features in three dimensions for ℓ = 1.0, 0.2 and 0.01. As the length scale shrinks and the model must tell finer angles apart, the embedding first develops kinks and then breaks into discrete clusters. The authors call this emergent discretization, say they are seeing hints of it across several problems, and suggest: "It may be that "emergent discretization" is the thing we mean when we talk about features." They add the caveat that the discretization at the smallest length scale may be an optimization failure.

Comments, publications and outside work

The rest of the update is a digest. It lists recent comments on earlier Transformer Circuits papers (including replications of Toy Models of Superposition by Spencer Becker-Kahn and Marius Hobbhahn, and a comment by Sharkey, Braun and Millidge on extracting features with sparse autoencoders), and the team's recent smaller publications, among them Superposition, Memorization, and Double Descent, Privileged Bases in the Transformer Residual Stream and the composition-and-superposition essay listed in this library as Distributed Representations: Composition & Superposition.

The section on research by other groups highlights, among other work:

  • Linear representations. The exchange between Li et al. and Nanda on Othello-playing models is described as perhaps the strongest evidence yet, "from a Popperian perspective", for the hypothesis that features are directions, because the hypothesis made a prediction that at first conflicted with the evidence and was then confirmed. Turner et al.'s steering of GPT-2 by adding activation vectors is cited as further evidence.
  • Automated interpretability. Bills et al.'s use of language models to explain neurons is described as a proof of concept for scaling interpretability, with a reservation the team states as a question: "Do we really want our auditing of AI models to depend on trusting an AI model to help us with auditing?" The same concern appears at greater length in Interpretability Dreams.
  • Learning dynamics and scaling. Michaud et al.'s "quanta" theory of scaling and work on mode connectivity are presented as making the team more hopeful of a bridge between circuits and large-scale training behavior. A separate section covers the reverse engineering of grokking and work on why it occurs.
  • Neuroscience parallels. Recent findings in mice and in primate visual area V4 resembling features first described in vision models are read as evidence for a strong version of universality, the idea that similar features form in different networks.

Limitations and open questions

  • Preliminary by design. Every item is labelled informal, and the update asks not to be read as a mature paper. Most results come from toy models or synthetic data.
  • Synthetic tests only for the information metric. The bounce was shown on synthetic data with a known number of factors; whether it appears in real activations was left open. The later paper Towards Monosemanticity (October 2023) reports in its section on whether the autoencoder is working that the team's total-information metric "did not generally correlate with subjective feature interpretability or activation sparsity", and its future-work section says the information-based proposal had not yet shown "compelling signs of life" on real data.
  • Autoencoder versus dictionary learning. The trade-off between the stronger dictionary-learning algorithms and the more fragile autoencoders was unresolved here. Towards Monosemanticity chose sparse autoencoders, repeating the concern that iterative methods might be "too strong", and its appendix reports that adding a learned bias was needed for its autoencoder to reproduce the "bounce plots" on toy problems.
  • Attention superposition. Towards Monosemanticity's future-work section says a clear example of extracting structure from attention layers had not yet been established, citing this update and a later one.
  • Feature manifolds. The same paper returns to the possibility of multi-dimensional "feature manifolds" in its discussion of superposition, linking back to this item.

Research questions

  1. Does the information bounce appear in real model activations? The test used synthetic data with known structure. Running the same sweep of dictionary sizes and penalties on a small language model's activations, and checking for a minimum in total information, would answer it; Towards Monosemanticity's report of no correlation with interpretability suggests the measurement choices matter.
  2. How much does the histogram choice drive the total-information measure? The 100-bin histogram was a placeholder. Repeating the synthetic experiment with other entropy estimates would show whether the bounce is robust to that choice.
  3. When does emergent discretization reflect the task rather than optimization? The authors suspect the smallest-scale clusters may be an optimization failure. Repeating the feature-manifold experiment with different optimizers, seeds and training lengths would separate the two.
  4. How many heads does a superposed attention circuit occupy as the number of skip-trigrams grows? The note shows each skip-trigram on at least two heads in a small case. Measuring the spread as a function of the ratio of incompatible skip-trigrams to heads would give a quantitative counterpart to the pressure results for features.
  5. Can constructive weight interference be observed in trained networks? The weight-superposition argument predicts that some stored weights help one another. Measuring the inner products between stored weight directions in a toy model that computes in superposition would test the prediction.

How it connects

References

  • Bricken, T., Batson, J., Templeton, A., Jermyn, A., Hume, T., Henighan, T., and Olah, C. (2023, May 24). Circuits Updates – May 2023. Transformer Circuits Thread. https://transformer-circuits.pub/2023/may-update/index.html. Different sections have different authors; section names in this note refer to this web version. Anthropic landing page: https://www.anthropic.com/research/circuits-updates-may-2023
  • Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread. https://transformer-circuits.pub/2023/monosemantic-features/index.html. Cited for its references back to this update ("How can we tell if the autoencoder is working?", "Future Work", the discussion of superposition theories, and the appendix on training sparse autoencoders).
  • Elhage, N., Hume, T., Olsson, C., et al. (2022). Toy Models of Superposition. Transformer Circuits Thread. https://transformer-circuits.pub/2022/toy_model/index.html