The Circuits Updates are informal posts on the Transformer Circuits Thread in which Anthropic's interpretability team shares work in progress. The July 2023 update, the second in the series after Circuits Updates – May 2023, holds three short items. The first returns to the toy model of Superposition, Memorization, and Double Descent (January 2023) and reports that the regime the team could not explain there was an artifact of poor optimization. The second, a conceptual piece with no experiments, lists the features and behaviors the authors would most like interpretability to find for safety purposes. The third revisits the May 2023 example of attention head superposition and describes an algorithm that makes the authors less sure the example deserves that name. As in every update, the team asks readers to treat the results "like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper" (introduction). The update is about 1,900 words with three figures; it has no numbered sections, tables or overall byline, so locations in this note are the item titles, and the authors are those listed under each item. The page prints no date; the date given here, 18 July 2023, is the one the Transformer Circuits index attaches to the entry in its markup (a data-date attribute), the rule this collection uses for Circuits Updates, and the earliest Internet Archive capture (11 September 2023) is consistent with that.
Terms used in this note
- Feature and superposition. As in Toy Models of Superposition: a feature is a property of the input that a network represents, and superposition is representing more features than there are dimensions by storing them in overlapping directions.
- Memorizing and generalizing solutions. In the toy model of the January 2023 paper, a network trained on few examples stores each training example as if it were a feature (memorization), while one trained on many examples stores the underlying features of the data-generating process (generalization).
- First-order phase transition. The update's term for a point at which the best solution switches abruptly from one kind to another, so that the minimum training loss has a kink (a discontinuous derivative) where the two solutions' losses cross.
- Skip-trigram and OV-incoherence. A skip-trigram is a pattern an attention layer can implement: when token A appears earlier and the current token is B, predict C (written [A]...[B] -> [C]). A set of skip-trigrams is OV-incoherent when they share the earlier token but need different predictions; one head cannot implement them all, because the part of the head that writes its output (the OV circuit) sees only the token attended to.
Finite data intermediate regime
Adam Jermyn, Chris Olah and Tom Henighan report further work on the one-layer toy model of Superposition, Memorization, and Double Descent. In the update's summary of that paper, the model memorized datasets of up to about 1,000 samples, learned generalizing features above about 10,000 samples, and did something the team could not determine in between. (The paper's text gives no such cut-offs; a replication in its Comments section put the onset of generalization at 20,000 samples against the paper's 5,000.) The intermediate regime worried the team because it was not clear it could be understood as linear features in superposition, which the item calls "our present framework for understanding neural networks".
What was found. The item reports that the intermediate regime "is in fact an artifact produced by optimization failure". By training models with a variety of learning rates, the authors found that a memorizing solution reaches the lowest training loss up to very large datasets, about 500,000 samples, beyond which a generalizing solution does better. They describe this as a "true first-order phase transition" around 500,000 samples. A footnote attributes the optimization failure to the toy model's low hidden dimension and says the authors do not expect such failures to be "nearly so prevalent in more realistic models". The update does not state the hidden dimension; its plots are two-dimensional, consistent with the two hidden dimensions the January paper mostly used (this note's inference).
The first figure plots one minus the training loss against dataset size for the two kinds of solution, with a panel showing each solution's geometry: a disk-like cloud with a ragged rim for the clustered memorizing solution and a five-armed star for the generalizing one. The figure has no legend, and its curves are matched to the solutions here by the colors of the regime labels (this note's reading). Read from the figure, which prints no values and appears to use logarithmic axes: the generalizing curve declines slowly from about 0.0016 to about 0.0014 across the plotted range, while the memorizing curve starts near 0.0038 at about 160,000 samples, falls unevenly (its points jump up and down from one dataset size to the next), crosses the generalizing curve at about 530,000 samples, where the figure draws its regime boundary, and lies between about 0.001 and 0.0014 beyond it. (Higher values mean lower loss, so the color matching agrees with the text: the memorizing solution has the lower loss below the crossover.) The item reports no number of seeds, no spread across runs and no test loss, so the double descent of the January paper is not re-examined. That paper had already asked whether its double descent was "an artifact of only having m=2 hidden dimensions, or difficulties with optimization" and answered no, from four models per setting at larger m (section "The Effect of m on Double Descent"). The July item's artifact claim concerns what the model represents in the intermediate regime, not the test-loss bump; the update does not relate the two.
Clustered memorization. The second finding concerns how memorizing solutions change as the dataset grows from about 1,000 to about 500,000 samples. Below about 1,000 samples, the item says, samples are mutually orthogonal with high probability, so the model can memorize each one exactly. Above that, correlations between samples make exact memorization impossible, and the model instead groups data points into clusters that share features. The second figure labels three regimes ("Basic Memorization", about 1 to 1,000 samples; "Cluster Memorization", about 1,000 to 500,000; "Generalization", about 500,000 and up) and zooms in on the rim of a clustered solution. Each cluster turns out to be a small "triangle" of lines, which the figure's annotation says the authors "don't yet understand". Coloring data points by sets of features whose embeddings have cosine similarity above 0.9999 and magnitudes within a factor of two shows that the clusters correspond to sets of correlated features.
The authors' overall reading is that the investigation "reassured" them that the model "can be fully understood as linear features in superposition". That is their interpretation of a toy model; the item shows the train-loss comparison and the geometry, not a decomposition that accounts for the whole model.
What would be the most safety-relevant features in language models?
Chris Olah and Adam Jermyn offer a list of targets for mechanistic investigation, motivated by the hope that interpretability "may ultimately contribute to the safety of AI systems". The item reports no experiments. It notes that lying and deception had been widely proposed as targets, and cites a "User and System model" framework (Viégas and Wattenberg, 2023) as a broader proposal. The targets fall into two classes.
Hypothesized mechanisms, framed as "units of computation" the authors speculate exist, which they call features. These can be properties of the input, intermediate knowledge, or actions:
- High-level actions. Some neurons near the end of a model perform simple actions such as completing multi-token words (citing the Softmax Linear Units paper); the item asks whether there are also abstract actions that steer behavior over many tokens. A figure gives what its caption calls "a speculative conceptual model" of how features might be arranged through a transformer's depth: low-level "detokenization" features, more abstract features, highly abstract concepts or planning, high-level actions and low-level "retokenization" actions, each marked with a question mark.
- Planning: how actions are coordinated, and whether the model represents goals, reasons instrumentally or searches.
- Social reasoning: what the model represents of people's mental and emotional states, with social actions and social planning named as the concerning forms.
- Personas: how a model tracks and switches the personas learned in pretraining; the item worries that if RLHF tries to "lock" a model into one persona, the persona-change logic might still be triggered.
- Situational awareness: whether the model knows it is a model, how it was trained, and that it can be turned off or modified.
Empirical behaviors that the authors would like explained mechanistically:
- Jailbreaks: why they work, and whether interpretability can find new ones (citing Carter et al., 2019, and Goh et al., 2021, for adversarial examples found that way).
- Instrumental reasoning: caginess around red-teaming prompts, sycophancy and self-preservation "may suggest" it, though "much more benign causes" are imaginable.
- In-context learning: which circuits implement it, beyond induction heads.
- Successful RLHF: whether RLHF elicits what is already present or creates new circuits, and how much it removes.
The list is a statement of research priorities; the item marks the mechanisms as hypotheses.
Further investigation of the skip-trigram model
Tom Conerly, Adam Jermyn and Chris Olah return to the May 2023 example of attention head superposition (see Circuits Updates – May 2023), in which a one-layer model trained on more OV-incoherent skip-trigrams than it had heads spread each skip-trigram over two or more heads. At the time the team did not understand the algorithm. Studying a simpler two-head model, the authors now report that "two attention heads suffice" to model a large number of OV-incoherent skip-trigrams with zero loss. In the item's example, with skip-trigrams such as [0]...[1] -> [4] and [0]...[2] -> [5]:
- Head 0 attends from the current token to itself and raises the logit of the right completion as if the current token were the second token of a skip-trigram (from [1] it predicts [4]; from [2] it predicts [5]).
- Head 1 attends to the shared first token [0] when it is present and does nothing; when [0] is absent, it attends to the current token and writes the negative of head 0's logits.
Together the heads implement a conditional: if [0] is present, head 0's prediction stands; if not, the two heads cancel and the output is uniform. The item adds that trained models do not split cleanly into one head per role; they divide the skip-trigrams between the heads, each taking a subset. The item reports no loss curves, model counts or figures for this result.
The item's conclusion revises the May example: the authors write that they "no longer feel certain that the example we found is best described as “attention head superposition”, at least in the strict sense of “features as directions in activation space”." The algorithm is a division of labor between heads with a cancellation mechanism, which need not involve features stored as overlapping directions.
Limitations and open questions
- Preliminary by design. All three items are short and informal; the safety-features item reports no measurements.
- Toy model, one criterion. The memorization result comes from a low-dimensional toy model and compares solutions by training loss, with no seeds, variance or test loss, and no check of the crossover at other hidden dimensions or sparsities. The "triangle" structure is not understood, and "can be fully understood as linear features in superposition" is the authors' reading, not an accounting of the model's loss by features.
- A revised example. The skip-trigram item withdraws confidence from the May 2023 example, the team's published toy example of attention head superposition. Towards Monosemanticity (October 2023, "Future Work", under "Attentional Superposition?") later wrote that "a clear example has not yet been established (e.g., see our May and July Updates)". Outside the team, Greenspan and Wynroe (LessWrong, 29 August 2023) answered both updates with a two-head toy model of what they call "OV-coherent" superposition, in which what a head should copy depends on tokens it does not attend to; they describe the July item as arguing for "building of high-level features out of low-level ones", wording the item does not use. The question stayed open in later notes: Progress on Attention (April 2025) reports evidence its authors read as attention superposition in an 18-layer model, and HeadVis (May 2026) lists the lack of a confirmed real example among its obstacles.
Research questions
- Does the memorization-to-generalization crossover stay near 500,000 samples when the hidden dimension, sparsity or number of features changes? The item reports one crossover in one toy setting. Repeating the learning-rate sweep across hidden dimensions and sparsities, with several seeds per point, would show how the crossover scales and how noisy the minimum train loss is.
- Where does the January 2023 paper's test-loss bump fall once optimization is improved? The paper argued that its double descent was not an optimization artifact; the item compares training losses only and moves the regime boundary to about 500,000 samples. Recomputing test loss for the best-optimized solutions across dataset sizes would show whether the bump moves with the crossover.
- What produces the "triangle" structure inside memorized clusters? The authors say they do not understand it. Relating each triangle's vertices to the shared and unshared features of the clustered samples would test a feature-level account.
- How often do trained models find the two-head cancellation algorithm, and how do they share skip-trigrams between heads? The item describes the algorithm and says trained models split the skip-trigrams between heads. Training many seeds and measuring how the skip-trigrams divide, and whether any head approximates the pure roles, would quantify it.
- Which of the listed safety-relevant features have since been found, and with what evidence? The list names hypothesized mechanisms and behaviors. Tabulating later feature and circuit studies against each item, with whether the evidence was correlational or interventional, would show which remain open.
- When is a spread of computation across heads better described as superposition than as a division of labor? The skip-trigram item turns on that distinction. A criterion that could be tested on toy models, such as whether the relevant directions overlap in activation space, would let the May and July examples be classified.
How it connects
- Superposition, Memorization, and Double Descent is the paper the first item follows up. Towards Monosemanticity (October 2023) cites this item in its related work as the monthly update that "found a sharp phase transition, as well as dataset clustering in superposition on the memorization side of the boundary".
- Circuits Updates – May 2023 has the attention head superposition example that the third item revisits; later work on attention is in Progress on Attention and HeadVis.
- The safety-features list is cited by Towards Monosemanticity ("speculation about safety-relevant features", in its section on what makes a good decomposition) and by Scaling Monosemanticity (May 2024), whose section on safety-relevant features opens by pointing back to "our previous discussion" before reporting features for unsafe code, bias, sycophancy, deception and power-seeking; see also Mapping the Mind of a Large Language Model.
- This note connects several other list items to later notes in the collection, links the update does not make: planning to the planned rhymes in On the Biology of a Large Language Model, which also studies jailbreaks; personas to The Persona Selection Model and Persona Vectors; and the question of what RLHF changes to the base-versus-chat model diffs in Sparse Crosscoders.
References
- Jermyn, A., Olah, C., Henighan, T., and Conerly, T. (2023, July 18). Circuits Updates – July 2023. Transformer Circuits Thread. https://transformer-circuits.pub/2023/july-update/index.html. Sections have separate authors: "Finite Data Intermediate Regime" (Jermyn, Olah, Henighan); "What would be the most safety-relevant features in Language Models?" (Olah, Jermyn); "Further investigation of our skip-trigram model" (Conerly, Jermyn, Olah). Date from the Transformer Circuits index. No anthropic.com landing page was found for this update. record
- Henighan, T., Carter, S., Hume, T., et al. (2023). Superposition, Memorization, and Double Descent. Transformer Circuits Thread. https://transformer-circuits.pub/2023/toy-double-descent/index.html. Cited for its regime description and the replication in its Comments section. record · 2 pages
- Bricken, T., Templeton, A., Batson, J., et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread. https://transformer-circuits.pub/2023/monosemantic-features/index.html. Cited for its references to this update ("Related Work", "Future Work", "What makes a good decomposition?"). record · 12 pages
- Templeton, A., Conerly, T., et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html. Cited for its reference to this update in "Safety-Relevant Features". record · 9 pages
- Greenspan, L., and Wynroe, K. (2023, August 29). An OV-Coherent Toy Model of Attention Head Superposition. LessWrong. https://www.lesswrong.com/posts/cqRGZisKbpSjgaJbc/an-ov-coherent-toy-model-of-attention-head-superposition-1 record