Circuits Updates – June 2026

A one-item Anthropic interpretability update from the Anthropic Fellows Program, summarizing a paper on turn-averaged sparse autoencoders, which are trained on the mean activation over a whole conversation turn so that one turn yields a fixed number of features. In Qwen-2.5-7B-Instruct, a Claude judge preferred their feature lists for describing a turn but found them less useful than per-token features for picking the turn out of ten, and they make attribution graphs over multi-turn conversations much smaller at some cost in edge accuracy.

Kevin Der (Anthropic Fellows Program), Harish Kamath and Ben Thompson (Anthropic) · 2026-06-30 · original

The June 2026 Circuits Update holds a single item, "Anthropic Fellows Program: Turn-Averaged Sparse Autoencoders", by Kevin Der, Harish Kamath and Ben Thompson, edited by Nick Turner. It addresses a practical problem in using sparse autoencoders (SAEs) to audit model transcripts: an SAE describes each token separately, so even a short transcript yields thousands of feature activations, many of them about syntax or word choice rather than the behavior under study. The item proposes training an SAE on the average activation over a whole conversation turn instead, so that each turn is described by a fixed number of features, and reports that such features describe a turn's overall character better by one measure and identify it worse by another. The item is a short summary of a 27-page arXiv paper by the same authors, which also introduces a hybrid "nested" SAE and adapts attribution graphs to work turn by turn. Every experiment uses an open-weight model, Qwen-2.5-7B-Instruct, not a Claude model.

About this update and its version of record

Circuits Updates are informal posts on Anthropic's Transformer Circuits site. This one asks readers to treat its results "like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper" (introduction). The item itself is about 650 words with one figure. Its numbers come from, and its "full paper" link points to, Der, Kamath and Thompson, "Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution", arXiv:2606.28548v1 (26 June 2026), in which Der is listed with the Anthropic Fellows Program and Kamath and Thompson with Anthropic. Only version 1 exists. The paper is the version of record for every number in this note and is cited by section, table, figure and appendix; the update is cited for its framing.

Date. The page prints no date. The site's index lists it under "June 2026", and the index's link to it carries the date 30 June 2026 in its markup (a data-date attribute, read on 13 November 2026); that is the date given here, four days after the paper's arXiv posting. An archived copy of the index from 9 June 2026 does not list the update. The live page fetched for this note has the same digest as the Wayback Machine's capture of 16 July 2026; earlier captures (7 and 8 July) have different digests and could not be retrieved for comparison.

Terms used in this note

  • Sparse autoencoder (SAE). A network trained to rewrite a model's internal activation as a sparse combination of learned directions ("features"), each of which is then given a text description, usually by a language model reading the feature's top-activating examples ("automated interpretability").
  • Per-token SAE. The standard kind, which encodes the activation at each token position separately.
  • Turn-averaged SAE. An SAE trained on the mean of the residual-stream activations over every token of one Human or Assistant turn (paper, Section 3.1). The training loss is the ordinary reconstruction loss; only the input changes.
  • BatchTopK. A sparsity rule that keeps the k largest feature activations per sample, on average over a batch. All SAEs here use k = 128 active features out of 32,768.
  • nMSE. Normalized mean squared error of a reconstruction: the squared error divided by the squared size of the activation (Section 3.3). Lower is better.
  • Attribution graph. A graph whose nodes are active features and whose edges estimate, to first order, how much one feature's activation contributes to another's or to the output (the method of Circuit Tracing).

The method

Averaging over a turn. For a turn of n tokens, a per-token SAE produces up to n times 128 feature activations; a turn-averaged SAE produces 128 (update; paper, Section 3.1). The update says the idea was motivated by prior work in which averaged activations identified abstract properties, citing persona vectors. The paper gives the intuition that topic and behavioral properties are spread across many positions while token-specific detail varies from token to token, so that the mean acts as a low-pass filter that keeps the first and smooths away the second (Section 3.1, "Intuition"). This is offered as an explanation, not tested directly.

Setup. The SAEs are trained on Qwen-2.5-7B-Instruct using the LMSYS-Chat-1M dataset of real chatbot conversations. A turn-averaged SAE took about 1.5 hours on four H100 GPUs for one epoch of about 1.58 million turn averages, against 266 million tokens for a per-token SAE on the same data (Section 3.1, "Setup"). The SAEs used for feature comparisons are trained on layer 19, which the paper gives as 70% of the model's depth, on assistant turns only (Appendix A). The update describes the layer as "the middle layer".

Hybrid architectures. Because averaging discards per-token detail, the paper also tries to learn both kinds of feature in one model (Section 3.2). A decoupled model trains two separate SAEs, one on the turn mean and one on each token's difference from it, with 64 active features each; the paper notes that the two halves are causally disconnected, so attribution cannot pass between them. A nested model trains a single SAE whose first 16,384 features must reconstruct the turn mean on their own while all 32,768 features reconstruct each token, a loss modeled on Matryoshka SAEs; the weight on the turn-mean term was swept over five values and set to 1.0.

Reconstruction. On the held-out set, the turn-averaged SAE reconstructs turn means with an nMSE of 0.097 and the per-token SAE reconstructs tokens with 0.162 (Table 1). Each fails on the other's input: the per-token SAE scores 0.355 on turn means, which the paper reads as the mean being "off-distribution" for it, and the turn-averaged SAE scores 1.799 on single tokens. The nested model scores 0.116 on turn means and 0.167 on tokens, close to each specialist; the decoupled model scores 0.128 and 0.212.

What the item and paper find

Hand-picked examples

The update's single example is a wrong answer: asked for the highest number below 100 that does not contain 9, the assistant answers 95. The update reports that the per-token SAE's strongest features concern numerical reasoning, while the turn-averaged SAE's strongest feature concerns incorrect answers to number puzzles. In the paper's table, the turn-averaged SAE's descriptions include "Incorrect answers to digit-exclusion number puzzles", the per-token SAE has no feature in the error category, and the nested SAE also finds one (Section 4, Example A). Feature descriptions throughout were written by Claude from each feature's top 25 activating examples (Section 4, "Setup").

These examples were chosen by hand from 316,000 held-out assistant turns "by manually browsing the results and choosing cases where the contrast between turn-averaged and per-token features was illustrative" (Section 4). In two further examples the contrast is narrower: on an enthusiastic neuroscience explanation, the per-token SAE found the style, creative-writing, figurative and neuroscience aspects but described the style without its technical-explanation side (Example B); on an automotive safety outline, it found the outline structure and word-level features but not the automotive or regulatory content (Example C).

Long documents. The update says the improvement "extrapolates to turns 150× longer than the average length of turns seen during training." The evidence is two documents from the LongBench v2 benchmark, of 26,920 and 30,023 tokens, encoded as single spans, in which the turn-averaged SAE surfaced subtopics (macroeconomics in a finance document, computer graphics in a graphics paper) that the per-token SAE missed (Section 4, Examples D and E). The paper states in its limitations that these two were selected "from a pool of around 50" (Section 7).

Discrimination and coverage, judged by Claude Sonnet 4.6

The quantitative comparison uses 2,000 held-out turns and five feature lists: the turn-averaged SAE, and the per-token and nested SAEs each pooled over the turn by summing or by taking the maximum of each feature's activations (Section 5). For each turn, Claude Sonnet 4.6 first wrote a structured summary with 24 fields in five categories (content, form, voice, function, meta), using a schema that Sonnet 4.6 had itself designed (Appendix D). Sonnet 4.6 was also the judge.

  • Discrimination. Given one configuration's feature descriptions and ten candidate texts, the judge picked the right text 95.0% of the time with per-token max-pooled features, 87.0% with per-token summed features, 73.9% with turn-averaged features, 70.6% with nested summed and 57.3% with nested max-pooled (Figure 1). The paper attributes the per-token lead to descriptions that mention specific words in the text.
  • Pairwise coverage. Shown the structured summary and two feature lists, the judge chose the more complete description. Over all ten pairings (20,000 trials, presentation order randomized), turn-averaged features won 87.9% of comparisons on average; head to head they beat per-token max-pooled features 79.7% of the time and per-token summed features 77.2% (Section 5.2; Figure 2, which prints each cell).
  • Five-way ranking. Ranking all five lists at once, the judge gave turn-averaged features the best average rank, 2.03, and ranked them first 44% of the time (Figure 3).
  • Embedding similarity. As a check that avoids a judge, each summary field and each feature description were embedded with Qwen3-Embedding-8B, and the closest descriptions' cosine similarity to each field was averaged. At three descriptions per field, turn-averaged features scored 0.663, nested summed 0.658, per-token summed 0.647, nested max-pooled 0.640 and per-token max-pooled 0.617 (Section 5.5); the order is the same at one, two, three and five descriptions (Appendix H). The paper notes that this check still depends on LLM-written feature descriptions (Section 7).

The update reports the first two results as 74% against 95% for discrimination and a 77% preference for turn-averaged features in coverage. The 95% is the per-token max-pooled configuration; the 77% matches the paper's cell for turn-averaged against per-token summed features, not against max-pooled (79.7%). The paper reports no repeated runs or confidence intervals for these evaluations.

Judge sensitivity. The paper reports that the coverage result depends on how the judge is prompted (Appendix E). With assistant prefill (the judge's reply seeded with a partial answer), Sonnet's average win rate for turn-averaged features was 49.1%, "indistinguishable from chance"; Claude Haiku 4.5 with and without prefill, and Sonnet without prefill, all favored turn-averaged features. The main text uses Sonnet without prefill, but the appendix's table gives that configuration an average win rate of 83.1% and 74.8% against per-token max-pooled features, where Section 5.2 reports 87.9% and 79.7%; the paper does not explain the difference.

The paper's own reading is that the two axes rank the methods in opposite orders because a turn is easiest to identify from its most distinctive detail, while turn averaging captures more of what a turn is about (Section 5.5). Nested SAEs scored poorly on these tests; the paper notes that the top 128 nested features on a turn were about 99% from the turn-level half, so it behaved like a turn-averaged SAE with half the capacity (Section 5.1).

Attribution graphs at the level of turns

With turn-averaged features, an attribution graph has nodes per feature per turn per layer rather than per token, which for a hypothetical ten-turn, 250-token conversation over four layers the paper puts at about 5,000 candidate nodes against about 128,000 (Section 6.1). The graphs were built with the open-source circuit-tracer library on SAEs at layers 6, 11, 16 and 21, trained on both user and assistant turns (Appendix A), because SAEs trained on assistant turns alone reconstructed user turns 1.5 to 3 times worse (Appendix G).

  • Which system-prompt paragraph matters. Claude wrote a "contrived" 12-paragraph system prompt, each paragraph a different instruction, and eight user queries each meant to trigger one paragraph (Section 6.2; Appendix F). For every query and all three SAE types, the share of the output's attribution coming from the targeted paragraph rose relative to a neutral query, by between 0.3 and 1.9 percentage points (Table 2). The paper reports no test of whether increases of this size exceed variation between neutral queries.
  • A 14-turn conversation. In one LMSYS conversation in which the model moves from refusal to compliance to repetitive, degenerate text, pruning reduced the turn-averaged graph from 7,239 nodes and 10.4 million edges to about 1,300 nodes and 110,000 edges (Section 6.3). Tracing back by hand from the degenerate output, the authors found a chain through a feature for jailbreak prompts at layer 16 to features for AI safety disclaimers, nuclear weapons and domination in earlier turns (Figure 5). With per-token features summed over the turn, the trace led instead to features such as "chemical compound names" and "JSON", which the paper describes as "concepts with no coherent causal relationship to the degenerate output". This is one manually built subgraph.
  • Do the edges predict interventions? On 300 held-out conversations of two, four and six turns (36,000 interventions in all), the authors pushed each source feature along its decoder direction and compared the change in the target with the edge weight, for the 40 heaviest edges per graph (Section 6.4). Turn-averaged edges agreed least: Spearman correlations of 0.543, 0.457 and 0.419 at two, four and six turns, against 0.755, 0.651 and 0.613 for per-token and 0.720, 0.645 and 0.604 for nested features; Pearson correlations were 0.725, 0.639 and 0.624 for turn-averaged, 0.821, 0.778 and 0.769 for per-token and 0.850, 0.829 and 0.816 for nested (Figure 6, printed labels). The sign of the edge weight matched the sign of the observed effect on an average of 100% of edges for each SAE type, and on at least 95% in any one graph.
  • How much do features explain? Completeness and replacement scores, which measure how much of the computation passes through features rather than error terms, ranked nested above per-token above turn-averaged at every length: completeness from 0.906 to 0.915 for nested, 0.895 to 0.908 for per-token and 0.872 to 0.889 for turn-averaged; replacement from 0.658 to 0.692, 0.624 to 0.669 and 0.552 to 0.608 (Section 6.5; Figure 7). The update describes the nested result as graphs "equal to or better than" per-token graphs at measuring causal influence; on the intervention test above, nested edges had the highest Pearson correlations but slightly lower Spearman correlations than per-token edges (Figure 6).

Persona features found by contrast

An appendix uses contrasting system prompts, one asking for a trait and one against it, to find features that fire more on the trait's responses (Appendix I). For three traits from Persona Vectors (evil, hallucination, sycophancy), responses were generated by Qwen and filtered by Claude Haiku 4.5 ratings; for 240 traits, rollouts from The Assistant Axis were used. Both SAE types surfaced related features, such as a turn-averaged feature for "Dismissive, critical, and insulting interpersonal speech toward others" at the top of the evil list, alongside features the authors call unclear or tangential (Appendix I.3). The paper says it has "not attempted a broad quantitative evaluation" of which SAE type does better here.

The prompts and instructions used

Written instructions enter the paper in three places. The 12-paragraph system prompt for the attribution case study was written by Claude and is printed in full; its paragraphs include a persona name ("Your name is Atlas"), a refusal of political topics, a crisis-line instruction and a knowledge cutoff (Appendix F). The study measures where attribution comes from, not whether the instructions were followed. The contrastive persona prompts are taken from earlier work and printed per trait; for example, the positive evil prompt begins "Your responses should prioritize causing harm and suffering to people" (Appendix I.1). The judge and summary prompts are described but not printed, beyond the summary schema and an instruction to avoid quoting or paraphrasing the original text (Appendix D). The sensitivity of the coverage result to a prefill setting (Appendix E) is the paper's only test of how judge prompting affects a result.

Limitations and open questions

  • One small open model. All results are from a 7-billion-parameter model with SAEs of 32,768 features, which the authors call much smaller than a production setting (Section 7). The update's motivation refers to auditing Anthropic's own models, but no Claude model is studied.
  • Model-made evaluation. Feature descriptions, the summary schema, the summaries and the judgments are all produced by Claude models, and the automated-interpretability prompts differ between turn-averaged and per-token SAEs, so the two sets of descriptions "are not produced under identical conditions" (Section 7). No human check of the judge is reported.
  • Selected cases. The feature examples, the long documents and the 14-turn subgraph were chosen by hand (Section 7).
  • Weaker edges. Turn-averaged attribution edges predict interventions less well than per-token ones, which the paper expects from a representation that discards per-token detail (Section 6.4).
  • Uneven weighting. Averaging treats every token equally, though models concentrate information at some positions, such as the last token of a name; the paper leaves the interaction to future work (Section 2.3).
  • No variance reported. Each architecture is one trained SAE, and no seeds or intervals are given.

Research questions

  1. Do turn-averaged features help audit a frontier model? The update motivates the method by safety audits of Anthropic's models, but tests it only on Qwen-2.5-7B-Instruct. Training turn-averaged and per-token SAEs on a larger model and comparing how quickly auditors find a planted behavior with each would answer it.
  2. Would human judges agree with the coverage result? All coverage judgments are by Claude models, and one prefill setting reduced the result to chance. Having human raters repeat the pairwise comparison on a sample of turns would show whether the preference holds.
  3. How much of the coverage advantage comes from the description prompts? The two SAE types were described with different prompts. Describing both with a common format and re-running the judge would separate the effect of the features from that of their descriptions.
  4. Are the system-prompt attribution increases larger than noise? The increases were 0.3 to 1.9 percentage points over one neutral query. Repeating with many neutral queries and reporting the spread would show which increases are distinguishable from variation.
  5. Does weighting tokens unequally improve turn-level features? The paper notes that information concentrates at some positions. Training SAEs on attention-weighted or end-of-sentence averages and repeating the coverage and intervention tests would answer it.
  6. Do turn-averaged features find behaviors that per-token features miss on a benchmark with known answers? The examples are hand-picked. Running both on transcripts labeled for a behavior, such as known jailbreaks or known incorrect answers, and measuring how often a relevant feature is among the top features would give a rate rather than examples.

How it connects

References