The September 2025 Circuits Update holds a single item, "Features & In-context learning", by Adam Jermyn and Wes Gurnee, edited by Joshua Batson. It returns to a measurement from On the Biology of a Large Language Model (March 2025): how many of the features active on a passage are also active on its translation into another language. Revisiting that measure, the authors noticed that the overlap grows as the passages get longer, and the item asks why. Two explanations are on the table: an artifact of the measure, or a model that builds a fuller representation across languages as it reads further into a text. The item reports two comparisons, each shown only as a histogram, and the authors conclude in favor of the second.
About the Circuits Updates format
Circuits Updates are informal posts on Anthropic's Transformer Circuits site, each a bundle of short, separately authored items. This one asks readers to treat its results "like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper" (introduction). The update is not peer reviewed and has no arXiv or other version; the web page, about 550 words, is the version of record. The page prints no date: 29 September 2025 is the date the Transformer Circuits index gives it (its data-date attribute), and the Internet Archive's first capture is from 2 October 2025. Its two results appear only as histograms, which print counts on the vertical axis but no summary statistics. Locations below refer to the item's title, since it has no numbered sections or figures.
Terms used in this note
- Feature. A direction in a model's activations learned by a sparse dictionary (a sparse autoencoder or transcoder), which the team interprets as representing a concept; see Towards Monosemanticity.
- Intersection over union (IoU). For two passages, the number of features active on both divided by the number active on either. An IoU of 1 means the same features fire on both; 0 means none are shared. "Sample length" is the length of the passage over which active features are collected.
The finding being revisited
The item summarizes three findings it attributes to the Biology paper's multilingual section: larger models represent different languages more similarly than smaller ones; the similarity peaks near the middle layers; and closely related languages are represented more similarly than distant ones ("Features & In-context learning"). In the Biology paper these came from per-layer IoU between paragraphs and their Claude-generated French and Chinese translations, compared with unrelated paragraphs as a baseline, with each paragraph's features counted if they were active "anywhere in the context" (Biology paper, "How General are Multilingual Features?"; see that note's "Multilingual circuits" section).
The new observation is that this IoU "increases with increasing sample length". The item shows no figure or numbers for this trend itself; it is stated in the text. Because the Biology measure pools features over a whole passage, a dependence on length bears directly on it; that connection is this note's.
Two explanations
The item sets out two readings of the trend:
- An artifact of what is counted. The IoU might be dominated by spurious activations (weak firing from low confidence, or imperfections in the dictionaries) or by activations unrelated to meaning, such as position-in-text features that "would be present regardless of any overlap in meaning".
- A representation built over the context. The model may build a fuller representation across languages in longer contexts. The authors call this "almost certainly true in the first few tokens", but note that it would have to continue over longer spans to explain the trend.
The authors say that on priors they suspected the second, because many of the original findings matched their intuitions, and ran the two checks below "just to be sure".
First check: first and last sentences of paired paragraphs
The team took paired English and French paragraphs and computed the IoU of the first sentence of the English paragraph with the first sentence of the French one, and likewise for the last sentences. Their reasoning: sentence length does not vary systematically across a paragraph, so under the artifact reading the two IoUs should be similar; under the representation reading, the last-sentence IoU should be higher.
The item plots a histogram of the difference, last-sentence IoU minus first-sentence IoU (titled "English-French"), and describes the distribution as "skewed strongly to the right", meaning the last sentence usually has the higher IoU. The histogram gives no mean or median. Read from it: the full range is about -0.11 to 0.12, the peak lies near 0.02 to 0.05, and by this note's count of the bar heights about three quarters of the differences are positive. "Skewed" here means shifted: the bulk lies right of zero, while the longer tail runs to the left. The authors take this as consistent with the representation reading and contrary to the artifact reading, and "think this is evidence in favor of the theory that the model just has a richer understanding later in the context".
Second check: unrelated sentences as a baseline
As a baseline, the team compared unrelated samples: first sentences of English paragraphs with first sentences of unrelated French paragraphs, and the same for last sentences. Under the artifact reading the two IoUs should again be similar. Under the representation reading, the item says, unrelated first sentences should overlap more than unrelated last sentences. It does not spell out why; a reading consistent with its argument, and this note's, is that a richer representation later in the context is more specific to each text's meaning.
The second histogram (titled "English-French: Unrelated samples!") skews the other way: unrelated first sentences overlap more than unrelated last ones. Read from the figure, about three quarters of the differences lie between about -0.04 and 0.01, and the full range, about -0.07 to 0.07, is narrower than in the paired case. The authors call this "again consistent" with the model developing a richer understanding.
What the item does not report
- The model, layer and dictionary. The item does not say which model the features come from, at which layer IoU was computed, or which dictionary was used. The Biology paper's measurement covered Claude 3.5 Haiku and a smaller 18-layer model at every layer; whether this item repeats that setup is not stated.
- Sample counts and statistics. The number of paragraph pairs is not stated. By this note's count of the histogram bars, each histogram holds about 1,000 samples, but that is our reading, not a reported number. No mean, median, test or interval is given.
- Other languages. Only English and French are shown, although the original measurement also covered Chinese, and the Biology paper reported the clearest gain from the larger model for pairs that do not share an alphabet.
- How paragraphs were chosen and translated. The item says nothing about the source of the paired paragraphs, their length, or how "first" and "last" sentences were segmented, nor whether they are the Biology paper's Claude-translated paragraphs.
Limitations and open questions
-
What it revises. The item leaves the Biology paper's three findings standing; the authors conclude that the possible confound is not the main driver. This note's inference, not stated by the item: comparisons of IoU between languages or models are fair only at matched positions and lengths.
-
Two coarse checks. Both compare only the first and last sentence: they show a difference between the beginning and end of a paragraph, not how IoU grows along it, and do not rule out differences in content between opening and closing sentences, which the item does not discuss.
- No interventions. The evidence is correlational: active-feature overlap, with no ablation or steering to test whether the shared later-context features carry meaning.
- Preliminary status. The format presents the work as a lab-meeting result, and no later paper in this collection is known to extend it.
Research questions
- How does cross-language IoU grow with position? IoU computed sentence by sentence along long paired documents would show whether overlap keeps rising or saturates, and over how many tokens.
- Does the effect depend on model size? The same comparison on models of several sizes would show whether the growth over the context is larger in larger models, as the original cross-language sharing was.
- Does the effect hold for pairs that share no script? The comparison for English and Chinese would test whether later context helps most where surface overlap is least.
- How much of the IoU comes from position or low-confidence features? Recomputing IoU after removing position-correlated features, or thresholding out weak activations, would test the artifact reading directly.
- Do sentence-type differences explain part of the result? First and last sentences matched by length and type would separate position in the context from the kind of sentence.
- Are the shared later-context features causally used? Ablating features that become shared only late in the context, and measuring the effect on translation or cross-lingual question answering, would show whether they carry meaning.
How it connects
- Within this library. The measurement revisited here comes from On the Biology of a Large Language Model (March 2025), summarized for a general audience in Tracing the thoughts of a large language model. Multilingual features in Claude 3 Sonnet's sparse autoencoders were reported in Scaling Monosemanticity. The next month's update, Circuits Updates – October 2025, reports features shared across another kind of format, ASCII and SVG drawings; the connection between the two is this note's.
- In the course wiki. IoU is the Jaccard coefficient applied to sets of active features, the same set-overlap measure used to compare documents by their shared shingles in Near-Duplicate Detection.
References
- Jermyn, A., Gurnee, W., and Batson, J. (ed.) (2025). Features & In-context learning. In Circuits Updates – September 2025. Transformer Circuits Thread, dated 29 September 2025 on the site index. https://transformer-circuits.pub/2025/september-update/index.html record · 2 pages
- Lindsey, J., Gurnee, W., Ameisen, E., et al. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread, 27 March 2025. https://transformer-circuits.pub/2025/attribution-graphs/biology.html record · 5 pages