The October 2025 Circuits Update has two unrelated items. The first, "Visual Features Across Modalities: SVG and ASCII Art Reveal Cross-Modal Understanding", follows the line-breaking paper When Models Manipulate Manifolds (October 2025) from low-level visual properties of text to higher-level ones: does a model recognize the eyes in an ASCII face or in SVG code, and can the features involved be used to change what it draws? The second, "Data Point Initialization for Dictionary Models", is a training method: start a sparse dictionary's weights from real activations rather than random values. The first item reports qualitative examples; the second reports four before-and-after comparisons.
About the Circuits Updates format
Circuits Updates are informal posts on Anthropic's Transformer Circuits site, each a bundle of short, separately authored items. This one asks readers to treat its results "like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper" (introduction). The update is not peer reviewed and has no arXiv or other version; the web page is the version of record. The page prints no publication date: 24 October 2025 is the date the Transformer Circuits index gives it (its data-date attribute), and the Internet Archive's first capture is from 25 October 2025. It carries one later edit, dated 3 November 2025, adding a related-work reference to the second item. It is about 2,400 words long. The visual-features results are shown as images of activations over code and of generated drawings, with no numbers beyond one steering strength; the initialization results are in one bar chart with printed percentages. Locations below are item and subsection titles.
Terms used in this note
- Sparse autoencoder (SAE), feature. An SAE rewrites a model's activations as a sparse combination of learned directions, each called a feature (see Towards Monosemanticity).
- SVG, ASCII art. SVG is a text format that describes a picture as code (circles, paths and their coordinates and colors). ASCII art draws a picture with keyboard characters arranged in lines.
- Cross-modal feature. The item's term for a feature that fires on the same concept in different text formats: prose in several languages, ASCII art and SVG code.
- "Motor neuron" feature. The item's term for a feature whose strongest effect on the output logits concerns one concept, for example a "say smile" feature that fires on tokens most often followed by "smile". The item contrasts these with perceptual features, which fire on input tokens about the concept.
- Steering. Pushing the model's activations along a feature's direction during generation, with a sign and strength chosen by the experimenter. The item gives one strength value but does not describe how steering was implemented.
- L0, MSE. The average number of features active per token, and the mean squared reconstruction error; lower is better for both.
- Weakly causal crosscoder (WCC). A dictionary whose features read one layer and reconstruct that layer and later ones, introduced in Sparse Crosscoders.
Visual features across modalities
Julius Tarng, Purvi Goel and Isaac Kauvar, edited by Joshua Batson and Adam Jermyn, report features that respond to the parts of text-based drawings. The models are Claude 3.5 Haiku and a Sonnet 4.5 base model; the introduction says such features exist in models "ranging from Haiku 3.5 to Sonnet 4.5", in SAEs trained on a middle layer. All evidence is activations shown on examples and generated drawings; the item reports no counts of features examined, no success rates and no repeated runs.
An "eyes" feature on ASCII, SVG and prose
The team had Claude generate ASCII and SVG smiley faces, removed every comment or label that could identify the parts or the whole as a face, and examined Claude 3.5 Haiku's feature activations ("Feature representations of visual depictions"). One feature, which the authors describe as representing "eyes across languages and descriptions", fired on the eye shapes in both the ASCII and the SVG face and on prose describing eyes in several languages. The item finds the same pattern for other parts (mouths, ears) and for whole depictions (dogs, cats).
Context determines whether the feature fires
The item reports that the "eye" feature depends on what comes before ("Feature representations of visual depictions"):
- An "@" alone does not activate it unless preceded by lines that establish ASCII art; in the example shown, a single underscore, a forehead of slashes and two "@" characters were enough.
- In SVG, the feature fires on a circle only after another circle establishing the face.
- Moving the eyes above the face's context removed the activation even when the full face (with yellow fill) was otherwise present.
- The activation was sensitive to the character counts of ASCII lines, the color of SVG circles, and the width and height of the enclosing SVG element. The item lists these as sensitivities without measuring them.
A dog in SVG, and pareidolia
On a model-generated SVG dog (the item calls it "LLM-generated"), a Sonnet 4.5 base model's SAE had features for many body parts, several of which also fired on the ASCII face, along with motor features such as "say smile", which fired on the ASCII face before the smile ("Feature representations of visual depictions"). The caption notes that features overlap across the code "as the model's confidence on a shape's meaning varies", but that each feature's strongest activations are on the right parts.
The features were robust to surface changes such as color and radius. Reordering the four lines that define the dog's eyes reduced activation only for the circle moved to the top of the SVG, which the authors read as the model lacking the context to treat it as an eye. Once a body (here, an ellipse for the torso) had been drawn, later shapes were read as parts of an animal; the authors call this "an LLM version of face pareidolia". A caption adds that the dog's ears and nose also activated the eye feature, "as potential eyes".
A "mouth and lips" feature fired most on the most mouth-like path after the eyes. It fired briefly on the path forming the tail, but stopped once the path's coordinates began, which the authors read as the model telling the tail from the mouth by its attributes. When the mouth and tail paths were moved away from the eyes, the feature fired most on the dog's red collar, throughout its definition and on to the bell. The authors leave open whether the collar's shape (a rounded red rectangle) or its position in code and space explains this.
A hand-drawn dog and a pelican on a bicycle
A dog drawn by a person, not a model, gave similar features for eyes, mouth, legs and head ("Feature representations of visual depictions"). Order mattered here too: the drawing originally defined the nose before the eyes and no nose feature fired; moving the nose after the eyes made one appear. On an SVG of a pelican riding a bicycle, a test of drawing ability popularized by Simon Willison, features for bike, wheels, feet, tail, eyes and mouth fired on the matching code, with a "soft/fluffy texture" feature on the wings and weak "bird" motor features.
Steering what the model draws
To test whether these features shape generation as well as perception, the team prompted the Sonnet 4.5 base model to "Make a simple SVG with a similar style to this one", followed by the code for a yellow smiley face with no labels ("Visual steering with semantic features"). The prompt is printed in full. Without steering, the model reproduced the face with small changes, the smile sitting lower. The steering features were found by looking for activations on plain-text sentences about concepts such as "smile", "eye" or "cat".
- Smile to frown. Steering negatively on the "say smile" motor feature produced a frowning face; the same feature also turned the earlier ASCII smiley into a frown, as the text and figure show (the item's introduction states the ASCII case the other way round, "turning ASCII frowns to smiles"). A figure labels the frown at strength -7.5 and shows a straight-mouthed neutral face about halfway between it and the unsteered smile, which the authors describe as a "smooth gradient". In that figure the steered faces are also light blue rather than yellow, a change the text does not mention.
- Adding elements. Steering positively on "say cat" added ears, whiskers and a muzzle; a "say wrinkles" feature added wrinkles. A summary figure also shows unicorn, eye, lion and owl variants. The outputs kept the example's style, bold colors and circular shapes.
Strengths are not given for any result except the frown, and the item does not say how many samples were drawn per setting or how outputs were chosen for display.
The authors' conclusions and questions
The item concludes that "many features activating for plain text descriptions of concepts also activate for, and can generate, text-based visual depictions of those concepts" ("Conclusion"). It reports one comparison between feature types as an observation: motor features "proved more effective for steering" than perceptual ones, without numbers. The authors list open questions: whether there are features for more abstract properties such as aesthetics or artistic style; what makes a feature useful for steering and how to find such features automatically; how visual output could be combined with other tools such as contrastive vectors; and what mechanisms turn low-level text into these features, including "to what extent does the model have an internal representation where it pictures, plans, and then draws an image".
Data point initialization for dictionary models
Nikhil Bhargava, Wes Gurnee, David Abrahams and T. Ben Thompson, edited by Adam Jermyn, report that how a dictionary's weights are initialized affects the quality of the trained dictionary, and introduce a simple change ("Data Point Initialization for Dictionary Models"). They cite prior evidence that SAE initialization affects which features are learned (Paulo and Belrose, 2025) and that constraining dictionaries to the convex hull of the data helped in computer vision (Fel et al., 2025). The 3 November edit, crediting Lee Sharkey for the pointer, adds an earlier LessWrong post on data-point initialization by Pierre Peigné (posted 23 September 2023, per LessWrong's record; not read for this note), and says this item extends it "by demonstrating the efficacy of this approach in real models".
The method
An SAE encodes an activation $x$ of dimension $d_{\text{model}}$ into $n_{\text{features}}$ latents with an encoder matrix and a nonlinearity, and decodes back with a decoder matrix: $\hat{x} = W_{\text{dec}} \cdot \text{nonlinearity}(W_{\text{enc}} x)$ ("Method"). Data point initialization (DPI) sets the starting weights as follows:
- Sample $n_{\text{features}}$ data points from the activation distribution being modeled.
- Subtract their mean $\mu$.
- Blend each centered point with a random vector $r$ from Kaiming initialization, $w = s(x - \mu) + (1 - s)r$, where the mixing parameter $s$ (the item's datapoint_scale) lies between 0 (pure random) and 1 (pure data).
- Stack the vectors $w$ to form $W_{\text{enc}}$, and set $W_{\text{dec}} = W_{\text{enc}}^T$.
The authors' intuition is that activations are not isotropic, so starting in their high-density region may give an early gain in sparsity and reconstruction; they add: "More importantly, this works empirically." Variations they tried (sampling in tranches, copying initialization blocks) did not consistently beat the base method.
Results
The dictionaries were trained on "a pretrain-only" version of Claude Haiku 4.5, with datapoint_scale 0.8 for SAEs and 0.4 for WCCs; the item says results "are consistent on other subject models" but shows none ("Results"). The text states that for the largest SAE (524k features) DPI gave "a ~17% reduction in L0 and a ~4% improvement in MSE", and for WCCs about 1% lower L0 and about 2.3% lower MSE. The bar chart prints the change for each of four runs:
| Dictionary (training iterations) | L0 change | MSE change |
|---|---|---|
| SAE, 262k features (12k) | -17.3% | -6.2% |
| SAE, 524k features (30k) | -16.7% | -4.2% |
| WCC, 262k features (9k) | -0.6% | -2.3% |
| WCC, 262k features (15k) | -1.3% | -2.3% |
The text's "~17%" and "~4%" match the 524k row; the 262k SAE's MSE gain of 6.2% is larger than the text's headline. Read from the chart's axes, the SAEs ran at an L0 of roughly 90 to 130 and the WCCs at roughly 225 to 270. The text says DPI helps "across model sizes"; the chart varies dictionary size on one subject model. The item does not say how many runs each bar represents; it reports no seeds, intervals or sweeps over datapoint_scale, and does not say whether the compared runs were matched in everything but initialization, or whether L0 and MSE were measured at the end of the stated iterations.
The prompts used
The only printed prompt is the steering task's request, "Make a simple SVG with a similar style to this one", with the SVG code of the face ("Visual steering with semantic features"). The item's point about it is that the code carries no labels: whatever the model reads as a face, it reads from structure. No variation of the prompt is reported, so the item does not show how the steering results depend on its wording. The item has no written principles or training documents.
Limitations and open questions
- Examples, not rates. The visual-features item shows selected examples on a handful of drawings; how many features or drawings were examined, and how often the patterns held, is not reported.
- Correlational perception results. Context dependence and pareidolia are shown by activations under edits to the input code; steering is the only intervention on the model's internals. The steered features shown are motor ("say") features; the item reports that perceptual features steered less effectively but shows no such attempts.
- Steering details. One strength is given; sample counts, selection of displayed outputs and failure cases are not.
- Two different models. The first smiley-face results use Claude 3.5 Haiku; the model-generated dog (whose figure also shows the ASCII face) and the steering results use a Sonnet 4.5 base model, not the deployed assistant. The model used for the hand-drawn dog and the pelican is not stated.
- No reported repeats for DPI. The initialization comparisons report no seeds or repeats, on one subject model, with gains for WCCs of about 1% to 2%, small enough that run-to-run variation could matter.
- Interpretability of DPI features. The initialization item measures sparsity and reconstruction only, not whether the resulting features are more interpretable or differ in kind.
Research questions
- How often are concept features cross-modal? Sampling features that fire on prose about concrete objects and measuring what fraction also fire on matching ASCII or SVG parts would turn the examples into a rate.
- Which context cues switch the eye feature on? Varying each listed cue (line lengths, circle color, SVG size) on a fixed face and recording activation would show which matter and by how much.
- Do perceptual features steer drawings at all? Steering perceptual and motor features for the same concepts at matched strengths, with outputs graded blind, would measure the difference the item reports without numbers.
- Is the steering result robust across prompts and samples? Repeating the smile-to-frown steering with several reference drawings and many samples per strength would give the share of outputs that change as described.
- Does DPI improve interpretability as well as L0 and MSE? Scoring features from DPI and randomly initialized dictionaries with the same automated evaluations, such as those of Circuits Updates – August 2024, would answer it.
- How large is run-to-run variation for the WCC gains? Several seeds of each WCC configuration with and without DPI would show whether 1% to 2% changes exceed noise.
- How does the best datapoint_scale depend on dictionary type? A sweep over the scale for SAEs (0.8 in the item) and WCCs (0.4) would show whether these values are near optimal.
How it connects
- Within this library. The visual-features item opens by referring to When Models Manipulate Manifolds (October 2025), on how Claude 3.5 Haiku perceives line lengths in text. Feature steering as a way to change outputs was measured on social-bias features in Evaluating feature steering and used on Claude 3 Sonnet in Scaling Monosemanticity; multilingual and multimodal features were reported there too. The initialization item tests on the weakly causal crosscoders of Sparse Crosscoders (October 2024), which the item cites. Earlier dictionary-training changes appear in Circuits Updates – April 2024 and Circuits Updates – January 2025. The previous month's Circuits Updates – September 2025 reports features shared across languages; linking the two is this note's.
- In the course wiki. The initialization item's use of Kaiming (He) initialization as the random component, and its argument from where activations are dense, can be read beside the treatment of weight initialization in Nonlinear Activation Functions.
References
- Tarng, J., Goel, P., Kauvar, I., Batson, J. (ed.), and Jermyn, A. (ed.) (2025). Visual Features Across Modalities: SVG and ASCII Art Reveal Cross-Modal Understanding. In Circuits Updates – October 2025. Transformer Circuits Thread, dated 24 October 2025 on the site index. https://transformer-circuits.pub/2025/october-update/index.html record · 2 pages
- Bhargava, N., Gurnee, W., Abrahams, D., Thompson, T. B., and Jermyn, A. (ed.) (2025). Data Point Initialization for Dictionary Models. In Circuits Updates – October 2025 (edited 3 November 2025). Transformer Circuits Thread. https://transformer-circuits.pub/2025/october-update/index.html record
- Paulo, G., and Belrose, N. (2025). Sparse Autoencoders Trained on the Same Data Learn Different Features. arXiv:2501.16615. (Cited by the item; not read for this note.) record
- Fel, T., et al. (2025). Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models. arXiv:2502.12892. (Cited by the item; not read for this note.) record
- Peigné, P. (2023, September 23). Taking features out of superposition with sparse autoencoders more quickly with informed initialization. LessWrong. https://www.lesswrong.com/posts/YJpMgi7HJuHwXTkjk/taking-features-out-of-superposition-with-sparse (Cited by the item's 3 November edit, which gives the surname only; author and date from LessWrong's record; not read for this note.) record