When Models Manipulate Manifolds: The Geometry of a Counting Task

A Transformer Circuits case study of how Claude 3.5 Haiku decides where to break lines in fixed-width text; the authors find character counts represented on curved low-dimensional manifolds tiled by families of sparse features, attention heads that rotate one count against another to sense the line boundary, an arrangement that makes the break decision close to linearly separable, and inserted character pairs such as "@@" that distract the counting heads and disrupt the prediction.

Wes Gurnee, Emmanuel Ameisen, Isaac Kauvar et al. (Anthropic) · 2025-10-21 · original

A language model reads text as a list of token numbers, with no picture of the page. Yet much of its training text (source code, email archives, scanned articles) wraps at a fixed width, and to predict where a line breaks the model must count characters it never sees as characters. "When Models Manipulate Manifolds: The Geometry of a Counting Task" (Gurnee, Ameisen, Kauvar, Tarng, Pearce, Olah and Batson; Transformer Circuits Thread, 21 October 2025) studies how Claude 3.5 Haiku does this. It is a single-task case study with a methodological aim: the authors argue that the same computation can be described either as many discrete features or as a few continuous geometric objects, and that the geometric description is simpler where it exists.

The version of record is the interactive web article. An archival copy was posted to arXiv on 8 January 2026 (arXiv:2601.04480v1), which calls itself "an archival version of a paper originally published on October 21, 2025 at Transformer Circuits". Only the arXiv copy numbers its sections, figures and appendices, so locations below use those numbers; the two texts agree on every number this note reports. The web page carries no changelog or update notice (checked 13 November 2026).

Terms used in this note

  • Feature and crosscoder. A feature is a direction in a model's activations, found by sparse dictionary learning, that is active on a recognizable set of inputs. All main-text features come from a 10-million-feature weakly causal crosscoder trained on Claude 3.5 Haiku, a dictionary whose features read and write across layers (§ 1.1; see Sparse Crosscoders). Feature labels are the authors' interpretations, checked mainly on synthetic data with known counts.
  • Attribution graph. For one prompt and one predicted token, a graph of which features drive which (Circuit Tracing); the paper's graphs add QK attributions, which explain where attention heads attend (Tracing Attention Computation Through Feature Interactions).
  • QK and OV circuits. An attention head's QK circuit sets where it attends; its OV circuit sets what it copies from the attended token (A Mathematical Framework for Transformer Circuits).
  • Feature manifold. A continuous family of directions, here a curve, with one intrinsic coordinate (the count) that may bend through several dimensions of the residual stream, the vector each layer reads and adds to. Following Olah, every point also has a magnitude, so the manifold includes its scalings toward the origin (Introduction, footnote 2; see Circuits Updates – July 2024).
  • Probe. A supervised model trained to read a known quantity from activations: a linear regression probe predicts the count as one number; the paper's logistic probes are 150 classifiers, one per count.
  • Ringing. The authors' signal-processing term for a similarity pattern that falls away from its peak, dips negative, and rises again further out (§ 2.5).
  • Character count, line width, characters remaining. Characters since the last newline, including the current token; the length of the previous line; and the second minus the first (§ 2, § 3).

What the paper does

Task and data. In the running "aluminum prompt", the previous lines have 49 and 47 characters and the current line, "element with atomic number 13, also called", has 42; " aluminum" would make it 51, so the correct next token is a newline, whereas " tin" after "50" fits (Introduction, Figure 2). For systematic analysis the authors took a prose corpus, removed its newlines, and reinserted one at the last word boundary at or before every k characters, for k = 15, 20, ..., 150 (§ 1.1). The corpus and the number of passages behind the probes and averages are not stated; the task-performance appendix uses 200 passages (Appendix C).

Approach. The authors used attribution graphs on single prompts to find which feature families mattered, characterized those families on the synthetic data, and then looked for a geometric description: averaging activations for each count, taking principal components (PCA), training probes, and multiplying probes through heads' QK and OV matrices to see how a head transforms one representation into another. They tested claims by ablating subspaces (setting them to zero) and by patching in the mean activation for a different count. A first attempt, probing and patching without the graphs, "turned out poorly": they did not know to separate line width from character count, did not expect line width to live only on newline tokens, and started with one-dimensional probes (§ 8).

Model. Claude 3.5 Haiku, a production model whose weights are not public. The authors also share, without analysis, transcoder attribution graphs for a line-break prompt on Gemma 2 2B and Qwen 3 4B via Neuronpedia (§ 8).

What it finds

Haiku does the task

Claude 3.5 Haiku adapts to every line width tested, predicting newlines correctly "with high probability by the third line" (§ 1.1). The supporting figure plots newline log-probability and top-1 accuracy by line number for 200 passages wrapped at k = 20, 40, ..., 140 (Appendix C, Figure 35). It prints no values; read from it, top-1 accuracy is near zero on the first line, roughly 0.6 to 0.75 on the third, and mostly 0.8 to 0.9 after that.

Character count lies on a curved manifold in six dimensions

  • A linear readout exists, but not a single direction. A linear regression probe after layer 1 predicts character count with R² = 0.985, which the authors say does not mean the count is represented along one line (§ 2).
  • Ten features tile the count. In layers one and two, ten features each respond to a range of counts, the ranges widening at larger counts (§ 2.1, Figure 5). The authors compare them to place cells and to vision models' curve detectors, and the widening to "dilation" in biological number perception.
  • The manifold. The top six principal components of the mean layer-2 residual stream for counts 1 to 150 capture 95% of their variance; in that subspace the points trace a twisting, helix-like curve, which the ten features' reconstruction follows closely, two or three features active at a time (§ 2.2, Figure 6).
  • The subspace is causal. Zero-ablating the top principal components at one early layer raises loss on tokens followed by a newline and barely elsewhere; random subspaces of the same size have no visible effect (§ 2.3, Figure 7; read from the figure, which prints no values, about 0.36 at six dimensions and 0.41 at ten). On the aluminum prompt, patching in the mean activation for counts from 20 to 50 instead of the true 42 (at three adjacent early layers, on the last two tokens) moved the newline's log-probability from about −3.4 to about −0.1 and " aluminum" from about −0.5 to about −3.2, and patching only the six-dimensional subspace matched patching the full vector (read from Figure 8). This intervention is shown on one prompt. A footnote cautions that a subspace found by PCA need not be dedicated to one variable, and argues that the design is more justified here because the count subspace is densely active.
  • Probes ring. Logistic probes after layer 1 have a root mean squared error of 5 characters, and six components capture 82% of the variance of the 150 probe vectors (§ 2.4). Each probe responds most to its own count and rebounds at a distance (Figure 9); mean activations and feature decoders show the same ringing (Figure 10).

Why the curve ripples

The authors propose that ripples are what to expect when a quantity that ideally needs many dimensions is packed into few (§ 2.5). Their evidence is from constructions, not from Haiku: the best five-dimensional approximation of 150 points on a circle, each similar only to its neighbors, rings and ripples (Figure 11; Appendix G relates this to truncated Fourier series and finds Fourier components explain at most 10% less variance than principal components, Figure 41); and a simulation of 100 points on a sphere, attracted to neighbors and repelled by the rest, settles into rippled curves, a "baseball seam" in three dimensions like shapes others reported for colors, dates and years (Figures 12 and 13, Appendix F). From the feature view, they read ringing as interference between features squeezed into too few dimensions. The Introduction calls the ringing pattern "optimal with respect to a simple physical model", and the Discussion puts it more broadly: "the counting manifolds are the result of an optimal tradeoff between capacity and resolution". This note observes that the support for optimality comes from the constructions; beyond the Fourier comparison, the paper does not test Haiku's manifold against alternative embeddings.

Boundary heads rotate one count against another

  • Line width lives on newlines. Newline tokens carry their own features for the length of the line that ended (§ 3.1; Appendix E).
  • The twist. An attention head found through the graph rotates the character-count manifold so that count i aligns with a line width slightly larger than i; it therefore attends to the previous newline when the current line is a little shorter than the last (§ 3.1, Figure 14). In the residual stream the two probe sets are most similar at i = k with maximum cosine similarity about 0.25; through this head's QK circuit, maximum similarity is about 1 and falls at i < k; through a random head there is almost no structure (Figure 15). The authors write that this lets the head detect approaching boundaries "for arbitrary line widths".
  • Several heads, several offsets. Several boundary heads, each twisted by a different offset, implement what the authors call a kind of "stereoscopic" algorithm; a footnote adds that they come in sets of about three at several layers ("so not actually “stereo”") (§ 3.2, Figure 16). In the first two principal components of a characters-remaining space (92% of its variance), head 0 varies most over 0 to 10 and 15 to 20 characters remaining, head 1 over 10 to 20, and head 2 over 5 to 15; each output is nearly one-dimensional, and their sum is an evenly spaced two-dimensional curve (Figures 17 and 18).
  • The characters-remaining subspace is causal. Ablating it raises loss on newline tokens (about 2.2 from four dimensions, read from Figure 19) and barely elsewhere; patching it on the aluminum prompt moves the prediction between newline and " aluminum", which needs ten characters (Figure 19).
  • Why extra dimensions and extra heads. The authors argue that comparing two counts needs a rotation, impossible in one dimension, and that 150 counts on one ray would force huge norms or poor resolution (§ 3.3). Because a head's output is a weighted mix of its inputs passed through a linear map, it cannot add much curvature, so several heads must each supply part. Having found one boundary head, they were able to "correctly predict that there would need to be additional heads to provide curvature in the output" (§ 8).

The break decision is nearly linearly separable

The most influential feature in the aluminum graph is a late "break predictor" whose top inputs are a "say aluminum" feature and a boundary-detecting feature driven by the boundary head (§ 4, Figures 21 and 22). Boundary detectors ignore the next word's length; break predictors fire only when it would not fit, and "break suppressors" when it barely would (Appendix I). Mean activations at about 90% depth, for characters remaining and next-token length from 1 to 15 each, place the two counts in nearly orthogonal subspaces, so that "break if the word is longer than the space left" is a separating plane (§ 4.1, Figure 23). On real data that plane separates tokens followed by a newline from others with an AUC of 0.91 (1 is perfect separation, 0.5 chance). The labels use the true next token, so the figure combines the plane's error with Haiku's uncertainty about the next word. The authors also checked for a more general mechanism that would move probability from every overlong word to the newline and report that "Claude 3.5 Haiku does not seem to leverage such a mechanism".

Many heads build the count

The count is built mainly by attention in the first two layers; the authors set MLPs aside because attention outputs affect the representation four times more, without saying how this was measured (§ 5.2). The embedding already places token lengths 1 to 14 on a circle with an oscillating third component (three components, 70% of variance; § 5.1, Figure 25). Each layer-0 counting head attends to the previous newline as a "sink" for a set number of tokens, then spreads attention over recent tokens; its OV circuit converts this into roughly tokens so far times average token length, corrected for whether the attended tokens are long or short (§ 5.2, Figures 28 and 29; Appendix K walks through head L0H1). Five key layer-0 heads predict the count with R² = 0.93, and eleven heads in the first two layers with R² = 0.97 (§ 5). Layer-0 heads each write along a nearly straight ray and their sum curves; layer-1 heads use that estimate to sharpen it (Figure 24, Appendix L). Line width appears to be computed by a similar, partly separate set of heads, but how it is aggregated over several previous lines is, in the authors' words, "one subtlety that we do not address in this work" (§ 5.3).

Visual illusions

Some counting heads, instead of attending from a newline to the previous newline, attend to "@@", a git-diff delimiter after which restarting a line count would make sense (§ 6). Replacing two words in the aluminum prompt with "@@", line lengths unchanged, dropped the newline's probability from 0.79 to 0.22 and raised " aluminum" from 0.12 to 0.66, with the head now also attending to the "@@" (Figures 31 and 32). Of 180 inserted two-character sequences, most lowered the newline probability moderately but left it on top; a few, mostly code or delimiter pairs (">>", "}}", ";|", "||", two backticks, a backtick and comma, "@@"), disrupted it substantially, and disruption was correlated with how much a pair distracted the counting heads (Figure 33, a scatter plot without a statistic). In an explicit question about which of two sentences is shorter, replacing two letters of the longer one with "@@" (length unchanged) flipped the answer from (a) at 0.80 to (b) at 0.78 (Figure 34); the authors call this "robust to multiple choice orderings" without numbers. They write that "we are not claiming any direct analogy between illusions of human visual perception and this alteration of line character count estimates, the parallels are suggestive", and offer the misapplication of a prior learned from git diffs as one possible reading.

Other findings in the appendices

Dictionaries of different sizes learn character-count features with very similar ranges, which the authors read as a "canonical" tiling of this manifold, although some features split by line width (Appendix D). Similar feature families and ringing appear for how many characters were left when a line broke early and for row and column positions in Markdown tables (Appendix N).

The authors' conclusions

The Discussion (§ 8) argues that tasks natural to pretraining and done well by the model may have crisper mechanisms; that it is "perhaps more evocative" to think of early layers as perception than as "detokenizing"; that geometry made several steps clearer, while the final decision has complexity they "were not able to see yet"; and that discrete features impose a complexity tax by fragmenting a computation that a manifold describes more simply. The search for manifolds, guided by features, was "an existence proof more than a general recipe", and they call for unsupervised methods that find such structure.

The prompts and data involved

The paper has no written principles or system prompts. Its inputs are prose rewrapped at fixed widths, hand-built prompts (the aluminum and tin pair, the Gettysburg Address at 40 characters), 180 inserted character pairs, and one length-comparison question in Human/Assistant format (§ 1.1, § 6).

Limitations and open questions

  • One model, one task, closed weights. All mechanistic results are for Claude 3.5 Haiku; the open-model graphs are shared without analysis (§ 8).
  • Scope the authors leave open. Line-width aggregation (§ 5.3); uncertainty in the count, variable-width documents, several plausible next tokens of different lengths, multi-token words, and special cases such as LaTeX footnotes or Markdown links (§ 8).
  • MLPs set aside in the counting analysis (§ 5.2).
  • Supervised geometry. The manifolds were found with known counts; the authors call the approach "expensive in researcher time and potentially limited in scope" (Introduction).
  • Measurement detail. Error bars in Figures 7 and 19 are not explained, no repeated runs are reported, the number of tokens behind the averages is not given, and the fine-grained interventions are on one prompt.
  • Partial account of the decision. The 0.91 AUC is for a three-dimensional plane fitted to averages (§ 4.1).
  • Analogies. Place cells, boundary cells and human illusions are offered as suggestive parallels, not tested claims (§ 6, § 7).
  • A polysemantic head. HeadVis (Luger, Kamath et al., May 2026) reports that one line-width head studied here shows three distinct behaviors in its attention patterns on dataset examples: attending between the two tokens of years 1000 to 1999, between pieces of multi-token words, and from newline to newline, the behavior studied here (HeadVis, "A Polysemantic Attention Head").
  • Outside reproduction. Sinii and Balagansky (T-Tech, 19 February 2026) repeated the behavioral, probe, PCA and sparse-autoencoder analyses on open models, with 1,000 web documents wrapped at a single width of 150 characters. Several families predicted wrapped newlines well and GPT-2 did not; probe R² peaked at 0.72 to 0.87 for five of seven models (0.49 for Gemma-3-4B, 0.36 for GPT-2), at layers 3 to 18; the first three principal components held most of the variance; and sparse-autoencoder features showed hill-shaped tuning curves. The reproduction contrasts its layers with "Anthropic’s original result, where peaks appeared in very early layers", but the paper reports one probe after layer 1, not a sweep over layers, and its widths of 15 to 150 differ from the single width, so the R² values are not directly comparable. The reproduction is a post without peer review; its work is correlational and does not test the attention mechanism.

Research questions

  1. Do open models use boundary heads that twist one count against another? The mechanism is shown only in Claude 3.5 Haiku, and the open-model reproduction stops at probes and PCA. Multiplying line-width and character-count probes through early heads' QK circuits in an open model, and patching the resulting subspace, would show whether it is general.
  2. How does the model combine several previous line widths? The authors leave open a maximum or a moving average (§ 5.3). Prompts whose earlier lines vary in controlled ways, with the next line's newline probability measured, would distinguish them.
  3. How much of the break decision does the plane explain once next-word uncertainty is included? Scoring it against the model's own predicted next token, and on multi-token words, would separate the plane's error from Haiku's.
  4. Can such manifolds be found without labels? Applying an unsupervised manifold-finding method to the same crosscoder and checking whether it recovers the count curve would test the authors' call for methodology.
  5. Is canonical tiling specific to bounded counts? The authors suggest the start of the line anchors it (Appendix D). Comparing dictionary sizes on a circular variable such as months would show whether finer splitting appears there.
  6. Do the illusions matter on real documents? Measuring newline accuracy on wrapped prose that naturally contains delimiter pairs, and on git diffs where restarting the count is correct, would show their practical effect.
  7. Do later, larger models count the same way? Repeating the probe, twist and ablation analyses on a later Claude model would show whether the geometry persists with scale and training.

How it connects

References