A month after Scaling Monosemanticity (summarized in Mapping the Mind of a Large Language Model) extracted tens of millions of features from Claude 3 Sonnet, Anthropic published this short post about what it took, in infrastructure, to get there from the one-layer model of Towards Monosemanticity (October 2023). The post is explicitly a recruiting piece aimed at engineers. Its claim is that early interpretability papers such as the transformer circuits framework and Toy Models of Superposition "required relatively little engineering", but that the newer work made engineering "one of the major bottlenecks" to progress. It gives two worked examples and an FAQ about the team, and is accompanied by a roundtable video.
Problem 1: shuffling 100 TB of activations
Sparse autoencoders (SAEs) are trained on activations recorded from a transformer, and the post explains that these must be shuffled so the SAE does not learn spurious order-dependent patterns ("Engineering Problem 1: Distributed Shuffle"). While the training data fit on one GPU, shuffling was trivial. The first scaled-up solution split the work into $K$ jobs, each reading all of the data in a stream and writing out its $1/K$ share of a fixed permutation. Because every job read the whole dataset, this took hours and then days. By the time of Towards Monosemanticity the team had 100 TB of training data, described as 100 billion data points of 1 KB each.
The fix, which the post says extends an approach from a Jane Street blog post, uses repeated passes. In one pass, $N$ jobs each read $1/N$ of the data, shuffle it, and write $K$ files of $1/(NK)$ of the data each. Taking the first file from every job gives the first $1/K$ of the final output, which still needs shuffling internally, and likewise for the second file and the rest. One pass therefore turns one large shuffle into smaller shuffles each $K$ times smaller; if those fit in one machine's memory they are done in memory, and otherwise another pass is run. The post's worked numbers: if a job holds 100 GB in memory and writes one hundred 1 GB files, each pass reduces the shuffle size 100-fold, so one pass handles 100 GB, two passes 10 TB, three 1 PB and four 100 PB.
The post presents this as a known problem rather than a research contribution: "There are certainly better approaches and faster implementations than ours", but the approach removed the bottleneck.
Problem 2: the feature visualization pipeline
The second example is computing the data behind the team's feature browsers, which show the text on which each feature activates ("Engineering Problem 2: Feature Visualization Pipeline"). For each of millions of features the team wants examples across the feature's whole range of activation, drawn from a 100 million token dataset whose transformer activations are already cached in cloud storage. The pipeline described has these passes:
- Sharded over both data and features. Each job scans its slice of the dataset and, for its slice of features, keeps the $K$ highest-activating tokens and $10K$ random activating tokens.
- Sharded over features. Results from the first pass are merged to give each feature's top examples and a random sample across the whole dataset.
- Sharded over data again. To show how a feature fires on the tokens around each example, a job needs activations scattered at random across the dataset. Rather than have each feature job read everything, a pass over the data writes the activations each group of features needs into its own file.
- Sharded over features. A final pass computes activations on the surrounding tokens and writes the output in a format the website reads.
In the video, one participant estimates that the full pipeline had grown to 10 or 12 distributed steps (at about 43:27).
How the team describes research engineering
The post's closing section says the team treats research and engineering as inseparable and invests in infrastructure for an idea only once experiments suggest it works, since "many research ideas don’t work out" ("What we’re looking for"). It asks for generalist engineers who can move between pipelines, machine learning experiments and GPU optimization. The FAQ states that the team had 18 members and was looking for at least five senior engineers and two managers; that it worked closely with the Alignment team; that it planned research with reference to Anthropic's Responsible Scaling Policy, with day-to-day work described as exploratory; that members came from neuroscience, mathematics, biology, physics, data visualization and software engineering; and that members worked in San Francisco, Boston, New York, Seattle and London, with remote staff expected to visit an office about 25% of the year.
The video adds anecdotes that the post does not, among them a bug in evaluation metrics that made results look too good and cost weeks, a slow default matrix multiplication for the unusual matrix shapes involved, and the observation that a one-GPU open-source SAE implementation is short because most of the team's code exists to scale. The auto-generated transcript in the source extract is garbled in places, so this note paraphrases the video and does not quote it.
Limitations
The post is a hiring piece, not a research report, and makes no empirical claims beyond the engineering descriptions above. It gives no timing or cost figures for either pipeline after the fixes, and it says alternative designs may be better. The team-size and hiring figures are as of June 2024 (an April 2024 update gave 17 members; see Circuits Updates – April 2024).
Research questions
- How much did infrastructure determine which interpretability methods were tried? The roundtable describes choosing sparse autoencoders partly because they scaled. A comparison of dictionary learning methods at matched compute, as in later SAE variant studies (August 2024), would show how much of the choice rested on scalability rather than quality.
- Do order effects actually harm SAE training? The post's case for the shuffle is that unshuffled data would teach spurious order-dependent patterns. Training SAEs on partially shuffled activations and measuring loss and dead features would show how large the effect is.
- How representative are the visualized examples? The pipeline shows top examples and a random sample of $10K$ activating tokens. Comparing interpretations made from those samples with interpretations from larger samples would show whether the sample size is enough for rare or broad features.
- What does it cost to visualize every feature at larger scale? The post gives the dataset size (100 million tokens) but no compute figures. Reported costs for each pass at several dictionary sizes would let others plan similar work.
How it connects
- Within this library. The post is about the work between Towards Monosemanticity and Scaling Monosemanticity (Mapping the Mind of a Large Language Model), both of which it links, and it contrasts them with the less engineering-heavy A Mathematical Framework for Transformer Circuits and Toy Models of Superposition. The SAE training recipe that this infrastructure served is described in Circuits Updates – April 2024.
- In the course wiki. The multi-pass shuffle solves the same problem as building an inverted index for a collection larger than memory: split the data into blocks that fit, process each in memory, and combine, or distribute the work with MapReduce (Index Construction).
References
- Anthropic (2024). The engineering challenges of scaling interpretability. Anthropic research post and video, 13 June 2024. https://www.anthropic.com/research/engineering-challenges-interpretability
- Templeton, A., Conerly, T., et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html
- Bricken, T., Templeton, A., Batson, J., et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread. https://transformer-circuits.pub/2023/monosemantic-features/index.html