Thread: Circuits

The Circuits thread on Distill (March 2020 to April 2021), a collection of nine short articles by a group based at OpenAI that tried to reverse engineer the vision model InceptionV1 neuron by neuron; Anthropic's Transformer Circuits Thread lists it as the "Original Distill Circuits Thread" and says its work on transformers builds on it.

Cammarata et al. (OpenAI, Mount Royal University, Stanford University) · 2020-03-10 · original

Before Anthropic existed, a group of researchers based at OpenAI asked what could be learned "if we invest heavily in reverse engineering a single neural network" (the thread's subtitle). Their subject was InceptionV1, an image classifier with, by the thread's count, "around 10,000 unique neurons", few enough that a group could try to understand each one and the connections between them. The work appeared on Distill, an online machine-learning journal, as the Circuits thread: a hub page opened on 10 March 2020 and nine short articles published between March 2020 and April 2021. The Transformer Circuits Thread (transformer-circuits.pub), where Anthropic publishes its interpretability research, lists the hub as its oldest entry, under the heading "Original Distill Circuits Thread" and the dates "March 2020 - April 2021", with the description "Our exploration of Transformers builds heavily on the original Circuits thread on Distill." This note describes the thread and lists its articles. It does not summarize each article in depth; several later notes in this collection draw on individual articles, and they are listed at the end.

The thread format

A thread was Distill's name for a format it was trying out: "collections of short articles, experiments, and critical commentary around a narrow or unusual research topic", meant to be "earlier stage than a full Distill paper" (hub page, "About the Thread Format"). The hub page calls threads "very much an experiment" and says Distill planned "to trial two such threads and then re-evaluate". Circuits, in the thread's usage, are features (what a neuron or direction responds to) together with the weights that connect them and compute new features; Zoom In sets out the program in these terms.

Who wrote it and when

The hub page lists nine authors with their affiliations at the time: Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert and Chelsea Voss (OpenAI), Ben Egan (Mount Royal University) and Swee Kiat Lim (Stanford University). Its suggested citation gives the thread as Cammarata et al. (2020), with DOI 10.23915/distill.00024. The page says the author order is "all participants in the thread in alphabetical order" and that the citation "may add additional authors as it evolves"; the two authors from outside OpenAI, whose only article in the thread is Visualizing Weights, are listed after the seven alphabetized names. Chris Olah, listed here at OpenAI, is the author or a co-author of many later Anthropic papers and essays in this collection, among them Interpretability Dreams.

The hub page says the thread was organized through an open channel, #circuits, on the Distill Slack, that articles could be suggested there and "will be included at the discretion of previous authors in the thread, or in the case of disagreement by an uninvolved editor", and that its content "should be seen as early stage exploratory research" (sections "Articles & Comments" and "Get Involved").

The articles

The hub page lists nine articles, "presented below in chronological order". Dates and DOIs below are from each article's page; the one-line descriptions paraphrase the hub page's own summaries.

  1. Zoom In: An Introduction to Circuits (Olah, Cammarata, Schubert, Goh, Petrov and Carter; 10 March 2020; DOI 10.23915/distill.00024.001). An essay proposing three claims which, if true, would justify studying individual neurons and their connections seriously: that meaningful features exist, that meaningful circuits connect them, and that features and circuits are universal across models. It also discusses how other sciences advanced by "zooming in" and whether qualitative research of this kind can be rigorous.
  2. An Overview of Early Vision in InceptionV1 (same six authors; 1 April 2020; 10.23915/distill.00024.002). A survey of every neuron in the first five layers, sorted into a taxonomy of "neuron groups", meant to set up later articles.
  3. Curve Detectors (Cammarata, Goh, Carter, Schubert, Petrov and Olah; 17 June 2020; 10.23915/distill.00024.003). The first of a series of three articles on one family of neurons, which the hub page says is found in every vision model the authors had explored in detail.
  4. Naturally Occurring Equivariance in Neural Networks (Olah, Cammarata, Voss, Schubert and Goh; 8 December 2020; 10.23915/distill.00024.004). Networks learn many transformed copies of one feature (rotated, for example), connected by symmetric weights.
  5. High-Low Frequency Detectors (Schubert, Voss, Cammarata, Goh and Olah; 27 January 2021; 10.23915/distill.00024.005). A family of early-vision neurons that respond to directional transitions from high to low spatial frequency.
  6. Curve Circuits (Cammarata, Goh, Carter, Voss, Schubert and Olah; 30 January 2021; 10.23915/distill.00024.006). The authors reverse engineer the algorithm behind the curve detectors from the weights and use it to build, by hand, a small network that reimplements it.
  7. Visualizing Weights (Voss, Cammarata, Goh, Petrov, Schubert, Egan, Lim and Olah; 4 February 2021; 10.23915/distill.00024.007). Techniques for visualizing and putting into context the weights between neurons.
  8. Branch Specialization (Voss, Goh, Cammarata, Petrov, Schubert and Olah; 5 April 2021; 10.23915/distill.00024.008). When a layer is split into separate branches, neurons organize themselves into coherent groups by branch.
  9. Weight Banding (Petrov, Voss, Schubert, Cammarata, Goh and Olah; 8 April 2021; 10.23915/distill.00024.009). The weights in the final layer of common vision models appear as horizontal bands; the article investigates how and why.

The hub page's heading is "Articles & Comments", and it invites "critical commentary by experts in adjacent fields", but the list on the page as read for this note (13 November 2026) contains articles only. Distill's archive page lists the same ten pages under 2020/circuits/ (the hub and nine articles).

Whether it is still maintained

The hub page still calls itself "a living document" and says to "Expect more articles on this topic, along with critical comments from experts." No article has been added since Weight Banding (8 April 2021). On 2 July 2021 Distill announced a "one year hiatus, which may be extended indefinitely", citing editorial burnout and doubts about running Distill as a journal; the announcement said that "published threads can continue to add to their exploration" ("Distill Hiatus"). The Transformer Circuits Thread's "About" section gives the hiatus as one reason Anthropic publishes its own site: "Previously we would have submitted to Distill, but with Distill on Hiatus, we're taking a page from David Ha’s approach of simply creating websites". The thread is best read as complete at nine articles, with its hub page left as it stood.

Limitations of the thread as a source

  • One model, mostly qualitative. Almost all of the evidence concerns one vision model, and much of it is presented through feature visualizations and inspected weights. Zoom In itself raises the question of whether such qualitative research can be rigorous. Later notes in this collection report how the team's methods changed when it turned to language models, where neurons proved harder to read (see Toy Models of Superposition).
  • Early-stage by its own description, without the planned commentary. The hub page asks readers to treat the content as exploratory research, and the critical comments from outside experts that the format called for do not appear on it.

Research questions

  1. How far did the thread's three claims carry over to language models? Zoom In proposed features, circuits and universality as working hypotheses. Comparing the evidence for each in the thread with what the Transformer Circuits papers later report for transformers would show which claims transferred and which needed new methods.
  2. Did the curve-detector account survive later tools? Curve Circuits hand-built a network from the authors' reading of the weights. Applying a later automated method (for example a sparse dictionary or attribution method) to the same InceptionV1 layers and comparing what it recovers would test the original account.
  3. Did the thread format work as intended? The editors planned to re-evaluate threads after two trials. Distill's own later statements, and the citation and participation record of the Circuits thread, would show whether the format produced the outside commentary and wider participation it was meant to.
  4. Which practices moved from the thread to the Transformer Circuits Thread? Comparing the two sites' formats and stated publishing aims would show what Anthropic kept and what it changed.

How it connects

References