In March 2025 Anthropic described a method for tracing the internal steps a language model takes on a given prompt, and applied it to its own Claude 3.5 Haiku (Tracing the thoughts of a large language model). The post "Open-sourcing circuit tracing tools", published on 29 May 2025, announces that Anthropic is open-sourcing the method "so that anyone can build on our research", with tools that apply it to open-weights models. It is a short announcement of under 400 words and reports no new findings.
What was released
The method builds attribution graphs: for one prompt and one output, a graph of the interpretable features active in the model and the direct effects between them, which the post says "(partially) reveal" the steps the model took to reach the output. (The earlier note explains the method, including the transcoders that supply the features.) The release has two parts:
- an open-source library,
circuit-tracer(github.com/safety-research/circuit-tracer), that generates attribution graphs on popular open-weights models; - an interactive frontend hosted by Neuronpedia, where users can generate graphs for prompts of their own choosing and explore them in the browser.
The post lists three uses: tracing circuits by generating attribution graphs; visualizing, annotating and sharing graphs; and testing hypotheses by changing feature values and observing how outputs change. It names two models, Gemma-2-2b and Llama-3.2-1b, on which the team has used the tools to study multi-step reasoning and multilingual representations, with examples in a demo notebook. The Gemma graphs use transcoders trained as part of the GemmaScope project (closing credits). The post also publishes attribution graphs that the team has not analyzed and invites others to find circuits in them.
Who built it and why
The library was developed by Anthropic Fellows Michael Hanna and Mateusz Piotrowski, mentored by Emmanuel Ameisen and Jack Lindsey; the Neuronpedia integration was built by Decode Research (Johnny Lin and Curt Tigges). The post ties the release to an essay by Anthropic's CEO, Dario Amodei, on the urgency of interpretability, summarizing its point that understanding of models' inner workings "lags far behind" progress in capabilities, and states the aim as making it easier for the wider community to study what happens inside language models.
Limitations and open questions
The post itself states no limitations beyond the word "partially": attribution graphs reveal only part of a model's computation. The March papers, as summarized in the earlier note, report that graphs gave satisfying insight on about a quarter of the prompts tried on Claude 3.5 Haiku. The post does not report how often graphs on Gemma-2-2b or Llama-3.2-1b give such insight, or how faithful the replacement models are on these models.
Research questions
- Do the case-study mechanisms reported for Claude 3.5 Haiku appear in small open models? The post says the demo notebook examines multi-step reasoning and multilingual representations in these models, but does not compare them case by case with the Haiku findings. Running the March case-study prompts through the library on Gemma-2-2b and Llama-3.2-1b would show which mechanisms recur.
- How often do graphs on open models give interpretable answers? The March work reports about a quarter of prompts for Claude 3.5 Haiku. A comparable count for the open models would show whether the rate depends on model size or on the transcoders used.
- Does community annotation of the unanalyzed graphs agree across annotators? The post invites outside analysis. Having several researchers label the same graphs independently would measure how much interpretation depends on the analyst.
- Can behavior defined by written instructions be traced? The tools support any prompt. Comparing graphs for the same request with and without a system-prompt instruction would show where in the model such an instruction takes effect.
How it connects
- Tracing the thoughts of a large language model introduces the method this post releases and is the page the post links to; that note lists this release as a later item.
References
- Anthropic. (2025, May 29). Open-sourcing circuit tracing tools. Library by M. Hanna and M. Piotrowski (Anthropic Fellows), mentored by E. Ameisen and J. Lindsey; Neuronpedia integration by Decode Research (J. Lin, C. Tigges). https://www.anthropic.com/research/open-source-circuit-tracing
- Library: https://github.com/safety-research/circuit-tracer (linked from the post; not reviewed for this note).