Alongside its first paper, A Mathematical Framework for Transformer Circuits, Anthropic's interpretability team released recordings of the informal talks in which it had worked out much of that paper. The Transformer Circuit Videos page is a short note on the Transformer Circuits Thread that introduces them and links a YouTube playlist. In the page's words, the team "recorded a couple videos discussing our early stage thinking on trying to reverse engineer neural networks" and made them "to share our very informal thoughts with colleagues at other institutions". This note describes the page and the playlist, says which papers link individual videos, and records the caveats the page attaches to them. It does not summarize the talks; it is based on the page, the playlist's titles, durations and descriptions, and the papers that link them, not on watching the recordings.
The page and its caveats
The page is about 140 words long, has no byline and no date, and consists of four short paragraphs. It asks readers to "treat these videos like talks one might give on early results at a research group meeting", in bold, and continues: "Our thinking is very rough and errors are very possible. Please take all of these videos with a big grain of salt." It expects them to be "primarily of interest to people actively thinking about how to reverse engineer neural networks". It adds that the team's thinking "evolved a lot" after the recordings began, and that "The first couple of these videos have since been superseded by our more polished paper", linking the framework paper.
The page's text has not changed since the earliest Internet Archive capture, of 22 December 2021; the only difference between that capture and the page fetched on 13 November 2026 is the address of a script it loads.
Date and who made them
The Transformer Circuits Thread's index lists the page under "December 2021", tagged "links, videos", with the description "Very rough informal talks as we search for a way to reverse engineering transformers"; the index's page source dates it 2021-12-17, which this note uses. On YouTube, the playlist is titled "Transformer Circuits [rough early thoughts]" and belongs to a channel named "Mechanistic Interpretability". YouTube gives 24 December 2021 (Pacific time) as the date of the first 17 videos and 26 June 2022 for the last two, and says the playlist was last updated on 26 June 2022. The YouTube dates are later than the page's first capture, which already linked the playlist; this note does not resolve the difference. Neither the page nor the video descriptions name the speakers.
The playlist
The playlist holds 19 videos. Titles and durations are as YouTube lists them (read 13 November 2026); all but the last two carry the suffix "[rough early thoughts]", omitted here.
- 0L - Theory (3:02)
- 1L Attention - Theory (18:03)
- 1L Attention - Results (16:03)
- 1L Attention - Eigenvalue Analysis (11:38)
- 2L Attention - Theory (9:21)
- 2L Attention - Term Importance (8:58)
- 2L Attention - Results (11:53)
- Virtual Attention heads (9:14)
- Attention - General - Theory, Info-Weighted Patterns, Attribution Patterns (31:37)
- Attention - General - Summarizing with NMF (36:43)
- Attention - General - Indirect & n-gram Attention Heads (28:30)
- Attention - General - Copying & Induction heads (30:56)
- "Induction Bump" Phase Change (46:19)
- Metalearning & Induction Heads (45:28)
- MLP Neurons - Privileged vs Non-Privileged Basis (17:22)
- MLP Neurons - 40L Preliminary Investigation (55:55)
- Infrastructure - Garcon (49:02)
- Softmax Linear Unit (softmax(x)*x) Neurons - Preliminary Results (1:45:46)
- Softmax Linear Unit #2: Update on Preliminary Results (26:29)
The durations sum to 9 hours 22 minutes (this note's arithmetic). Eight video descriptions (numbers 1 to 7 and 15) begin "This video has been superseded by this paper" and link the framework paper; the Garcon video links the Garcon article. The first SoLU video's description says that switching the MLP activation to softmax(x)*x "seems to greatly help" with understanding MLP layers, and gives a chapter list running from one-layer models to phrase neurons and motifs. The 40-layer MLP video's chapters name neurons it discusses, among them short-phrase, "install", copied-text and alphabetically-sorted-list neurons, and end with a chapter titled "Challenges". The second SoLU video's description reads only "TODO".
Where the papers link them
- A Mathematical Framework for Transformer Circuits (December 2021) opens its sections on zero-layer and one-layer attention-only transformers with "Watch videos covering similar content to this section", and its two-layer section with "Videos covering similar content to this section", linking videos 1, 2, 3, 5, 6 and 7. A "Summary of Follow-Up Research" added to the paper later, which describes the state of research "as of February 2023", says "We have a video on some rare interpretable neurons we've found in MLP layers", linking video 16.
- Softmax Linear Units (June 2022) links video 18 as an "earlier video discussing our preliminary findings with SoLU", for further qualitative investigation of neurons (Section 6.3), and video 16 as an "informal video" (Section 7.1) showing that the team, too, had found individual MLP neurons hard to interpret before SoLU. The paper says it mentions this "because negative results are often not formally represented in the literature".
In-context Learning and Induction Heads (March 2022) does not link the playlist, although videos 12 to 14 concern induction heads and the "induction bump" in training loss that the paper studies. This connection is the note's.
Maintenance and limitations
No videos have been added since June 2022, and the page has not changed since December 2021. The talks are early and informal by the page's own description, and eight are marked superseded by the framework paper. The remaining talks on attention, induction heads, MLP neurons and SoLU preview work later published as papers, but neither the page nor the descriptions say which claims changed, and this note has not compared the talks with the papers. Where a talk and a later paper differ, the paper sets the claim.
Research questions
- Which early claims changed before publication? Comparing the talks on induction heads and the "induction bump" (videos 12 to 14) with In-context Learning and Induction Heads would show which claims the paper kept, qualified or dropped.
- What did the 40-layer neuron survey find that the papers did not report? Video 16 is cited as evidence that MLP neurons were hard to interpret. A list of the neurons it describes, checked against Softmax Linear Units, would show how much of the negative result entered the literature.
- How was the Garcon talk reflected in the written Garcon article? Comparing video 17 with the article would show what the written description omits.
- How were the videos used? View counts, citations and course syllabi that link the playlist would show whether informal recorded talks reached the audience the page names.
How it connects
- The paper that superseded the first talks. A Mathematical Framework for Transformer Circuits (December 2021) links six videos by section and, in a follow-up summary added later, a seventh on MLP neurons.
- Later papers. Softmax Linear Units (June 2022) links two videos. In-context Learning and Induction Heads (March 2022) covers the subject of videos 12 to 14 without linking them.
- Other December 2021 resources. The same index section lists Transformer Circuit Exercises, PySvelte and Garcon, the subject of video 17. For the earlier vision-model work the site builds on, see the original Distill Circuits thread.
References
- Anthropic. (2021). Transformer Circuit Videos. Transformer Circuits Thread. https://transformer-circuits.pub/2021/videos/index.html. No byline or date on the page; date from the site index. Fetched 13 November 2026 and compared with the Internet Archive capture of 22 December 2021. record
- Transformer Circuits [rough early thoughts]. YouTube playlist, channel "Mechanistic Interpretability". https://www.youtube.com/playlist?list=PLoyGOS2WIonajhAVqKUgEMNmeq3nEeM51. Titles, durations, dates and descriptions read 13 November 2026. record
- Elhage, N., Nanda, N., Olsson, C., et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread, 22 December 2021. https://transformer-circuits.pub/2021/framework/index.html record · 8 pages
- Elhage, N., Hume, T., Olsson, C., Nanda, N., et al. (2022). Softmax Linear Units. Transformer Circuits Thread. https://transformer-circuits.pub/2022/solu/index.html. Sections 6.3 ("Qualitative Exploration of SoLU Models") and 7.1 ("Understanding Transformer MLPs"). record · 3 pages
- Transformer Circuits Thread, index page, entry "Videos". https://transformer-circuits.pub/ (read 13 November 2026). record