Most researchers are trained in mature fields, where experiments test one well-defined hypothesis against another using agreed measurements. This essay, published on the Transformer Circuits Thread in March 2024 (the Thread's index gives only the month; Anthropic's page for the essay gives 8 March 2024), argues that those habits can mislead in interpretability, which it describes as a field without an established paradigm, where even whether a model should be thought of in terms of "features" is open to debate. Its authors present it as "opinionated thoughts" and offer heuristics for research taste in qualitative work. It reports no experiments. Its argument is that qualitative results should be treated "as first class citizens" alongside quantitative ones, and it gives a criterion for when to trust them. Because the essay is short, this note follows its sections closely and merges the remaining sections.
The argument
Early sciences are qualitative
The opening section observes that young sciences usually begin with qualitative results and become quantitative over time. Its examples are the discovery of cells, which only decades later became quantitative tools such as white blood cell counts, and spectral lines, which became quantitative when Bohr connected them to electron orbitals. The authors state that they are not opposing quantitative work; the two are often complementary, with qualitative work helping to confirm that the right quantities are being measured, and the line between them can be blurry.
Summary statistics and their dangers
The section "Summary Statistics and Their Dangers" uses Anscombe's quartet, four very different datasets with the same means, standard deviations and correlation, to argue that reducing rich data to a single number hides most of what is happening. In a mature field, standard measures are well understood; in interpretability they are not.
The essay criticizes a pattern it sees in some interpretability and adjacent papers: defining a metric said to track a property of interest and then measuring that metric very rigorously. It compares this to Feynman's "Cargo Cult Science": the work looks rigorous, with error bars and many plots, but the weak point is whether the metric reliably tracks the property, not how carefully it is measured. The authors give an example from their own team: in work on tanh-regularization in dictionary learning, reported in the February 2024 Circuits Update, summary statistics at first looked promising, and only later qualitative inspection of the features showed that the team had been led astray.
The section gives three reasons summary statistics dominate in mature fields and can mislead in young ones:
- Constrained hypothesis spaces. A mature field can design an experiment to separate two well-defined theories, such as general relativity and Chern-Simons gravity, and one number can do that. In a pre-paradigmatic field the right hypotheses are not yet known; a nineteenth-century physicist explaining the Sun's energy would not have considered nuclear physics. Single numbers do not give enough signal to find one's way in a large space of possibilities.
- Interfaces. Equations, line plots and terminology are interfaces for thinking about data, and the most reusable ones need a scalar statistic. Working with interpretability's volume of data without reducing it requires custom, often interactive visualizations, which the authors compare to the custom glassware of early chemistry.
- Defensibility. Presenting data in standard formats signals that one is at least doing science. In a young field that culture can push researchers toward numbers that do not mean what they seem to.
The signal of structure
The section "Rigor and The Signal of Structure" asks how qualitative work can be rigorous without statistical significance. It notes that some of the most important discoveries in science, such as cells under the microscope, stellar spectra and superconductivity, were qualitative and needed no error bars. Its proposed criterion is defined as follows: "The signal of structure is any structure in one's qualitative observations which cannot be an artifact of measurement or have come from another source, but instead must reflect some kind of structure in the object of inquiry, even if we don't understand it." The essay calls it an informal, "unsupervised" counterpart to statistical significance: instead of testing a pre-specified hypothesis against a null, one observes an unpredicted, high-dimensional pattern and rejects noise or artifact as an explanation because the pattern is far too complex and organized to arise that way.
The examples given are Hooke's drawings of cells in Micrographia, DeepDream images (too complex to be anything but a reflection of structure inside the network), and the weights between curve detectors in InceptionV1. The authors argue that such results remain convincing even if cherry-picked, and that interpretability, unlike mature fields, still has much "low hanging fruit" of this kind. An aside notes that the same signal appears in quantitative work, for instance in the clean straight lines of scaling laws on log-log plots, which point to a new abstraction.
The criterion has a stated condition: the structure must not come from another source. Saliency maps for image models can look highly structured, but much of that structure comes from the input image rather than the network, so the signal of structure cannot vouch for them. Wherever several objects interact, such as a network applied to an input, each source must be considered before invoking the signal.
Good and bad qualitative work
The section "The Success and Failure of Qualitative Research" admits that qualitative research makes it very easy to fool oneself, which it suggests is one reason scientists distrust it. It lists two signs of good work, of which it wants at least one: the signal of structure, and a principled reason for looking at what one is looking at (a magnifying glass enlarges things; a network's weights are its basic computational substrate). It lists three warning signs: looking at something unprincipled (which very clear structure can overcome), structure that is not striking, especially combined with cherry-picking, and other possible sources for the structure, such as the input image in a saliency map.
Additional reading
The essay closes with a reading list: The Structure of Scientific Revolutions (on competing early schools and on how choosing what to treat as an abstraction enables quantitative work: "Once you recognize cells, you can count them"), Drawing Theories Apart (chapter 1, on "paper tools"), the talk Media for Thinking the Unthinkable (on interfaces in scientific thinking), Lakatos's Proofs and Refutations (on the interplay of examples and definitions, which the authors liken to that of qualitative investigation and summary statistics), and Feynman's Cargo Cult Science.
Limitations and open questions
- An opinion piece. The essay is presented as opinionated heuristics, not as a result, and gives no procedure for deciding when structure is striking enough.
- One worked example from the team's own work. The tanh-regularization case is described in one sentence; the details are in the February 2024 update, not in the essay.
- Judging "too complex to be chance". The signal of structure rests on a judgment that a pattern is far past what noise or artifact could produce. The essay does not offer a way to quantify that judgment, and its saliency-map example shows that striking structure can come from an unexpected source.
Research questions
- How often do interpretability metrics disagree with qualitative inspection? The tanh-regularization case is one instance. Collecting cases in which a dictionary-learning or feature-quality metric improved while human inspection judged features worse would show how common the failure is.
- Can the signal of structure be made checkable? One test is to compare observed structure with what the same method produces on randomly initialized networks or shuffled inputs. Towards Monosemanticity used a related control, training an autoencoder on a randomly initialized model; systematizing such controls would give the criterion an operational form.
- Which visual interfaces most change interpretability conclusions? The essay ties the field's dependence on visualization to the limits of scalar statistics. Comparing conclusions drawn by researchers with summary plots versus custom feature browsers on the same data would measure the effect.
- When do qualitative findings later become quantitative measures in interpretability? The essay predicts that metrics grow out of qualitative work. Tracing which of the field's current metrics began as qualitative observations would test the prediction for this field.
How it connects
- The essay cites no other item in this library by name. Its one example from Anthropic's own work, tanh-regularization in dictionary learning, links to the February 2024 Circuits Update, which is not in the collection's list.
- For the dictionary-learning method that example concerns, see Towards Monosemanticity, which describes in its own words why the team found no single satisfactory metric for judging an autoencoder and relied partly on manual inspection ("How can we tell if the autoencoder is working?").
References
- Olah, C., and Jermyn, A. (2024). Reflections on Qualitative Research. Transformer Circuits Thread. https://transformer-circuits.pub/2024/qualitative-essay/index.html. Section names in this note refer to this web version. Anthropic landing page (source of the 8 March 2024 date): https://www.anthropic.com/research/transformer-circuits
- Bricken, T., Templeton, A., Batson, J., et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread. https://transformer-circuits.pub/2023/monosemantic-features/index.html. Cited for its randomly initialized model control ("Do features tell us about the model or the data?").