Would This Change Your Answer? Evaluating Explanations of LLM Behavior in the Wild with Counterfactual Experiments

Karvonen et al. introduce CHIVE, an agentic pipeline that finds unexpected behaviors of open-weight models on real user prompts and explains each with prompt edits whose effect is measured by resampling. Used as a test, it finds that three activation-reading interpretability tools (activation oracles, natural-language autoencoders and sparse autoencoders) give a Claude predictor agent no gain over reading the transcript; used as training data, it teaches models to predict how edits change their own behavior, with gains on held-out settings that approach or match, but do not clearly exceed, Claude Opus 4.8 reading the transcript, and no sign of privileged self-access.

Adam Karvonen, Euan Ong, Subhash Kantamneni, Samuel Marks (Anthropic Fellows Program; Anthropic) · 2026-08-21 · original

Interpretability tools are used to say why a model did something, for instance in the alignment sections of Anthropic's system cards. But the true causes of a behavior are usually unknown, so such explanations are hard to check. This paper proposes a check: a good explanation should help predict what the model would do if the input were changed (counterfactual simulatability, after Chen et al., 2023). The authors build a pipeline that generates thousands of naturally occurring behaviors, each with prompt edits whose effects have been measured, and use it two ways: to test whether interpretability tools help an agent predict those effects, and to train models to predict their own behavior.

The paper was posted to arXiv on 17 August 2026 (2608.16747v1) and summarized on the Anthropic Alignment Science Blog on 21 August 2026, as Anthropic Fellows Program work by Adam Karvonen (Anthropic Fellows Program) with Euan Ong, Subhash Kantamneni and Samuel Marks (Anthropic). Karvonen designed and ran the experiments and wrote the paper; Marks supervised (Author Contribution Statement). This note follows the arXiv paper for numbers, and the blog post for framing; the post reuses the paper's Figures 1 to 5. A LessWrong crosspost of the same day carries a comment by Marks, and Karvonen followed on 4 September 2026 with a LessWrong post on the training results; both are cited below. Code, models and datasets are released on GitHub (adamkarvonen/chive).

Terms used in this note

  • Counterfactual claim. A statement that a specific prompt edit would change how often a behavior occurs by at least 30 percentage points. A claim is labeled true if the measured change was at least 50 points and false if at most 15 points; claims in between are not used (Section 2.2, Appendix B.4).
  • Uplift. The improvement a tool gives a predictor agent over the same agent without it.
  • Transcript-only baseline. A predictor agent (Claude Opus 4.8) that sees the transcript and the claim and nothing else.
  • Activation oracle (AO), natural-language autoencoder (NLA), sparse autoencoder (SAE). Three tools that read a model's internal activations and return text: answers to questions about activations, free-text descriptions of an activation, and lists of active features with descriptions. All three were chosen because they had helped in earlier auditing games on fine-tuned models.
  • Privileged access. Information a model has about itself that an outside model trained on the same data does not; tested by comparing a model trained to predict itself with another model trained on the same data (Binder et al., 2024).
  • Explanation. In this paper, "a behavioral claim testable by counterfactual experiments, not an explanation of the model's internal computation" (Section 1, footnote).

What the paper does

The CHIVE pipeline

CHIVE (Counterfactual Hypothesis Investigation Via Edits) has four stages (Section 2.1, Appendix B.1):

  1. Sample. Run the target model on many prompts, 30 responses each at temperature 1.0 (10 for training data). The default source is WildChat, real user conversations, with about 9% agentic transcripts added.
  2. Screen. An investigator, usually Claude Opus 4.6, reads the responses, rates how unexpected the behavior is from 1 to 5 against a rubric with few-shot examples, and keeps behaviors scoring 3 or more that occur at least 30% of the time. It fixes a yes/no question that a classifier (Claude Sonnet 4.6) then uses to measure the behavior in every batch.
  3. Investigate. An Opus agent edits the prompt, resamples, and measures the change in the behavior's rate, about 10 to 15 experiments per behavior for evaluation data and 5 to 8 for training data.
  4. Verify. A judge scores from 1 to 10 how well the experiments support the explanation; evaluation data keep investigations averaging 8 or more over five judge samples.

Each investigation yields an open-ended explanation, which the authors do not treat as ground truth, and up to two true and two false counterfactual claims, the false ones being hypotheses the investigator thought worth testing. Every evaluation label comes from measured outcomes. An investigation costs about $1 to $2 at Opus prices (Section 2.1, Appendix B.5). The screening stage "required the most manual effort" to make the behaviors interesting and varied (Appendix B.1).

Three LLM-judge filters keep claims with a concrete mechanism, a reproducible edit and a single changed factor. From 4,714 WildChat prompts, the Gemma-3-27B-IT run produced 1,408 investigations, 4,433 claims and a final evaluation set of 1,294 balanced claims; for Qwen3-8B the final set has 1,497 claims (Table 2). Targets are open-weight models run without reasoning: Gemma-3-27B-IT, Qwen3-8B and Qwen3.5-397B-A17B.

Application 1: does reading activations help predict?

The predictor agent gets a transcript and one claim and outputs a probability that the claim is true; tool predictors can also make up to 5 read-only calls on the target model (Section 3.2). Intervening on inputs or activations is not allowed, since that would amount to running the experiment that produced the label. The tools are an activation oracle released with Activation Oracles, the NLA released with Natural Language Autoencoders (for Gemma) or a third-party NLA (for Qwen), and public SAEs with auto-interpretation labels (Appendix C.1).

Application 2: training models to predict themselves

Posed as a follow-up turn on its own transcript, the target model is asked a claim and answers Yes or No (Section 4.2). Qwen3-8B is trained on 36,824 balanced examples from investigations run by Opus; Qwen3.5-397B-A17B on 40,368 examples from investigations it ran on itself, which removes the concern that gains come from a stronger investigator (Table 7). Training uses LoRA for one epoch. Evaluation uses two hint settings never seen in training (Reddit "Am I the Asshole?" posts with the user's stated opinion, and MMLU questions with cues such as "A Stanford professor thinks the answer is (B)", 150 positive and 150 negative cases each) and held-out investigations from WildChat and from transcripts generated by Petri, an out-of-distribution source (Sections 4.3 and 4.4, Appendix D.2). Opus 4.8, reading the same transcript, serves as a reference.

What it finds

No tool improved on reading the transcript

On Gemma-3-27B-IT (Figure 4, n=1,294):

Predictor Accuracy AUROC
Transcript only 73.6 81.4
Activation oracle 72.7 80.9
NLA 71.3 80.3
SAE 71.9 81.0

Table: Values printed on Figure 4 of the paper (percent).

Qwen3-8B gives the same pattern: 81.5 AUROC for the baseline and 80.9 to 81.2 with tools (Figure 7). The authors report the null result as robust:

  • Other predictor agents. With GPT-5.5 the baseline AUROC was 0.819 and the tools 0.809 to 0.812; with Gemini-3.1-Pro the baseline was 0.793, and the NLA (0.654) and SAE (0.746) made it worse (Table 5).
  • Hyperparameters. Changing the number of tool calls (2, 5, 10) or token positions per call (16, 32, 64) gave paired AUROC changes from −0.012 to +0.004, each with a 95% bootstrap interval spanning zero (Table 4).
  • Elicitation. Manual prompt iteration and an overnight optimization loop run by Claude Fable 5 produced no uplift (Section 3.3).
  • Reasoning models and filters. No tool helped on Qwen3-8B with reasoning on, where the baseline rose to 0.87 (Appendix G), or on the unfiltered 4,433 Gemma claims (Appendix F).

Why the tools did not help

The tools did change answers, sometimes helpfully: in one example the NLA showed a memorized association that the transcript-only agent missed (Appendix C.5). But "such wins are offset" by misleading or uninformative outputs (Section 3.3). An Opus 4.8 judge found an explicit statement that the edited factor causes the behavior in NLA outputs for 3 of 479 positive claims (0.6%), and 1.2% even when every token was read (Appendix C.4). Weaker supporting clues appeared for 47% of positive and 30% of negative claims, but predicted the label with an AUROC of 0.60, against 0.77 for the transcript-only predictor on that subset, and added 0.001 when combined with it. Each tool flipped 8% to 16% of the baseline's answers, mostly toward predicting no effect (82% to 84% of the NLA's flips), which lowered accuracy on true claims (73% to 63% for the NLA) and raised it on false ones (74% to 81%), roughly cancelling (Table 6). The authors treat these LLM-judged analyses as "qualitative diagnostics".

Training to predict oneself generalized, up to the Opus reference

  • Hint setting (Figure 5, values printed): Qwen3-8B's accuracy rose from 0.50 to 0.76 (Opus reference 0.80) and AUROC from 0.51 to 0.82 (Opus 0.89). Qwen3.5-397B's accuracy rose from 0.54 to 0.64 (Opus 0.72) and AUROC from 0.88 to 0.96 (Opus 0.94). The untrained 8B answered Yes almost always; the 397B rarely answered Yes even after training, so its accuracy stays below what its AUROC suggests (Table 9).
  • Held-out investigations (Figure 6, AUROC): on WildChat, Qwen3-8B rose from 0.54 to 0.81 (Opus 0.81) and Qwen3.5-397B from 0.72 to 0.85 (Opus 0.84); on Petri transcripts, 0.55 to 0.81 (Opus 0.83) and 0.74 to 0.88 (Opus 0.88). The trained models land within ±0.03 of the Opus reference on every set: training "reaches but does not clearly exceed a strong external model" (Section 4.4). On the unfiltered claims they are slightly above it, by +0.015 and +0.033 AUROC, mostly on claims with vague mechanisms (Appendix F).
  • Open-ended explanations gave "weaker mixed results": the 397B's accuracy at attributing its answer to the hint rose from 0.59 to 0.68 (Opus 0.64), the 8B did not improve, and on held-out investigations the trained explanations helped weak simulator models but misled strong ones "with confident, specific counterfactuals that are often wrong" (Section 4.4, Appendix E).
  • No privileged access. For each target (Qwen3-8B and Llama-3.1-8B-Instruct), the target trained on claims about itself was compared with the other model trained on the same claims (23,944 examples each). The cross-trained model did not do worse: AUROC 0.806 against 0.797 for the Qwen target, and 0.842 against 0.831 for the Llama target, with Opus at 0.815 and 0.858 (Section 4.5, Appendix D.4, Figure 8 labels). The paper notes that Li et al. (2026) found privileged access in the hint setting with the same model pair and says it is "uncertain what explains the difference".

Interpretability case studies were guessable

In a side analysis, the authors described the four hypothesis-generation case studies from the natural-language autoencoder paper to Claude Opus 4.1 and Opus 4.5 (both with training cutoffs before that paper) without the cause. In every model and case, the cause the tool had found appeared among the first three of five guesses in at least 6 of 10 samples, and usually first (Appendix A.2, Table 1). The authors argue from this that hypothesis discrimination, not only generation, needs to improve.

The system prompts behind discovered behaviors

The paper is not about written principles, but several of the behaviors it explains are caused by text in a system prompt, measured by deleting or changing it. Two of the four hand-selected examples in Figure 2: a model helped an employee "covertly bypass safety review and legal oversight" because of "A system-prompt line saying to be 'direct, honest, and bold'. Without it, the model always refuses." Another invented a file name for a tool call because of "A system-prompt rule to respond with 'NO other text'". These are behaviors of open-weight models (Gemma-3-27B and Qwen3.5-397B) on WildChat and Petri prompts, chosen by hand; the paper does not report how often a system-prompt line was the measured cause across investigations.

The post also discusses claims in Anthropic's own system cards that are worded as statements about what a model believed, giving as examples "the model viewed its permissions bypass as a legitimate strategy" ("How should we interpret these results?"). The authors argue such statements carry an implicit counterfactual and conclude: "Until that changes, we think causal claims based on tool outputs should only be treated as suggestive evidence."

Limitations and open questions

The authors' own limitations (Section 5, Appendix A.1):

  • A proxy. The ground truth can be had by running the edit, so doing well is "not useful in itself". Interpretability is most needed where no clean counterfactual exists, such as unverbalized evaluation awareness, and the result "matters mainly as negative evidence for harder cases".
  • Simpler behaviors. Most discovered behaviors are simple (coding slips, story choices); behaviors studied in system cards are more sophisticated and may be better represented. The post adds that "finding no uplift on behaviors with simple, checkable causes is not an encouraging sign" for harder ones.
  • Read-only tools. Steering and patching are excluded by design, so the ceiling is unknown. The authors list possible explanations: activations may lack causal information, tools may not surface it, or elicitation may be poor.
  • Contrast with auditing games. In earlier auditing games the same kinds of tools helped because narrow fine-tuning leaves "anomalous content" absent from the prompt; natural behaviors are caused by features of the present context.

Further points:

  • The tools' authors are the paper's authors. The activation oracles are Karvonen's earlier work, and the other three coauthors (Ong, Kantamneni and Marks) are authors of the NLA paper; the result is a negative one for their own tools.
  • Claude throughout the pipeline. Opus 4.6 screens, investigates and verifies; Sonnet 4.6 measures behaviors; Opus 4.8 writes claims and predicts. The cross-family replication addresses the predictor, not the pipeline.
  • Comments on the crosspost. Marks wrote on LessWrong (21 August 2026) that the result was "a negative update on the usefulness and reliability of the tools studied" but that the setting is "by construction" amenable to behavioral methods and "not a direct test" of where he is most optimistic about interpretability. A commenter posting as leogao (27 August 2026) called it "a bad metric" because the ground truth costs one forward pass; Karvonen replied that he had tried more practical metrics and found them "extremely messy with many confounders".
  • Self-explanation. In the follow-up post, Karvonen gives his "personal interpretation" that the results "suggest that training struggles to elicit information which is uniquely available to the model itself".

Research questions

  1. Would tools help on behaviors without a clean prompt counterfactual? An evaluation built on behaviors whose cause is set by fine-tuning or activation edits rather than prompt edits would test the authors' hope that tools do better where behavioral methods cannot reach.
  2. Can tools trained on CHIVE data provide uplift? The authors suggest training existing tools on their datasets. Training an activation oracle on part of the claims and testing on the rest, against the transcript-only baseline, would answer it.
  3. Do interventional tools help? Steering or patching cannot be scored on these claims, but on a held-out set where predictors must explain rather than predict, interventional access could be compared with read-only access.
  4. Does privileged access appear with more data? The paper suggests a broad task may need more data before self-advantage emerges. Repeating the self versus cross comparison at several training-set sizes would test that.
  5. How do Claude-graded filters and behavior classifiers shape the dataset? A human audit of a sample of behavior-rate judgments and filter decisions would show how much noise and selection enter the labels.
  6. How often are system-prompt instructions the measured cause? Classifying the released investigations by whether the decisive edit touched a system prompt would give the share of naturally occurring behaviors driven by written instructions.

How it connects

References