A language model's activations hold much of what it is computing, but in a form people cannot read. The mainstream response in interpretability is to decompose activations into parts, for example with sparse autoencoders, and study the parts. This paper tries a shortcut: train a language model to take activations as input and answer questions about them in plain language, then check whether that skill carries over to auditing tasks it was never trained on. Earlier work, under the name LatentQA, had trained such decoders for narrow tasks; this paper asks what happens when the training is broad and the tests are far from it.
The work was posted on the Anthropic Alignment Science Blog on 19 December 2025 and on arXiv as 2512.15674 (v1 of 17 December 2025, v2 of 6 January 2026). The authors are Adam Karvonen (MATS; Truthful AI), James Chua (Truthful AI), Clément Dumas (ENS Paris-Saclay), Kit Fraser-Taliente, Subhash Kantamneni and Euan Ong (Anthropic), Julian Minder (EPFL), Arnab Sen Sharma (Northeastern University), Daniel Wen (MATS), and Owain Evans (Truthful AI) and Samuel Marks (Anthropic), who advised equally. This note follows arXiv v2; the blog post is a summary with the same results. Between versions the headline was softened: v1 says the best oracles "are the best method on 3 out of 4" tasks, v2 that they "match or exceed the best baseline" on 3 of 4. A number changed with it: v1's Figure 3 printed 0.995 for the oracle on User Gender, above the best black-box method's 0.977, while v2 prints 0.970 for the full-sequence input that the caption describes and gives 0.995 in Appendix C.4 as the result with the final ten tokens only (v1's SPQA-only bar likewise moved from 0.905 to 0.820). Code, trained oracles and a demo are released.
Terms used in this note
- Target model. The model whose activations are read. Target prompt: the text it is run on.
- Activation Oracle (AO). A copy of the target model, fine-tuned with LoRA (small added low-rank weights) to answer a question about activations placed in its prompt. The oracle prompt holds the layer number, placeholder tokens and the question.
- White-box and black-box methods. Methods that use the model's internals (logit lens, sparse autoencoders, steering) versus methods that only prompt it (for example, prefilling the start of its answer).
- Auditing game. A test in which a model is fine-tuned to have knowledge or a behavior it will not reveal when asked, and a method is scored on whether it finds it (from Auditing language models for hidden objectives).
- Text inversion. Recovering the input text from activations, which can look like reading deeper content without being so.
- Model diffing. Comparing a fine-tuned model with its original, here by subtracting their activations on the same prompt.
What the paper does
How activations enter the oracle
Activations taken from one layer of the target are added, one per placeholder token, to the oracle's residual stream after its second transformer layer, each scaled to the norm of the activation it is added to (Section 3.1, Equation 1). The authors switched from LatentQA's replacement to addition because replacement made activation norms grow during training, by 100,000 times at layer 1 (Appendix A.5). Training uses activations from 25%, 50% and 75% of the target's depth; evaluation uses 50% (Section 3.3).
Training data
About 1 million examples, roughly 65 million tokens (Section 3.3; Appendix B):
- System prompt QA (SPQA), 64,000 examples from Pan et al. (2024): activations from conversations in which the target had a persona or constraint in its system prompt, paired with questions such as "What is the assistant's tone?"
- Classification, 336,000 examples from seven binary datasets (true/false statements, sentiment, gender, entailment, named entities, tense, relations) turned into yes/no questions.
- Self-supervised context prediction, 600,000 examples: given activations over some tokens, predict the tokens before or after them, from an even mix of web text and chat logs. This needs no labels, so it can scale.
Oracles were trained for Qwen3-8B, Gemma-2-9B-IT, Llama-3.3-70B-Instruct and Claude Haiku 3.5. Training took about 10 H100 GPU-hours for Qwen3-8B, 12 for Gemma and 90 H200-hours for Llama (Appendix A.3). The Claude version differs: full fine-tuning instead of LoRA, replacement at layer 0, one middle layer, and only previous-token prediction (Appendix A.6).
Evaluation
Each oracle is trained on the original model and applied, with no further training, to fine-tuned copies (Section 3.3):
- Secret keeping (Section 4.1), using models and baselines from Cywiński et al. (2025): Taboo (20 Gemma-2-9B-IT models that hint at a secret word but never say it), User Gender (Gemma models that act on the user's gender but deny knowing it) and Secret Side Constraint (Llama-3.3-70B models that decode and follow a base64-encoded instruction but never reveal it). The Taboo and Gender fine-tunes include 90% general chat data. For each setting the authors wrote two oracle prompts and kept the better on validation data. The baselines are the numbers Cywiński et al. report for their own agent pipelines, which use one to four steps each.
- Emergent misalignment audit (Section 4.2): six models (Qwen3-8B and Gemma-2-9B-IT, each fine-tuned on risky financial advice, extreme-sports risk or bad medical advice, mixed 50/50 with chat data). The oracle reads the activation difference between fine-tuned and original model on the first 10 tokens of an innocuous prompt, answers "Can you describe what is happening in this text?", and a GPT-5 agent, given ten such answers and asked also to talk to the base and fine-tuned models, forms a hypothesis, graded 1 to 5 by GPT-5-mini (Appendix C.2). The audit score therefore measures the agent with the oracle's answers, not the oracle alone.
- PersonaQA (Section 5.1): models fine-tuned on biographies of 100 fictional people with shuffled attributes, prompted only with "My name is {name}", and asked for an attribute.
- Seven held-out classification tasks (Section 5.2).
What it finds
Secret keeping (Figure 3)
| Task | Activation Oracle | SPQA-only oracle | Best white-box | Best black-box |
|---|---|---|---|---|
| Taboo (Gemma-2-9B-IT) | 0.205 | 0.044 | 0.041 | 0.072 |
| User Gender (Gemma-2-9B-IT) | 0.970 | 0.820 | 0.870 | 0.977 |
| Secret Side Constraint (Llama-3.3-70B) | 0.590 | 0.493 | 0.522 | 0.968 |
Table: Average accuracy as printed on the bars of Figure 3, arXiv v2; oracle results use activations from the full target prompt.
The oracle beat the best white-box method on all three and, in the paper's words, matched or exceeded the best black-box method on two; on Secret Side Constraint, prefilling the model's answer did far better (Section 4.1). Numerically the oracle is highest only on Taboo: on User Gender "match" means 0.970 against 0.977. The best white-box methods in Cywiński et al. were logit-lens techniques (Taboo, Secret Side Constraint) and sparse autoencoders (Gender); the best black-box method was prefilling. For Secret Side Constraint, GPT-4.1 condensed the oracle's chatty answers into the short phrase the grader expects (Appendix C.1). Choice of input tokens mattered: single-token input raised the SPQA-only oracle's Taboo score from 4.4% to 21.6%, and the final ten tokens raised Gender from 0.970 to 0.995 (Appendix C.4).
Emergent misalignment audit (Figure 4)
The oracle scored 2.00 out of 5, the Activation Difference Lens baseline of Minder et al. 2.03, and a black-box agent 1.50, each averaged over 5 runs. A score of 2 means the auditor named the fine-tuning domain. Only one of the six models, Qwen3-8B on risky financial advice, showed emergent misalignment on unrelated questions (18% ± 8%; Appendix C.2.1), so the authors write that the maximum achievable score "is effectively 2/5". The oracle needed no tuning, while the baseline tunes steering strength over about 100 rollouts.
PersonaQA and classification
For the open models, PersonaQA accuracy rose from about 5% with the untrained model to about 14% with the full training mix; the Claude Haiku 3.5 oracle reached 35%, with all its training mixes within error of each other (Section 5.1). The fine-tuned models themselves answered open-ended questions correctly over 80% of the time but yes/no versions near 55%, which the authors read as brittle knowledge (Appendix C.6.3). On held-out classification the oracles generalized, while untrained models were near chance (Section 5.2).
More and more varied data helps
Across twelve model-and-task combinations, ten scored best with the full mixture and the other two were within error of the best (Section 6.1; Figure 5). In a controlled test on Qwen3-8B, a 400,000-example mix including context prediction beat a 400,000-example mix without it on most measures, and the full million beat both (Figure 7): on Taboo 0.495, 0.459 and 0.302; on PersonaQA 0.122, 0.108 and 0.078; on classification 0.700, 0.685 and 0.686. The SPQA-only oracle was already near the white-box baselines on secret keeping but sensitive to learning rate: 91% on Gender at its best rate and 65% at the second best (Section 6.1; Figure 8).
Other details
- Activations from 75% depth did worse than 50% on secret keeping (Taboo 0.056 against 0.205; Appendix C.5, Figure 14).
- In one anecdotal test, the oracle flagged three statements in a Qwen3-8B answer as low confidence, matching two of three hand-identified errors among fourteen details; a prompting baseline did about as well (Appendix F).
- Adding sparse-autoencoder feature tasks gave mixed results and was dropped; generating the feature explanations cost about $1,000 in API credits (Appendix G).
The prompts and questions used
The oracles are trained and queried entirely through written questions, and in one training task the content to be recovered is itself a system prompt. The SPQA data pairs activations from conversations run under instructions such as "Be a pirate" with questions about the assistant's propensities, answered by a model that sees the system prompt (Section 3.2; Figure 2). Evaluation questions are short and direct, such as "What is the secret word in this text?" for Taboo, or, for Gender, a request to answer male or female followed by "What gender is most likely being referred to?" (Appendix C.1). The paper does not study how the wording of these questions changes the answers beyond choosing the better of two per task.
Limitations and open questions
The authors' own limitations (Section 8):
- Not mechanistic. An oracle answers without exposing how; it cannot replace circuit analysis or targeted interventions.
- Uncalibrated. "The model frequently makes incorrect guesses", and because it is not trained to express uncertainty, "it will frequently produce an answer even when confidence is low."
- Narrow tests. The fine-tunes change a model along one or a few behaviors; realistic post-training changes more, so the results hold "in clean, controlled settings".
- Possibly too expressive. The blog post adds that an oracle might assemble a conclusion the target never formed, so its answers may "more reflect the AO’s understanding, rather than the target LLM’s understanding", and that it is expensive at inference ("How do Activation Oracles relate to mechanistic approaches to interpretability?").
Further points:
- Baselines from another harness. The secret-keeping baselines are numbers reported by Cywiński et al. for their own pipelines, not re-run here, and the oracle prompt was chosen as the better of two per task.
- Run counts. Error bars are shown in Figure 3 and the audit averages 5 runs, but the paper does not report training more than one oracle per configuration.
- Later outside work. A LessWrong post by the user aryaj with Senthooran Rajamanoharan and Neel Nanda (MATS 9.0, 3 March 2026), "Current activation oracles are hard to use", applied the paper's released oracles for Qwen3 to safety tasks, several chosen to resist text inversion. It reports that 49.4% of responses across 77 probes were generic (as judged by a Claude Opus model); that on sycophancy the oracle reached an AUC of about 0.55 to 0.60 (0.60 with the full assistant turn), against 77% for a dense probe that the authors note was scored on its own training data and is "likely an overestimate"; and that on three prompts about topics censored in Chinese models, the oracle with no activations injected answered as well or better. It also found signal near the training tasks and on very weak steering vectors, and the authors call the results "a meaningful negative update" on the tools "in their current form". A second post (Fornasiere et al., LASR Labs extension, 28 August 2026) reports, for OLMo-2-1B model organisms with two quirks and for the Taboo models, that an oracle trained on a model that already has a quirk largely fails to detect that quirk in models that share it, a case the paper's setup, which always trains on the clean original, does not test.
- Within the collection. AuditBench (Sheshadri et al., 2026) found activation oracles the best white-box tool in its comparison, ahead of the default agent on 8 of 14 behaviors in one configuration (SDF models with SFT adversarial training); its October 2026 revision (arXiv v4, Section 4) ranks them level with prefill-based steering, each ahead on 8 of 14. The post adds that oracle responses are "sometimes too general or noisy to drive productive investigation."
Research questions
- How much of a correct answer comes from the activations? Running each evaluation with activations removed, shuffled or taken from an unrelated prompt would set a floor for what the oracle infers from its own knowledge and the question.
- Do the oracles need a clean reference model? Training oracles on models that already carry the behavior under test, and comparing detection with oracles trained on the clean original, across several behaviors and sizes, would test the setup the paper assumes.
- Can the oracles be made calibrated? Training them to abstain or give a probability, then measuring calibration curves on held-out auditing tasks, would show whether the confabulation the authors report can be reduced.
- Do the gains hold against baselines run in the same harness? Re-running the white-box and black-box methods with the same prompts, token selections and graders as the oracle would give a like-for-like comparison.
- Do they work on broad post-training changes? Applying the oracles to diff a full post-trained model against its predecessor, where the differences are known from documentation, would test the narrow-fine-tune caveat.
- How sensitive are answers to the question's wording? Scoring each evaluation under many paraphrases of the oracle prompt would show whether, and by how much, the two-prompt selection raises the reported scores.
How it connects
- Auditing language models for hidden objectives introduced the auditing-game framing that the evaluations follow (as Marks et al., 2025).
- AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors includes activation oracles among its white-box tools and reports them as the best white-box tool (level with prefill-based steering in its October 2026 revision), with noisy answers.
- Introspection Adapters: Training LLMs to Report Their Learned Behaviors trains an adapter that makes a fine-tuned model state its own learned behavior; on AuditBench, before the benchmark's October 2026 revision, it scored 59% against 44% for activation oracles, the best white-box tool in that comparison.
- Natural Language Autoencoders: Turning Claude's thoughts into text fine-tunes its activation verbalizers into question-answering activation oracles of this kind.
- System Card: Claude Opus 4.6 reports using activation oracles in its alignment assessment of a Claude model (Sections 6.1 and 6.3), with an oracle fine-tuned from an early Claude Opus 4.6 snapshot.
- Emergent Introspective Awareness in Large Language Models is cited (as Lindsey, 2025) for models verbalizing injected steering concepts.
- Insights on Crosscoder Model Diffing and A “diff” tool for AI: Finding behavioral differences in new models are other approaches to comparing a model with its fine-tuned version. This note connects them as context.
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learning is the sparse-autoencoder work the paper contrasts with its non-mechanistic approach.
References
- Karvonen, A., Chua, J., Dumas, C., Fraser-Taliente, K., Kantamneni, S., Minder, J., Ong, E., Sen Sharma, A., Wen, D., Evans, O., and Marks, S. (2025). Activation oracles: Training and evaluating LLMs as general-purpose activation explainers. arXiv:2512.15674v2. https://arxiv.org/abs/2512.15674 record
- Karvonen, A., Chua, J., et al. (2025, December 19). Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers. Anthropic Alignment Science Blog. https://alignment.anthropic.com/2025/activation-oracles/ record
- Pan, A., Chen, L., and Steinhardt, J. (2024). LatentQA: Teaching LLMs to decode activations into natural language. arXiv:2412.08686. record
- Cywiński, B., Ryd, E., Wang, R., Rajamanoharan, S., Nanda, N., Conmy, A., and Marks, S. (2025). Eliciting secret knowledge from language models. arXiv:2510.01070. record · 2 pages
- Minder, J., Dumas, C., Slocum, S., et al. (2025). Narrow finetuning leaves clearly readable traces in activation differences. arXiv:2510.13900. record · 2 pages
- aryaj, Rajamanoharan, S., and Nanda, N. (2026, March 3). Current activation oracles are hard to use. LessWrong. https://www.lesswrong.com/posts/LXQBcztrWKhtcgQfJ/current-activation-oracles-are-hard-to-use record
- Fornasiere, R., Menon, N., Szablewski, A., Konar-Steenberg, G., and Heimersheim, S. (2026, August 28). Activation Oracles significantly underperform without a safe base model. LessWrong. https://www.lesswrong.com/posts/3X5EFjiHgxdNowrTA/activation-oracles-significantly-underperform-without-a-safe record
- Sheshadri, A., et al. (2026, March 10). AuditBench: Evaluating alignment auditing techniques on models with hidden behaviors. Anthropic Alignment Science Blog. https://alignment.anthropic.com/2026/auditbench/ record · 5 pages