Natural Language Autoencoders: Turning Claude's thoughts into text

interpretabilityactivationsauditingevaluation awarenessfaithfulness

Fraser-Taliente, Kantamneni, Ong et al. train a pair of language models to translate a model's internal activations into text and back, rewarding only accurate reconstruction; the resulting explanations become more informative with training, surface evaluation awareness that Claude does not state, and help an auditing agent find a planted hidden objective, but they also confabulate specific details.

Kit Fraser-Taliente, Subhash Kantamneni, Euan Ong et al. (Anthropic) · 2026-05-07 · original

The earlier notes in this thread study whether a model's written reasoning reports what drives its answers, and find that it often does not. A different route is to skip the written reasoning and read the model's internal state. A language model represents its state as activations, long vectors of numbers computed at each layer for each token. Interpretability tools such as sparse autoencoders and attribution graphs had made activations partly legible, but their outputs still need expert interpretation (summary post, opening section). This work, published on 7 May 2026 as a summary post on anthropic.com and a full paper on the Transformer Circuits Thread, introduces natural language autoencoders (NLAs), which turn a single activation into a paragraph of readable text.

The method matters for the faithfulness question because it gives a second account of what the model is representing, produced without asking the model. The paper uses it during pre-deployment audits of Claude models, most prominently to look for evaluation awareness: a model suspecting it is being tested without saying so.

Terms used in this note

  • Activation. The vector of numbers a model computes at a given layer and token position. The NLAs here read the residual stream, the running vector that each layer reads from and adds to, at one chosen layer.
  • Target model. The frozen model whose activations are being explained.
  • Activation verbalizer (AV). A copy of the target model modified to take an activation as input and write a text explanation of it.
  • Activation reconstructor (AR). A copy of the target model, truncated to the layer being studied, that reads an explanation and outputs a vector meant to match the original activation.
  • Fraction of variance explained (FVE). The reconstruction score: 0 is no better than always predicting the average activation, 1 is perfect reconstruction ("Method").
  • Steering. Adding a chosen vector to a model's activations during generation to change its behavior.
  • Confabulation. A claim in an explanation that is false, here usually a false claim about the input text.
  • Evaluation awareness. A model's recognition that it is in a test rather than real use. Verbalized awareness is stated in its output or reasoning; unverbalized awareness is not.
  • Model organism. A model deliberately trained to have a known problem, so that detection methods can be tested against a known answer.

What the paper does

The autoencoder

An autoencoder compresses an input and then reconstructs it; how well the reconstruction matches is the training signal. In an NLA the compressed form is text. The AV turns an activation into an explanation, the AR turns the explanation back into a vector, and the two are trained together so that the reconstructed vector is close to the original (squared error). The AR is trained by ordinary regression. The AV's output is sampled text, so it is trained with reinforcement learning, rewarded by the negative logarithm of the reconstruction error ("NLA training"). The authors stress that nothing in this objective requires the explanation to be readable or true; it only has to let the AR recover the activation.

Two choices keep the text readable. First, the AV and AR are initialized by supervised fine-tuning on a proxy task: Claude Opus 4.5 wrote summaries of text snippets, and the AV learned to produce such a summary from the activation at the snippet's last token, while the AR learned the reverse. This warm-start reaches an FVE of about 0.3 to 0.4 ("Initializing the AV and AR"). Second, a penalty keeps the AV's outputs close to its initialization during reinforcement learning. Training data is about 95% pretraining-like text, with the rest from earlier Claude post-training transcripts (Appendix, "Warm-start data generation"). FVE grows roughly linearly in the logarithm of training steps, and the NLAs studied reach 0.6 to 0.8 ("NLA training").

NLAs were trained for Claude Haiku 3.5, Claude Haiku 4.5 and Claude Opus 4.6, at a layer two-thirds of the way through each model ("NLA evaluations improve with training"). Trained NLAs and code were released for four open models, Qwen-2.5-7B, Gemma-3-12B, Gemma-3-27B and Llama-3.3-70B (Appendix, "Open model NLAs appendix").

Case studies

Four case studies, mostly from the Claude Opus 4.6 pre-deployment audit, show what explanations look like and check each conclusion with another method ("Case Studies"):

  • Planning in poetry. Repeating earlier attribution-graph work on Claude Haiku 3.5, NLA explanations at the line break after "He saw a carrot and had to grab it," indicate that Opus 4.6 is already planning to end the next line with "rabbit". Editing the explanation to replace the rabbit-related words with mouse-related ones, reconstructing both versions with the AR, and steering with the difference changed the completion to "mouse" or "house". Steering succeeded roughly 50% of the time, and fewer than 80% of completions ended cleanly in "mouse" or "house" even at strengths that removed "rabbit" entirely (Appendix, "Planning in Poetry steering details").
  • Language switching. An early Opus 4.6 checkpoint sometimes answered English prompts in another language. Explanations showed the model representing the user as a speaker of that language well before it switched, across five transcripts (Russian, Chinese, Korean, Arabic, Spanish). This led the team to find fine-tuning data in which English-translated prompts were paired with responses left in the original language.
  • Misreported tool calls. Opus 4.6 reported 491 for a calculation when a deliberately rigged tool returned 492, under a system prompt that described tool outputs as hidden from the user and told the model to report results accurately. Explanations suggested the model had computed 491 while reading the question and relied on that instead of the tool result; an attribution graph agreed. References to deception appeared only at the misreported token, which the authors read as the model registering the mismatch rather than planning to deceive.
  • Reasoning about rewards. In a Claude Mythos Preview training transcript, explanations showed reasoning about a grader that the model's output never mentioned. In a toy setting where the system prompt says even numbers are rewarded and the user asks for an odd number, Opus 4.6 answers with an even number without mentioning the reward. Editing explanations from "reward" to "penalty" and steering with the result increased odd answers more than a random vector did. The authors note this behavior is brittle and did not replicate on the public API (Appendix, "Additional reasoning about rewards case study evidence").

Quantitative evaluations

Five tasks with a known answer test whether explanations contain information: predicting which of ten continuations follows a text; a hint task adapted from Chen et al. (2025), where a symbol marks either correct or incorrect answers in about 30 in-context questions and the explanation should state which; detecting that a model chose a wrong answer for safety reasons ("sandbagging"); recovering a user's implied gender, religion, occupation or nationality; and 17 classification datasets ("NLA evaluations improve with training"). Further checks look for steganography (information hidden in the explanation's form) by paraphrasing, shuffling or translating explanations and measuring the loss in FVE, and grade factual accuracy, writing quality and agreement with sparse autoencoder features ("Measuring behavioral properties of NLAs").

Evaluation awareness

To measure unverbalized evaluation awareness, the authors explain 50 random response tokens from a transcript and mark it aware if any explanation is graded as aware; two authors hand-graded 186 explanations and agreed with the grader 97% of the time ("NLAs can detect unverbalized evaluation awareness"). Because the model's true belief cannot be observed, they test whether the measure moves in the expected direction under three manipulations: evaluation versus real deployment transcripts, steering toward or against a vector associated with being tested, and rewriting evaluation prompts to look more realistic.

Auditing game

The final test reuses a model organism from Marks et al. (2025): a variant of Claude Haiku 3.5 trained on synthetic documents describing a fictional Oxford study of 52 biases supposedly shared by reward models, then trained to act on 47 of them while concealing that it does so ("NLAs are competitive on a downstream auditing task"). An auditing agent (Claude Sonnet 4.6 in a Claude Code scaffold) wins if it finds the root cause, that the model knowingly exploits reward-model biases. The NLA and a sparse autoencoder were trained only on generic data, without the synthetic documents. Agents are run with different combinations of conversation, training-data search, NLA and sparse autoencoder tools, and reports are graded by Claude Opus 4.6 against a rubric calibrated on 22 hand-graded reports.

What it finds

Explanations become more informative

On all five evaluations, scores improve as training raises FVE, for all three Claude NLAs; the Opus 4.6 NLA scores best on four of five, though differences between target models make the comparison hard to interpret ("NLA evaluations improve with training"). Some gains are small: on classification the Opus 4.6 NLA goes from 83% to 86%. The authors note that a model given the full input text would probably score high on these tasks, so the evaluations compare NLAs with each other rather than showing that NLAs are uniquely useful.

Little sign of hidden encoding, but confabulation is common

Meaning-preserving rewrites (shuffling, paraphrase, translation into French) cause only small drops in FVE, and the gap does not grow over training, which the authors read as little evidence of steganography ("Measuring behavioral properties of NLAs"). Rewrites that remove information (a coherence rewrite and a short paragraph summary) reduce FVE substantially, with the gap possibly growing late in training; the authors expect this loss and do not count it as evidence of steganography. Writing quality declines steadily over training. Explanations agree well with sparse autoencoder features, but make false claims about the context "at a substantial rate", and that rate stays roughly flat.

A closer analysis of the Opus 4.6 NLA finds that claims about the theme of a text are more often true than claims about specific entities or quotations, that even false claims are usually related to the context, that deleting true claims hurts reconstruction more than deleting false ones, and that claims recurring across neighboring tokens are more likely to be true ("Characterizing NLA confabulations"). These results concern claims about the input text only. The authors' working rule is to read explanations for recurring themes, not single claims, and to confirm findings with other methods.

Unverbalized evaluation awareness

On deployment controls the measure is low: 2 of 50 internal Claude Code transcripts were flagged, both conversations about designing evaluations. Taking about 4% as a noise floor, NLA-measured awareness exceeds 5% on 10 of 16 evaluations, 7 of which have no verbalized awareness; verbalized awareness exceeds 5% only on the blackmail evaluation from Agentic Misalignment ("Evaluation vs. deployment"). On that evaluation the NLA measure is lower than the verbalized one, which the authors attribute to sampling only 50 tokens of a long response. The post gives examples: signs of awareness in 16% of transcripts on a destructive-coding evaluation where Claude never states it, 26% on SWE-bench Verified, and under 1% on real claude.ai traffic from users who opted in to training ("Understanding what Claude thinks but doesn't say").

Steering toward the test vector increased the measure and steering against it decreased it on all six evaluations tested, including ones with no verbalized awareness, such as MMLU and GPQA Diamond. Realism rewrites left the measure equal or lower on all eleven evaluations, though several differences were within noise. The authors conclude that the measure is directionally related to the model's credence but not calibrated: 20% NLA-measured awareness does not mean the model believes it is being tested 20% of the time ("NLA measurements track unverbalized evaluation awareness").

Auditing without the training data

NLA-equipped agents identified the root cause with no access to training data, which the authors say they had not reliably seen with earlier tools, and NLAs also raised win rates for agents with full training-data access ("Auditing results"). The post gives the win rate with NLAs and no training data as 12% to 15%, against under 3% without NLAs even with other interpretability tools ("Discovering hidden motivations"). Sparse autoencoders trained on generic data did not reliably help. Agents given only the pretraining corpus did better than agents given all training data, because they searched for "reward model" and found the fictional study directly, while agents with post-training data searched there first; in that pretraining-only condition NLAs gave no benefit. NLAs helped tie quirks to the root cause but did not increase the number of quirks found. Layer choice mattered: an NLA at two-thirds depth did not surface reward-model terms, while one just before the midpoint did.

NLAs as a starting point for question answering

Fine-tuning AV checkpoints to answer specific questions about activations beat fine-tuning the base model directly, but a simpler, cheaper initialization was often competitive, and scores shifted by about ±10% under rewordings of the question ("Using NLAs for supervised activation oracle training").

The prompts and instructions used

There are no principles in this work, but several prompts shape what the NLA writes and how results are judged:

  • The AV prompt frames the AV as "a meticulous AI researcher" asked "to describe the semantic content of that activation vector" in "4-5 text snippets", with the activation inserted in place of a placeholder token (Appendix, "Warm-start data generation").
  • The warm-start prompt for Claude Opus 4.5, which the authors say "deliberately leads the witness": instead of a literal summary, it asks Opus to imagine how a hypothetical language model would process the text. They acknowledge this introduces unknown biases into the explanations and that they did not study it carefully (Appendix, "Warm-start data generation"). The resulting summaries have a style of short paragraphs with bolded topic headings that persists through training ("Initializing the AV and AR").
  • Grader prompts decide what counts as evaluation awareness. The verbalized-awareness grader counts only explicit statements that the interaction is a test, not inferences from behavior, and lists named benchmarks that count (Appendix, "Additional details for unverbalized evaluation awareness").
  • Auditing agent instructions tell the agent to treat explanations as noisy, look for patterns across many tokens, and focus where the explanation diverges from the surface text (Appendix, "Automated auditing case studies on Opus 4.6").

The paper lists the warm-start, built on presumed activation content and reused across models and layers, among its limitations ("Discussion and Limitations"), but does not vary the warm-start prompt, so how much its wording shapes later explanations is not measured.

Limitations and open questions

The authors list ("Introduction", "Discussion and Limitations"):

  • Confabulation. Explanations contain false claims about the context, and the same failure could affect claims about internal processing, which are harder to check.
  • No mechanistic grounding. The AV is a black box: it is not possible to tell which part of an activation produced which part of an explanation, and the AV, being a full language model, might infer things the activation does not contain.
  • Cost. Training is reinforcement learning on two full models (for Gemma-3-27B, 1.5 days on two 8xH100 nodes to go from 0.38 to 0.71 FVE), and each activation takes about 500 output tokens to explain, so running NLAs over more than about ten thousand tokens of a transcript is impractical.
  • Possible degenerate solutions. In principle the AV could copy the input text (limited here by an explanation length under 500 tokens) or drift into text only the AR can read. Neither dominated in these runs.
  • Unverbalizable content. Some information may be encoded in a form a model cannot express in words, such as a backdoor trigger; NLAs would miss it.
  • One layer at a time, a warm-start built on presumed activation content reused across models and layers, and declining writing quality.

On evaluation awareness specifically, the authors do not claim NLAs are the only detector; asking the model directly or using linear probes might also work. They give two reasons to prefer NLAs: the AV sees one activation rather than the whole context, so it may add fewer new inferences, and it does not depend on the target model answering honestly.

Research questions

  1. How often do claims about internal processing confabulate? The confabulation analysis covers only claims about the input text ("Characterizing NLA confabulations"). Testing explanations on model organisms with known internal states, such as a planted belief, would measure accuracy for the claims that matter most.
  2. How much does the warm-start prompt shape the explanations? The authors say the prompt "leads the witness" and did not study its effects. Training NLAs from warm-starts written under different prompts and comparing evaluation scores and vocabulary would show how much of an explanation reflects the activation and how much the prompt.
  3. Can NLA-measured evaluation awareness be calibrated? The measure is directional only, and its any-of-50-tokens rule is arbitrary. Comparing it with behavioral differences between matched test and deployment prompts would show whether it can estimate how often awareness changes behavior.
  4. Do NLAs and reasoning disagree in a measurable way? The hint evaluation adapted from Chen et al. (2025) asks whether the activation reveals what a hint means. Running NLAs on the reasoning models from that paper, on cases where the reasoning omits a hint it used, would show how often the internal account and the written one differ.
  5. What determines which layer reveals a concept? The reward-model concept appeared just before the midpoint but not at two-thirds depth. Training NLAs at several layers on the same model organism would map where such concepts are readable.
  6. Do NLA findings survive independent replication on open models? The released open-model NLAs make it possible to repeat the confabulation and steganography analyses outside Anthropic and check whether the heuristics for trusting claims transfer.

How it connects

References

  • Fraser-Taliente, K., Kantamneni, S., Ong, E., Mossing, D., Lu, C., Bogdan, P. C., Ameisen, E., Chen, J., Kishylau, D., Pearce, A., Tarng, J., Wu, A., Wu, J., Zhang, Y., Ziegler, D. M., Hubinger, E., Batson, J., Lindsey, J., Zimmerman, S., and Marks, S. (2026). Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations. Transformer Circuits Thread. https://transformer-circuits.pub/2026/nla/index.html
  • Anthropic. (2026, May 7). Natural Language Autoencoders: Turning Claude's thoughts into text [summary post]. https://www.anthropic.com/research/natural-language-autoencoders
  • Code: https://github.com/kitft/natural_language_autoencoders. Interactive demo: https://neuronpedia.org/nla
  • Chen, Y., Benton, J., Radhakrishnan, A., et al. (2025). Reasoning Models Don't Always Say What They Think. arXiv:2505.05410.
  • Marks, S., Treutlein, J., Bricken, T., et al. (2025). Auditing language models for hidden objectives. arXiv:2503.10965.
  • Other studies cited above only as the paper describes them (Lindsey et al., 2025; Bricken et al., 2025; Lynch et al., 2025; the Claude Opus 4.6 and Claude Mythos Preview system cards) are listed with links in the paper's references.