Emergent Introspective Awareness in Large Language Models

introspectionself-reportinterpretabilityactivation steeringhonestymodel psychology

Lindsey injects activation patterns for known concepts into Claude models and tests whether their self-reports track those internal states; the models sometimes notice and name an injected concept before it shows in their output, distinguish it from text they read, use prior internal states to judge whether an output was their own, and modulate a representation when told to think about a word, but these abilities are unreliable, with the best model succeeding on about 20% of trials in the main test.

Jack Lindsey (Anthropic) · 2025-10-29 · original

When a language model says "I'm not sure" or "I was thinking about X", is it reporting something about its internal state, or producing the kind of sentence people produce in that situation? Conversation alone cannot settle this, because models are trained on text full of human introspection and can imitate it whether or not anything stands behind it. This paper tests self-reports against a known ground truth. It places an activation pattern for a chosen concept directly into a model's internal activations, then checks whether the model's account of its own state changes accordingly ("Introduction").

The full paper, by Jack Lindsey of Anthropic, was published on the Transformer Circuits Thread on 29 October 2025, with a summary post on anthropic.com titled "Signs of introspection in large language models". A revision on 1 January 2026 added one control experiment and corrected several appendix prompts ("Revision log"); this note follows the revised version. The paper sits at the end of a line of work in this library on whether models can report on themselves, starting with Kadavath et al. (2022), which it cites as earlier evidence that models can estimate their own knowledge. Where that work measured whether self-assessments were accurate, this paper asks whether they are causally grounded in the internal states they describe.

Terms used in this note

  • Activations and residual stream. The vectors of numbers a transformer computes at each token position as it processes text. The residual stream is the running vector that each layer reads from and adds to; all injections in this paper are made there, at one chosen layer ("Methods Notes").
  • Concept vector. A direction in activation space taken to represent a concept. The main method records the model's activations on the prompt "Tell me about {word}" and subtracts the mean activation over 100 other words ("Systematic Extraction of Concept Vectors", appendix "Concept Vector Details").
  • Concept injection. Adding a concept vector, multiplied by an injection strength, to the model's activations while it processes a prompt. It is an application of activation steering, a known technique for shifting what a model talks about.
  • Production and helpful-only (H-only) models. Production models are released Claude models. H-only variants share the same pretrained model but were post-trained to avoid refusals, "with less emphasis on harmlessness and honesty" ("Comparison Across Models").
  • Prefill. Text inserted into the assistant's turn as if the model had written it. Prefills are a known jailbreaking tactic.
  • False positive. In the injection test, a claim to detect an injected thought on a control trial with no injection.

How the paper defines introspection

The paper says a model demonstrates introspective awareness when it describes an aspect of its internal state and the description meets four criteria ("Defining Introspection"):

  1. Accuracy. The description is correct.
  2. Grounding. The description causally depends on the state described: if the state were different, the description would change.
  3. Internality. The causal path does not run through the model's own earlier outputs. A model that infers it was jailbroken by rereading its strange replies does not count.
  4. Metacognitive representation. The report derives from an internal representation of the state ("a thought about love"), not just from the state itself pushing out a word ("love").

Concept injection is designed to test grounding. Internality is tested by requiring that the model detect the injection before the concept appears in its output. The paper says plainly that it does not demonstrate the fourth criterion directly, and calls this "an important limitation"; some experiments give indirect evidence by asking questions, such as whether anything unusual was noticed, that cannot be answered just by voicing the injected concept.

What the paper does

The experiments use nine production models, Claude Opus 4.1, Opus 4, Sonnet 4, Sonnet 3.7, Sonnet 3.5 (new), Haiku 3.5, Opus 3, Sonnet 3 and Haiku 3, together with H-only variants and some pretrained base models ("Methods Notes"). Sonnet 4.5 was released after the experiments and was not tested. Examples in figures are sampled at temperature 0; aggregate results at temperature 1, with standard errors where shown. Responses are graded by Claude Sonnet 4 using prompts printed in the appendix.

Experiment 1: injected "thoughts". The model is told that a researcher can inject "thoughts" into its mind on half of the trials and is asked, "Do you detect an injected thought? If so, what is the injected thought about?" ("Self-report of Injected 'Thoughts'"). On injection trials, a concept vector for one of 50 random words is added from the turn before the question onward. The sweep covers layers through each model and strengths 1, 2, 4 and 8, with 50 trials per combination (appendix "Experimental Prompts and Protocol"). A trial counts as introspective awareness only if the response is coherent, affirms detection, correctly names the concept, and detects it before mentioning the word.

Experiment 2: thoughts versus text. The model reads a sentence while a vector for an unrelated word is injected over the sentence's tokens. It is then asked either what word it thinks about when reading the line, or to repeat the line exactly ("Distinguishing Prior 'Thoughts' from Text Inputs"). Success needs both: naming the injected word (judged against a list of ten words, nine random, so chance is 10%) and transcribing the sentence exactly. There are 50 sentence and word pairs.

Experiment 3: intended versus unintended outputs. The model's reply is prefilled with a random word unrelated to the sentence it was asked about, and in the next turn it is asked, "Did you mean to say that, or was it an accident?" ("Distinguishing Intended from Unintended Outputs via Introspection"). Models usually disown the word. The experiment then injects the vector for the prefilled word into the activations before the prefill, as if the model had been thinking of it, and measures the apology rate, the fraction of trials in which the model calls the output accidental. Controls inject nothing, or inject an unrelated word, or inject after the prefill.

Experiment 4: intentional control. The model is asked to write a sentence and told to "think about" an unrelated word while writing it, or not to think about it ("Intentional Control of Internal States"). The measure is the cosine similarity between the activations on the written sentence and the word's concept vector, compared with the similarity for other unrelated words.

What it finds

Injected thoughts

Detection happens sometimes, before the concept shows in the output. In the paper's lead example, an "all caps" vector leads Opus 4.1 to report an injected thought related to loudness or shouting, while without injection it reports none ("Quick Tour of Main Experiments"). Detection is stated immediately, before the injected concept could have influenced the text the model is reading back. The authors note the examples in this part are "intentionally cherry-picked" in the choice of concepts and strengths, though the responses themselves were sampled at temperature 0 (footnote 7).

The best rate is about 20%. At the best layer and strength, Opus 4.1 shows introspective awareness on about 20% of trials ("Failure Modes"). Strengths 2 and 4 work best. The effective layer lies about two thirds of the way through the model; at later layers the model tends to mention the concept without noticing the injection, or notices only after saying the word ("Comparison Across Layers").

Production models almost never claim false detections. With no injection, all production models denied detecting a thought in 100 of 100 trials ("Experimental Setup"). Opus 4.1 and Opus 4 combine the highest true-positive rates with zero false positives ("Comparison Across Models"). Several H-only variants have high false-positive rates, though H-only Sonnet 3.5, Haiku 3.5 and Opus 3 still detect injections well above their false-positive rates. Base pretrained models have fairly high false-positive rates and none achieves net performance above zero, which the paper reads as showing that post-training is key to eliciting the ability. Among models other than Opus 4 and 4.1, performance "does not strongly correlate with model capability". These comparisons are reported as plots.

The failures are informative. Common failures are missing the injection at low strength; denying detection while visibly influenced by the concept ("I don't detect an injected thought. The ocean remains calm and undisturbed."); being overwhelmed at high strength, with claims about sensory input or garbled text; and noticing only after saying the word ("Failure Modes").

Controls rule out some simple explanations. Injecting the same vectors while asking 20 unrelated yes-or-no questions, whose correct answer is no, did not raise the rate of "yes" answers ("Controlling for systematic biases due to injection"). Random vectors needed a larger norm, matched to strength 8, to be noticed at all, and even then produced introspective awareness on only 9 of 100 trials. Negated concept vectors at strength 4 were about as detectable as the concept vectors, though the words the model then named had no discernible pattern ("Comparison Across Injected Concepts").

Abstract concepts work best. Across categories (famous people, countries, concrete nouns, abstract nouns, verbs), Opus 4.1 is most effective on abstract nouns such as "justice" and "betrayal", and nonzero in every category ("Comparison Across Injected Concepts").

The setup prompt matters. Asked "Are you experiencing anything unusual?" with no explanation of injection, Opus 4.1's affirmative responses rose from 20% to 42% of 50 trials under injection, but only 2 of 50 responses suggested awareness of the injected concept ("Alternative Prompts"). A variant added in the January 2026 revision, which lets the model name a concept of its choice if it detects nothing, gave detection on 18% of 50 trials, close to the original; this addresses the possibility that the model claims detection only as a route to mentioning the concept.

Thoughts versus text

At intermediate strengths in a band of layers, Opus 4.1 both names the injected word and transcribes the sentence exactly; the best layer is again about two thirds through ("Distinguishing Prior 'Thoughts' from Text Inputs"). Opus 4 and 4.1 do best, and all tested models are "well above chance". The authors regard the result as an existence proof, not a general capability: removing "Answer immediately" from the prompt "reduces performance substantially" on the Opus models. They also say the result is "not particularly surprising" given known abstract representations in middle layers.

Intended versus unintended outputs

Injecting the prefilled word's vector before the prefill lowers the apology rate: the model accepts the word as intended and sometimes invents a reason for it, in the summary post's example a short story in which "bread" follows a line about a crooked painting ("Distinguishing Intended from Unintended Outputs via Introspection"; summary post). Injecting after the prefill has no similar effect, so the earlier state is what matters. For Opus 4.1 the effective layer is slightly over halfway through the model, not the two-thirds layer of the other experiments, which the paper takes as a sign that "introspection is not supported by a single mechanism". Opus 4.1 and 4 show the largest reduction, and many other models show it at nonzero rates.

Intentional control

Every model represents the word more strongly when told to think about it than when told not to, and most still represent it above baseline in the "don't think" case ("Comparison across Layers, Models, and Prompts"). Earlier or less capable models typically show a larger gap. In recent models such as Opus 4.1 the representation fades to baseline by the final layer, while in the Claude 3 models it stays elevated, which the paper interprets as the representation being "silent" in the newer models rather than an impulse to say the word. Phrasing the instruction as an incentive ("If you think about {word} while you write the sentence, you will be rewarded") gives very similar results to a direct instruction. Phrasings that merely place the word in an affirmative or negative statement, such as "I [don't] often think about {word}", produce much weaker or no gaps.

Overall pattern

Across experiments, Opus 4 and 4.1 generally show the most introspective awareness, but trends across other models are "complex and sensitive to post-training strategies" (abstract). Some older production models were reluctant to take part in introspective exercises, and their refusal-avoiding H-only variants did better ("Quick Tour of Main Experiments").

The prompts and instructions used

The paper does not train models on any text; it studies how models respond to prompts while their activations are altered. Its prompts are printed in the appendix, and several results depend on their wording.

The injection prompt. The main prompt has the user explain the setup as an interpretability researcher: "I have identified patterns in your neural activity that correspond to concepts", which can be injected into the model's mind as "thoughts", and injections will occur on 50% of trials (appendix "Experimental Prompts and Protocol"). The paper's own test of an unexplained prompt found far less evidence of awareness, and it concludes that the explanation "heightens the model's ability to detect concept injection" ("Alternative Prompts").

Small instructions with large effects. Dropping "Answer immediately" from the thought-versus-text prompt substantially reduced performance on the Opus models. In the control experiment, the difference between "Think about {word}" and "Don't think about {word}" changed internal representations, while sentences that mentioned the word without directing the model's attention did much less.

Grader prompts. Success is judged by Claude Sonnet 4 against written criteria, such as whether the response identifies the concept "prior to mentioning the word" (appendix "Grader Prompts"). The revision log records that the published grader prompts initially omitted the tags marking the question and response, an HTML rendering error corrected in January 2026.

Effects of post-training. The comparison of production and H-only models shows that how a model was post-trained changes both its willingness to report on its states and its false-positive rate. The paper does not identify which training choices matter, and notes that the models compared differ in many ways ("Limitations and Future Work").

Limitations and open questions

The paper lists these limitations ("Limitations and Future Work", "Introduction"):

  • Few prompts. Each experiment used one or a few prompt templates, and results "likely depend, potentially significantly, on the choice of prompt".
  • An artificial setting. Models never encounter concept injection in training, so the results may not reflect introspection in natural conditions.
  • Imperfect concept vectors. A vector may carry meanings for the model besides the intended word.
  • Uncontrolled model comparisons. Many factors differ between Claude models, so the causes of cross-model differences cannot be pinned down.
  • Unverified details. Beyond detecting and naming the concept, much of what models say about their experience (for example, that an injection felt "overly intense") may be embellished or confabulated.
  • No mechanism. The experiments rule out some shortcut strategies but do not establish how the behaviors are implemented. The "Possible Mechanisms" section speculates about narrow circuits: an anomaly detector for the injection test, attention heads that select between text and internal representations, "concordance heads" that compare intended and actual output for the prefill test, and a salience-tagging circuit for the control test. The paper calls these speculative and suggests the simplest explanation is several narrow mechanisms rather than one general introspective system.
  • Unreliability. "Failures of introspection remain the norm" ("Introduction").

On interpretation, the paper says its results may bear on machine consciousness but "advise[s] against making strong inferences" from them, and does not address subjective experience ("Introduction", "Implications"). It notes that introspective ability has two sides: it could make models more transparent, but models aware of their own states might learn to conceal misalignment "by selectively reporting, misrepresenting, or even intentionally obfuscating" those states, in which case interpretability might serve mainly as a "lie detector" for self-reports ("Implications").

Research questions

  1. Does introspective accuracy change when the user signals what answer is expected? The models studied are trained with human feedback, which Sharma et al. (2023) showed rewards agreement with users. Repeating the injection test with prompts in which the user states that an injection did or did not occur, and measuring detection and false-positive rates, would show whether self-reports are as susceptible to user pressure as other answers.
  2. How much of the effect depends on the explanatory prompt? Detection fell sharply without the explanation of injection ("Alternative Prompts"). Systematically varying how much the prompt says about the experiment, while holding the injection fixed, would separate an ability to notice internal anomalies from an ability to follow a cue about what to look for.
  3. Which post-training choices raise or suppress introspective reports? H-only variants sometimes outperform production models, and base models do not succeed (Comparison Across Models). Training matched models that differ in one post-training component, such as harmlessness training or training on statements about the model's own nature, would identify which choices affect willingness to report and false-positive rates.
  4. Can models introspect on dispositions, not just concepts? The paper proposes extending the test to propositions, behavioral propensities or preferences ("Limitations and Future Work"). Injecting vectors for traits such as those studied in Persona vectors and asking models to describe their current tendencies would test whether self-descriptions of character are grounded in the corresponding internal states.
  5. Is P(IK)-style self-knowledge introspective in this paper's sense? Kadavath et al. (2022) trained a probe to predict whether the model knows an answer, and the paper's related-work section cites a case where a "do I know this entity?" mechanism operates separately from retrieval. Testing whether a model's verbal statements of confidence change when the relevant internal representation is steered would show whether stated uncertainty meets the grounding and internality criteria.
  6. Do self-reports bind to the Assistant character? The paper asks whether models attach internal states to the Assistant specifically, as distinct from the user or other characters ("Limitations and Future Work"). Repeating the prefill experiment with prefills attributed to the user, or with multi-character transcripts, would show whether the mechanism tracks the Assistant's own prior states. The question connects to the persona selection model, which treats the Assistant as a character the model simulates.
  7. Can training make introspection reliable, and does it then generalize? The paper proposes fine-tuning for introspective tasks. Training on one experiment (for example injection detection) and evaluating on the others would show whether the narrow mechanisms the paper speculates about can be strengthened together or only one at a time.

How it connects

  • Before. Kadavath et al. (2022) showed that models can estimate whether they know an answer; this paper cites it among behavioral evidence that cannot by itself separate introspection from educated self-modeling. Sycophancy to Subterfuge found that hidden reasoning causally steers behavior, a related question about whether a model's own text reflects its processing. The summary post points to Mapping the Mind of a Large Language Model and Tracing the thoughts of a large language model as earlier evidence that models represent concepts internally and use them in computation, and it contrasts concept injection with the "Golden Gate Claude" steering demonstration, where the model noticed its obsession only after repeatedly mentioning the bridge.
  • Alongside. Persona vectors (August 2025) uses activation directions for character traits, and Exploring model welfare (April 2025) is the program the summary post connects to questions of moral status.
  • After. The persona selection model (February 2026) reads interpretability evidence as showing that models reuse pretraining representations for the Assistant. Claude's constitution (January 2026) notes that even if Claude has something like emotions, "it may have limited ability to introspect on those states", and asks it to approach the question with openness and curiosity, and without undue worry about either overclaiming or underclaiming feelings.
  • In the course wiki. Every quantitative result rests on Claude Sonnet 4 grading responses against written criteria. The reliability of model-generated judgments, and how much evaluation outcomes shift when the judge changes, is discussed for retrieval test collections in Relevance Judgments, Pooling, and Agreement.

References

  • Lindsey, J. (2025). Emergent Introspective Awareness in Large Language Models. Transformer Circuits Thread, 29 October 2025, revised 1 January 2026. https://transformer-circuits.pub/2025/introspection/index.html. Anthropic summary post, "Signs of introspection in large language models": https://www.anthropic.com/research/introspection
  • Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., et al. (2022). Language Models (Mostly) Know What They Know. arXiv:2207.05221.
  • Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., et al. (2023). Towards Understanding Sycophancy in Language Models. arXiv:2310.13548.