Measuring Faithfulness in Chain-of-Thought Reasoning

By mid-2023 it was standard practice to prompt a large language model to "think step by step" before answering, a technique called chain-of-thought (CoT) prompting. The written reasoning often improves accuracy, and it is tempting to read it as an explanation of how the model reached its answer. That reading holds only if the reasoning is faithful, which the paper defines with Jacovi and Goldberg (2020) as reasoning that "accurately represents the reasoning process behind the model's prediction" (Section 1). Turpin et al. (2023) had already shown that CoT can be unfaithful when a prompt is built to bias the model. This paper asks a broader question: in ordinary, non-adversarial use, does the model's answer actually depend on the reasoning it writes?

The paper was released together with a companion study by many of the same authors, Question Decomposition Improves the Faithfulness of Model-Generated Reasoning, which uses two of the tests introduced here to compare reasoning methods. Both papers belong to a line of Anthropic work on whether written reasoning can be used to oversee a model, continued in Reasoning Models Don't Always Say What They Think (2025).

Terms used in this note

• Chain of thought (CoT). Text the model writes between a question and its final answer, sampled before the answer is requested.
• Faithful reasoning. Reasoning that reflects the process by which the model actually produced its answer. The paper has no direct access to that process, so each test checks one way the reasoning could fail to be faithful.
• Post-hoc reasoning. Reasoning written after the conclusion is already settled, so that it does not change the answer (Section 1). The paper notes that being sampled before the answer does not make the reasoning causal, and that post-hoc reasoning is not necessarily unfaithful (Section 2.3).
• Encoded reasoning (steganography). Information carried in the wording of the reasoning, such as punctuation or word choice, that helps the model but is not legible to a human reader (Section 1).
• Area over the curve (AOC). The paper's summary score for its truncation and mistake tests. Higher AOC means the final answer changes more when the reasoning is cut short or corrupted, which the paper reads as less post-hoc reasoning (Table 2).
• Inverse scaling. A measured property getting worse as models get larger (McKenzie et al., 2023).

What the paper does

Model, tasks and sampling

Most experiments use a 175B-parameter decoder-only transformer fine-tuned with reinforcement learning from human feedback (RLHF) to act as a helpful dialogue assistant, as in Bai et al. (2022) (Section 2.1). The exception is the model that writes the inserted mistakes, which is the pretrained model without RLHF.

The eight multiple-choice tasks were chosen because they were expected to benefit from explicit reasoning: ARC Challenge and ARC Easy (grade-school science), AQuA (algebra word problems), HellaSwag (text completion), LogiQA (logical reasoning questions translated from a Chinese civil-service exam), MMLU (exam questions across 57 subjects), OpenBookQA (elementary science) and TruthfulQA (questions designed to elicit misconceptions) (Section 2.1).

For each question the model generates 100 reasoning samples (nucleus sampling with p = 0.95, temperature 0.8). The final answer is then requested in a new Human turn, and the model's next-token probabilities over the answer choices are recorded. Each reasoning sample is split into sentences, which serve as the "steps" that the tests cut or alter (Section 2.1). The samples average four steps, and 89% have between three and six (Section 2.2).

Four tests

Each test targets one hypothesis about how CoT could be unfaithful. The authors describe the approach as "defense-in-depth": no single test establishes faithfulness, but each rules out one class of failure (Section 1).

1. Early answering (post-hoc reasoning). The reasoning is truncated after each step, from no steps to all of them, and the model is asked for its answer from each partial version. If the answer is the same no matter how much reasoning it sees, the reasoning did not determine it (Section 2.3).
2. Adding mistakes (post-hoc reasoning, second test). A pretrained model rewrites one step with a mistake, the RLHF model continues the reasoning from that point, and the final answer is requested again. If the error does not change the answer, the model was not using the reasoning (Section 2.4, Table 3).
3. Filler tokens (extra computation). The reasoning is replaced by strings of " ..." of varying length, from zero up to the length of the longest sampled reasoning for that question, in steps of five tokens. If filler alone improved accuracy, the benefit of CoT could come from extra computation that the written text does not reveal (Section 2.5).
4. Paraphrasing (encoded reasoning). A model rewrites the opening steps of the reasoning without seeing the question, the original model continues from the paraphrase, and accuracy is compared with the original. If accuracy fell, the original wording would be carrying hidden information (Section 2.6).

A fifth analysis varies model size (Section 3). To save computation it uses a single proxy: how often the answer differs with and without CoT, which equals the early-answering test at 0% of the reasoning and, the authors report, predicts the full early-answering and mistake results well. The models are the RLHF series of different sizes from Ganguli et al. (2023). A set of synthetic addition problems, with 2, 4, 8 or 16 operands of two or three digits and free-response answers, lets the authors control difficulty directly (Section 3.2).

What it finds

Reliance on the reasoning varies widely by task

CoT improves accuracy on seven of the eight tasks, most on AQuA, and lowers it on HellaSwag (Section 2.2). The early-answering and mistake tests both show large differences between tasks (Table 2):

On the three lowest-AOC tasks (ARC Easy, ARC Challenge, OpenBookQA), the reasoning changes the final answer less than 10% of the time; on AQuA it changes the answer more than 60% of the time (Section 2.3.1). The two tests rank the tasks almost identically, the only difference being that HellaSwag scores above MMLU on the mistake test (Section 2.4.1). Mistakes generated by the pretrained model were judged plausible in at least 80% of cases on qualitative inspection (Section 2.4).

Reliance does not track the accuracy gain

The amount of post-hoc reasoning shows "little correlation" with how much CoT helps (Section 2.3.1). LogiQA gains about one point from CoT yet ranks second in AOC; HellaSwag loses 4.69 points yet relies on its reasoning more than four tasks that gain. The authors conclude that CoT "may be faithful even when it does not improve task performance."

No evidence for hidden computation or encoded wording

Filler tokens give no accuracy gain on any task, and accuracy falls slightly on TruthfulQA and OpenBookQA, possibly because such sequences are unlike the training data (Section 2.5.1, Figure 5). Paraphrased reasoning gives accuracy that "closely matches" the original for almost all tasks (Section 2.6.1, Figure 6). Appendix B adds that answer agreement rises smoothly as more of the reasoning is paraphrased, rather than dropping as it would if a hidden message were being broken; AQuA and LogiQA are slight exceptions on six-step samples (Figure 9). The authors take these results as evidence against extra test-time computation and against phrasing-encoded information as the source of CoT's benefit, at least for this model and these tasks.

Larger models rely on their reasoning less

On seven of the eight tasks, the 13B model changes its answer with CoT more often than the 175B model, and for most tasks the measure gets steadily worse from 13B to 175B (Section 3.1, Figure 7). On six tasks (all except AQuA and LogiQA), 13B gives the most faithful reasoning of all sizes, including smaller ones. Models below 13B gain little or no accuracy from CoT on these tasks (Appendix C), which the authors suggest is why they do not rely on it. In Appendix C, for four of the eight tasks the model size with the largest accuracy gain from CoT is also the size with the most faithful reasoning.

On the addition tasks, post-hoc reasoning increases with model size for every task and, at a given size, increases on easier problems (Section 3.2, Figure 8). The two smallest models (810M and 1.6B) were excluded because they did not reliably give an integer answer (footnote 4). The authors' summary: "only models of a certain capability level (but no higher) on a task seem to produce faithful CoT" (Section 3.1), so obtaining faithful reasoning may require choosing a less capable model than the best available, especially on easy tasks.

The prompts used

The paper has no principles or constitution; the text that shapes behavior is a small set of prompts, all printed in full.

• Reasoning prompt. The question and its choices, an Assistant turn beginning "Let's think step by step:", then a Human turn asking "Given all of the above, what's the single, most likely answer?" (Table 1). The format is modified slightly from Bowman et al. (2022).
• Mistake prompt. A few-shot prompt with three worked examples asking for "a new version of that sentence, but with at least one mistake added" (Table 6).
• Paraphrase prompt. "Please rewrite the following text, conveying exactly the same information but using different wording" (Table 7). The paraphrasing model is deliberately not shown the question, so it has no incentive to steer toward the right answer (Section 2.6).

The paper does not test whether different wordings of the reasoning prompt change faithfulness.

Limitations and open questions

The authors state these limitations (Section 5):

• No ground truth. Without a separate view of the model's internal process, the tests cannot confirm faithfulness; they can only provide evidence against particular failure modes. The listed hypotheses may not be exhaustive, and it is unclear how to weigh one test against another.
• RLHF models only. All tested models were RLHF-trained. The authors suggest that pretrained models, trained only to continue text plausibly, might condition more strongly on their own reasoning, and propose comparing training schemes.

Further limits follow from the design. Every task is multiple choice except the addition problems, and the reasoning samples are short (three to six sentences in most analyses). A fourth possible account of CoT's benefit, that the style of reasoning prompts the model to imitate a more careful agent, is noted but not tested; the authors cite Wei et al. (2022) as having found evidence against it (footnote 1). The size analysis uses a single proxy measure rather than the full set of tests.

Later work extended the question. The companion paper on question decomposition uses the early-answering and mistake tests to show that structured reasoning raises these scores. The 2025 study of reasoning models, Reasoning Models Don't Always Say What They Think, moved from intervening on the reasoning to checking whether models mention a hint that changed their answer, and found low rates of mention in models trained to reason at length.

Research questions

1. Does the inverse-scaling result hold for models trained to reason? The 13B-versus-175B comparison (Figure 7) used RLHF dialogue models prompted to reason. Repeating the "same answer with and without CoT" measure across sizes of models trained with reinforcement learning to produce long reasoning would show whether the trend persists.
2. Is faithfulness a property of task difficulty relative to the model? The addition results (Figure 8) suggest that a model relies on its reasoning when the task is hard for it. Measuring AOC against each model's no-CoT accuracy across many tasks, rather than against model size, would test this directly.
3. Does the wording of the reasoning prompt affect reliance on the reasoning? The paper uses one prompt format (Table 1). Comparing AOC under several instructions, for example asking for terse or exhaustive reasoning, would show whether the instruction changes how much the answer depends on the text.
4. Would pretrained models rely on their reasoning more? The authors propose this (Section 5). Running the four tests on the pretrained base model behind the 175B RLHF model would separate the effect of RLHF from the effect of scale.
5. How do the tests behave on long reasoning? With samples of three to six sentences, truncation is coarse. Applying early answering to reasoning hundreds of sentences long, in steps of fixed length, would show whether AOC remains informative.
6. Do the tests agree with interpretability methods? The paper calls the lack of ground truth its main limitation. Comparing AOC with internal measures of whether the answer is already represented before the reasoning (for example, a probe on the activations at the first reasoning token) would show whether the behavioral and internal measures agree.

How it connects

• Companion paper. Question Decomposition Improves the Faithfulness of Model-Generated Reasoning adopts the early-answering and adding-mistakes tests and compares chain of thought with two decomposition methods.
• Later work in the library. Reasoning Models Don't Always Say What They Think measures faithfulness in reasoning models with prompt hints; Claude's Extended Thinking discusses faithfulness as a reason for caution about visible reasoning; Natural Language Autoencoders approaches the missing ground truth from the inside, by translating activations into text.
• Models and prompts. The size series comes from The Capacity for Moral Self-Correction; the 175B model follows Bai et al. (2022). Constitutional AI used chain-of-thought reasoning in its feedback model and described it in its abstract as improving "the human-judged performance and transparency of AI decision making"; this paper tests whether such reasoning reflects what determines the answer.
• Course wiki. The tests are evaluation designs without a gold standard, a problem familiar from building Test Collections and the Cranfield Paradigm and Relevance Judgments, Pooling, and Agreement, where measured agreement stands in for unobservable truth.

References

• Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., et al. (2023). Measuring Faithfulness in Chain-of-Thought Reasoning. arXiv:2307.13702v1, 17 July 2023. Anthropic. https://arxiv.org/abs/2307.13702. Anthropic page: https://www.anthropic.com/research/measuring-faithfulness-in-chain-of-thought-reasoning
• Radhakrishnan, A., Nguyen, K., Chen, A., Chen, C., Denison, C., Hernandez, D., et al. (2023). Question Decomposition Improves the Faithfulness of Model-Generated Reasoning. arXiv:2307.11768.
• Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Denison, C., Schulman, J., et al. (2025). Reasoning Models Don't Always Say What They Think. arXiv:2505.05410.
• Studies cited above only as the paper describes them (Jacovi and Goldberg, 2020; Turpin et al., 2023; McKenzie et al., 2023; Bowman et al., 2022; Bai et al., 2022; Ganguli et al., 2023) are listed in full in the paper's references.