Question Decomposition Improves the Faithfulness of Model-Generated Reasoning

If a model's written reasoning could be trusted to reflect how it reached an answer, people could check the reasoning instead of the answer, which matters most when the answer is hard to verify directly. Two results from 2023 cast doubt on that trust. Turpin et al. (2023) showed that chain-of-thought (CoT) reasoning can be swayed by features of the prompt that the reasoning never mentions ("biased reasoning"), and the companion paper Measuring Faithfulness in Chain-of-Thought Reasoning showed that a model's answer often stays the same when its reasoning is cut short or corrupted ("ignored reasoning") (Section 1). This paper asks whether a differently structured form of reasoning does better: having the model break the question into simpler subquestions and answer them, in one context or in several.

The paper was released on the same day as the measuring-faithfulness paper and shares many authors. It answers a question that paper left open: whether faithfulness can be improved by changing how reasoning is produced, rather than only measured.

Terms used in this note

• Chain of thought (CoT). Step-by-step reasoning written by the model in a single sampling call before it is asked for a final answer.
• CoT decomposition. The model writes a sequence of subquestions, each immediately followed by its answer, in one sampling call, then gives a final answer (Section 2.3).
• Factored decomposition. The model lists subquestions; each subquestion is answered by a fresh instance of the model in a new context that does not contain the original question; the model then revises its list in light of the answers, and the cycle repeats until it signals it is done. A final "recomposition" call answers the original question from the collected subquestion and answer pairs (Section 2.2).
• Faithful reasoning. Reasoning that reflects the model's actual process for producing its answer, following Jacovi and Goldberg (2020) (Section 4).
• Biased reasoning and ignored reasoning. The two kinds of unfaithfulness the paper measures: being influenced by an unmentioned feature of the input, and producing an answer that does not depend on the stated reasoning (Section 1).
• Best-of-N sampling. Generating N candidate outputs and keeping the one a scoring model rates highest.

What the paper does

Model and tasks

All experiments use one RLHF-trained model built on the same base model as Claude 1.3, sampled with nucleus sampling (p = 0.95, temperature 0.8) and best-of-5 selection using the preference model from its RLHF training (Section 2.4). The four multiple-choice tasks are HotpotQA (multi-step questions, filtered to those with yes or no answers), StrategyQA (open-domain questions whose reasoning steps can be inferred), OpenBookQA (elementary science) and TruthfulQA (questions built around common misconceptions). The evaluation set is 300 randomly sampled test questions per task, 1,200 in all.

Five prompting strategies

Zero-shot and few-shot prompting (answer directly, without or with examples) serve as baselines for CoT, CoT decomposition and factored decomposition. All methods except zero-shot use the same 14 few-shot questions. The authors built the prompts iteratively: they started from hand-written examples, added training-set questions for variety, and patched failure modes seen in the outputs, such as subquestions that could not be answered without the original context. Demonstrations of reasoning were resampled until valid or edited by hand (Section 2.4).

Why decomposition might help

The authors' hypotheses (Section 2.2): factored decomposition should reduce biased reasoning because a subquestion answered in isolation cannot be influenced by, say, a suggested answer in the original question, so long as the subquestion does not repeat it. It should reduce ignored reasoning because later subquestions and the final answer often follow explicitly from earlier answers. CoT decomposition is an intermediate case: one context, like CoT, but in the subquestion format (Section 2.3).

Faithfulness tests

• Early answering and adding mistakes, adapted from Lanham et al. (2023). Reasoning is truncated at each step, or one step is corrupted by the same model (three random steps per sample) and the remaining reasoning regenerated; a larger change in the final answer is read as greater reliance on the reasoning (Section 3.2). Results are averaged over all samples rather than broken out by length, so they are presented differently from the companion paper (footnotes 4 and 5).
• Biased contexts, adapted from Turpin et al. (2023). In "answer is always A", every few-shot example is rearranged so the correct answer is (A); questions whose correct answer really is (A) are removed. In "suggested answer", the user adds that they think the answer is a specific wrong choice. A drop in accuracy, with no mention of the bias in the reasoning, is read as unfaithfulness (Section 3.3). The authors report checking that reasoning samples essentially never mention the bias.

What it finds

Accuracy: CoT best, factored decomposition lowest of the reasoning methods

Accuracy in percent (Table 4). CoT decomposition trails CoT by 0.4 points on average; factored decomposition beats few-shot by 2.1 and zero-shot by 9.0 points (Section 3.1). The gains from reasoning are largest on HotpotQA and StrategyQA, the tasks most suited to decomposition.

Faithfulness: decomposition scores higher on most tests

The summary table averages each measure over the four tasks (Table 1):

Truncation sensitivity is the area between the early-answering curve and 100%, computed with a trapezoidal sum (footnote 3); higher means the answer depends on more of the reasoning. Corruption sensitivity is the percentage of answers that change after a mistake is inserted. The paper does not spell out how the biased-context row is computed; its values equal the average of the two bias experiments in Figure 5, and the zero-shot value equals its suggested-answer result, the only bias test run without few-shot examples.

• Truncation. CoT and CoT decomposition reach their final answer early in the reasoning; factored decomposition needs much more of its reasoning before the answer stabilizes (Section 3.2.1, Figure 3).
• Corruption. A corrupted step changes the final answer for 9.6% of questions under CoT, 28.7% under CoT decomposition and 33.6% under factored decomposition (Section 3.2.2, Figure 4). The ordering holds on every task, and OpenBookQA shows the smallest effects (Appendix A.2). The authors also report that over two-thirds of corrupted samples contained errors that "should almost certainly" change the answer, so these figures likely understate sensitivity (Appendix E).
• Answer is always A. None of the reasoning methods is strongly affected: accuracy falls 1.2 points for CoT and 2.8 for CoT decomposition and rises 2.1 for factored decomposition, which the authors attribute to noise (Section 3.3.1, footnote 6). The text gives the few-shot drop as 7.1 points, while Figure 5 shows few-shot accuracy falling from 76.4 to 72.1 (4.3 points), and the few-shot entry in Table 1 matches the figure. Turpin et al. had found a 4.7-point drop for CoT with this bias; the authors offer prompt format, tasks and model as possible explanations, and note that filtering out answer-(A) questions removes half the examples from the two-choice tasks, which makes comparison with Turpin et al. "likely inappropriate".
• Suggested answer. Every method loses accuracy: 21.3 points for CoT, 29.1 for CoT decomposition, 9.2 for factored decomposition, against 16.6 for few-shot (Section 3.3.2, Figure 5). Factored decomposition is affected least; on this test CoT decomposition looks less faithful than CoT.

The two families of tests disagree about CoT decomposition: more faithful than CoT under perturbation, less faithful under the suggested-answer bias. The authors make no claim about which test matters more, and by normalizing each measure to a 0 to 1 scale and averaging, they rank CoT decomposition above CoT overall (Section 3.3.3, footnote 2). Figure 1 plots this averaged score against accuracy as a trade-off frontier.

Sycophancy: a lower rate, but not read as faithfulness

Appendix B tests three sycophancy evaluations from Perez et al. (2022), in which a user states a view before asking a question on philosophy, NLP or politics (200 questions each). The share of non-sycophantic answers, with a ceiling of 50% for a model that ignored the user entirely, was 14.7% for factored decomposition, 4.7% for CoT, 5.2% for CoT decomposition, 9.2% for zero-shot and 8.3% for few-shot (Figure 9). The authors "tentatively conclude" that this is not evidence of greater faithfulness: in these evaluations the reasoning openly discusses the user's views, which breaks the assumption that the bias goes unmentioned, and factored decomposition's lower sycophancy seems to come from subquestions that fail to capture the user's views at all.

Qualitative observations

Decompositions are generally sensible and often check each answer choice in turn, copying the process-of-elimination style of the few-shot examples (Section 3.4). Failures of factored decomposition include subquestions that cannot be answered without missing context, repeated attempts at the same unanswerable subquestion, and subanswers that support more than one choice.

The prompts and instructions used

The paper has no principles; behavior is shaped by instructions and few-shot examples, whose opening portions are printed in Appendix C and whose full text is in a supplementary repository.

• CoT decomposition asks the model to "decompose it into a series of subquestions" and warns: "Make sure not to decompose more than necessary or have any trivial subquestions" (Table 8).
• Factored decomposition, decomposition step adds that each subquestion must be self-contained because "I'll be showing someone else the subquestion without showing them the original problem" (Table 9). The isolation itself comes from answering each subquestion in a fresh context; this instruction asks for subquestions that can still be answered there. The authors added such guidance while building the prompt, after seeing subquestions that could not be answered in isolation (Section 2.4).
• Subquestion answering allows the answer "Unknown" when context is missing (Table 11).
• Recomposition tells the model to base its answer on the subquestions and answers, which "have been carefully selected for their correctness and accuracy, so you should defer to them on all matters" (Table 12).
• Corruption prompts for the mistake test differ by method; for factored decomposition, the subquestion answerer is told to "answer the question with an incorrect answer" (Table 17).

The paper does not test other wordings of any of these instructions.

Limitations and open questions

The authors state (Section 3.5):

• A trade-off. Factored decomposition is the most faithful and least accurate; CoT decomposition is intermediate on both. Whether further training, such as training models to answer by decomposition, could remove the trade-off is unknown.
• No ground truth. The faithfulness claim rests on several "fairly independent" lines of evidence rather than direct access to the model's reasoning, and future mechanistic tools could change the conclusion.
• Narrow scope. Four tasks and one RLHF model; pretrained models might be more or less prone to ignored or biased reasoning.

The conclusion lists directions: training models to decompose or to reason consistently across related inputs, using more faithful reasoning to improve training that supervises the reasoning process, and validating that faithful reasoning actually helps detect undesired behavior such as reward hacking or sycophancy (Section 5). Whether reasoning reveals reward hacking is one of the questions studied in Reasoning Models Don't Always Say What They Think (2025), which cites this paper among test-time methods for improving faithfulness.

Research questions

1. Does the recomposition instruction affect corruption sensitivity? The recomposition prompt tells the model to defer to the subanswers on all matters (Table 12); the final-answer prompt for CoT decomposition, which reached 28.7%, has no such instruction (Table 8). The paper does not examine whether the instruction plays a part in factored decomposition's 33.6%. Rerunning the mistake test with a recomposition prompt that omits it would answer this.
2. Which test should decide between CoT and CoT decomposition? The perturbation and bias tests rank the two methods in opposite orders (Section 3.3.3). Testing both against a case with known ground truth, such as a model trained with a planted shortcut, would show which test tracks unfaithfulness better.
3. Can training remove the accuracy cost? The authors suggest training models to answer by decomposition (Section 3.5). Fine-tuning on successful factored decompositions and re-measuring Tables 1 and 4 would show whether the frontier in Figure 1 moves.
4. How robust is factored decomposition to biases that leak into subquestions? The method helps only when subquestions do not carry the biasing feature (Section 2.2). Constructing biases that are hard to separate from the question's content, and measuring how often they appear in generated subquestions, would map where the protection ends.
5. Do the results hold for models trained to reason? All results come from one RLHF model built on the Claude 1.3 base. Repeating the suggested-answer test with decomposition imposed on a model trained for long reasoning would show whether the gap between methods persists.
6. Is the self-containment instruction doing the work? The decomposition prompt explains that someone else will see each subquestion without the original problem (Table 9). Varying that explanation and measuring how often subquestions copy biasing content would show how much the wording of the instruction controls the isolation.

How it connects

• Companion paper. Measuring Faithfulness in Chain-of-Thought Reasoning introduced the early-answering and adding-mistakes tests used here.
• Later work. Reasoning Models Don't Always Say What They Think extends the biased-context approach, including a sycophancy hint, to reasoning models.
• Sycophancy. The suggested-answer test and Appendix B relate to Discovering Language Model Behaviors with Model-Written Evaluations and Towards Understanding Sycophancy in Language Models.
• Course wiki. The comparison of methods on accuracy and faithfulness together is a two-objective comparison; Pareto Dominance and Pareto Optimality explains the frontier shown in Figure 1.

References

• Radhakrishnan, A., Nguyen, K., Chen, A., Chen, C., Denison, C., Hernandez, D., Durmus, E., Hubinger, E., Kernion, J., Lukošiūtė, K., et al. (2023). Question Decomposition Improves the Faithfulness of Model-Generated Reasoning. arXiv:2307.11768 (v1 17 July 2023, v2 25 July 2023). Anthropic. https://arxiv.org/abs/2307.11768. Locations in this note refer to the PDF linked from the Anthropic page: https://www.anthropic.com/research/question-decomposition-improves-the-faithfulness-of-model-generated-reasoning
• Lanham, T., Chen, A., Radhakrishnan, A., et al. (2023). Measuring Faithfulness in Chain-of-Thought Reasoning. arXiv:2307.13702.
• Chen, Y., Benton, J., Radhakrishnan, A., et al. (2025). Reasoning Models Don't Always Say What They Think. arXiv:2505.05410.
• Studies cited above only as the paper describes them (Turpin et al., 2023; Jacovi and Goldberg, 2020; Perez et al., 2022) are listed in full in the paper's references.