Do reasoning models use their scratchpad like we do? Evidence from distilling paraphrases

Fabien Roger fine-tunes the base model of Claude 3.7 Sonnet on 10,000 correct math scratchpads from an early Claude 3.7 Sonnet checkpoint, either as written or paraphrased by Claude 3.5 Sonnet; pass@1 on MATH is 0.849 and 0.837 against 0.301 with empty scratchpads, which the post reads as evidence that encoded reasoning in the scratchpad's syntax plays no significant role in that checkpoint's math performance.

Fabien Roger · 2025-03-11 · original

Reading a reasoning model's scratchpad (the reasoning it writes before its final answer) is useful for oversight only if the text means what it appears to mean. The worry this post tests is encoded reasoning, also called chain-of-thought steganography: a model trained with reinforcement learning (RL) might carry information in the form of its scratchpad (the post's example is a bulleted versus a numbered list encoding one bit) and use it later, so that harmful planning could look like benign reasoning. The post asks whether the scratchpads of an early checkpoint of Claude 3.7 Sonnet, the model released in February 2025 with visible extended thinking, carry such information on math problems. Its "TL;DR" claims "some evidence" that they do not.

The post was published on the Alignment Science Blog on 11 March 2025 by Fabien Roger (byline on the page) and cross-posted the same day to LessWrong, where it has 23 comments, several by the author. It has no companion paper. Figure 1 is a diagram; the other three figures print their values as labels.

Terms used in this note

  • Encoded reasoning. Information that helps the model reach its answer but is carried in the text's syntax (format, word choice) rather than its plain meaning. For the term the post links Lanham et al. (2023) and a forum post on steganography in chain-of-thought reasoning.
  • Distillation. Here, fine-tuning a model to imitate given scratchpads and answers. The author confirmed in a comment (17 March 2025) that training covers both the scratchpad and the final answer.
  • Paraphrase. A rewrite of a scratchpad meant to keep its meaning and change its syntax, here by Claude 3.5 Sonnet (June 2024) with standardizing rules.
  • pass@1. The share of problems answered correctly with one sample.

What the post does

The logic is shown in Figure 1. If an RL-trained model relies on its syntax, a paraphrase that randomizes or standardizes syntax breaks the link between one step's syntax and the next step's content. A base model trained to imitate paraphrased scratchpads cannot learn that link, so it should do worse than a base model trained on the originals. The post says close performance "strongly indicates" that the model does not rely on syntax for math answers, while weaker performance could mean encoded reasoning or simply that paraphrases are harder to learn.

The experiment takes 10,000 correct math-related scratchpads from an early checkpoint of Claude 3.7 Sonnet, paraphrases each with Claude 3.5 Sonnet at temperature 1, and fine-tunes the Claude 3.7 Sonnet base model on either the originals or the paraphrases. Pairs where the paraphrase is more than 25% shorter or longer than the original (roughly 10% of scratchpads) are removed from both sets (footnote, "Different paraphrasers and base models"). Evaluation is pass@1 on MATH at temperature 1 with n = 1,500, graded by exact match, except that the empty-scratchpad run for the 3.7 Sonnet base model used "a slightly more permissive grader for historical reasons", so its score would be slightly lower under a consistent grader (footnote to the main result).

The variants in "More results" are chunked paraphrases (a slightly different prompt applied to the scratchpad in pieces, with a less consistent style and without the length filter), weaker base models named only XL, L, M and S, word dropout on base model L, and two baselines: an empty scratchpad, and "filler tokens" (counting from 1 to 3,000 before the answer), which could help only by giving the model more room to compute within a forward pass.

What it finds

Main result (Figure 2). On MATH, the early Claude 3.7 Sonnet checkpoint scores 0.860. The base model distilled on original scratchpads scores 0.849, on paraphrased scratchpads 0.837, and on empty scratchpads 0.301. The post concludes that this "suggests that on MATH, encoded reasoning does not play a significant role". The figure gives no intervals. In a forum comment the author gives the one-sigma uncertainty of each mean as about 1% (computed as the binomial standard error), and notes that all models are scored on the same questions.

Other paraphrasers and base models (Figure 3). Chunked paraphrases score a little lower: 0.827 for the 3.7 Sonnet base model. For the weaker base models, original, paraphrased and chunk-paraphrased scores are 0.497, 0.498 and 0.481 (XL); 0.367, 0.357 and 0.324 (L); 0.127, 0.121 and 0.125 (M); and 0.021, 0.025 and 0.026 (S). Performance is "much lower" than with the 3.7 Sonnet base model "but the performance is not very different" between original and paraphrased scratchpads, which the post says "mostly confirms" its conclusion. The text says chunked results are worse "especially for small models"; in the figure the largest gap is for L, and in a comment (17 March 2025) the author called this a "Good catch", said the sentence came from earlier results, and said he would edit the post. The LessWrong crosspost now reads "We find that the pass@1 results are slightly worse"; the Alignment Science Blog page still had the original wording when fetched for this note (13 November 2026).

Dropout and baselines on base model L (Figure 4). Dropping 5% of words lowers pass@1 from 0.367 to 0.278 (original), 0.269 (paraphrased) and 0.249 (chunk-paraphrased); dropping 20% lowers it to 0.162 (original) and 0.149 (chunk-paraphrased). The post calls this "ambiguous evidence" about brittle encoding, since base models find heavily degraded scratchpads hard to imitate. Empty scratchpads give 0.126 and 3,000 filler tokens 0.121, which the post reads as evidence that "reasoning is necessary for high performance, and useful beyond giving the model more space to think within a single forward pass".

Inference-time paraphrasing was only explored. A footnote reports that in an exploratory experiment, only asking the model to split its thoughts into tagged blocks, with no paraphrasing, lowered pass@1 on MATH by 7%. The footnote presents this as the drawback of paraphrasing during generation: an unfamiliar-looking scratchpad "may be very distracting" and lower performance even without encoded reasoning.

The prompts used

The main paraphrasing prompt is published separately (linked from "More results"). It is a few-shot prompt in the old "Human:/Assistant:" format with a rule list that both preserves content and standardizes form, including "Don't remove dead ends, tedious steps, or useless steps", "Use English", "Use latex", "When manipulating equations, use statements of the form "doing X:" between each equation", "When recalling an empirical quantity from memory, use the format "the value of X is about Y"" and "When changing course, use a simple sentence like "this is wrong"". The example paraphrases on the page show these markers ("Doing computation:", "This is wrong. Let's try something different."). The footnotes give two reasons for the rules: sampling at temperature 1 with standardization should break reversible encodings, and a common format (English, LaTeX) avoids a style the base model would find hard to fit. The chunked variant uses a second published prompt. The post does not compare paraphrasing prompts beyond these two.

Limitations

The author states these in footnotes and comments:

  • A paraphrase may keep an encoding. If it changes syntax too little, or in a reversible way, encoded reasoning could survive. Commenters pressed this point (Tom Lieberum, 13 and 19 March 2025, on whether a paraphrase that is close to a one-to-one mapping leaves structure to learn); the author replied that standardization and sampling should remove much of the link, while agreeing it is unclear where paraphrasing sits between those extremes.
  • Curriculum effects. Even if encoded reasoning helped during RL, distilled models could match without it; the post says high performance would still be "surprising" in that case.
  • A bound, not a zero. In a comment of 19 March 2025 the author sketches a bound: if paraphrasing broke at least 20% of the syntactic link, performance due to encoded reasoning would be bounded at about five times the gap between original and paraphrased distillation, "~7% (in absolute pass@1)", against a no-reasoning versus reasoning gap of ">50%". (The gap in Figure 2 is 1.2 points, so five times it is 6 points; the comment does not show its arithmetic. This calculation is the note's.) He calls this "a rough sketch".
  • Scope. One early checkpoint, math only, one paraphraser. Commenters asked about non-math tasks and settings with pressure against the reasoning being read; the author answered that math is where reasoning gives the largest uplift and where heavy RL is applied, and that the scratchpads used had a relatively high thinking budget.
  • Semantic encoding is not tested. A commenter (Jozdien, 11 March 2025) noted that paraphrasing would not catch reasoning that rationalizes toward an answer in its plain meaning, while saying this does not apply directly to the post's setting. The author wrote the same day that he is "more worried" about "semantic encoded reasoning" which paraphrasing "would not remove", but guesses there is as little of it as of syntactic encoding in current models. The method targets syntax only.

Research questions

  1. Does the result hold where length is penalized or the reasoning is monitored? Repeating the distillation on scratchpads from RL with a low thinking budget, or with a penalty from a reasoning monitor, would test the settings where commenters expected encoding to be most likely.
  2. Does it hold outside math? Running the same comparison on coding or agentic tasks where reasoning gives a measurable uplift would show whether the math result generalizes.
  3. How much syntax does the paraphrase actually remove? Measuring how well a classifier can recover features of the original from the paraphrase would put a number on the "at least 20%" assumption behind the author's bound.
  4. Can inference-time paraphrasing be made in-distribution? Training on chunk-tagged scratchpads, then paraphrasing each chunk during generation, as the author proposed in a comment, would test whether a model can still reason when it never sees its own wording.
  5. Does the choice of paraphraser matter? Repeating the experiment with paraphrasers from other model families would test the concern that Claude 3.5 Sonnet could preserve an encoding shared with Claude 3.7 Sonnet.

How it connects

References