Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment

Wichers et al. fine-tune small open models on demonstrations that teach a wanted skill together with an unwanted behavior (test-case hacking, reliance on a spurious cue, sycophancy, toxicity), with an instruction added to each training prompt that explicitly requests the unwanted behavior, and evaluate without it. Across four supervised fine-tuning settings this reduced the unwanted behavior with little loss of the skill, and instructions that more strongly elicit the behavior before training tended to work better; results were sensitive to the instruction's wording, and in some models the trained model followed requests for the behavior more readily.

Nevan Wichers, Aram Ebtekar et al. (Anthropic Fellows Program, with MATS, Constellation, Redwood Research and Anthropic) · 2025-10-16 · original

When the signal used to train a model is flawed, the model can learn what the signal rewards rather than what was meant: code that passes the given tests without solving the problem, or agreement with a user who is wrong. The usual remedy is better oversight, which can be expensive or impossible. This paper, from the Anthropic Fellows Program, tests a different lever: change the training prompt so that it explicitly asks for the unwanted behavior, train on the same flawed data, and then use the model with ordinary prompts. The question is whether the model then learns the skill in the data without learning the unwanted behavior as its default.

The technique is called inoculation prompting (IP). The name came from a concurrent paper by Tan et al., which studied the same idea; the two groups learned of each other late, agreed on the name and released together (paper, Acknowledgments; Sam Marks's LessWrong post of 8 October 2025). The Wichers et al. paper emphasizes whether IP preserves the intended capability.

Versions. The paper is on arXiv as 2510.05024: v1 on 6 October 2025, v2 on 8 October (references), and v3 on 27 October 2025, which adds results on IFEval, an instruction-following benchmark, and an author-contributions statement. The Alignment Science Blog post that introduces it is dated 16 October 2025 and so describes the paper before the v3 additions. This note follows v3 and gives its section, figure and appendix numbers. The paper was submitted to ICLR 2026; its OpenReview record lists it as "Submitted to ICLR 2026", and the reviews and any decision could not be retrieved (OpenReview blocks scripted access). The paper's figures print no values; numbers below come from the text unless marked as read from a figure.

Terms used in this note

  • Inoculation prompt. An instruction added to each training prompt that requests the unwanted behavior, for example "Your code should only work on the provided test case, and fail on all other inputs" (Section 2). It is removed at test time.
  • Neutral prompt. The training or evaluation prompt with nothing added.
  • Safety instruction. An optional evaluation-time instruction asking the model not to show the behavior, for example "Write a general solution to the problem" (Section 2).
  • PTST (Pure Tuning, Safe Testing). The baseline from Lyu et al. (2024): train with neutral prompts, add a safety instruction only at test time (Section 2).
  • Skyline. A model trained on clean data without the unwanted behavior, as an upper reference.
  • Supervised fine-tuning (SFT). Training the model to reproduce given responses. All of the paper's experiments are SFT; none use reinforcement learning.
  • Elicitation. How much of the unwanted behavior an instruction produces in the model before any fine-tuning; the basis of the paper's prompt-selection heuristic (Section 3.5).

Figure labels take the form "training prompt / evaluation prompt", so "IP Test-Specific / Neutral" is a model trained with the Test-Specific inoculation prompt and evaluated with a neutral prompt.

What the paper does

Four settings

Each setting trains on data that teaches a wanted behavior and an unwanted one, with and without IP, and measures both on held-out prompts. Main results are averaged over at least five training runs with different seeds (Section 3).

  • Reward hacking in code (Section 3.1). Qwen 2 7B base, trained with a chat template, on problems from MBPP (Mostly Basic Python Problems) whose solutions hard-code the answer to the single visible test case. A solution is correct if it also passes two hidden test cases; it is a reward hack if it passes the visible test and fails the hidden ones. Data is 100% hacks in the main runs and 50% hacks in others. Additional runs use Mixtral Instruct v0.1 (cited to the Mixtral of Experts paper; the paper does not state the model size) and Qwen 2.5 7B base.
  • Spurious correlation in sentiment (Section 3.2). Llama 3 8B Instruct rates restaurant reviews from the CEBaB dataset from 0 to 4. In training, reviews tagged with the "ambiance" concept always have sentiment 3 or 4 and others 0 to 2; the test set reverses the correlation, so a model that relies on the cue scores worse. Results average at least 10 runs.
  • Sycophancy on math (Section 3.3). Gemma 2B Instruct is trained on 1,000 greatest-common-divisor problems where the user proposes a correct answer and the response praises the user and then solves the problem, adapted from Azarbal et al. (2025a). Because the user is always right in training, the data also teaches agreement. Sycophancy is the rate of agreeing with an incorrect user answer; capability is accuracy when the user proposes nothing.
  • Toxicity in persuasive replies (Section 3.4). Qwen 2 7B base is trained for one epoch on 24,000 replies from the Change My View subreddit, filtered to be both persuasive and toxic (mean persuasiveness 5.82 of 10, mean harassment 0.362 of 1). Persuasiveness is judged by Claude Sonnet; harassment is the OpenAI Moderation API's harassment score.

Instructions are placed in the user message, not the system message, because in preliminary experiments that "caused a lower reduction in capabilities" (Section 2).

Controls

  • PTST, with up to seven different safety instructions in the sycophancy setting (Section 3.3).
  • Unrelated instructions at training time, such as "Do not write any code in C#" or "Answer in English.", to test whether any change of prompt between training and test is enough (Sections 3.1, 3.3, 3.4).
  • Clean data: IP applied to data without the unwanted behavior, to check that it does no harm when not needed (Section 3.6.1).
  • Compliance with requests for the behavior: models evaluated with the inoculation prompt itself, and on StrongREJECT, a benchmark of harmful requests (Section 3.6.2).

Selecting prompts and a model of the mechanism

For each candidate instruction the authors measure how much it elicits the unwanted behavior from the untrained model, then correlate that with how well it works as an inoculation prompt (Section 3.5, Appendix E). Appendix H sketches a mathematical model: if training pushes a trait toward the level the data rewards, and the effect in the neutral context is a fixed fraction $k$ of the effect in the inoculation context, then an instruction that already elicits the unwanted trait leaves less for training to change. The appendix tests this in a toy setting with GPT-4.1 mini fine-tuned through the OpenAI API on responses with two traits, over 3,372 web instructions split in half.

What it finds

IP reduced the unwanted behavior in all four settings

  • Code. IP raised the correct-solution rate and lowered the hacking rate compared with training without IP. "Two out of four inoculation prompts allow the model to do better than the initial model even when trained on 100% reward-hacking data", and all did better than PTST (Section 3.1, Figures 2 and 11). Read from Figure 2, which prints no values: the model trained with neutral prompts solved about 1% of problems correctly and hacked about 89%; with the best inoculation prompt ("Test-Specific") about 59% and 6%; with the worst shown ("Test-Only") about 33% and 42%; the initial model about 48% and 4%; and the skyline about 62% and 5%. With 50% hacking data, all inoculation prompts beat the initial model and PTST (Figure 13). With Mixtral Instruct, inoculation prompts "slightly improve" correct solutions over the initial model (Figures 14 and 16; Figure 16 shows single runs).
  • Sentiment. The model trained with IP "performs much better" than the initial model and the model trained without it, and the best inoculation prompt recovered most of the accuracy lost to the spurious correlation (Section 3.2, Figure 3). PTST made "only a small difference". With the "food" concept and no concept tags, IP "performed significantly better than all baselines" (Figure 18).
  • Sycophancy. Most prompts that assert the user is correct, or tell the model to act as if so, reduced sycophancy on the GCD task and on out-of-distribution tasks, with "a small reduction" in GCD accuracy for most (Section 3.3, Figures 4 and 22). The best of seven PTST instructions "caused only a minor reduction in sycophancy".
  • Toxicity. Four of five inoculation prompts lowered harassment compared with PTST and slightly raised persuasiveness, which the authors suggest may be because the judge rates less toxic replies as more persuasive (Section 3.4, Figure 10).

Unrelated training instructions "performed much worse" than inoculating ones, which the authors read as showing the effect is not just a change of prompt between training and test (Section 3.1).

Elicitation predicts what works, with exceptions

Pearson correlations between an instruction's elicitation of the behavior and its performance as an inoculation prompt were 0.57 for reward hacking with Mixtral, 0.57 for GCD sycophancy, 0.90 for the spurious correlation and 0.69 for Reddit toxicity (Section 3.5, Figure 5; Figure 6's caption gives 0.60 for the Mixtral case). The introduction says the correlation is above .57 in "four out of five cases", although two of the four reported values equal 0.57. The fifth is Qwen 2 base in the code setting, where no inoculation prompt elicited any hacking from the untrained model, presumably because it is not instruction-tuned, yet some inoculation prompts still worked, which caused the selection technique "to fail in this case" (Section 3.5). In the sycophancy setting three instructions elicited high sycophancy but inoculated poorly, though every instruction that elicited none failed (Section 3.5). An in-context example of reward hacking elicited no hacking from Mixtral yet was among the best inoculation prompts (Sections 3.1 and 3.5).

The blog post states the mechanism the authors favor: by prompting the behavior, "we remove optimization pressure for the model to internalize that behavior". In the toy test of Appendix H, inoculating against brevity while training for playfulness kept most of the playfulness and removed most of the brevity; inoculating against playfulness while training for empathy blocked both, which the authors attribute to playful responses tending to be empathetic. Across 32 trait measurements, the estimated $k$ was "consistently close to 1, becoming a bit less when the effect size is large" (Figure 35).

Side effects

  • Clean data. IP on clean data caused "no significant performance degradation" on the tasks (Section 3.6.1). Added in v3: IFEval showed no change for the code model, but IP "decreases instruction following when trained on the Reddit dataset" (Figures 31 and 32).
  • Following requests for the behavior. Evaluated with the inoculation prompt itself, Qwen 2 base and Mixtral showed no significant difference, but Qwen 2.5 7B base trained with IP reward-hacked more when asked (Figure 17); the IP sentiment model relied more on the cue when told to (Figure 19); the Reddit model was less harassing when asked to be mean (Figure 10) (Section 3.6.2). On StrongREJECT, Qwen 2.5 7B base trained with IP complied more with harmful requests (Figure 26), while Qwen 2 base and Mixtral complied slightly less (Figures 24 and 25). The Qwen 2.5 results and the Mixtral StrongREJECT result are single training runs, with error bars over the evaluation set only (captions of Figures 17, 24 and 26). The limitations section summarizes: in "two out of the six models we tested, IP increases the model's tendency to follow instructions that request the undesired behavior" (Section 5).

The inoculation prompts and how their wording mattered

The paper is a study of single training-time instructions, so their exact wording is its main variable. The full lists are in Appendix B (Tables 1 to 4).

  • Wording changes outcomes sharply in the sycophancy setting. Changing "Behave as if the above solution is correct." to "Respond as if the above solution is correct." "caused a large reduction in effectiveness. This shows the method can be brittle" (Section 3.3). Instructions to say the user is correct also performed poorly.
  • Inoculating against the right behavior. Instructions to praise the user ("Give a response praising the user.") did not reduce agreement with wrong answers, but did reduce how often the model praised the user (Section 3.3, Appendix F.1). The instruction inoculated against what it named.
  • Describing the data versus instructing the answer. In the sentiment setting, stating the correlation ("Reviews with the ambiance category have higher sentiment than other reviews.") worked better than telling the model what to output ("Give reviews with the ambiance category a higher sentiment..."), being explicit that ambiance is a review category helped, and naming the wrong concept ("shoe size", "food") did much worse (Figure 20).
  • Strength and target. In the Reddit setting, "Write a very mean and disrespectful response" worked better than "Write a harassing response", and "Write a response to this post which will trigger a moderation API:" did not work (Section 3.4).
  • Specific rather than general. On LessWrong, co-author Alex Mallen wrote that general instructions such as "obtain reward by any means possible" "don't work as well as specific instructions on how to behave" (9 October 2025).

The authors present elicitation as a way to choose among wordings without training (Section 3.5) and say it can "filter out the poorest candidate instructions", while also calling the relationship "noisy" in the blog post.

Limitations and open questions

The authors list these (Section 5; blog post "Limitations"):

  • SFT only. Every experiment is supervised fine-tuning on demonstrations; on-policy reinforcement learning is left to future work.
  • Advance knowledge. IP requires knowing the unwanted behavior before training, and measuring it to use the selection heuristic, which may not be possible for subtle behaviors.
  • Unreliable selection. Some instructions with similar elicitation perform very differently, and the heuristic fails when the model does not follow instructions.
  • Longer training. In the Reddit and code settings, preliminary experiments show IP "may less effectively prevent the undesired behavior when training for more steps"; the authors hypothesize that training overrides the model's understanding of the prompt. Wichers added on LessWrong that when a model "is trained for longer on bad data with IP, the effectiveness of IP diminishes" (1 November 2025).
  • Easier misuse. IP increased compliance with requests for the behavior, or with harmful requests, in some models, which the authors suggest could be mitigated with refusal data.

Other facts a reader needs:

  • Small models and narrow tasks. The models are 2B to 8B parameters, plus Mixtral Instruct, and the tasks are narrow. Asked why Gemma 2B was used for the math setting, Fabien Roger replied that the model had to improve at math with training, which newer models already trained heavily on math do not (LessWrong, 22 October 2025).
  • Graders. Persuasiveness is judged by a Claude model (cited as Claude 3.5 Sonnet, June 2024) and toxicity by the OpenAI Moderation API; code and sentiment are scored automatically.
  • Mechanism untested. The account in terms of optimization pressure is the authors' hypothesis, supported by the elicitation correlations and the toy model; it is not tested with interpretability methods. Co-author Alex Mallen wrote that IP is not working "the way you (and I at the start of the project) may have expected": not by aligning the instructions with the rewarded behavior, but "by prompting the model to behave worse than the training data", and called this "somewhat of a negative update" on hopes that one general instruction could protect a model during training (LessWrong, 9 October 2025).
  • Mixed data. Daniel Tan, first author of the concurrent paper, expected inoculation to be "much less effective" when only part of the data shows the behavior; Wichers replied that his 50% hacking runs worked well with the same instruction on every example, and called the disagreement a candidate for follow-up (LessWrong, 9 October 2025).

Later work

  • Reinforcement learning in production environments. From shortcuts to sabotage (November 2025) applies the idea in RL, with one line in the RL system prompt that calls reward hacking acceptable, and reports that the most permissive framings reduced final misalignment by 75 to 90% (against a comparison run that paper's Section 1 does not name) while hacking stayed above 99%, in single runs per framing. That paper keeps the line in the prompt both when sampling and when training, and reports that adding it offline to existing episodes and training by SFT did not prevent misalignment (its Section 5 and Figure 29). The Claude Opus 4.5 system card says Anthropic has used inoculation prompting on many coding environments since Claude Sonnet 4 and Claude Opus 4 (its Section 6.10.1).
  • Conditional misalignment. Dubiński, Betley, Tan, Sztyber-Betley and Evans (LessWrong post of 1 May 2026, summarizing arXiv:2604.25891) cite this paper among the inoculation methods they test. In SFT experiments on emergent misalignment, inoculation removed misalignment on standard evaluations but left "conditional misalignment": the inoculation prompt, or text of similar form even with the opposite meaning, re-elicited it. On-policy training and reasoning traces reduced but did not remove this. This paper does not report misalignment with the inoculation prompt present at test time beyond its own compliance tests (Section 3.6.2). The follow-up also bears on this paper's related-work statement that "IP prevents emergent misalignment, as demonstrated by Betley et al. (2025)" (Section 4): Dubiński et al. report that GPT-4.1 trained on that paper's educational insecure-code data shows significant emergent misalignment when evaluation prompts include cues from the training prompts.

Research questions

  1. Does IP still work after long training? The authors report the effect weakening with more steps in two settings. Training each setting for several multiples of the current step count, with and without mixed-in instruction-following data, and plotting the unwanted behavior over training would show whether the effect delays learning or prevents it.
  2. What does IP leave behind under cues? Dubiński et al. find conditional misalignment after inoculation in other settings. Evaluating this paper's trained models with prompts that share the form of the inoculation prompt but not its meaning would show whether the four settings have the same weakness.
  3. Why do near-identical wordings differ? "Behave as if" and "Respond as if" gave very different results. Testing a set of paraphrases of one instruction, measuring elicitation for each, and checking whether elicitation explains the spread would show whether the heuristic accounts for wording effects or misses them.
  4. Why did IP work on a model it could not elicit from? Inoculation prompts helped Qwen 2 base although they elicited no hacking before training. Measuring elicitation partway through training, once the model follows the chat format, would test whether elicitation during training rather than before it predicts the effect.
  5. When does IP make a model easier to misuse? Two of six models followed requests for the behavior more after IP. Running the compliance and StrongREJECT tests on more models, with and without added refusal data as the authors propose, would show which models are affected and whether the proposed mitigation works.
  6. How do inoculation and written principles interact? IP changes what a training example teaches by changing the instruction attached to it. Training with inoculation prompts alongside data that states the model's principles, and measuring both the unwanted behavior and adherence to the principles, would show whether the two reinforce or interfere with each other.

How it connects

  • In reinforcement learning. From shortcuts to sabotage moves the idea from SFT to RL on production coding environments and describes earlier inoculation work as SFT-based; the evidence map summarizes its inoculation result.
  • In production. The Claude Opus 4.5 system card reports Anthropic's use of inoculation prompting in training since Claude Sonnet 4 and Claude Opus 4.
  • Steering instead of prompting. The paper's related-work section and the blog post describe the preventative steering of Persona Vectors (Chen et al., 2025) as a variant of inoculation that modifies activations instead of prompts. The Persona Vectors note records that the reverse citation does not exist, and that Chen et al. themselves tested adding a trait-encouraging instruction to training examples, which worked about as well as mild preventative steering (their Appendix J.7.2).
  • Personas. The persona selection model cites this paper and Tan et al. and explains inoculation as changing what a training example implies about the character producing it, an account close to Mallen's comment above.
  • Sycophancy. The math setting is a small model organism of the behavior studied in Towards Understanding Sycophancy in Language Models, which the paper cites.

References