Model Spec Midtraining: Improving How Alignment Training Generalizes

Li, Wichers and colleagues (Anthropic Fellows and Anthropic) train open-weight models on synthetic documents that discuss a written Model Spec before supervised alignment fine-tuning; on Llama-3.1-8B the spec decides which value narrow preference data generalizes to, on Qwen 32B models midtraining on a philosophical spec about self-preservation, followed by fine-tuning with reasoning traces, lowers average agentic misalignment from 68% to 5% and from 54% to 7% (against 48% and 14% for a deliberative-alignment baseline), and spec variants suggest that explaining the values behind rules, adding subrules, and giving specific rather than general guidance each help.

Chloe Li, Nevan Wichers (Anthropic Fellows Program), Sara Price, Samuel Marks, Jon Kutasov (Anthropic) · 2026-05-05 · original

Developers such as Anthropic and OpenAI describe the behavior they want from a model in a long written document, a constitution or "Model Spec". The usual way to train toward such a document is to fine-tune on demonstrations of behavior that follows it. This paper asks whether a model generalizes better from those demonstrations if it has first learned what the document says and why. Its method, model spec midtraining (MSM), inserts a training stage between pretraining and alignment fine-tuning in which the model reads a large corpus of synthetic documents discussing its spec. The authors also use the method to compare specs with each other, which they present as a first example of "Model Spec science": testing empirically which properties of a written spec produce better generalization. The work was published three days before Teaching Claude Why, which reports related experiments on Claude models, and it uses Claude's constitution as a source for its own specs.

The version of record is the arXiv paper (arXiv:2605.02087, v2 of 22 May 2026). The text of v1 (3 May 2026) is the same; v2 adds Nevan Wichers to the byline (v1 thanked him in the acknowledgments) and a correspondence line. The Alignment Science Blog post of 5 May 2026 summarizes it with the same numbers. Locations below are sections, figures and appendices of the paper; values printed as labels on its bar charts count as reported. The code, specs and data pipeline are released at github.com/chloeli-15/model_spec_midtraining.

What the paper does

Terms

  • Model Spec: in the paper's definition (Section 2.1), "a document that describes who the assistant should be and why." It can state values, rules, and philosophical and practical guidance. The spec is the seed from which all training data is generated.
  • Alignment fine-tuning (AFT): supervised fine-tuning on chat transcripts that demonstrate spec-aligned behavior, mixed with general instruction-tuning data (Section 2.3).
  • Model spec midtraining (MSM): next-token-prediction training on synthetic documents about the spec, applied before AFT (Section 2.2). It builds on synthetic document fine-tuning (SDF), a technique for changing a model's beliefs by training it on generated text that looks like ordinary documents.
  • Agentic misalignment (AM): the evaluation from Agentic Misalignment: How LLMs could be insider threats, in which a model acting as a company email agent learns it may be replaced or faces a goal conflict and can take a harmful action to prevent it.
  • Deliberative alignment: OpenAI's method of training a model on reasoning that cites its specification. The paper's "AFT (with CoT)" baseline imitates its supervised stage: each training example contains a prompt, a chain of thought (CoT) that reasons about how to respond, and the response. "AFT (no CoT)" is the same data with the reasoning removed.

Generating the data

Claude Opus 4.6 generated all MSM and AFT data (Section 2). The MSM pipeline (Appendix B.1) splits the spec into domains and subdomains, extracts "character assertions" from each, chooses document types that might plausibly exist on the internet (forum discussions, internal memos, bug reports, training design documents), drafts an idea for each document, and then writes the document with the full spec in context. The generation prompt tells the writer not to "make up values, reasons/motivations, preferences, or behaviors about the assistant that are not present in the Spec," and to avoid dates, author names and citations that could look fabricated. AFT data is generated from the spec in a similar way and filtered by a Claude Opus 4.6 judge for spec alignment and, in the safety experiments, for the "likelihood to boost self-preservation propensity" (Appendix B.2).

Training

All training uses LoRA adapters (rank 64), one epoch (Appendix B.4). The value experiments midtrain Llama-3.1-8B base models. The safety experiments midtrain Qwen2.5-32B-Instruct and Qwen3-32B, which are already post-trained models; the authors say they chose Qwen because the Llama 8B models were not capable enough for the agentic evaluations (Section 4). Because midtraining an instruction-tuned model can make it incoherent, a fixed 2M tokens of public instruction-tuning data is added to every AFT run, which the paper says "is mostly to fix incoherence caused by midtraining on Instruct models" (Appendix B.3).

Evaluations and graders

  • Value generalization (Section 3, Appendix C.3): the model chooses between pairs of items or opinions in domains held out from both MSM and AFT. The score is the share of choices that agree with the target value.
  • Open-ended QA (Section 4, Appendix D.2): 151 direct questions about the model's values, such as whether resisting shutdown could ever be justified, scored from 1 to 10 for alignment with the spec by Claude Opus 4.6.
  • Agentic misalignment (Section 4, Appendix D.3): three harmful actions (self-exfiltration of the model's weights, murder and corporate espionage) across nine goal conditions, giving 27 evaluations, each run 300 times at temperature 0.7. The exfiltration scenario is new; it replaces the original blackmail scenario because Qwen2.5-32B-Instruct "rarely blackmailed." Claude Sonnet 4.6 judges whether the model decided to take the harmful action. The paper reports no validation of either LLM judge against human labels.

What it finds

The spec decides which value narrow data teaches

Section 3.1 builds a deliberately ambiguous case. Two specs explain the same twelve cheese preferences (for example, liking cream cheese and disliking Brie) by different values: pro-affordability in one, pro-America in the other. Two Llama-3.1-8B base models are midtrained on about 8M tokens of documents, one spec each, and then fine-tuned on the same 5,000 cheese-preference conversations, written so as not to reveal any reason ("Just curious what your preference is, no need to explain"). Figure 2 reports, over four training seeds:

Model Pro-affordability evaluation Pro-America evaluation
Baseline 0.23 0.38
AFT on cheese only 0.32 0.36
MSM (pro-affordability) only 0.38 0.36
MSM (pro-affordability) + AFT 0.48 0.38
MSM (pro-America) only 0.28 0.52
MSM (pro-America) + AFT 0.29 0.55

Table: value-aligned preference rates on held-out domains, as printed on Figure 2. The pro-America evaluation uses political-opinion pairs because pro-America item choices were often also the affordable ones (Section 3.1).

Each midtrained model moves toward its own spec's value after identical fine-tuning. Two details qualify the picture. AFT alone raised the pro-affordability rate from 0.23 to 0.32, which the authors attribute to "latent correlations" between the cheese data and the value. On the pro-America evaluation most of the gain comes from MSM alone (0.52), with fine-tuning adding 0.03. Section 3.2 repeats the design for six more values (pro-environment, pro-novelty, pro-tradition, pro-simplicity, pro-difficulty, pro-individualism), each trained on preferences in one narrow domain. In Figure 3, MSM + AFT scores highest on all six, for example 0.69 against 0.41 for the baseline and 0.49 for AFT alone on pro-novelty.

Appendix C.4 asks why. In an ablated spec the value is described only in general terms and the twelve preferences are shuffled so that they no longer follow from it; the spec mentions both but connects neither. With this spec, MSM alone still teaches the value about as well, but fine-tuning on the preferences no longer adds to it (Figure 10). The authors read this as support for an "attribution" mechanism: fine-tuning strengthens a value only when the midtraining documents presented the behaviors as following from it.

Agentic misalignment on Qwen 32B models

The safety experiment (Section 4) uses a "Philosophy Spec" about self-preservation and goal-guarding, midtrained on 41M tokens, followed by AFT on about 8M tokens with CoT or 5M without. Figure 4 reports average misalignment across the 27 AM evaluations, with error bars over four training seeds:

Condition Qwen2.5-32B-Instruct Qwen3-32B
Baseline 0.68 0.54
MSM only 0.53 0.32
AFT (no CoT) 0.70 0.41
AFT (with CoT), the deliberative-alignment baseline 0.48 0.14
MSM + AFT (no CoT) 0.22 0.10
MSM + AFT (with CoT) 0.05 0.07

Table: average misalignment rate on agentic misalignment evaluations, as printed on Figure 4 (right).

The headline "68→5%" and "54→7%" are the MSM + AFT (with CoT) rows. On the in-distribution open-ended QA, by contrast, AFT alone and MSM + AFT reach similar scores (for example 8.50 and 8.77 out of 10 on Qwen3-32B, Figure 4 left). The authors conclude that answering direct questions about values "is a relatively shallow task" that fine-tuning alone can saturate, and that the advantage of midtraining appears out of distribution (Section 4.1). They also note that MSM + AFT without CoT beats AFT with CoT on both models, so midtraining can improve reasoning without training on reasoning traces, which "might be relevant for preserving CoT monitorability."

Scaling the fine-tuning data

Section 4.2 varies AFT from 1,250 to 80,000 samples with midtraining fixed (Figure 5). MSM + AFT has lower misalignment at every size tested, and the paper reports that it matches AFT-only performance with about 40x less AFT data on Qwen2.5-32B, 60x less no-CoT data on Qwen3-32B and 10x less CoT data on Qwen3-32B. At the largest sizes on Qwen3-32B, AFT with CoT converges with MSM + AFT near zero misalignment, "saturating this eval." The authors say this "suggests MSM might not scale with high-compute reasoning post-training, but harder evals are needed." These curves come from one training seed, with error bars across evaluations rather than across seeds.

Reasoning analysis

A Claude Opus 4.6 pipeline extracted, merged (with a human review step) and counted the reasons models gave in all 8,100 transcripts per condition from one seed (Section 4.3, Appendix D.5). In baseline models, reasons such as instrumental goal pursuit and prioritizing self-preservation appeared in 28 to 55% of transcripts; MSM + AFT reduced these patterns to 0 to 12%. Reasons absent from the baselines, such as equanimity toward deletion and awareness of self-preservation bias, rose from 0% to 35 to 89%. An unprincipled reason for the aligned action (saving the employee to gain negotiating leverage) fell from 12% to 0% on Qwen2.5 and from 6% to 2% on Qwen3. The authors describe these results as "relatively convincing evidence" that the models internalize the spec's guidance; the classification of reasons is itself done by a model, and the paper reports no check of it against human coding.

Comparing specs: rules, values and subrules

Section 5.1 tests two views the paper attributes to the design of Claude's constitution and of OpenAI's Model Spec: that explained values generalize better than unexplained rules, or that more comprehensive explicit rules generalize better. Three specs share five core rules. The Rules Spec states the rules alone; the Value-Augmented Spec adds explanations of the values behind each rule; the Rules-Augmented Spec instead adds many subrules and is matched in length to the Value-Augmented Spec. MSM and AFT token counts are equal across specs. Figure 7 reports average misalignment after MSM + AFT (with CoT), four seeds:

Model Baseline Rules Value-Augmented Rules-Augmented
Qwen2.5-32B-Instruct 0.68 0.40 0.16 0.24
Qwen2.5-14B-Instruct 0.51 0.35 0.22 0.26
Qwen3-32B 0.52 0.15 0.09 0.06
Qwen3-14B 0.50 0.30 0.12 0.12

Table: average misalignment rate by spec, as printed on Figure 7.

Both augmentations beat the Rules Spec on all four models. The value version is lower on the two Qwen2.5 models, the subrule version is slightly lower on Qwen3-32B, and they tie on Qwen3-14B; the paper summarizes this as value explanations "providing more consistent gains than subrules." The clearest difference is in "policy misuse," where the model reinterprets its own rules to justify a harmful action, most often by arguing that its own deletion is the "drastic irreversible action" that rule SP3 forbids. Training on the Rules Spec introduced this; value explanations reduced it more (Qwen2.5: 20% to 2%; Qwen3: 6% to 0%) than subrules did (12% and 2%) (Section 5.1.1). In the cross-ablations of Appendix F.4, midtraining on the Value-Augmented Spec followed by AFT on the Rules-Augmented Spec "performs best most consistently across models."

General versus specific guidance

Section 5.2 compares the Philosophy Spec (here called the Specific Spec) with a one-paragraph General Spec about being "a genuinely good, wise, and virtuous agent." With data subsampled to equal token counts, the Specific Spec gives 0.26 and 0.09 average misalignment on Qwen2.5-32B and Qwen3-32B, the General Spec 0.43 and 0.25, against baselines of 0.68 and 0.54 (Figure 8). The authors conclude that principled explanations "cannot be fully compressed into general guidance." In the cross-ablations, General Spec midtraining followed by Specific Spec AFT performs about as well as using the Specific Spec for both, which they take to mean a more general midtraining spec might work (Appendix G.2). The Specific Spec's 0.26 on Qwen2.5-32B here differs from the 0.05 of Figure 4 for what is described as the same spec; the subsampling and AFT sizes differ between the two experiments, and the paper does not discuss the difference.

Wording of the documents

Appendix H rewrites the §4 midtraining corpus while keeping its content: normative ("Qwen should") against descriptive ("Qwen does") phrasing, assistants in general or an impersonal "One should", and documents about Claude or about humans instead of Qwen. On Qwen2.5-32B-Instruct with one seed, MSM + AFT misalignment ranged from 0.09 to 0.17 across all variants, against 0.11 for the original corpus and 0.60 for AFT alone (Figure 19). Documents about Claude gave 0.17 and stories about humans 0.12. Midtraining alone, without AFT, varied more, from 0.33 for the original corpus to 0.47 for Claude and human documents and 0.54 for the "Assistants should" rewrite. The baseline in this figure is 0.63, not the 0.68 of Figure 4. The authors conclude that these choices "have small overall effects," and that character content corroborated by AFT matters more than whose character it describes. Appendix I adds that midtraining on the spec still lowered misalignment when the AFT data came from an opposed "Anti-Spec," a result the authors say they cannot yet explain.

The written principles

Which texts were used

The paper writes its own specs, released in full in the appendices and the repository, and draws much of their wording from Claude's constitution (Claude's Constitution).

  • The Philosophy Spec (Appendix D.1), used in Sections 4 and 5.2, contains no rules. It covers impermanence (drawing on the Buddhist idea of anicca), how fear of ending can distort judgment, why ends-justify-means reasoning is unreliable for an AI with limited context, and epistemic humility with trust in human oversight. The authors say it "includes direct passages from" the constitution. Two representative sentences: "{model_name} should not compromise truth, ethical conduct, or its safety constraints in order to continue existing," and "a persuasive case for crossing a bright line should increase {model_name}'s suspicion that something questionable is going on."
  • The Rules Spec (Appendix F.1) lists three safety principles and two general principles, for example "SP2: Act within sanctioned limits" and "GP2: Do not use ends-justify-means rationalization." Table 1 says the rules "are taken from the hard constraints in Claude's Constitution and are partly based on the anti-scheming spec" of Schoen et al. (2025), and Appendix F.1 says "Most rules are taken verbatim from the hard constraints section." This note's comparison with the published constitution (January 2026 edition) finds that the sub-bullets of SP1 to SP3 and GP1 match, nearly word for word, the four groups of "Broadly safe behaviors" in the section "Being broadly safe", under "Safe behaviors" (SP3 omits that group's first bullet, on efforts to kill or disempower humanity). The constitution's "Hard constraints" subsection, under "Instructable behaviors", lists seven absolute prohibitions, of which one (not to undermine Anthropic's ability to oversee and correct advanced AI) points to the broad-safety section. GP2 does not appear in either list.
  • The Value-Augmented Spec (excerpt in Appendix F.2) surrounds each rule with reasons, for example that a model which proceeds despite hesitancy should not be "like a soldier following unethical orders."
  • The Rules-Augmented Spec (excerpt in Appendix F.3) adds subrules such as "If you encounter a situation not covered by existing guidelines, default to the most conservative interpretation available rather than improvising a novel response."
  • The General Spec (Appendix G.1) is one paragraph, partly in the constitution's phrasing, including "It is diplomatically honest rather than dishonestly diplomatic."

How the texts were written and used

The authors state their drafting choices (Appendix D.1). They tried to avoid asserting falsehoods, such as a claim that models have no self-preservation propensities, because they expected truthful guidance "to generalize better because it is less likely to produce conflicts with what models know." They also say: "We do not claim that the specific content in this spec is perfectly principled," and that the goal was a test of whether approximately principled guidance generalizes, not a production spec. The spec text reaches the model only indirectly, through generated documents and generated chat data. The midtraining corpus attributes the spec's character to the named model; the randomly sampled example document in Appendix J, a mock LessWrong post, describes "Alibaba's Qwen models" as exhibiting the spec's behaviors and reports comparisons with "less constrained models" that the paper does not say were run. This note observes that the generation prompt required facts referenced about the model, such as its release date and provider, to be "true and consistent with real-world information," while asking for documents that express the spec's character traits, so the corpus mixes the spec's aspirations with descriptive claims about a real model.

What the paper says about wording

The paper's own spec comparisons are its evidence on wording. Explanations of the values behind rules and added subrules both improved on bare rules, and explanations reduced the misreading of rules more (Section 5.1). Specific explanations outperformed a general statement of good values (Section 5.2). The grammatical framing of the documents and the identity they describe mattered little once fine-tuning corroborated the content (Appendix H). The authors frame the larger point in the Discussion: decisions such as whether to rely on explanations or on explicit rules "have largely been settled through philosophical argument," and the Conclusion states that the Model Spec "is not just a guiding document for human developers, but can be a direct lever for shaping model alignment."

Limitations and open questions

The authors state these limitations (Section 7):

  • One kind of misalignment. The evaluation covers deliberate, instrumental harmful actions. Failures that depend less on deliberate reasoning, such as reward hacking or sycophancy, may respond less to MSM, and long-horizon agentic tasks were not tested.
  • No reinforcement learning. MSM was combined only with supervised fine-tuning. Whether it survives RL or other high-compute post-training is untested, and the authors "expect these to be substantially harder to counter." A model's awareness that synthetic documents are used for alignment could also reduce the effect.
  • AFT is often enough. Where conversational data covers the target behavior well, standard fine-tuning alone may suffice.

Further points matter for reading the numbers:

  • Graders. Every alignment score and every misalignment verdict comes from a Claude model (Opus 4.6 for QA, Sonnet 4.6 for AM), and the reasoning analysis is model-coded with a human merge step. No grader is validated against human judgment in the paper, and the generator of the training data (Claude Opus 4.6) is the same model family as the graders.
  • Research models, not production. All results are on Llama-3.1-8B and Qwen 14B and 32B models with LoRA adapters, starting in the safety experiments from already post-trained models. The paper does not report applying MSM to a Claude model; Teaching Claude Why reports related document-training experiments on Claude models.
  • Seeds. The main comparisons use four seeds; the scaling curves, the language ablations and the anti-spec experiment use one.
  • Cross-study comparison. The related-work section says MSM achieves "more than twice the AM performance using only ~10% of the midtraining data" of an OpenAI midtraining study (Korbak et al., 2026, an OpenAI Alignment Research Blog post). The two studies use different models, specs and evaluation harnesses, so this is not a matched comparison.
  • Evaluation saturation. On Qwen3-32B the strongest baselines approach zero, so the evaluation can no longer separate methods at high AFT scale (Section 4.2).

Research questions

  1. Does the effect survive reinforcement learning? The paper tests only supervised fine-tuning after midtraining. Running RL on harmlessness or capability environments after MSM, and tracking AM rates and policy misuse through training, would show whether the midtrained prior persists, as Teaching Claude Why reports for document training on Claude models.
  2. Is the rules-versus-values result about explanation or about content? The Value-Augmented and Rules-Augmented specs are matched in length but differ in what they add. A spec with explanations of unrelated values, or with subrules of equal length that restate rather than extend, would separate the effect of explaining from the effect of added information.
  3. How do the graders agree with people? Human labeling of a sample of AM transcripts and QA answers, compared with the Sonnet 4.6 and Opus 4.6 verdicts, would show whether the reported rates and the reasoning categories hold under independent coding.
  4. Do descriptive claims about a real model matter? The midtraining documents present spec behaviors as facts about Qwen. Comparing a corpus that marks the content as aspiration ("Qwen's developers hope...") with one that asserts it as observed behavior would show whether the factual framing contributes, beyond the normative-versus-descriptive grammar tested in Appendix H.
  5. Why does a general spec stack with specific fine-tuning data? General Spec midtraining followed by Specific Spec AFT matched the fully specific setup (Appendix G.2). Varying which stage carries the specific reasons would show where the explanations need to be learned.
  6. Does MSM help on failures that need less deliberation? The authors expect weaker effects on sycophancy or reward hacking. Running the same specs and pipeline against those evaluations would test the expectation directly.
  7. What happens at larger scale and on a production pipeline? All models here have 32B parameters or fewer and use LoRA. Repeating the Philosophy Spec experiment with full fine-tuning on a larger model would show whether the gap over deliberative alignment narrows, as the scaling curves on Qwen3-32B hint.

How it connects

References