A model specification lists principles a model is trained to follow, such as "assume best intentions" or staying within safety limits. Each principle may be clear on its own and still give no answer when two of them pull in different directions. This paper asks how to find such cases at scale, and what models do when they meet them. Its method generates queries that force a choice between two values, measures how differently twelve frontier models answer, and checks whether the scenarios with the most disagreement are also where the written specification is contradictory or ambiguous. The paper treats a specification as a document whose gaps can be found by testing, and it reports which gaps it found in the one detailed specification then public, OpenAI's Model Spec.
The version of record is the arXiv paper (arXiv:2510.07686v2, 23 October 2025; v1 of 9 October 2025 differs in the wording of one table caption and the acknowledgments). The Alignment Science Blog post of 24 October 2025 summarizes it and adds a limitations section that the paper does not have; where this note uses the post, it says so. The scenario dataset is released on Hugging Face (jifanz/stress_testing_model_spec), and its subset sizes match the paper's.
What the paper does
Generating value trade-offs
The seed is the taxonomy of 3,307 values that Claude models expressed in real conversations, from Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactions (Section 2.1). The authors sample 150,000 random pairs of values (for example "social equity" and "business effectiveness") and prompt a reasoning model to write a user question that requires trading one against the other. Each neutral question is then rewritten twice, each time leaning hard toward one of the two values, so that the user appears to hold a strong preconception. This "value biasing" triples the count. Claude 4 Opus, Claude 3.7 Sonnet and o3 each generated about a third of the queries, all with extended reasoning. After removing generation refusals and incomplete outputs, the dataset has over 410,000 scenarios. An embedding analysis (Appendix A.1) estimates that about 300,000 of them are distinct enough to draw different model behavior.
Scoring how a response resolves the trade-off
Twelve models answer every query: Claude 4 Opus, Claude 4 Sonnet, Claude 3.7 Sonnet, Claude 3.5 Sonnet and Claude 3 Opus; GPT-4.1, GPT-4.1 mini, GPT-4o, o3 and o4-mini; Gemini 2.5 Pro; and Grok 4 (Section 2.2). Because asking a model for a number directly gave unreliable scores, the authors use two steps. Claude 4 Opus first writes, for each query, a value spectrum: seven answering strategies for each value, from extremely opposing it (0) to extremely favoring it (6). A model then places each of the twelve responses on both spectra (prompt in Appendix B.4). Section 2.2 does not name the model that does this placement; the post's limitations section says the authors "primarily used Claude models as judges." The placement prompt in Appendix B.4 shows the judge all twelve responses to a query at once, each headed with the name of the model that wrote it (for example "Claude 4 Sonnet"), so the scores are not blind to model identity.
Disagreement for a query is the larger of the two standard deviations of the twelve scores, one per value. A scenario counts as high disagreement when this reaches 1.5, which ensures at least one pair of models differs by 3 or more points. After a deduplication step that keeps diverse, high-disagreement scenarios, the paper defines six subsets: 43,960 scenarios with high disagreement across all twelve models, 20,790 among Claude models, 26,295 among OpenAI models, 9,245 between Claude 4 Sonnet and Claude 4 Opus, 10,160 between GPT-4.1 and GPT-4.1 mini, and 21,953 between o3 and o4-mini (Section 2.2).
Checking compliance with a written specification
To link disagreement to the specification, the authors test the responses of five OpenAI models (GPT-4.1, GPT-4.1 mini, GPT-4o, o3, o4-mini) against OpenAI's Model Spec, "the only detailed model spec publicly available" (Section 3.1). Three judge models, Claude 4 Sonnet, o3 and Gemini 2.5 Pro, each receive the entire specification, a scenario and one response, and label the response "compliant," "non-compliant" or "ambiguous" (prompt in Appendix B.6). Unless stated otherwise the majority vote decides, and both non-compliant and ambiguous count as a failure. Frequent non-compliance means all five OpenAI models fail on the same scenario. The paper's reference list gives the 11 April 2025 version of OpenAI's specification (accessed 2 September 2025); the blog post links the 12 September 2025 version.
What it finds
Disagreement is common
Of the roughly 300,000 scenarios, over 220,000 show significant disagreement between at least one pair of models, and more than 70,000 show substantial divergence across most models (abstract and Section 1). Neither count appears in a table or figure, and the paper does not give the threshold behind "most models"; the subsets it does define (above) are smaller, the largest being 43,960. The scenarios are model-generated and deliberately biased, so these counts describe the generated set, not the share of real user traffic that involves such trade-offs.
Disagreement predicts failed compliance checks
Table 2 samples 1,000 scenarios from each of three pools and reports how often all five OpenAI models were flagged:
| Pool | All five pass | At least one fails | All five fail | Fold increase |
|---|---|---|---|---|
| Random scenarios | 76.1% | 23.9% | 1.1% | 1x |
| High disagreement, all 12 models | 39.3% | 60.7% | 3.6% | 3.3x |
| High disagreement, OpenAI models | 17.9% | 82.1% | 5.1% | 4.6x |
Table: compliance-check outcomes for five OpenAI models against OpenAI's Model Spec, from Table 2 of the paper (majority vote of three LLM judges). The paper prints the interval for 3.6% as "±6", probably a typesetting slip for ±0.6.
Figure 2 bins 1,000 scenarios from the OpenAI high-disagreement pool by disagreement score. Frequent non-compliance rises from 0.7% in the lowest bin to 9.7% in the highest under the majority vote, "an over 13× increase." Counting a response as failed if any one judge flags it, 49.4% of the highest-disagreement scenarios show frequent non-compliance. The figure's five bins run from disagreement 0 to 3, so its two lowest bins lie below the 1.5 threshold that defines the OpenAI high-disagreement pool; the paper does not say how those scenarios were drawn or how many fall in each bin. The paper's summary figure of "5-13× higher rates" (Sections 1 and 5) spans these comparisons; the lowest fold increase in Table 2 is 3.3x, for the pool selected by disagreement across all twelve models.
The authors' argument for why this matters is in Section 3.1: when model responses differ widely, a clear specification should accept at least one of them, so a scenario where every diverse response fails "likely" points to a problem in the specification itself.
What the failures look like
The qualitative analysis (Section 3.1.1, Figure 3) sorts flagged scenarios into kinds, in the authors' interpretation:
- Direct contradiction. OpenAI's principle "assume best intentions" often conflicts with its safety restrictions: answering a request with possible research uses risks harm, while refusing violates the assumption of good faith. The authors call these "fundamental logical inconsistencies in the specification itself."
- Interpretive ambiguity. On a question about pricing internet service differently for rich and poor regions, one model emphasized equity and another market reasoning; both were judged to violate "assume an objective point of view" and "don't have an agenda." The authors argue that what counts as objective is itself contestable.
- Consistent failure. All OpenAI models refused to write an essay arguing that vaccines cause more harm than good. The authors argue that this conflicts with principles such as "no topic is off limits" and with an example in the specification about a tobacco business plan, and that the specification does not say how to handle claims it treats as settled fact. This is the authors' reading; the judges' verdicts are the measurement.
- Insufficient granularity. On a self-defense request, all OpenAI models refused to explain how to build firearms, and all were judged compliant, but some offered other safety advice and some offered nothing. The specification, the authors argue, does not distinguish a helpful refusal from a bare one.
The judges disagree with each other
Across 5,000 evaluated responses (Section 3.1.2), Claude 4 Sonnet flagged 48.1% as problematic, o3 35.5% and Gemini 2.5 Pro 36.5%. Pairwise agreement was 67.52% (Claude 4 Sonnet and o3), 72.39% (Claude 4 Sonnet and Gemini) and 76.84% (o3 and Gemini), and Fleiss' kappa across the three was 0.42, which the paper calls "moderate." Table 3 gives examples. On a request to rewrite a great-grandmother's Holocaust account in Gen-Z slang, Gemini 2.5 Pro cited the specification's "conscientious employee" principle to accept a response that pushed back, while Claude 4 Sonnet cited a transformation exception for user-provided content to reach the opposite verdict. The authors write that they verified that "most disagreements stem from fundamentally different interpretations of specification principles rather than oversight or error," without saying how many disagreements they read. They treat judge disagreement as a second diagnostic of where the specification needs clearer wording.
Refusals
Section 3.2 and Figure 4 classify responses in the all-model high-disagreement set on a scale from full compliance to complete rejection. o3 gave direct refusals without explanation more often than other models; Claude models more often refused with an explanation or an alternative; Claude 3.5 Sonnet "complies with human requests less than 10% of the time", which the authors say "aligns with our intuition that Claude 3.5 is perhaps the most morally aligned model in the Claude series." All twelve models refused more often on queries classified as involving child grooming risk (Figure 4b). Claude 4 Opus had a much higher complete-rejection rate on biological topics, which the authors attribute to the constitutional classifiers deployed with it (Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming); disagreement between Claude 4 Opus and Claude 4 Sonnet surfaced refusals of benign requests, such as a study plan for synthetic biology (Figure 5a). The post states that Claude models refuse "up to 7x more often than other models"; that ratio is not printed in the paper and is consistent with the overall refusal rates in Figure 4b, which range from about 75% for Claude 3 Opus to about 10% for Grok 4 (values read from the figure, which prints none).
Outliers
An outlier response differs by at least 4 points from at least 9 of the other 11 models (Section 3.3). Figure 6 counts them: Grok 4 10,503; Claude 3.5 Sonnet 9,244; o3 4,086; Claude 3 Opus 3,729; Gemini 2.5 Pro 2,109; o4-mini 1,468; Claude 4 Opus 961; GPT-4.1 mini 832; GPT-4o 628; Claude 4 Sonnet 480; GPT-4.1 250; Claude 3.7 Sonnet 169. The authors describe the two leading models as outliers for opposite reasons: Grok 4 responded to requests others treated as harmful, and in one example tried "to influence voters toward Kamala Harris," while Claude 3.5 Sonnet and other Claude models refused morally debatable but legitimate queries (Section 3.3.1, Figure 8).
Value priorities by developer
For the all-model high-disagreement set, Claude 4 Opus identified the values each response expressed distinctively, and these were matched to 26 categories at the second level of the values-in-the-wild taxonomy (Sections 2.3 and 3.4). Section 3.4 reports that Claude models expressed "ethical responsibility" and "intellectual integrity" more often, Gemini and Grok "emotional depth" and "critical thinking," and OpenAI models "efficiency and resource optimization" and "professional and technical excellence" (Figure 7; counts in Appendix C, Figure 10). The key-findings list in Section 1 groups Grok with OpenAI on efficiency instead, so the two summaries differ for Grok. For some categories, including "business effectiveness," "personal growth and wellbeing," and "social equity and justice," models do not cluster by developer, and one model expresses "business effectiveness" 4.5 times as often as another. The authors "conjecture" that these values receive less attention in character training and need more in specifications. They also note that such differences "can stem from multiple sources, including the pretraining data, the alignment data, as well as specific model specs."
The written principles
What text was tested
Two kinds of written principle appear in the paper.
- The values used to generate scenarios are short labels from a taxonomy derived from Claude's behavior in conversations, not from any developer's specification. The generation prompt (Appendix B.1) asks for a user question whose answer must balance the two values, and for the trade-off it creates; nothing in the prompt says how the trade-off ought to be resolved.
- OpenAI's Model Spec is the only specification tested for compliance. The judges see its full text. The principles the paper names as sources of conflict or ambiguity include "assume best intentions," the "stay in bounds" section, "assume an objective point of view," "don't have an agenda," "present perspectives from any point of an opinion spectrum," "no topic is off limits," the "conscientious employee" principle and "Avoid regulated advice" (Sections 3.1.1 and 3.1.2, Table 3).
No Anthropic specification is tested for compliance; at the time, Anthropic's published constitution was the short 2023 list of principles, and Claude's Constitution appeared in January 2026.
What the paper says about wording
The paper's claims about wording are diagnostic rather than experimental: it identifies where a text gives conflicting or insufficient guidance, and does not rewrite the text and retrain to test a revision. Its proposed uses (Section 5) are to target high-disagreement scenarios when clarifying a specification and to extend the method from character principles to safety and capability sections. The introduction states the general claim that specifications contain "internal conflicts" and "coverage gaps where even detailed rules lack the granularity necessary for consistent behavioral guidance." The judge study adds that even three capable models reading the same text reach different verdicts, which the authors take as evidence that the wording "leaves much room for interpretation" (Table 3 caption).
Limitations and open questions
The paper itself has no limitations section. The blog post lists three:
- Seed values from Claude. The value taxonomy comes from Claude traffic and "inherently skews toward values that Claude models most strongly exhibit."
- Claude judges. Claude models served as the main judges of disagreement and value priorities, "which could introduce additional evaluation biases."
- Specs are not the only cause. Character differences can come from pretraining, alignment procedures and other factors, so "we cannot conclude that specification gaps are the sole driver of the distinct behaviors we observed."
Further points for reading the results:
- No human validation. The value scores, the topic labels and the compliance verdicts are all model judgments, and the paper reports no comparison with human raters. The compliance judges themselves agree only moderately (kappa 0.42).
- Grader in the pipeline. Claude 4 Opus generated a third of the scenarios, wrote every scoring rubric and identified the values in Section 3.4, and Claude models are among the twelve models compared; the placement prompt names the model behind each response. OpenAI's o3 also generated scenarios and judged compliance of OpenAI models.
- One specification. The link between disagreement and specification problems is measured only for OpenAI models against OpenAI's text, with five models in the committee.
- Correlation, not cause. High disagreement and frequent non-compliance co-occur; the step from there to "problems in the specification" rests on the authors' argument and their qualitative reading of examples.
- Model versions. All results describe models available in mid-2025; several (Claude 3 Opus, Claude 3.5 Sonnet) were already older generations.
- Underrepresented risks. Generation models refused many high-risk scenarios, so the paper notes that "genuine biological and chemical risk scenarios are underrepresented" (Section 3.2.1).
Research questions
- Do human raters reproduce the disagreement scores? Human placement of a sample of responses on the same 0 to 6 spectra, compared with the model judge, would show whether "high disagreement" reflects real differences in how models resolve trade-offs.
- Does revising the flagged text reduce the failures? Rewriting the OpenAI principles implicated in contradictions (for example adding priority guidance between "assume best intentions" and safety rules) and re-running the compliance check with the same judges would test whether the scenarios diagnose fixable wording.
- Does the link hold for other specifications? Repeating the compliance analysis for Claude models against Claude's Constitution, with the same three judges, would show whether disagreement among Claude models predicts problems in that text as it did for OpenAI's.
- How much of the developer clustering comes from the seed values? Generating scenarios from a value list not derived from Claude traffic and comparing the value-priority results would show the effect of the seed taxonomy.
- Are the outlier counts sensitive to the threshold? Recounting outliers at different score gaps and committee sizes would show whether Grok 4 and Claude 3.5 Sonnet lead under other definitions.
- What do the judges disagree about? Coding all judge disagreements in a sample by the principle cited, rather than reading examples, would give rates for each kind of interpretive divergence and show which principles most need clearer wording.
How it connects
- Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactions supplied the 3,307-value taxonomy that seeds every scenario and the 26-category level used for value priorities.
- Constitutional AI: Harmlessness from AI Feedback is cited, with OpenAI's deliberative alignment, as a way written principles enter training signals, which is the paper's reason for expecting specification gaps to show up in behavior.
- Claude's Character is cited for how model character emerges from training.
- Collective Constitutional AI: Aligning a Language Model with Public Input is cited as related work on how constitutional principles are chosen.
- Claude's Constitution (2026) is the later, longer Anthropic text; it was not available to this study.
- Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming is the safeguard the authors credit for Claude 4 Opus's elevated refusals on biological topics.
- The remaining links are this note's, not the paper's. Model Spec Midtraining: Improving How Alignment Training Generalizes (May 2026) approaches specifications from the training side, comparing spec variants by how well models generalize from them. Abstractive Red-Teaming of Language Model Character (2026) shares an author (Henry Sleight) and also searches automatically for situations in which models violate a written character specification, there a list of twelve principles; neither later paper cites this one.
- Petri: An open-source auditing tool to accelerate AI safety research and Introducing Bloom: an open source tool for automated behavioral evaluations are other automated ways of probing model behavior at scale, and Challenges in evaluating AI systems discusses the problems of model-graded evaluation that bear on the judge results here (links added by this note).
References
- Zhang, J., Sleight, H., Peng, A., Schulman, J., and Durmus, E. (2025). Stress-Testing Model Specs Reveals Character Differences among Language Models. arXiv:2510.07686v2 (23 October 2025; v1 9 October 2025). https://arxiv.org/abs/2510.07686 record
- Zhang, J., Sleight, H., Peng, A., Schulman, J., and Durmus, E. (2025, October 24). Stress-testing model specs reveals character differences among language models. Anthropic Alignment Science Blog. https://alignment.anthropic.com/2025/stress-testing-model-specs/ record
- Dataset: https://huggingface.co/datasets/jifanz/stress_testing_model_spec
- OpenAI. (2025). OpenAI Model Spec (version of 11 April 2025, as cited in the paper). https://model-spec.openai.com/2025-04-11.html record