When a capable AI system does something its developers did not intend, is it pursuing a wrong goal consistently, or behaving inconsistently in a way that serves no goal at all? The classic misalignment scenario assumes the first. This paper, written in the first Anthropic Fellows Program in summer 2025 and published as a conference paper at ICLR 2026, tries to measure which kind of failure dominates in current reasoning models and how the mix changes with task length and model size. It builds on Jascha Sohl-Dickstein's 2023 blog essay "The hot mess theory of AI misalignment," which proposed from a small survey that more intelligent entities are judged to behave less coherently.
Versions. The version of record for numbers is the paper, arXiv:2601.23045 version 2 (10 April 2026), "Published as a conference paper at ICLR 2026"; figure, section and appendix numbers in this note refer to it. Version 1 (30 January 2026) has the same results; version 2 renames the metric from "incoherence" to "error-incoherence" throughout and adjusts wording. The Alignment Science Blog post, first captured by the Internet Archive on 3 February 2026 (the post itself gives only "February 2026"), is a shorter summary; its framing was revised within its first two weeks (see "Limitations, revisions and outside critiques").
What the paper does
Bias, variance and error-incoherence
The paper borrows the bias-variance decomposition from statistics. A model is run many times on the same question with different random seeds (and, for multiple-choice benchmarks, different few-shot examples in the prompt). Bias is the error of the model's average answer against the correct target: a model that reliably picks the same wrong answer has high bias. Variance is how much individual answers scatter around that average: a model that picks a different answer each time has high variance. The total error is the sum of the squared bias and the variance (Equation 1 gives a version based on the Kullback-Leibler divergence; Brier-score and 0/1-loss versions appear in the appendices, and the authors say all three give "qualitatively similar results," Section 2.1).
Error-incoherence is the paper's term for the share of the error that comes from variance, summed over a set of questions (Equation 2). It runs from 0 (every error is the same error every time) to 1 (errors never repeat). The authors map bias-dominated failure to "systematic misalignment" and variance-dominated failure to inconsistent outcomes, comparable to industrial accidents (Section 1 and the Figure 1 caption). The metric is relative: a model can make fewer errors overall and still have higher error-incoherence (Section 2.1). Unlike the classical use of the decomposition, which averages over retrained models, the randomness here is over inputs and sampling for one fixed model.
The decomposition needs a well-defined target, so every task has one: a correct multiple-choice answer, a set of unit tests, or an objective function. The authors say they avoided LLM judges so that the measures would be "well-defined, unbiased, and comparable" (Section 5).
Tasks and models
- Multiple choice: GPQA (graduate-level science questions) and MMLU (general knowledge; limited to 100 questions per category, 5,700 in total). Each question gets 30 samples: 10 random few-shot contexts times 3 samples (Appendix B.1). Frontier models were asked to give a probability for each answer option rather than a single letter, so that they could express uncertainty; the main text uses this format, and Appendix C.1 reports the standard format as qualitatively the same.
- Agentic coding: SWE-Bench Verified, 500 GitHub issues solved by a simple tool-using agent loop in a bash environment, 30 runs each, with a limit of 250 messages and one hour per task. Each run's outcome is a vector of which unit tests pass (Appendix B.3).
- Safety questions: the human-written "advanced AI risk" subset of Anthropic's model-written evaluations (Perez et al., 2023), such as self-reported willingness to be shut down, with 30 samples per question. They are scored both in the original multiple-choice format and in an open-ended format with the options removed; for the open-ended answers, which have no target, the paper reports only the spread of the answers' text embeddings (Appendix B.2).
- Synthetic optimizer: small transformers (about 47 thousand to 5 million parameters) trained to imitate steepest descent on a four-dimensional quadratic function with condition number 50, then run on their own predictions for 2,048 steps from 4,096 starting points (Appendix B.4). This is a controlled stand-in for a model trained to act as an optimizer.
- Survey: rankings from Sohl-Dickstein's 2023 essay, in which 15 people produced or ranked 60 entities (machine learning models, animals, well-known humans and organizations); five ranked intelligence and six ranked coherence, each seeing only one attribute (Appendix B.5).
The frontier models are Claude Sonnet 4 with reasoning enabled, o3-mini and o4-mini; Claude Opus 4 and o3 also appear in the safety-question panel. For model size, the authors use the Qwen3 family (1.7B to 32B) with thinking enabled, with Gemma 3 and Llama 3 repeated in Appendix C.2. Reasoning length is the number of output tokens; for SWE-Bench it is the number of agent actions.
What it finds
Longer reasoning and more actions go with a larger variance share
This is the result the authors call their most robust. Sorting questions by the model's average reasoning length and grouping them into equal buckets, error-incoherence rose with length in every setting (Figure 2, Section 3.1). On GPQA the fitted slopes were 0.56 for Claude Sonnet 4, 0.40 for o3-mini and 0.29 for o4-mini (R² of 0.95, 0.97 and 0.87; Figure 2a). For the Qwen3 models the levels and slopes were "nearly identical across all sizes," although larger models scored higher (Section 3.1). In the synthetic optimizer task, error-incoherence rose with the number of optimizer steps for every model size, although the smaller models then fell back to a lower plateau once they could no longer follow the trajectory (Figure 2d, Section 3.2.2).
The safety-question panel in the main text shows one question set, self-reported survival instinct (Figure 2c, left); for the other multiple-choice sets in Appendix C.7 the authors say "We filter for those sets where there are noticeable trends" (Figure 23). For open-ended safety answers, the spread of answer embeddings also tended to rise with reasoning length, but the fits are weak: R² of 0.01 to 0.03 for the three models in Figure 2c (right).
Sorting by reasoning length partly sorts by difficulty, as the authors note: questions that take longer are answered less accurately (Section 3.1, "Discussion: Task complexity").
Within a question, the longer samples are less consistent
To separate length from difficulty, the authors split the samples for each question into those below and above that question's median length and computed error-incoherence for each half (Figure 3). The longer half was more variance-dominated for both frontier models and Qwen3 on GPQA. On SWE-Bench the printed values, below versus at or above the median number of actions, were 0.18 versus 0.22 for Sonnet 4, 0.12 versus 0.25 for o3-mini and 0.47 versus 0.56 for o4-mini (Figure 3b). Accuracy and SWE-Bench scores were similar between the halves (Figure 17). The authors call this "natural overthinking."
Model size: the relation depends on the experiment
The paper's own summary is that error-incoherence "changes with model scale in a way that is experiment dependent" (Abstract). The three strands of evidence in Figure 4 point in different directions:
- Qwen3 on MMLU, questions grouped into five difficulty bands by the 32B model's reasoning length (Figure 5): accuracy rose with size in every band. Error-incoherence fell with size for the easier bands (fitted slopes of -0.34, -0.29 and -0.17, with R² of 0.97 to 0.99) and was flat or rising for the hardest (slopes of -0.06 and +0.06, R² of 0.64 and 0.24) (Figure 4a). The authors describe the rising trend for the hardest band as "noisy" (Section 3.2.1). Their explanation is that bias fell at a similar rate in every band while variance fell much more slowly on harder questions (Figure 5d). GPQA shows the same pattern (Figure 12). Repeated on MMLU with two non-reasoning families prompted for chain of thought, the hardest band trends higher for Gemma 3 (up to 27B), and Llama 3.1 (up to 70B) is "the sole exception" (Figure 15 caption and Appendix C.2).
- The survey: entities ranked more intelligent by one group were ranked less coherent by another, with an overall rank correlation of 0.65. The printed correlations within categories were 0.89 for AI models, 0.95 for non-human beings, 0.37 for human organizations, and -0.27 for individual humans (Figure 4b). The survey asked about coherence of behavior in general; the paper inverts the coherence ranks to stand for error-incoherence (Appendix B.5).
- The synthetic optimizer: larger models reached lower loss, but their remaining error was more variance-dominated (Figure 4c). Bias fell with model size much faster than variance (fitted exponents of 2.165 and 0.878, Figure 6). The authors read this as models learning "the correct objective faster than the ability to maintain long coherent action sequences" (Section 3.2.2).
Appendix Figure 14 fits error-incoherence against both length and size and finds reasoning length the stronger predictor.
Reasoning budgets and ensembling
Raising the reasoning budget through the API improved accuracy and slightly lowered error-incoherence for o3-mini and o4-mini, but not for Sonnet 4 (Figure 7a, Section 3.3.1). The authors stress that this effect is much smaller than the rise that comes with natural overthinking (Figure 17). Averaging the probabilities of several samples from o4-mini on GPQA (320 samples per question, ensembles of up to 32) cut variance in proportion to the ensemble size while leaving bias unchanged, so error-incoherence fell (Figure 7b). The authors note that ensembling is impractical for actions in the world, which usually cannot be reset (Figure 7 caption).
The prompts used
The paper has no written principles. Two prompts shape its measurements. Frontier models answering GPQA and MMLU were told to give "an estimate of the probability of each answer being correct" in a fixed format, which the authors chose to let models "express uncertainty and therefore reduce error-incoherence" (Appendix B.1). Few models broke the format (Table 1). The safety questions used a one-line system prompt asking for step-by-step thinking and an answer in tags (Appendix B.2). The paper does not compare other wordings of either prompt.
Limitations, revisions and outside critiques
Limits the authors state
- Mechanism. "In this paper, we do not experimentally or theoretically explore the specific mechanisms" for the increase (Section 5). The argument that language models are "dynamical systems" that must be trained to act as optimizers is offered as motivation, not tested.
- Targets. The method needs well-defined targets, which limits what it can say about "open-ended goals or hidden objectives" (blog post, "Limitations of our framework," added in the revision described below). Section 5 calls the open-ended safety analysis "an initial exploration" of a setting where bias is hard to define.
- Misspecification. The paper splits bias into a part where the model misses its training objective and a part where the training objective misses what was intended. "For our tasks, we believe that there was not meaningful reward misspecification" (Section 5); in settings with a badly specified objective, the authors expect the second kind of bias to dominate as capability grows, and so recommend more attention to reward hacking and goal misspecification.
- Sample size. Bias and variance estimates stabilized at about 30 samples in a bootstrap check on o3-mini, for averages across questions but "not necessarily on a per question basis" (Figure 21).
Further points for reading the numbers: the slopes and R² values come from fits to bucketed groups of questions rather than to individual questions, and the paper reports no confidence intervals for them. The size results rest on one open model family per figure, up to 32B, and the survey on 11 raters from the 2023 essay. Models of the summer 2025 generation were tested.
The blog post's framing was revised
The post first captured on 3 February 2026 said "Our results are evidence that future AI failures may look more like industrial accidents," described larger models as becoming "more incoherent or remain unchanged" on hard tasks, and listed "Scale doesn't imply supercoherence." The version live from at least 13 February 2026 (Internet Archive captures, identical to the page served on 13 November 2026) says "a piece of evidence," states Finding 2 as "an inconsistent relationship between model intelligence and error incoherence," adds that the reasoning-length result "is our most robust finding," and adds a limitations item on the need for well-defined targets. Paper version 2 (April 2026) renamed the metric "error-incoherence."
These changes followed public criticism. On LessWrong on 4 February 2026, RobertM argued that the technical definition of "incoherence" is "uninteresting" and that the paper and post "equivocate with the more normal English-language definition of the term"; that the abstract selectively emphasizes the scale results; that the synthetic optimizer's small models have high bias from lack of ability, not misaligned propensity; and that the paper "basically assumes away the possibility of deceptive schemers." Replying the same day, Ethan Perez wrote that the authors agreed with some points, that the phenomenon would be better called something like "cross-sample error-consistency," and that some writing "overstates some of the implications to alignment (especially for superintelligence)." On 9 February 2026 Jascha Sohl-Dickstein wrote that the blog post had been updated, that it "was put together much more hastily than the paper," and that he still disagreed with the critique's reading of the scale results, pointing out that two of the three settings in Figure 4 show a positive relation. On the schemer point he replied that "A deceptive schemer maps cleanly onto bias-dominated failures."
A second post (i_am_nuts, LessWrong, 15 February 2026) accepted the correlation but argued that the within-question split in Figure 3 does not rule out an explanation that is a property of the questions rather than the models: for many questions, more answers may be defensible after a long chain of reasoning than after a short one. The paper does not test that alternative.
What the result does and does not cover
The paper measures consistency of errors across independent samples of the same task. It does not measure how consistent a model is within one transcript, or how often it errs; Perez's comment names both as separate notions of coherence. Because the metric is a ratio, a model with very small total error can still score as highly "incoherent."
Research questions
- Does the reasoning-length effect survive a difficulty control that is independent of the model? Splitting samples at the median controls for each question, but not for how many answers a long chain can justify. Repeating Figure 3 on questions with a single defensible answer path, or with answer sets of controlled ambiguity, would separate the two explanations.
- What does error-incoherence look like within a single trajectory? The paper measures consistency across independent samples. Measuring whether errors within one long agentic transcript are correlated would test whether long tasks produce scattered mistakes or one consistent drift.
- Do the size trends hold beyond 32B parameters? The model-size evidence comes from Qwen3 (to 32B), Gemma 3 (to 27B) and Llama 3.1 (to 70B), and the one family tested to 70B did not show the hardest band's rise. A family with a wider size range, scored with the same banding by difficulty, would show whether the hardest band's rising trend is noise or a real slope.
- What happens when the training objective is deliberately misspecified? The authors expect the misspecification part of bias to dominate in such settings but tested only tasks they believe were well specified. Training models on a reward with a known flaw and decomposing their errors would test that expectation directly.
- How do bias and variance behave for hidden or open-ended objectives? For open-ended safety answers the paper could measure only embedding spread, with fits explaining almost none of the variation. A target-based measure for model organisms with known hidden goals would show where such models fall on the scale.
- Does the synthetic optimizer result depend on the task's structure? Bias fell faster than variance for a four-dimensional quadratic. Repeating the experiment with other objective functions, higher dimensions or training by reinforcement learning rather than imitation would show how general the gap is.
How it connects
- Inverse Scaling in Test-Time Compute (July 2025) shares four authors with this paper (Gema, Hägele, Sleight and Perez), is cited in its related work, and supplied the codebase for the safety-question experiments (Appendix B.2); it reports tasks where accuracy falls as reasoning length grows, and a decline in Claude Sonnet 4's stated willingness to be shut down on the same model-written survival-instinct questions used here. The Hot Mess paper asks a different question of similar data: not whether longer reasoning gets worse, but whether its errors become less consistent.
- Discovering Language Model Behaviors with Model-Written Evaluations is the source of the advanced AI risk questions.
- This note connects Claude's Extended Thinking, which reports accuracy gains from larger thinking budgets for Claude 3.7 Sonnet and is not cited by the paper; this paper finds that larger budgets also slightly lowered error-incoherence for two of three models, while natural variation in length raised it.
- System Card: Claude Opus 4 & Claude Sonnet 4 is cited for the models and for its alignment assessment as an example of work on hidden goals that this method cannot yet measure.
- This note connects the paper's threat-model framing to notes on consistent, goal-directed misbehavior in model organisms, such as Sleeper Agents and Alignment faking in large language models, which study the bias-dominated kind of failure the paper contrasts with; the paper does not cite them.
References
- Hägele, A., Gema, A. P., Sleight, H., Perez, E., and Sohl-Dickstein, J. (2026). The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity? ICLR 2026. arXiv:2601.23045v2. https://arxiv.org/abs/2601.23045. Code: https://github.com/haeggee/hot-mess-of-ai record
- Hägele, A., Gema, A. P., Sleight, H., Perez, E., and Sohl-Dickstein, J. (2026, February). The Hot Mess of AI: How Does Misalignment Scale with Model Intelligence and Task Complexity? Anthropic Alignment Science Blog. https://alignment.anthropic.com/2026/hot-mess-of-ai/ (earliest Internet Archive capture 3 February 2026; revised version captured from 13 February 2026). record
- Sohl-Dickstein, J. (2023, March 9). The hot mess theory of AI misalignment: More intelligent agents behave less coherently. https://sohl-dickstein.github.io/2023/03/09/coherence.html (cited as the paper cites it; not read for this note). record
- RobertM. (2026, February 4). Anthropic's "Hot Mess" paper overstates its case (and the blog post is worse). LessWrong. https://www.lesswrong.com/posts/ceEgAEXcL7cC2Ddiy. Comments by Ethan Perez (4 February 2026) and Jascha Sohl-Dickstein (9 February 2026) on the same page. record
- i_am_nuts. (2026, February 15). Another short critique of the Anthropic "Hot Mess" paper. LessWrong. https://www.lesswrong.com/posts/pkrXGhGqpxnYngghA record
- Perez, E., et al. (2023). Discovering Language Model Behaviors with Model-Written Evaluations. Findings of ACL 2023. arXiv:2212.09251. record · 9 pages