Introducing the Conceptual Reasoning Index

Redwood Research and Anthropic authors introduce three benchmarks of reasoning where answers cannot be checked empirically (rating philosophical and AI-risk arguments against expert ratings, consistency of probability judgments, and decision-theory questions) and a weighted index of them; as of 10 August 2026 Claude Opus 5 scored highest at 73.6 against an estimated ceiling of about 91, with frontier scores rising roughly linearly since late 2024.

Emery Cooper, Caspar Oesterheld, Chi Nguyen, Alex Kastner (Redwood Research); Joe Benton, Ethan Perez (Anthropic) · 2026-08-12 · original

Much of the work needed to reduce risks from advanced AI has no practical feedback loop: reasoning about systems more capable than any human, decisions whose consequences arrive too late to learn from, and questions such as which values AI systems should have that may lack a ground truth. The authors call reasoning in such domains, which relies on argument because evidence and verifiable answers are scarce, "conceptual reasoning". They argue that models trained mainly on verifiable tasks may lag at it, and that improving it requires measuring it. This post introduces three benchmarks for that purpose and an index that combines them.

The post appeared on Anthropic's Alignment Science Blog on 12 August 2026. Its byline lists Emery Cooper, Caspar Oesterheld, Chi Nguyen and Alex Kastner of Redwood Research and Joe Benton and Ethan Perez of Anthropic, and the post says the work "was done in collaboration with Anthropic". The post is the version of record for the results reported here, with the scoring rules taken from the methodology page of the project's site, conceptualreasoning.ai, which the post points to for details. That site is a living leaderboard: when it was read for this note (13 November 2026) it already listed models released after the post. The site's methodology may have changed since August; this note marks which document each statement comes from. The main dataset, LMCA, is available only on request through a form. Two papers sit behind the benchmarks, and the post links both. "A dataset of rated conceptual arguments" (Cooper, Oesterheld, Nguyen, Kastner and Perez; arXiv:2607.27499, v1 posted 29 July 2026, PDF dated 19 September 2025) describes an earlier release of LMCA, with 951 critiques of 442 positions and 1,458 ratings by six raters (Table 1), scored with its own loss functions rather than the index's correlation; it was read for this note. The DTBench questions come from Oesterheld et al., "A dataset of questions on decision-theoretic reasoning in Newcomb-like problems" (arXiv:2411.10588, 2024, revised June 2025), which was not read in full.

The three benchmarks

LMCA (Language Model Conceptual Argumentation)

LMCA holds 560 position texts (statements of a view) and 1,461 arguments against them, on topics including decision theory, philosophy and risks from advanced AI. Nearly all arguments were rated by Emery Cooper; the 16 that were not were rated by Caspar Oesterheld (footnote 2), and some were rated by at least one other researcher, for 2,140 ratings in total. Ratings follow a detailed rubric. A validation set of roughly 50 arguments was rated independently by four to six people and then discussed for seven to eight hours in total. The post says inter-rater agreement is "high compared to agreement between humans and models" but does not give the agreement figure. The LMCA paper does, for its earlier release and on its own loss (lower is better; Appendix C). On the 209 critiques both rated, Oesterheld's ratings had a loss of 0.139 ± 0.016 against Cooper's first ratings, against 0.208 ± 0.018 for GPT-5, the best model then tested. Its validation set ("rating test set") was 52 critiques across 19 positions, rated by four researchers in full and by two more in part (43 and 23 critiques); after discussion each rater's final ratings differed from the average by 0.044 to 0.065 on the critiques everyone rated, initial ratings by 0.064 to 0.149, and GPT-5 by 0.153. The paper also says many LMCA critiques were written by models and some were added because the best model judges disagreed on them, so that "we automatically generate critiques that are difficult for the best judges (and best models)", before final hand selection (Section 2; 478 of the 951 critiques were model-written, Table 1).

Models are scored on judging arguments, not writing them. According to the methodology page, a model's "overall" ratings are compared with the average of the human "overall" ratings using the Pearson correlation, multiplied by 100, with correlations at or below zero scored 0; the page reports that Spearman's rank correlation gave a nearly identical ordering. The post also describes a way to score argument writing (one model writes a new argument, another rates it from the rubric and few-shot examples of rated arguments) and says it "produces fairly accurate ratings", without numbers. Argument writing is not yet in the index.

ACCoRD (Assessment of Consistency in Conceptual Reasoning Domains)

ACCoRD asks separate instances of a model for numeric probabilities or preference orderings and checks whether the answers satisfy consistency constraints, logical relations such as P(A) ≥ P(A and B). Of close to 14,000 model-generated constraints across 18 constraint types that passed an automated checker, 567 were checked and approved by the authors, and only those 567 count toward the CRI. The methodology page describes the score as the average size of violations, normalized by the average distance between probabilities of unrelated propositions in the dataset, converted to consistency and scaled so that 0 is the consistency of uniform random answers and 100 is perfect consistency. It uses one sample per probability, and says that most measured inconsistency comes from differences between propositions rather than sampling noise, without showing the estimate.

DTBench capabilities (Decision Theory Benchmark)

DTBench capabilities is 407 handcrafted multiple-choice questions on decision problems involving faithful predictions of the model's own behavior or interactions with near-copies of itself. Most were written by Caspar Oesterheld and all were independently validated by Emery Cooper. A further 130 questions measure decision-theoretic attitudes rather than capability and are excluded from the index. The methodology page scales accuracy so that 40% (the random-guessing rate) is 0 and 100% is 100.

What the post finds

  • Index and weights. The CRI is currently 60% LMCA, 20% ACCoRD and 20% DTBench capabilities, with each score running from 0 (random guessing) to 100 (the highest possible score). The authors plan to add benchmarks, retire saturated ones and possibly change the weights (section "Results").
  • Estimated ceiling. Because human ratings are noisy, the authors estimate from expert inter-rater agreement that the best possible LMCA score is about 85, and put the ceiling on DTBench and ACCoRD at or very near 100, giving an estimated CRI ceiling of about 91. That figure equals 0.6 × 85 + 0.2 × 100 + 0.2 × 100 (our arithmetic).
  • Leading scores. As of 10 August 2026 (first chart, with 95% intervals drawn as whiskers): Claude Opus 5 73.6 (± 2.1, stated in the text), Claude Fable 5 72.7, GPT-5.6 Sol Pro 70.0, Gemini 3.1 Pro and Muse Spark 1.1 63.8, Kimi K3 62.8, Muse Spark 1.2 62.7, Grok 4.5 60.3, Gemini 3.6 Flash 59.5, and lower scores down to Command A at 20.7. The chart shows Anthropic's best models and the top model from each other developer evaluated, plus Fable 5, Muse Spark 1.2 and Gemini 3.6 Flash, which the post says lead their developers on many external benchmarks though not on the CRI.
  • Component scores. The site's score file, read on 13 November 2026, gives Opus 5 63.3 on LMCA, 81.9 on ACCoRD and 96.0 on DTBench, and Fable 5 60.3, 85.0 and 97.3; the 60/20/20 weighting of these reproduces the post's 73.6 and 72.7 (our arithmetic). LMCA, the most heavily weighted benchmark, is where both models stand furthest below the estimated ceiling.
  • Trend. Frontier scores "have been increasing roughly linearly since late 2024, with no signs of flattening". The second chart fits a line to each lab's most generally capable model at release, from GPT-4 (mid-2023) to Opus 5, with r = 0.95; the per-benchmark chart gives r = 0.93 for LMCA, 0.89 for ACCoRD and 0.94 for DTBench. These are fits over a few dozen models from three developers, not forecasts with stated uncertainty beyond the band drawn.
  • Saturation. DTBench capabilities is close to its ceiling, with Fable 5 answering 98% correctly. The authors "loosely estimate" that LMCA will start to saturate about a year after the post, and are "very uncertain" about ACCoRD.

Conditions on the scores

  • All models were run at their maximum token limit and effort setting, with each lab's default sampling parameters (methodology page).
  • Where Fable 5 refused a question, Opus 5's answer was used in its place. The post does not say how many questions this affected.
  • GPT-4 refused to fully answer 18% of ACCoRD items, so its ACCoRD score is partial.
  • The intervals reflect sampling over benchmark items, as the methodology page states; the post reports no repeated runs.

The prompts and rubric used

The benchmarks use written material in two places. The LMCA rubric guides both the human raters and the models being scored, and models judge each argument with that rubric in context; in the argument-writing variant, the rating model also sees the rubric and few-shot examples. The post does not print the rubric or the prompts, and does not test how the wording of either affects scores.

Limitations and open questions

  • One main rater. LMCA's reference ratings come almost entirely from one researcher, with agreement checked on a subset. The post gives no agreement figure; the paper's figures are for an earlier release and a different metric, so how the ceiling of 85 (a correlation) was derived from them cannot be checked from either document.
  • Adversarially chosen items. Part of LMCA consists of model-written critiques picked where the best model judges disagreed (LMCA paper, Section 2), which makes the benchmark harder for models by construction; the post does not mention this.
  • Consistency is not correctness. ACCoRD measures whether answers cohere, and the post presents inconsistency as a sign that a model's reasoning on a topic cannot be trusted; a consistent model could still be wrong.
  • Model-generated items. The ACCoRD constraints were generated by models and filtered; only the 567 the authors approved are scored.
  • Developers in the trend. The trend chart covers OpenAI, Anthropic and Google DeepMind models only, and the Anthropic co-authors work for the developer whose model ranks first.
  • A changing index. Weights, benchmarks and the model list are expected to change, so scores from different dates may not be comparable.

Research questions

  1. How reliable are the LMCA reference ratings in the current release? Reporting inter-rater agreement as a Pearson correlation for the current release's double-rated arguments, and deriving the ceiling from it, would show how firm the estimate of 85 is.
  2. Does the ranking depend on the weights? Recomputing the ranking under other plausible weights, or under each benchmark alone, would show whether the leading models' order holds.
  3. Does the CRI predict useful conceptual work? Comparing CRI scores with expert judgments of models' output on real research or policy questions would test whether the index measures what it is meant to.
  4. Is LMCA scoring sensitive to the rubric's wording? Re-running the judging task with paraphrased rubrics would show how much scores depend on the instructions given.
  5. How much do refusal fallbacks affect scores? Reporting the number of substituted answers for each model, and scoring with and without them, would bound their effect.

How it connects

References

  • Cooper, E., Oesterheld, C., Nguyen, C., Kastner, A., Benton, J., and Perez, E. (2026, August 12). Introducing the Conceptual Reasoning Index. Anthropic Alignment Science Blog. https://alignment.anthropic.com/2026/conceptual-reasoning-index/ record
  • Conceptual Reasoning Index. Methodology page and leaderboard data. https://conceptualreasoning.ai/methodology.html (accessed 13 November 2026; a living page). record
  • Cooper, E., Oesterheld, C., Nguyen, L. C., Kastner, A., and Perez, E. (2026). A dataset of rated conceptual arguments. arXiv:2607.27499 (v1, 29 July 2026; describes an earlier LMCA release). record
  • Oesterheld, C., Cooper, E., Kodama, M., Nguyen, L. C., and Perez, E. (2024). A dataset of questions on decision-theoretic reasoning in Newcomb-like problems. arXiv:2411.10588 (source of DTBench; not read in full for this note). record
  • METR. Frontier Risk Report (February to March 2026). Cited by the post (footnote 1) for agents' weaker judgment on tasks that are hard to verify; not read for this note.