TASTE: Can AI Models Judge AI Safety Research Proposals?

Baig, Joren and Benton build TASTE, 92 pairs of model-written AI safety research proposals labeled with the preferences of experienced researchers, kept only where a rater was strongly confident after discussing disagreements with a peer and scored the two proposals at least two points apart; held-out researchers agree with the labels an estimated 77% of the time. Of 19 frontier models, the best, Claude Fable 5, scores 60% and the rest 41% to 59%, all within two standard deviations of chance by the paper's account; the paper traces part of the gap to models over-weighting how well a proposal answers its prompt.

Hasan Baig, Hailey Joren, Joe Benton (Anthropic Fellows Program; Anthropic) · 2026-08-28 · original

If AI safety research is to be partly automated, models will need to do the parts that cannot be checked by running a test: forecasting risks, attributing beliefs to models, and deciding which research is worth doing. This paper measures one of those parts, judging research proposals, and confronts the problem that defeats most such measurements: experienced researchers often disagree, so there is no obvious ground truth. Its answer is to build a small benchmark only from judgments that other researchers reproduce, and then ask how often models pick the same proposal.

The work is an Anthropic Fellows Program project by Hasan Baig (Anthropic Fellows Program), Hailey Joren and Joe Benton (Anthropic), posted on the Anthropic Alignment Science Blog on 28 August 2026 with a 61-page paper hosted by Anthropic (PDF created 26 August 2026). This note follows the paper, cited by section, figure, table and appendix; the post is a summary, and its differences are noted where they matter. The benchmark and the human-feedback data are available to AI safety researchers on request, not publicly.

Terms used in this note

  • Motivating-question prompt. A short research problem posed in the voice of a researcher, from which proposals are generated.
  • Within-prompt and distinct-prompt pairs. Two proposals written for the same prompt, or for different prompts.
  • Anchor rater and gold label. For each benchmark pair, the one researcher whose preference defines the correct answer.
  • Held-out agreement. How often researchers who did not discuss the proposals with the anchor rater prefer the same proposal. It is the paper's estimate of human performance.

What the paper does

Proposals

The authors gave Claude Opus 4.6 each of 93 human-written proposals from the Anthropic Fellows Program (late 2025 and early 2026 cohorts) and had it reverse-engineer the question that might have motivated it, discarded weak questions and added their own, giving 45 prompts. A scaffold on Opus 4.6, with paper summaries in context and Opus 4.6 graders filtering ideas, then wrote three proposals per prompt, 135 in all, each about 280 words (Section 2.1; Appendix A; Appendix F.1).

Human judgments

Ten AI safety researchers from MATS and Constellation, with six months to four years of experience, rated the proposals, four per prompt, three proposals at a time (Section 2.2). Each gave every proposal an overall score from 1 to 5 (1 is "Typical LLM idea", 5 is "AFP mentor level idea"), two sub-scores, comments, and one confidence level (weak, medium or strong) per prompt (Table 1; Appendix B.1). Researchers then paired up, discussed disagreements, and revised their scores (Figure 4). The discussions were recorded and transcribed. The authors liken this protocol to IDEA, a structured method for eliciting expert judgments (Section 5).

Building the benchmark

From all pairs of proposals scored by the same researcher, the authors kept those where (Section 2.3; Appendix C.1 to C.2):

  1. the researcher reported strong confidence after discussion;
  2. their overall scores differed by at least 2 points;
  3. neither proposal appeared in more than 10 pairs.

The first two steps left 166 pairs; capping left 92 pairs (74 distinct-prompt, 18 within-prompt) over 50 proposals. Human performance is estimated by sampling, for each proposal, an overall score from one of the two researchers in the other discussion pair, which allows comparison across prompts no single researcher saw together.

Testing models

Nineteen models from eight developers were tested, at their highest available reasoning effort, with the human rubric in context. Each sees two proposals and gives a probability for each, averaged over both orders and over versions with the motivating question shown and hidden, then turned into one verdict per pair, with half credit for a probability of exactly 0.5 (Section 3.1; Table 3; Appendix D.1). The headline results use a "situational" framing that asks the model to model the preferences of empirical AI safety researchers from the raters' organizations; a neutral framing that asks for the model's own preference gave model-dependent differences (Appendix D.2, Figure 14). A harder variant scores one proposal at a time.

What it finds

How good the human labels are

  • Discussion plus confidence filtering. Agreement with held-out researchers rose from 53% for all pre-discussion preferences to 68% for strong-confidence, post-discussion ones (Section 4.1; Figure 7). Only that cell was clearly above chance. In the other cells discussion did not help: weak-confidence agreement fell from 56% to 46%, medium from 49% to 42%, and all preferences together from 53% to 49% (Figure 7 labels).
  • Score gaps. Among strong-confidence, post-discussion within-prompt pairs, requiring a two-point gap raised agreement from 68% to 83%, on 18 pairs by the text's count (19 in the figure); across prompts, from 64% to 79% (Section 4.2; Figure 7).
  • The benchmark. Held-out agreement on TASTE is 77.4%, measured on the 85 pairs where both held-out raters gave a strict preference (Appendix C.2); model accuracy, by contrast, is scored on all 92. When the cap's second removal step is simulated at random instead of by the fixed rule, agreement ranges from 70% to 82%, median 77%. The authors believe the sampled-score estimate slightly underestimates human agreement (Section 2.3).
  • Why researchers disagreed (Section 4.3; Appendix E.1): different empirical priors (23%), different weighting of importance, tractability and novelty (21%), misreading the proposal (17%), not knowing prior work (9%), confusing text (6%), ambiguity about fit to the prompt (5%), other (19%). These shares come from Fable 5 summarizing the transcripts, deriving the taxonomy and classifying each disagreement.
  • Averaging is not a substitute. Averaging researchers' pre-discussion scores did not raise agreement, which the authors read as discussion adding signal (Appendix E.2). For all seven researchers who gave strong post-discussion preferences, filtering raised their agreement (Appendix E.3).

How models perform (Table 3)

Fable 5 scored 60%, Kimi K3 59%, Grok 4.5 58%, Gemini 3.5 Flash, Sonnet 4.6 and Haiku 4.5 56%, Opus 4.8 55%, Opus 5 51%, the three GPT-5.6 variants 48%, Opus 4.6 (which wrote the proposals) 45%, and Gemini 3.1 Pro 41%. The paper describes Fable 5 as "narrowly above chance" and every other model as within two standard deviations of it (Figure 1). The post gives per-model intervals of "roughly ±10 percentage points", so the ranking among models is not established. Where Fable 5 refused, the paper filled 9 of its 184 cells by a fixed rule that can fall back to Opus 4.8's answer (Appendix D.1).

What helps and what fails

  • Test-time compute. Fable 5 rose from 48% at low to 60% at maximum effort, and from 46% to 58% in single-proposal scoring; it was the only model to improve in both settings (Section 3.2; Figure 5).
  • Rubric and examples. Results vary by model. With the rubric in context, Fable 5 went from 46% (zero-shot) to 60%, Opus 4.8 from 40% to 55%, and Kimi K3 from 58% to 59%; adding researchers' past judgments as examples helped some models (Kimi K3 to 66%, GPT-5.6-sol to 54%) and hurt others (Opus 5 to 41%) (Section 3.3; Figure 6 labels, on test folds). Reading model justifications, the authors found that with examples models leaned on heuristics such as which techniques a given researcher liked, and they chose the rubric-only setting for the benchmark.
  • Position. Swapping the order of a pair changed models' probabilities by up to 25 points (Section 3.4).
  • Answering the prompt. The rubric tells raters that every proposal answers its prompt well enough and to judge content instead. Models still over-weighted fit to the prompt: by a Fable 5 classifier, 42% of justifications on within-prompt pairs were prompt-driven against 5% on distinct-prompt pairs, and hiding the prompt raised five models' within-prompt accuracy by 17 points on average over 18 pairs (Section 3.4; Appendix D.6). Fable 5 scored 69% on distinct-prompt pairs with the question shown and 31% on within-prompt pairs (Appendix D.5, Table 5).
  • Subsets. Without the four most frequent proposals (52 pairs), Fable 5 scored 62% against 70% human agreement (Appendix D.3). On pairs where held-out raters agreed with the gold label, Fable 5 scored 64%, and on contested pairs 56% (Appendix D.4).

An anecdote about proposals

In December 2025 the authors entered two proposals from an earlier LLM scaffold into the first selection round for the Fellows Program's January 2026 cohort, without telling the selectors. One ranked last and the other 37th of 74; neither was recognized as model-written (Appendix F.2). The authors offer this as partly a limitation of time-limited review and partly evidence that models can write good proposals; it is one trial of two proposals chosen by the authors from 40.

The rubric and prompts

The rubric given to researchers is also the main text given to models (Appendix B.1). It frames the task as ranking ideas "as if you were an Anthropic Fellows Program (AFP) mentor", defines the five-point scale by reference classes, asks raters to skip areas outside their expertise, and says proposals all answer the prompt "sufficiently well by our standards that you should focus on the content of the ideas". Two findings concern how models used this text: adding it to the context raised accuracy for some models and not others (Figure 6), and models given its instruction to discount prompt fit did not follow it on within-prompt pairs (Section 3.4). The paper reports these as model behavior under prompting; no model is trained on the rubric.

Limitations and open questions

The authors' limitations (Section 6):

  • Few data points. 92 pairs from 50 proposals, some proposals in up to 10 pairs, which limits independence and what ablations can show.
  • Few raters. Ten researchers from two organizations, who sometimes rated outside their expertise; their views may not represent the field.

Further points:

  • Scope of the labels. The proposals are short, model-written and generated from Fellows-style prompts; human-written proposals are not in the benchmark, and an earlier comparison of scaffold against human proposals gave about 50% win rates with near-chance rater agreement (Appendix F.1).
  • Construction choices. The strong-confidence filter and the two-point gap were validated by their effect on held-out agreement (Section 4); the cap's removal rule is fixed and does not look at agreement, and Appendix C.2 shows how the estimate moves when that rule is varied. Some numbers differ between text and figure: Section 4.2 gives n = 58 and 18 pairs and 135 distinct-prompt pairs, while Figure 7 prints n = 65, 19 and 147.
  • Model as analyst. The disagreement taxonomy and the prompt-driven classification were produced by Fable 5 (AI use statement; Appendix E.1); the paper reports no human check of either.
  • The post's agreement figure. The post says that comparing only pairs where another researcher scored both proposals "gives 83% agreement but only validates 50 of the 92 pairs". This note found no 50-pair figure in the paper; the nearest figures there are 82% on 17 within-prompt pairs (Appendix D.5) and 83% on the gap-filtered within-prompt pairs before capping (Section 4.2).

Research questions

  1. Does the benchmark ranking hold on a larger set? Adding pairs with the same protocol until per-model intervals are a few points wide would show whether the differences among the top models are real.
  2. Is the human estimate stable across raters? Recruiting a second, independent panel to label the same 92 pairs would test the 77% estimate without the sampling procedure.
  3. Does training on preferences like these improve research judgment? The authors propose TASTE as a held-out test for such training; training a model on separately collected preference data and scoring it here would answer this.
  4. What explains the prompt-fit bias? Varying how prominently the motivating question and the rubric's instruction appear, and measuring within-prompt accuracy, would show whether the bias responds to wording or persists regardless.
  5. Do model and human failures overlap? Comparing which pairs models get wrong with the categories of human disagreement in Appendix E.1 would show whether models fail where humans also disagree.
  6. Does judging human-written proposals give the same picture? Building a matched set of human-written proposals with the same protocol and comparing model accuracy would test whether model-written proposals are a special case.

How it connects

References