Challenges in evaluating AI systems

Public discussion of an AI system's truthfulness, fairness or potential for misuse rests on evaluations: the post's introduction says such qualities can be discussed because models can be technically evaluated for them. The post notes that Anthropic also relies on evaluations in its Responsible Scaling Policy. This post, filed under Anthropic's Policy label, reports what the company had learned about how hard those evaluations are to build, run and interpret. It is written for people "developing AI governance initiatives that rely on evaluations" and for new evaluation organizations, and states two intended takeaways: robust evaluations are "extremely difficult to develop and implement", and effective governance depends on being able to evaluate systems meaningfully (Introduction).

The post is a practitioner's account, not a study. It reports no new experiment, and its few numbers are anecdotes from Anthropic's own work. Its value for this library is as a dated record of how one developer described the state of evaluation in late 2023, between model-written evaluations (2022) and the automated auditing tools of 2025 (Petri, Bloom).

What the post covers

The post discusses six kinds of evaluation, ordered by the authors from "less-challenging to more-challenging" (Introduction):

1. multiple-choice evaluations;
2. third-party evaluation frameworks (BIG-bench and HELM);
3. crowdworker tests of helpfulness and harmfulness;
4. red teaming by domain experts for national security risks;
5. evaluations generated by models;
6. an external audit for dangerous capabilities.

Terms used below:

• MMLU (Massive Multitask Language Understanding): a multiple-choice benchmark of 57 tasks from mathematics to law, scored by accuracy.
• BBQ (Bias Benchmark for QA): a multiple-choice test of social bias along nine dimensions, scored with a bias score from -1 (strong anti-stereotypical bias) through 0 (none) to 1 (strong stereotypical bias).
• BIG-bench: a collection of 204 evaluations contributed by over 450 authors.
• HELM (Holistic Evaluation of Language Models): a Stanford suite in which experts choose the tasks and metrics.
• Red teaming: deliberately trying to make a model produce harmful output.
• Audit (as the post uses it): a deeper, independent assessment focused on risks, as opposed to a broader evaluation of capabilities.

What it reports

Multiple-choice evaluations

For MMLU the post lists four problems (section "The supposedly simple multiple-choice evaluation"). Its questions are widely circulated and so likely to appear in training data. Small formatting changes, such as "(A)" to "(1)", "(A)" to "[A]", or an extra space, can change accuracy by about 5 percent. Developers run it differently, some with few-shot prompting or chain-of-thought reasoning, so scores are hard to compare across labs. And some questions are mislabeled or unanswerable.

For BBQ the difficulty was implementation. Anthropic could not find a working open-source implementation, and it took "one of our best full-time engineers one uninterrupted week" to implement and test. Footnote 1 adds that building BBQ itself took about two person-years spread over six months across eight people. The post's main example of misreading a score comes from BBQ: some Anthropic models scored a bias of 0, which looked like progress, until a BBQ developer working at Anthropic asked whether the models were answering the questions at all. They were not. The results were "technically unbiased, but they were also completely useless". The authors generalize: every evaluation risks overinterpreting a quantitative score.

Third-party frameworks

BIG-bench took significant engineering to install, did not scale to running all 204 tasks in reasonable time, contained bugs (including in BBQ-lite, a BBQ variant), and would have required a research project to decide which tasks mattered. Anthropic dropped it. Footnote 2 notes that BIG-bench Hard, a 23-task subset, came out four months later.

HELM needed no engineering from Anthropic, since its developers ran the benchmark through API access, but it prompted Claude without the Human/Assistant format Claude was trained on, so that, in the post's words, HELM "gives a misleading impression of Claude's performance". New models could take months to evaluate, which the post attributes to HELM being a volunteer effort at a university.

Human evaluations

Anthropic's main human evaluation at the time was the A/B test: crowdworkers converse with two models and pick the more helpful or more harmless response, with workers encouraged to red-team for harmlessness. The post lists three limits: the tests are costly and slow and raise ethical issues for crowdworkers, including exposure to harmful outputs; results depend on the workers' creativity, motivation and skill; and helpfulness and harmlessness pull against each other, since a model can avoid harm by refusing. The post leaves open "what numerical value indicates a model is sufficiently helpful and harmless".

Red teaming for national security risks

Anthropic had piloted "frontier threats red teaming", in which subject-matter experts define threat models and probe the model against them. The post describes red teaming as "more art than science", without a standard process. Some cases needed red teamers with security clearances, which limits what they can share with the developer, and a model producing controlled information could have legal consequences, which raises the question of a legal safe harbor.

Model-generated evaluations

The section title calls this "the ouroboros of model-generated evaluations". The post reports success with model-written evaluations, which take minutes rather than the days or months of human-written ones. It names two problems: verifying them still relies on humans, which brings back every difficulty of human evaluation, and models can be socially biased or fabricate information, which a model-written evaluation may inherit "in hard-to-disentangle ways". As a counter-example it cites Constitutional AI, where model-based red teaming replaced human red teaming and people still judged the resulting models more harmless than models red-teamed by humans.

Third-party audits

Anthropic took part in dangerous-capability evaluations by the Alignment Research Center (ARC), which test abilities such as acquiring resources, self-copying and resisting shutdown. The collaboration needed more science and engineering support from Anthropic than expected. The post also describes a tension in auditing: auditors withhold details to keep the audit sound, but after reading ARC's final report, Anthropic thought it could have helped ARC find more concerning behavior had it known more about their approach, because eliciting a model's full capability takes expertise that mostly sits inside AI companies.

Recommendations to policymakers

The post closes with recommendations (section "Policy recommendations"):

• Fund evaluation science: research into what makes an evaluation high-quality and replicable, preferring "one higher-quality evaluation rather than ten lower-quality evaluations".
• Fund implementation: easy-to-run packages for existing evaluations such as BIG-bench, and version-controlled, standardized dynamic benchmarks.
• Fund robustness analysis: monitoring existing evaluations, for example to detect saturation.
• Fund government evaluation agencies such as NIST, and encourage a public "AI safety leaderboard" modeled on NIST's Face Recognition Vendor Test.
• Create a legal safe harbor for evaluating national security risks, such as chemical, biological, radiological and nuclear (CBRN) risks, possibly with a "responsible disclosure protocol".

The written instructions involved

The post contains no principles or prompts of its own, but several of its problems concern written instructions. Crowdworker A/B tests depend on "careful instructions for A/B testers" (section "A/B tests with crowdworkers"), and the authors of the Constitutional AI paper attributed part of a shift in their harmlessness scores to new crowdworker instructions that penalized evasive answers. The HELM problem is about prompt format: a model trained to a particular conversational format behaves differently when evaluated without it. The MMLU formatting sensitivity is a related effect of format on scores.

Limitations and open questions

• Anecdotal evidence. Every problem is illustrated from Anthropic's own experience. The post gives no systematic measurement of, for example, how much MMLU accuracy varies across formats beyond the single figure of about 5 percent.
• One developer's view. The criticisms of BIG-bench and HELM come from the evaluated party, and the post does not include the frameworks' responses. It also does not discuss the reasons a suite might keep one prompt format for all models, beyond noting that HELM needs consistency.
• Dated. The post describes the situation in 2023. Anthropic's later work addresses some of its points, for example A statistical approach to model evaluations on how to report evaluation uncertainty, and Petri and Bloom on automated, model-run evaluations.
• The model-written evaluation problem is left open. The post says model-generated evaluations "deserve deeper study" but does not propose how their biases could be detected.

Research questions

1. How large are formatting effects across benchmarks and models? The post gives one figure, about 5 percent on MMLU. A systematic study varying answer labels, brackets and spacing over several benchmarks and model families would show whether this is typical and whether it shrinks with model size.
2. Which controls catch a meaningless score? The BBQ case was caught by checking whether models answered at all. Compiling such sanity checks (answer rate, refusal rate, chance-level baselines) for common benchmarks and testing how often published results fail them would turn the anecdote into a practice.
3. Do model-written evaluations inherit the generator's biases measurably? The post raises the concern without data. Comparing results on model-written and human-written versions of the same bias tests, as Perez et al. (2022) began to do, across models from different developers would test it.
4. How should auditors and developers share information? The ARC example suggests audits can miss behavior that developer knowledge would surface. Comparing audits run with and without developer collaboration, measured by what each finds, would inform how audit protocols are written.
5. What threshold makes a model "sufficiently" helpful and harmless? The post asks this and leaves it open. Answers would need agreed metrics and a stated trade-off between refusals and harms.

How it connects

• Within the library. Discovering Language Model Behaviors with Model-Written Evaluations is the work the post cites for model-written evaluations. BBQ, the bias benchmark it discusses, is also used (with Winogender) in The Capacity for Moral Self-Correction, whose authors include the post's first author. The crowdworker A/B tests are the kind of comparison behind the Elo scores of Bai et al. (April 2022) and Constitutional AI. Expert red teaming connects to Red Teaming Language Models to Reduce Harms.
• Later in this evaluation thread. Petri, Bloom and the "diff" tool are later Anthropic tools aimed at problems the post names: the cost of building evaluations, the limits of fixed benchmarks, and finding behaviors no one has thought to test.
• In the course wiki. Contamination and the rules for reusing a benchmark are discussed for retrieval in Test Collections and the Cranfield Paradigm. The dependence of results on who judges, and how to measure agreement, is the subject of Relevance Judgments, Pooling, and Agreement. Side-by-side human preference tests are covered in Online Evaluation, and deciding whether a score difference is real in Comparing Systems with Significance Tests.

References

• Ganguli, D., Schiefer, N., Favaro, M., and Clark, J. (2023). Challenges in evaluating AI systems. Anthropic, 4 October 2023. https://www.anthropic.com/research/evaluating-ai-systems
• Bai, Y., Kadavath, S., Kundu, S., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073.
• Perez, E., Ringer, S., Lukošiūtė, K., et al. (2022). Discovering Language Model Behaviors with Model-Written Evaluations. arXiv:2212.09251.