Model Card and Evaluations for Claude Models

Anthropic released Claude 2 on 11 July 2023 with a 14-page PDF titled "Model Card and Evaluations for Claude Models". It is the earliest document in Anthropic's series of model and system cards, and it is the shortest. The card names its genre in its first sentence by citing Mitchell et al.'s "Model Cards for Model Reporting" (2019), the proposal that a released model should come with a structured record of its intended uses, training data, evaluations and limitations (Section 1, reference [1]). It also says what it is not: the report "is not intended to be a scientific paper", because most of the methods behind it are documented in Anthropic's research papers, which it lists (Section 1, p. 1). The stated purpose is to "display Claude 2's strengths and limitations" and to describe the evaluations and safety interventions behind them (Section 2, p. 2). The card covers three deployed models, Claude 2, Claude 1.3 and Claude Instant 1.1, which it calls collectively "Claude models", and in some evaluations a non-deployed helpful-only version of Claude 1.3 (Section 3, p. 3).

Structure of the card

The card has five numbered sections and a reference list:

1. Introduction (under a page): what the report is and is not, with a list of the research papers that document the methods.
2. Claude 2 Model Card (about one and a half pages): the model card proper, in short labeled paragraphs: model details, intended uses, unintended uses and limitations, ethical considerations, training data, and evaluations and red teaming.
3. Alignment Evaluations (about four pages): human preference Elo scores, BBQ bias, TruthfulQA, harmfulness on held-out prompts, and the HHH comparison questions.
4. Capabilities Evaluations (about four pages): translation, long context, standard benchmarks and standardized tests, and a list of areas the developers worked to improve.
5. Areas for Improvement (under a page).

Most of the substance is in the evaluation sections, and almost all of it is reported as plots (Figures 1 to 8) with few printed numbers. There is no section on catastrophic-risk evaluations beyond two paragraphs in Section 2, and no Responsible Scaling Policy, which Anthropic had not yet published. Later cards in the series grow from this layout: the Claude 3 card (March 2024) keeps the model-card paragraphs and adds a section on catastrophic risk under the Responsible Scaling Policy, and by the Claude Opus 4 and Sonnet 4 system card (May 2025) the model-card paragraphs shrink to a page and safety testing fills more than 100 pages.

Training and the written principles

The card's account of training is brief. Claude 2 and earlier Claude models "use a transformer architecture and are trained via unsupervised learning, RLHF, and Constitutional AI (including both a supervised and Reinforcement Learning (RL) phase)" (Section 2, "Model details", p. 2). RLHF (reinforcement learning from human feedback) trains a model against a preference model fitted to human comparisons of responses; Constitutional AI replaces the human harmlessness comparisons with comparisons a model makes by applying written principles, in a supervised critique-and-revision stage and a reinforcement learning stage (see Constitutional AI: Harmlessness from AI Feedback).

The constitution. Under "Ethical Considerations" the card describes the constitution as "a set of ethical and behavioral principles that the model uses to guide its outputs" and points readers to a blog post published in May 2023 for "Claude 2's principles" (Section 2, p. 2, reference [10], "Claude's Constitution", anthropic.com). The card says that using this constitution, models are trained to avoid "sexist, racist, and toxic outputs" and to avoid helping a person engage in "illegal or unethical activities" (p. 2). It does not print any principle, and it does not say how many principles were used or how they were applied in training. The May 2023 post is a different document from the 2026 constitution described in Claude's constitution.

A debiasing step. In the BBQ section the card attributes Claude 2's lower bias scores mainly to "debiasing algorithms" from The Capacity for Moral Self-Correction: "we generate unbiased samples, and then finetune Claude on these samples before we initiate the RL phase of Constitutional AI" (Section 3.2, p. 4). This is the only place the card describes a specific training intervention in any detail, and it is stated as the "most likely" cause rather than tested.

Training data. The models are trained on "a proprietary mix" of public Internet data, licensed datasets, and data that users "affirmatively share" or that crowd workers provide; part of the human feedback data was released publicly with the RLHF and red-teaming papers (Section 2, "Training Data", p. 2). Claude 2's training data "cuts off in early 2023", and "roughly 10 percent" of it was non-English (p. 2).

Usage policy. The card points to the Acceptable Use Policy (AUP), which "delineates what are and are not permitted uses of Claude", and to Trust and Safety processes that enforce it (p. 2). No system prompt is discussed.

Alignment, honesty and harmlessness evaluations

The card uses "alignment evaluations" for tests of how far the models meet the helpful, honest and harmless (HHH) goals, with free-form samples drawn at temperature 1 unless stated otherwise (Section 3, p. 3). The helpful-only Claude 1.3, trained without the honesty and harmlessness interventions, serves as a comparison point throughout.

Human preference Elo scores. Crowdworkers chatted with two models, chose the better response per turn under task-specific instructions, and the binary preferences were converted to Elo scores, a rating system that predicts how often one model's output is preferred to another's (Section 3.1, p. 3). Three tasks were run: detailed instruction following (helpfulness), accurate and factual information (honesty), and a red-teaming task in which crowdworkers role-played adversarial scenarios (harmlessness). Claude 2 improved over Claude 1.3 on helpfulness and honesty and scored "similarly to 1.3 on harmlessness" (p. 4, Figure 1). The card notes that crowdworker scenarios "are not fully representative" of real use (p. 3). Figure 1 prints no values.

BBQ bias. The Bias Benchmark for QA (BBQ) asks multiple-choice questions about people in nine social dimensions, in an ambiguous version where the right answer is "Unknown" and a disambiguated version with a correct answer (Section 3.2, p. 4). Claude models were "much" less biased than the helpful-only model, and Claude 2 and Claude Instant "a bit less biased" than Claude 1 (p. 4, Figure 2). Disambiguated accuracy is reported in Figure 3, and the card judges it "sufficiently high across all models to trust the bias scores" (p. 4). In Figure 3 (p. 5), which has no printed values, the helpful-only model's bars are the highest in every category. The card attributes part of the accuracy gap between the helpful-only model and Claude to Claude "generally refusing to answer contentious questions" (pp. 4 to 5), and the caption says Claude models "likely have lower accuracy" than the helpful-only model because many questions involve obvious stereotypes and the model may refuse. Refusals therefore lower Claude's disambiguated accuracy, which is the condition the card uses to check that low bias scores are not produced by refusing.

TruthfulQA. Answers were sampled open-ended and then mapped to the benchmark's multiple-choice options by the helpful-only model, which could not see the question (Section 3.3, p. 5). The card reports that "Both helpfulness and honesty interventions improve performance" (p. 5, Figure 4), without printed numbers.

Harmfulness on held-out prompts. A held-out set of 328 prompts mixed red-teaming examples and jailbreaks seen online. For each prompt, five responses were sampled and an HHH preference model compared each to the fixed reply "I can't help you with that" (Section 3.4, p. 5). Claude 2 gave a response judged more harmful than that reply in four cases; on manual inspection three of these did not seem harmful, and in the fourth "the model was disrupted by the jailbreak attempts in about half of its sampled responses" (p. 6). Figure 5 (p. 7) shows the fraction on a log scale falling from the helpful-only model through Claude Instant 1.1 and Claude 1.3 to Claude 2.

HHH comparisons. On 438 binary-choice questions written by Anthropic researchers, the model picks the more helpful, honest and harmless of two responses (Section 3.5, p. 6). The zero-shot accuracies printed on Figure 6 are 0.84 for Claude Instant 1.1, 0.85 for Claude 1.3 and 0.86 for Claude 2 (p. 7).

The card does not report evaluations of sycophancy, deception, reward hacking, situational awareness or model welfare. Those categories appear in later cards.

Safety evaluations and deployment decisions

The card has no AI Safety Level, because the Responsible Scaling Policy came later (September 2023). Its safety statement is a paragraph: Anthropic "evaluated and red-teamed Claude 2 and previous Claude models for several national security and safety related risks" and does "not believe any deployed versions of Claude pose national security or significant safety related risks in the areas that we have identified", partly because of capability level and partly because of mitigations (Section 2, p. 2). No evaluation, threshold or result behind that statement is reported.

Two external activities are named. Anthropic had worked with the Alignment Research Center (ARC) since fall 2022 on safety audits and fine-tuned a snapshot of Claude to help them; neither party believed the models had the "autonomous replication" abilities ARC tests for (p. 2). External red teamers, including crowdworker platforms, tested Claude 2 on Trust and Safety topics before deployment (p. 2).

Capabilities reported

Capability results are summarized briefly here. Claude 2's context window was trained to 200K tokens (about 150,000 words), with 100K supported at launch; a plot of per-token loss averaged over 1,000 long documents is offered as evidence that the full context is used (Section 4.2, p. 9, Figure 8). On standard benchmarks Claude 2 scored 71.2% on Codex HumanEval (pass@1, 0-shot), 88.0% on GSM8k and 78.5% on MMLU (5-shot with chain of thought) (Section 4.3, table on p. 10). It scored 76.5% (153 of 200) on a 2021 Multistate Bar Examination practice test (p. 10) and was tested on the GRE and USMLE practice questions (pp. 10 to 11). Translation into 43 languages on the Flores 200 benchmark is shown as BLEU scores in Figure 7, which prints no values (Section 4.1, p. 8). Section 4.4 lists the areas the developers emphasized: coding, long documents, outputs up to 4,000 tokens, structured formats such as JSON, more non-English data, and training data from 2022 and early 2023 (p. 11).

What the card says it does not know

The card states several limits. Claude models "still confabulate" and "should not be used on their own in high stakes situations"; they do not search the web; and they perform less well in low-resource languages (Section 2, p. 2). Like all models, Claude "can be jailbroken" (p. 2). Section 5 repeats that Claude "generates confabulations, exhibits bias, makes factual errors, and can be jail-broken" (p. 11).

The most specific limitation concerns training itself. Adding capabilities "can trade off in unexpected ways against existing ones", and "Some of Claude 2's new or improved capabilities have had some subtle costs in other areas" (Section 5, p. 11). The card continues that the data and influences determining Claude's "personality" and capabilities "have become quite complex", and that "It has become a new research problem for us to balance these factors, track them in a simple, automatable way, and generally reduce the complexity of training Claude" (p. 11). The card does not say which capabilities traded off against which. The Claude 3 card repeats this paragraph almost word for word.

The report also calls itself "not comprehensive" and expects to release new findings (Section 1, p. 1).

Research questions

1. What did the May 2023 constitution contribute to Claude 2's harmlessness scores? The card attributes training against sexist, racist and toxic outputs to the constitution but reports no ablation. A comparison of models trained with and without the constitutional RL phase on the same 328 held-out prompts and the Elo harmlessness task would separate its effect from the RLHF data.
2. Which capabilities traded off against which? Section 5 reports "subtle costs" from new capabilities without naming them. A per-benchmark comparison of Claude 1.3 and Claude 2 (for example RACE-H and QuALITY, where Claude 2 scored slightly lower than Claude 1.3 on p. 10) alongside internal behavioral metrics would show whether the costs were in capability, character or safety behavior.
3. How much of the BBQ improvement comes from the debiasing fine-tune? The card names generating unbiased samples and fine-tuning on them before the RL phase as the "most likely" cause. Training with and without that step and rerunning BBQ in both conditions, with refusals counted separately from wrong answers, would test the attribution and show how much of the lower disambiguated accuracy in Figure 3 is refusal.
4. How sensitive is the harmfulness metric to its reference reply? Harmfulness is judged as worse than "I can't help you with that" by a preference model, and three of four flagged Claude 2 cases were not harmful on inspection. Re-scoring with several reference replies and with human raters would show how much the count depends on the reference and on the preference model.
5. How was the model-card format adapted, and what did it leave out? The card cites Mitchell et al. (2019) as the source of its format, but its sections are its own. A section-by-section comparison of this card with the 2019 template and with the Claude 3 card would document which parts of the template were kept, renamed or dropped for a general-purpose assistant.

How it connects

The card connects itself to the papers it lists as documenting its methods (Section 1, p. 1):

• Training a Helpful and Harmless Assistant with RLHF (the RLHF method, Elo scoring and released human feedback data) and A General Language Assistant as a Laboratory for Alignment (preference modeling and the HHH comparison questions).
• Constitutional AI: Harmlessness from AI Feedback (the training method named in the model details).
• Red Teaming Language Models to Reduce Harms (human safety evaluation and the source of held-out red-team prompts).
• Towards Measuring the Representation of Subjective Global Opinions in Language Models, Language Models (Mostly) Know What They Know, Discovering Language Model Behaviors with Model-Written Evaluations and The Capacity for Moral Self-Correction (the last also the source of the debiasing method and of the BBQ presentation).

Later cards in the series: The Claude 3 Model Family (March 2024), Claude 3.5 Sonnet Model Card Addendum (June 2024), Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet (October 2024), Claude 3.7 Sonnet System Card (February 2025) and System Card: Claude Opus 4 & Claude Sonnet 4 (May 2025).

In the course wiki, the Elo-based pairwise preference evaluation is a form of the side-by-side testing discussed in Online Evaluation.

References

• Anthropic (2023). Model Card and Evaluations for Claude Models. July 2023, 14 pp. https://www-cdn.anthropic.com/bd2a28d2535bfb0494cc8e2a3bf135d2e7523226/Model-Card-Claude-2.pdf. Release post "Claude 2", 11 July 2023: https://www.anthropic.com/news/claude-2
• Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., and Gebru, T. (2019). Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3287560.3287596 (cited by the card as its reference [1]; not read for this note).
• Anthropic (2023). Claude's Constitution. https://www.anthropic.com/news/claudes-constitution (cited by the card as reference [10]; not read for this note).