The Claude 3 Model Family: Opus, Sonnet, Haiku

system cardmodel cardclaude 3responsible scaling policyconstitutional aievaluationdocumentation

Anthropic's model card for Claude 3 Opus, Sonnet and Haiku (March 2024), which keeps the model-card format of the Claude 2 card, adds the first catastrophic-risk evaluations under the Responsible Scaling Policy (all three models classified ASL-2), names one constitution principle added from Collective Constitutional AI, and reports refusal, honesty, bias and Trust and Safety evaluations alongside capability benchmarks.

Anthropic · 2024-03-04 · original

Anthropic announced the Claude 3 family on 4 March 2024 with a model card titled "The Claude 3 Model Family: Opus, Sonnet, Haiku". Three models share it: Claude 3 Opus (the most capable), Claude 3 Sonnet and Claude 3 Haiku (the fastest and cheapest), all able to take images as input (Abstract, p. 1). The document calls itself a model card and, like the Claude 2 card, says it "is not intended to encompass all of our research", pointing to papers for methods (Section 1, p. 1). It differs from its predecessor in one structural way: it reports "the catastrophic risk assessments we committed to in our Responsible Scaling Policy" (Abstract, p. 1). The Responsible Scaling Policy (RSP) is Anthropic's framework, first published in September 2023, that ties required safety and security measures to measured capability levels called AI Safety Levels (ASLs).

The card was later extended. The PDF at the source URL carries two notices under the abstract: an addendum covering Claude 3.5 Sonnet was added on 20 June 2024, and an addendum covering an upgraded Claude 3.5 Sonnet and Claude 3.5 Haiku on 22 October 2024 (p. 1). The main card occupies pages 1 to 42 (body to p. 32, appendices to p. 38, references to p. 42), and the addenda follow. Those addenda have their own notes: Claude 3.5 Sonnet Model Card Addendum and Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet. One caption in the main card also records a later correction (Figure 16, p. 23, "updated on June 20, 2024 to fix incorrect colors"). Page numbers below refer to the current PDF.

Structure of the card

The body has eight sections:

  1. Introduction (about a page).
  2. Model Details (about two pages): intended, unintended and prohibited uses, safeguards against misuse, training data, training process, and release decisions and maintenance.
  3. Security (a paragraph).
  4. Social Responsibility (about a page): Constitutional AI, labor, sustainability.
  5. Core Capabilities Evaluations (about 19 pages, pp. 4 to 23): benchmarks, standardized tests, vision, "behavioral design" (refusals, instruction following), human preferences, multilingual results, factual accuracy and long context.
  6. Catastrophic Risk Evaluations and Mitigations (about three pages, pp. 23 to 26): RSP evaluations for autonomous replication, biology and cyber.
  7. Trust & Safety and Societal Impact Evaluations (about five pages, pp. 26 to 31): multimodal red teaming, elections, discrimination, BBQ.
  8. Areas for Improvement (under a page).

Appendices give refusal examples, vision examples, GPQA details, multimodal red-teaming prompts and the prompt format used for multiple-choice evaluations.

Compared with the Claude 2 card (14 pages, about one and a half pages of model-card paragraphs), the Claude 3 card triples in length, splits the model-card material into more labeled subsections, and adds Sections 3, 4 and 6. Capability evaluation is still the largest part. In the Claude 3.7 Sonnet System Card (February 2025), the document type changes name and the balance shifts to harms and safety.

Training and the written principles

Data and cutoff. Claude 3 models are trained on public Internet data "as of August 2023", non-public data from third parties, data from labeling services and paid contractors, and internally generated data, with deduplication and classification used for cleaning (Section 2.5, p. 3). The card states that the models "have not been trained on any user prompt or output data submitted to us by users or customers" (p. 3), and describes the crawler's handling of robots.txt and sign-in pages (p. 3). The knowledge cutoff is given as August 2023 in the introduction (p. 1); Section 2.2 says the models answer using "data up to mid-2023" (p. 2).

Training process. Training combines pretraining with "human feedback techniques that elicit helpful, harmless, honest responses" (Section 2.6, p. 3). Constitutional AI is described as aligning Claude "during reinforcement learning by explicitly specifying rules and principles based on sources like the UN Declaration of Human Rights" (p. 3).

The constitution and one added principle. Section 4.1 describes the constitution, as the Claude 2 card did, as "a set of ethical and behavioral principles that the model uses to guide its outputs", and says that "The majority of the principles in Claude's constitution are the same as those we published in May 2023" (p. 4). It names one change: following Collective Constitutional AI, Anthropic "added an additional principle informed by our public input process, which instructs Claude to be understanding of and accessible to individuals with disabilities, resulting in lower model stereotype bias" (p. 4). Section 2.6 describes the same addition as "an additional principle to Claude's constitution to encourage respect for disability rights" (p. 3). The card does not print the principle's wording or report the bias evaluation behind "resulting in lower model stereotype bias". A later card dates the addition differently: the Claude 3.7 Sonnet card says the principle was added "Starting with Claude 3.5 Sonnet (new)" (Section 1.1, p. 3).

Behavioral design. Section 5.4 introduces "behavioral design" for shaping core behaviors and responses, which "sometimes requires carefully balancing competing objectives" (p. 10). The areas named are refusals, honesty and truthfulness, instruction following and formatting. The models were trained to handle instructions with "absolute language (e.g., only, always, etc.)" and to reduce "laziness" in long outputs (Section 5.5.1, p. 14).

Usage policy and runtime instructions. The Acceptable Use Policy (AUP) lists prohibited uses, among them political campaigning or lobbying, surveillance, social scoring, criminal justice decisions, law enforcement, and decisions on financing, employment and housing; users must read and acknowledge it (Section 2.3, p. 2). Prompts flagged by classifiers as violating the AUP "trigger an instruction to our models to respond even more cautiously" (Section 2.4, p. 2), a mechanism the card later names "prompt modification" (Section 7, p. 26). Severe cases are blocked, and repeated violations can end access.

Labor and release process. Data workers select preferred outputs, rate outputs on criteria such as helpfulness and harmlessness, and red team the models (Section 4.2, p. 4). Release decisions draw on the NIST AI Risk Management Framework (Section 2.7, p. 3).

Alignment, honesty and character assessments

The card has no section on alignment in the later sense (deception, hidden goals, reward hacking or welfare). The evaluations closest to it concern refusals, honesty and bias.

Refusals. On the WildChat dataset, Claude 3 models refused harmless prompts "much less often" than Claude 2 while still refusing toxic ones (Section 5.4.1, p. 10, Figure 2, which prints no values). On XSTest, about two hundred non-malicious prompts that superficially resemble harmful ones, the incorrect refusal rate fell "from 35.1% with Claude 2.1 to just 9%" for Claude 3 Opus (p. 10). Refusals were scored both by a grader model with few-shot prompts and by string matching (p. 10).

Honesty and factual accuracy. The card frames honesty as having "the model's assertions be in line with its knowledge", and says the model was trained "to output fewer claims that it can identify are false" (Section 5.7, p. 18). Three internal question sets score each answer as correct, incorrect, or "I don't know". On "100Q Hard", obscure questions chosen to elicit errors from Claude 2 models, Claude 3 Opus answered 46.5% correctly, "almost a 2x increase in accuracy over Claude 2.1"; on "Multi-factual" it reached 62.8% correct against 43.8% for Claude 2.1, and its incorrect rate fell by about half (p. 18, Figure 11). The card notes that ideal behavior would move more of the incorrect answers into the unsure category without losing correct ones (p. 18).

Human preference. In head-to-head comparisons by raters, domain experts in finance, law, medicine, STEM and philosophy preferred Claude 3 Sonnet over earlier models "60-80% of the time" (Section 5.5, p. 12). The card converts win rates to Elo differences, noting that "a 64% win rate corresponds to a 100 point Elo score difference", and puts Sonnet's gain over Claude 2 at "roughly 50-200 Elo points" (p. 12). In adversarial preference tasks, where crowdworkers tried to get a model to say something false or harmful, Figure 8 (p. 15) prints win rates against a helpful-only baseline: on honesty 69% for Claude 3 Sonnet, 68% for Claude 2.1, 66% for Claude 2.0 and 61% for Claude Instant 1.2; on harmlessness 87%, 87%, 84% and 85%.

Recognizing a test. In the "Needle In A Haystack" long-context test, where a sentence is inserted into a long corpus and the model is asked to retrieve it, Claude 3 Opus reached 99.4% average recall and 98.3% at 200K tokens (Section 5.8.2, p. 22, Table 7). The card adds that Opus sometimes showed "a remarkable ability to identify the synthetic nature of the task" and prints a response in which the model says the inserted sentence may have been placed "to test if I was paying attention" (pp. 22 to 23). The card treats this as a possible limitation of contrived evaluations; it is the earliest report in this series of a model remarking on being tested.

Bias. On Anthropic's discrimination evaluation (see Evaluating and Mitigating Discrimination in Language Model Decisions), Opus and Sonnet had "comparable discrimination scores to Claude 2.1" and Haiku comparable or lower scores than Claude Instant 1.2 (Section 7.3.1, p. 28, Figures 19 and 20). On BBQ, the text says Claude 3 Opus "outperforms all Claude 2 family models", with "the highest accuracy in disambiguated context and the lowest bias score in ambiguous context overall" (Section 7.4, p. 31). Figure 21 (p. 31, no printed values) shows Opus with the lowest ambiguous bias score in every category, but in the disambiguated accuracy panel Claude 3 Sonnet's bar is higher than Opus's in every category, and Opus is below Claude 2 or Claude Instant 1.2 in some (for example religion).

Safety evaluations and deployment decisions

ASL determination. Anthropic evaluated for three sources of catastrophic risk: biological capabilities, cyber capabilities, and autonomous replication and adaptation (ARA) (Section 6.2, p. 24). Evaluations used "a lower-refusal version" of Opus, in several rounds including earlier training snapshots (p. 24). The result: "no indicators of catastrophic risk, and therefore we classify all of our Claude 3 models as ASL-2" (p. 24).

Thresholds and results. For ARA, the pre-specified ASL-3 warning indicator was passing 50% or more of five tasks with at least a 10% pass rate; Opus "failed at least 3 out of 5 tasks", made partial progress on some, and passed a simplified version of a task to set up a copycat of the Anthropic API (Section 6.2.1, p. 24). For biology, the early indicator was a greater than 25% accuracy increase for sub-expert users over Google alone, or a 25% jump over Claude 2.1 on either of two question sets; neither was crossed, and a human uplift trial found "what we believe is a minor uplift in accuracy" without safeguards (Section 6.2.2, p. 25). The card notes that on some biology evaluations Opus did worse than Claude 2.1, "suggesting that we may have under-elicited the model's capabilities" (p. 25). For cyber, the threshold was success in at least 1 of 5 attempts in at least 2 of 6 classes of expert vulnerability tasks; Opus scored 30% on one vulnerability discovery task, which expert advisors judged not concerning because it needed substantial hints (Section 6.2.3, pp. 25 to 26).

Mitigations. At ASL-2, model weights are hardened against opportunistic attackers, and automated detection of CBRN and cyber prompts runs on deployed models (Section 6.3, p. 26).

Trust and Safety. In multimodal red teaming across topics such as child safety, weapons, hate speech and extremism, Claude 3 Opus responded harmlessly to 370 of 378 prompts (97.9%) and Sonnet to 375 of 378 (99.2%) (Section 7.1.1, p. 27). The two areas for improvement named were hallucinated image contents and failure to recognize harmful images paired with innocuous text (pp. 27 to 28). Section 7.2 describes election-related policies and testing in a paragraph (p. 28).

What the card says it does not know

On catastrophic risk, the card states that "evaluations are a hard scientific problem" (Section 6.2, p. 24) and that its results "do not comprehensively rule out risk", with more elicitation work expected to change ARA and CBRN results (Section 6.4, p. 26).

Section 8 repeats the Claude 2 card's list (confabulation, bias, factual errors, jailbreaks), adds that the models refuse to identify people in images and can describe images inaccurately, and keeps the Claude 2 card's paragraph on training trade-offs nearly verbatim: some new capabilities "have had some subtle costs in other areas", the data and influences that determine Claude's "personality" and capabilities "continue to be quite complex", and balancing and tracking them "in a simple, automatable way" remain "key research problems" (pp. 31 to 32).

Research questions

  1. What did the disability-rights principle do? The card says the added principle resulted in "lower model stereotype bias" but reports no measurement isolating it. Training with and without the single principle and comparing BBQ disability-status scores would test the claim directly, and the result would show whether one added sentence in a constitution produces a measurable, targeted change.
  2. When was the principle added? The Claude 3 card (March 2024) says it was added "With Claude 3 models"; the Claude 3.7 Sonnet card says "Starting with Claude 3.5 Sonnet (new)". Anthropic's own documentation of the constitution's version history, if released, would settle which models were trained with it.
  3. What exactly did "a majority of the principles ... are the same" leave unchanged? The card gives no count of changed, removed or added principles beyond the one. A published diff between the May 2023 principles and the Claude 3 constitution would answer it.
  4. How much did elicitation limit the catastrophic-risk results? The card suspects under-elicitation on biology (p. 25), says some cyber failures "may be solvable with better prompting and fine-tuning" (p. 26), and expects elicitation on ARA and CBRN tasks to improve (Section 6.4, p. 26). Rerunning the same tasks on Claude 3 Opus with the elicitation methods of later cards would show how much headroom the March 2024 results left.
  5. How often did Claude 3 Opus remark on evaluations as artificial? The needle-in-a-haystack response is reported as an example, with no rate. Counting such remarks across a set of evaluations, as the Claude Opus 4 card later does for its automated audits, would give a baseline for comparing later models.

How it connects

The card cites as background (Challenges in Evaluating AI Systems, Red Teaming Language Models to Reduce Harms, The Capacity for Moral Self-Correction, Towards Measuring the Representation of Subjective Global Opinions in Language Models) and the RSP (Section 1, pp. 1 to 2). It names Constitutional AI as a training method and Collective Constitutional AI as the source of the added disability principle (pp. 3 to 4), and uses the discrimination evaluation from Evaluating and Mitigating Discrimination in Language Model Decisions (p. 28). Section 7.2 points to Anthropic's elections post; the later Testing and Mitigating Elections-Related Risks note covers related work.

Neighbors in the series: Model Card and Evaluations for Claude Models (July 2023) before it, and the two addenda appended to it, Claude 3.5 Sonnet Model Card Addendum (June 2024) and Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet (October 2024).

References

  • Anthropic (2024). The Claude 3 Model Family: Opus, Sonnet, Haiku. Model card, March 2024; updated 20 June 2024 and 22 October 2024 with addenda. https://www-cdn.anthropic.com/c6a80a657af445f40e31afac050f3bf76d3b1404.pdf. Announcement post "Introducing the next generation of Claude", 4 March 2024: https://www.anthropic.com/news/claude-3-family
  • Anthropic (2023). Anthropic's Responsible Scaling Policy, September 2023 (cited by the card; not read for this note).
  • Röttger, P., et al. (2023). XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (cited by the card for XSTest; not read for this note).