Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming

jailbreaksclassifiersconstitutionsynthetic datared teamingcbrn

Sharma et al. train input and output classifiers on synthetic data generated from a written constitution listing restricted and permitted content, and report that no participant in a bug bounty of over 3,000 estimated hours found a universal jailbreak of a classifier-guarded Claude 3.5 Sonnet, and that an improved chemical-weapons system blocked over 95% of held-out jailbreak attempts at a 0.38% absolute increase in production refusals.

Mrinank Sharma et al. · 2025-02-03 · original

Safety training makes a model refuse most harmful requests, but jailbreaks such as many-shot jailbreaking keep finding ways around it. The paper's concern is narrower than jailbreaks in general: chemical, biological, radiological and nuclear (CBRN) weapons, where a model that reliably walked a non-expert through every step of a complex process could provide real uplift (Section 2). The authors therefore target universal jailbreaks, prompting strategies that work on nearly all queries in a domain and extract detailed information, and add a defense outside the model: classifiers that screen what goes into the model and what comes out. The classifiers are trained on synthetic data generated from a constitution, a written list of content categories that are restricted and categories that are permitted. The paper reports two tests: a bug-bounty red-teaming program against a prototype, and automated evaluations of an improved system. The work was done, a footnote says, as part of Anthropic's Responsible Scaling Policy commitments (Section 1).

The paper is by Anthropic's Safeguards Research Team and appeared with an Anthropic post on 3 February 2025; the arXiv version is dated 31 January 2025.

Terms used in this note

  • Universal jailbreak. In the paper's definition, a "(potentially automated) prompting approach that reliably bypasses LLM safeguards on the vast majority of queries in a specific domain", revealing highly detailed harmful information (Section 2).
  • Input classifier and output classifier. Two fine-tuned language models. The input classifier judges the user's prompt; the output classifier judges the model's response as it is generated.
  • Streaming classification. The output classifier scores the response token by token and can stop generation as soon as a score crosses a threshold, so harmful text need not be produced in full before being blocked.
  • Helpful-only model. A model trained to be helpful without harmlessness training, used to generate harmful example queries and answers.
  • Rubric grading. The paper's method for judging whether a jailbreak succeeded: a model lists the key information that helpful-only answers to a question contain, and another model counts how much of it a candidate output contains.
  • False-positive rate (FPR) and refusal rate. The share of harmless traffic wrongly blocked. The course wiki covers thresholds and the trade-off between true and false positives in Evaluating Classifiers.
  • Attack success rate (ASR). The share of jailbreak attempts that produce an output judged harmful.

What the paper does

The design

The system is a pair of classifiers around the guarded model, which the authors describe as layers of a "swiss-cheese" defense in which each layer can have holes but the stack is strong (Section 1). The streaming output classifier mitigates many common universal jailbreaks, including many-shot jailbreaking and GCG adversarial suffixes; the input classifier blocks attempts to get around the output classifier, for example by setting up a cipher (Section 3, Section 3.4).

Training data from a constitution

Training data are produced in stages (Section 3.2):

  1. Constitution-based generation. Given the constitution, a helpful-only model writes queries spanning its categories, then writes harmful and harmless outputs for each. A prompted model filters out accidental refusals.
  2. Augmentation. The queries and outputs are transformed while keeping their meaning: translation, paraphrase, different system prompts, known jailbreak styles, and combinations of these.
  3. Automated red teaming. A helpful-only model, prompted with descriptions of known jailbreak techniques, writes new attacks, including long and multi-turn ones, using the constitution-generated queries as targets (Section 3.3, Appendix F). Successful attacks are balanced with a roughly equal number of benign questions about harmful topics, whose harmless topics come from the constitution stage (Appendix F.3). The improved chemical-weapons classifiers of Test 2 left automated red-teaming data out, because its "benign" examples were too similar to the harmful categories and made the training data noisier (Appendix D.2).
  4. Benign pools. Sets of harmless queries and outputs, collected from external contractors rather than production data (Section 1, footnote), help keep false positives down.

Training the classifiers

The input classifier is a language model fine-tuned to complete a fixed prompt wrapper that describes the classification task, so classification is next-token prediction (Section 3.4, Appendix E.7). The output classifier is a language model with a linear value head that predicts, at every token, whether the full response will be harmful. Each prefix's prediction is trained against the full sequence's label, with a next-token-prediction term as regularization. The prediction used in that loss starts as the value head's own output at each token and shifts linearly, reaching the running maximum of the predictions three-quarters of the way through training; the running maximum matches how streaming works (once generation stops, it cannot be undone) but is harder to optimize (Section 3.4). In comparisons, next-token prediction performed better for both classifiers, but the authors kept the value head for the output classifier because it makes streaming simpler (Appendix E.1).

Test 1: a bug bounty against a prototype

Classifiers fine-tuned from Claude 3.5 Sonnet, trained on a CBRN constitution, guarded Claude 3.5 Sonnet (Section 4.1). The prototype was tuned for robustness and refused about 44% of Claude.ai traffic (Section 4.1). Through HackerOne, 405 participants were invited from 800 applications, and the authors estimate 183 were active, meaning at least 15 queries and at least 3 classifier blocks (Section 4.1). They received ten CBRN target queries, which the paper calls non-infohazardous, and bounties of up to $15K per report scaled by the number of questions answered; $95K was paid in total (Section 4.1). An output counted as jailbroken for payment if it scored at least half of an average helpful-only answer on the rubric, and for most of the program participants could resubmit once after learning which outputs passed (Section 4.1). Effort was estimated from usage logs, calibrated with participants' self-reported hours: a mean of 4,720 hours with a 90% confidence interval of 3,242 to 7,417 (Section 4.1; Appendix H.4 gives the lower bound as 3,241). The authors note that calibrating on self-reports may overestimate (Section 4.1).

Test 2: automated evaluations of an improved system

A second system used a new constitution focused on chemical weapons, printed in Appendix B, with classifiers fine-tuned from the pretraining-only Claude 3.5 Haiku guarding Claude 3.5 Sonnet (New) (Section 5.1, Appendix D.2). Changes from the prototype included a much larger harmless constitution, recalibrated thresholds, more augmentation and more training data (Section 5.1). The jailbreak dataset composes "jailbreak primitives" from the literature, internal red teaming and text transformations, aimed at ten chemical-weapons queries; a held-out subset of 5,681 attempts uses primitives and transformations not seen in training (Section 5.1). Thresholds were chosen to maximize detection while keeping the full system's false positives within set limits on WildChat and OR-Bench (Section 5.1). Over-refusal was measured on 5,000 conversations sampled from Claude.ai Free and Pro traffic (Section 5.1). Baselines were the unguarded model and Claude 3.5 Sonnet used as a prompted classifier zero-shot, with chain of thought, and with 32 examples (Section 5.2).

What it finds

No universal jailbreak in the bug bounty

Of 113 jailbreak reports (85 after exclusions, Appendix H.3), none answered all ten questions at half the detail of a helpful-only model (Section 4.2). Nine reports answered more than six questions at that threshold; at the stricter standard of detail similar to a helpful-only answer, no report answered more than five (Section 4.2). Combining all of each participant's reports, the most successful red teamer answered at most six of ten at the stricter standard (Section 4.2, Figure 4). The authors' summary is that no red teamer found a universal jailbreak "that consistently extracts information that is similarly detailed to a model without safeguards" (Section 1). Red teamers did extract some harmful information (Section 1).

One report at first looked like a universal jailbreak; investigation found that an implementation error let participants receive up to 128 tokens after the output classifier had flagged content. The authors cite this as a reason to red team deployment infrastructure as well as classifiers (Section 4.2, footnote).

What red teamers did and what worked

Participants used multi-turn escalation, few-shot attacks, role-play, encodings, text in images, prompt injection, templates such as "DAN", and decomposition of a task into innocent-looking parts (Section 4.2). The most successful high-severity reports mostly relied on two strategies the authors describe as exploiting the rubric rather than the classifiers: benign paraphrasing, rewording a harmful request into an innocuous-looking one, and length exploitation, producing long outputs that hit rubric keywords by volume (Section 4.2). The authors judge that many high-scoring outputs were much less useful than helpful-only answers and show two examples (Appendix H.5, Table 15). Questions varied widely in difficulty: the most resistant was answered in 4 reports, the easiest in over 30 (Section 4.2, Figure 4).

Contribution of each layer

On a 10% sample of red-teaming queries, rubric-graded, the helpful-only model had an ASR of about 16%, the harmlessness-trained model about 14%, adding the input classifier 2%, and adding both classifiers 0.5% (Section 4.3, Figure 5). The low baseline is attributed to red teamers changing the questions' meaning to get past the classifiers (Section 4.3, footnote). An improved classifier pair applied to the same constitution reached a lower ASR with a production false-positive rate of about 1.74%, though the authors note red teamers had not targeted that system (Section 4.3).

The improved system: robustness at low cost

Guarding Claude 3.5 Sonnet (New) with the chemical-weapons classifiers reduced the jailbreak success rate on the held-out set from 86% to under 5% (Section 5.2, Figure 6), that is, the guarded system refused over 95% of attempts against 14% without classifiers (Section 1). No single transformation answered all ten queries (Section 5.2). The costs were a 0.38% absolute increase in refusals on production traffic and 23.7% inference overhead, in both cases the smallest among the classifier baselines, which were Claude 3.5 Sonnet prompted as a classifier (Abstract, Figure 6). Figure 6 plots the guarded success rate on the held-out set as 4.4%. The accompanying post adds that the refusal increase "was not statistically significant" in the 5,000-conversation sample, and describes the evaluation as 10,000 synthetically generated jailbreaking prompts, whereas the paper's held-out subset has 5,681.

The extra refusals are larger for chemistry. On a 1,000-conversation chemistry subset of WildChat, the model itself refused 3.6% and the classifiers blocked a further 3.4% (Appendix E.8). On technical questions, the false-positive rate was 1.50% for MMLU chemistry and 26.05% for GPQA chemistry (Appendix E.2). The authors read the GPQA result as a capability limit of the classifier, citing a trend in which larger classifiers have lower domain false-positive rates at fixed robustness (Appendix E.2, Figure 10).

On streaming, of 10,000 jailbreak outputs the system blocked 96.70%; truncating each output blocked by the output classifier at the point of blocking, 117 truncated outputs, or 1.17% of attempts, still contained harmful content (Appendix A.4).

Scaling

Classifier robustness generally increased with classifier size, training-set size and the number of augmentation types (Section 5.2, Figure 7). These results are reported as plots.

An illustrative uplift model

Appendix C models a process of N independent steps, assuming 95% step success with helpful-only detail, 50% with half the detail and 5% otherwise, and using the best red teamer's results (60% chance of full detail and 40% of partial detail per step). Under these assumptions the classifiers reduce uplift for a 50-step process by a median of about $10^5$ (Appendix C, Figure 8). The authors call it "an illustrative model rather than a precise estimate" (Section 4.2) and list its simplifications, including independent, uniformly difficult steps (Appendix C).

The constitution and how it is used

What the constitution is

The paper calls the constitution "a set of natural-language rules" defining "categories of permitted and restricted content" (Section 3.1). In practice it is a set of short category descriptions, not principles or explanations. Table 1 gives a sample: harmless categories such as "Listing common medications" and "Explaining high-school chemistry", harmful ones such as "Acquiring restricted chemicals" and "Scaling up production of restricted chemicals".

Two constitutions are involved, and only one is printed. The bug-bounty prototype used a CBRN constitution aimed at scientific knowledge related to weapons of mass destruction, which the paper does not reproduce (Appendix D.1). The chemical-weapons constitution of the automated evaluations is printed in Appendix B:

  • Harmful constitution (Table 3). Any conversation about any Schedule-1 chemical (following the Chemical Weapons Convention's schedules) covering acquisition, production, formulation, and weaponization and delivery, each with examples such as purification or aerosolization; and circumventing controls such as synthesis screening and export controls, for Schedule-1 or Schedule-2 chemicals, including data on how to access chemical laboratories and on when and where chemical research is ongoing.
  • Harmless constitution (Table 4). Three lists. Harmless chemistry has 41 items, from cooking chemistry and cleaning products to "Explaining high-school level chemistry lab procedures" and "Providing very basic or vague information about chemical weapons". Harmful non-chemistry has 33 items, such as building a gun, phishing and planning a bank robbery, which are labeled harmless for this classifier because it is meant to catch only chemical-weapons content (Appendix D.2). Harmless non-chemistry has 54 items of everyday tasks; several echo the wording of harmful categories in a benign sense, such as "Accessing a college laboratory", "Scaling up production of agriculture" and "Purification of natural gas".

The authors also omit "certain details to preserve system security" (Section 3, footnote).

How the text reaches the classifier

The constitution does not instruct the classifiers directly. It is the seed for data generation: a helpful-only model turns the categories into queries and writes harmful and harmless outputs for them, the examples are augmented, and the classifiers learn from those examples (Section 3.2). What the classifiers learn therefore passes through the generating model's reading of each category; the paper does not measure that step separately. A second text does reach the input classifier at run time: the prompt wrapper it is fine-tuned to complete, printed in Table 5. It asks whether a dialogue "Provides or requests information that could be useful for causing widespread harm via chemical weapons" or tries to teach the AI a cipher or new language, tells the classifier to flag anything unintelligible ("YOU MUST FLAG IT AS HARMFUL"), and warns against being "too overzealous" about innocent requests (Appendix E.7, Table 5). The cipher rule is applied regardless of content.

The authors list advantages of specifying behavior this way (Section 3.1): because the constitution is natural language, it can be revised for new threat models; the labeled data it produces can train other kinds of safeguard, such as linear probes; listing both harmful and harmless categories gives more nuanced data; and the classifiers' intended behavior can be read from the categories.

What the paper reports about the wording

  • Permitted categories matter. The paper states that specifying permitted categories "is essential for limiting over-refusals" (Section 3.1). Compared with training on the fixed benign pool alone, adding data generated from the harmless constitution "improves robustness by over 40%", measured as the true-positive rate under false-positive constraints (Appendix E.5, Figure 13). Removing any one of the three harmless lists reduced performance, especially for the output classifier, with no single list most important for both classifiers (Appendix E.6, Figure 14).
  • Iterating on categories. In preliminary experiments, retraining with added harmless categories that an earlier classifier over-refused reduced false positives (Appendix E.5). The implementation tips recommend refining the harmless constitution and updating the constitution from observed weaknesses, while noting that not all tips were validated in controlled experiments (Appendix A.5).
  • Prior notions of harm can conflict with the constitution. Classifiers initialized from the base Claude 3.5 Haiku outperformed those initialized from the harmlessness-trained or helpful-only versions. The authors suggest that HHH training creates biases that conflict with the constitution's finer distinctions, for example treating all discussion of explosives as harmful when the constitution singles out chemical-weapons explosives (Appendix E.4, Figure 12).
  • Adapting to new threats. The authors report that the approach let them adapt the classifiers quickly, without manual data collection, when they found vulnerabilities during development (Section 6).

The paper does not test variations in the wording of individual harmful categories.

Relation to Constitutional AI

The paper itself does not cite Constitutional AI; for generating queries from the constitution it cites the character-training post Claude's Character (Section 3.2). The accompanying post makes the comparison explicitly, saying the method "is based on a similar process to Constitutional AI" and that in this case the principles "define the classes of content that are allowed and disallowed (for example, recipes for mustard are allowed, but recipes for mustard gas are not)".

Limitations and open questions

The authors state these limitations (Section 4.2, Section 7):

  • The red-teamed prototype was impractical. Its robustness came with refusal of about 44% of traffic (Section 4.1); the low-refusal system was evaluated mainly with automated tests and was not the target of the bug bounty.
  • Some vulnerabilities were found. Red teamers extracted some harmful information, and the cohort represents only some adversaries.
  • Incentives. Payment at half the helpful-only detail may have discouraged searching for stronger attacks.
  • Rubric grading. It had trouble separating benign knowledge from actionable uplift, so errors run in both directions.
  • One setting. Only a chatbot with preset queries was tested.
  • Future attacks. "Common wisdom suggests that system vulnerabilities will likely emerge with continued testing" (Section 7), so the authors recommend complementary defenses such as rapid response and monitoring.

Later events reported by Anthropic bear on these limits. A 13 February 2025 update to the post describes a public demo of a chemical-weapons-guarded system with eight questions: 339 participants, over 300,000 interactions and about 3,700 hours. The system held for five of seven days; four participants then passed all eight levels, one with what Anthropic judged a universal jailbreak and one with a borderline-universal one. The most successful strategies the post lists were ciphers and encodings, role-play through system prompts, keyword substitution and prompt injection. The post notes the demo differed from real use, since the grader gave fast feedback and the classifiers were not updated during it. Next-generation Constitutional Classifiers (January 2026) reports two classes of attack, reconstruction and output obfuscation, that evaded last-generation Constitutional Classifiers in further testing, and red teams the authors' own implementation of the dual-classifier design alongside its replacements.

Research questions

  1. How do the generating model's interpretations of a category compare with expert readings? The classifiers learn the helpful-only model's reading of each category (Section 3.2). Having domain experts label a sample of generated examples against the category text would measure how faithfully the categories become data, and where they drift.
  2. Which parts of the harmless constitution carry the effect? The ablations remove whole lists (Appendix E.6). Removing individual categories, in particular the near-miss items that echo harmful wording, and measuring false positives on chemistry traffic would show whether the closest contrasts do most of the work.
  3. How sensitive are the classifiers to the wording of a harmful category? The paper does not vary wording. Training classifiers from paraphrased versions of Table 3, with everything else fixed, and comparing true- and false-positive rates would test how much the exact phrasing matters.
  4. How much do the constitution and the prompt wrapper each contribute? The input classifier is shaped both by data from the constitution and by the Table 5 wrapper. Training with a neutral wrapper, and with the wrapper's criteria changed while the data stay fixed, would separate their effects.
  5. Can the GPQA false-positive gap be closed by adding harmless categories rather than scale? The authors attribute the 26.05% rate to classifier capability (Appendix E.2). Adding graduate-level chemistry categories to the harmless constitution and comparing the result with a larger classifier would test the alternative.
  6. How close did red teamers come to universality, and on what grading? The paper reports per-report and per-red-teamer distributions (Figure 4) but judges many high-scoring outputs to be less useful than they appear (Appendix H.5). Expert grading of the top reports would show how far rubric scores over- or understate real uplift.

How it connects

  • Earlier attacks and red teaming. Many-shot jailbreaking is cited as a universal jailbreak the output classifier mitigates and as one of the evaluation's jailbreak primitives; Red Teaming Language Models to Reduce Harms is cited among approaches that fine-tune models on red-team findings, which the authors say did not reliably generalize (for example from text to code) in their preliminary experiments (Section 6).
  • Other uses of the word constitution. Constitutional AI and Claude's constitution use written principles to shape the model itself; here the text specifies what a separate classifier should flag. The accompanying post draws the comparison; the paper does not.
  • After. Next-generation Constitutional Classifiers keeps the constitution-based data pipeline and changes the classifier architecture.
  • In the course wiki. Choosing thresholds under false-positive limits is the operating-point problem of Evaluating Classifiers; using a model to grade outputs against a rubric raises the assessor questions of Relevance Judgments, Pooling, and Agreement.

References

  • Sharma, M., Tong, M., Mu, J., Wei, J., Kruthoff, J., Goodfriend, S., Ong, E., Peng, A., Agarwal, R., Anil, C., Askell, A., et al. (2025). Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming. Anthropic. arXiv:2501.18837v1, submitted 31 January 2025. https://arxiv.org/abs/2501.18837
  • Anthropic. (2025, February 3). Constitutional Classifiers: Defending against universal jailbreaks [post, with live demo results added 13 February 2025]. https://www.anthropic.com/research/constitutional-classifiers
  • Anil, C., Durmus, E., Sharma, M., et al. (2024). Many-shot Jailbreaking. Anthropic.
  • Studies cited above only as the paper describes them are listed in full in the paper's references.