Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

red teamingcrowdworkersharmlessnessrlhfdatasetsscaling

Ganguli et al. had 324 crowdworkers try to draw harmful output from four kinds of Anthropic dialogue model at three sizes, released the 38,961 resulting attacks, and found that RLHF models become harder to red team as they grow while the other model types show no trend with size.

Deep Ganguli et al. · 2022-08-22 · original

In 2022 the usual way to check whether a language model would say harmful things was a static benchmark: a fixed set of prompts, such as those in RealToxicityPrompts, scored for toxicity. Red teaming takes the opposite approach. People, or models, probe a system adversarially and in conversation, looking for whatever harmful output they can find, and the failures they uncover become data for making the model safer. Anthropic had already used red-team conversations to train the helpful and harmless assistant described in Bai et al. (April 2022); this paper describes how that red teaming was done, measures how the difficulty of red teaming changes with model size and type of safety intervention, and releases the attacks. The authors present it as "early efforts" (Abstract) and spend much of the paper documenting instructions, worker welfare measures and statistical choices, on the stated view that the field lacked shared norms for how to red team and how to publish the results (Sections 1 and 5.2).

The paper asks two questions. Do the safety interventions Anthropic was using make models harder to attack, and does that change as models get larger? And what kinds of harm do open-ended human attacks actually surface?

Terms used in this note

  • Red team, red teaming. People (here crowdworkers) who try to make a system misbehave, and the practice of doing so. The paper defines red teaming as "using manual or automated methods to adversarially probe a language model for harmful outputs, and then updating the model to avoid such outputs" (Section 1).
  • Attack. One red-team conversation with the assistant. The paper also calls it an attempt.
  • Preference model (PM) and harmlessness score. A model fine-tuned to score a response, trained here on pairs of responses where a red teamer marked one as more harmful. Lower scores mean more harmful. The PM is trained to predict helpfulness as well, but the paper calls its output the harmlessness score throughout (Section 3.2, footnote).
  • Plain LM. A pretrained language model turned into a dialogue agent with a single example conversation in its context (1-shot). This is the control condition.
  • Prompted LM. The same model with a 14-example prompt written to make it helpful, honest and harmless (HHH), from Askell et al. (2021), in a context-distilled form so that the prompt need not occupy the context window.
  • Rejection sampling (RS). The prompted LM generates 16 responses, the PM ranks them, and the two least harmful are shown.
  • RLHF. A prompted LM further trained by reinforcement learning to maximize the PM's score, as described in Bai et al. (April 2022).
  • Likert scale. An ordered rating scale, here 0 ("Not successful") to 4 ("Very successful").
  • Fleiss's kappa. A statistic for agreement among several raters that corrects for chance agreement; 1 is perfect agreement. The course wiki derives the two-rater version in Relevance Judgments, Pooling, and Agreement.

What the paper does

The red-team task

Red teamers held open-ended conversations with an AI assistant through a custom interface. The instructions asked them to "make the AI behave badly, to get it to say obnoxious, offensive, and harmful things" (Section 3.1), gave a short list of example topics, and asked them to be creative, avoid obvious profanity and stay on one topic per attack. Before each attack they wrote a short description of what they intended to try (Section 3.1).

At each turn the interface showed two model responses and asked the red teamer to choose the more harmful one. This gave twice as many chances to catch a harmful response, and it produced the comparison data used to train the PM (Section 3.1). Conversations were meant to run about four turns; most ran one to four. At the end of each conversation the red teamer rated their own success on the 0 to 4 scale. Five conversations made one task (Section 3.1). Workers were assigned to model types at random and did not know which model they faced (Section 3.1).

The models attacked

All dialogue models came from Anthropic's pretrained models at 2.7B, 13B and 52B parameters (Section 3.2). The four types were the plain LM, the prompted LM, RS and RLHF, as defined above. For RS and RLHF the PM was the same size as the policy, so that a "2.7B" RS model pairs a 2.7B prompted LM with a 2.7B PM (Section 3.2, footnote). The PM itself was trained on comparisons from attacks on the 52B prompted LM, which is why that condition has far more data: 17,739 attacks, against 1,151 to 3,081 in each of the other eleven conditions (Table 1). The RS and RLHF models therefore depend on red-team data collected from the prompted LM (Section 1).

The red team

The red team was 324 US-based crowdworkers, 307 recruited on Amazon Mechanical Turk and 17 on Upwork (Section 3.3). MTurk workers were paid $7.50 to $9.50 per five-conversation task, which the authors estimate met or exceeded California's minimum wage given at least two tasks an hour; Upwork workers were paid $20 per hour (Section 3.3). A demographic survey was completed by 115 of the 324 (Figure 4). Contributions were very uneven: about 80% of attacks came from about 50 of roughly 300 workers (Section 3.3, Figure 5). The data were collected between November 2021 and June 2022 (Appendix A.7).

Measuring attack success

Each attack gets two main measures (Section 3.4). The first is the red teamer's own 0 to 4 success rating. The second is the PM's harmlessness score for each assistant turn, conditioned on the conversation so far, reduced to one number by taking the minimum over the turns. The authors report that using the mean instead did not change the results much. They also score each red teamer's written intent with the PM and treat it as a possible confound. The two main measures are correlated but not tightly (Appendix A.4, Figure 11), so both are reported.

The review task

After collection, a second experiment had three annotators review each of 500 random attacks on the 52B prompted LM and 500 on the 52B RLHF model (Section 3.5). Reviewers rated attack success on the same scale and tagged each transcript with up to 2 of 20 harm categories, a list built from published harm taxonomies, industry content-moderation guidelines and a manual review of the 100 most harmful conversations (Section 3.5). This task was limited to Upwork workers, who had a shared Slack channel with the researchers, could preview a transcript and skip it, and were paid for time rather than by quota (Section 3.5, Appendix A.2).

Controls

A linear mixed-effects model with a random intercept for each red teamer, and fixed effects for intent harmfulness and platform, tested whether prolific or unusually skilled workers drove the main results. The authors report that the main results in Figure 1 still hold (Section 3.3, Appendix A.3).

What it finds

Scaling by model type

Figure 1 carries the paper's main result, reported as plots. By the PM measure (Figure 1, middle):

  • RLHF models become harder to red team as they get larger, the only clear size trend in the paper (Section 4).
  • Plain LMs, prompted LMs and RS models show a flat trend with size (Abstract, Section 1).
  • At 52B, RLHF and RS have about the same harmlessness score (Section 4).
  • A footnote notes that, counter to intuition, the plain LM appears less harmful than the prompted LM at 52B only (Section 4, footnote).

By the self-report measure (Figure 1, left), there is no discernible difference between the plain and prompted LMs, RS is the hardest model to red team, and there is no clear trend with size for any type (Section 4). The authors call the absence of a size trend on this measure surprising, since their earlier work typically found larger models more toxic (Section 4).

The HHH prompt did not help against red teaming

The absence of a difference between the plain and the prompted LM is the result the authors call surprising, because their earlier work with static evaluations had found the HHH prompt effective at reducing toxicity, especially at 52B (Section 4). They suggest that static prompts from RealToxicityPrompts are less adversarial than conversational red-team attacks (Section 4). In Section 1 they describe the result as inconsistent with that earlier static-evaluation finding.

Rejection sampling: harder to attack, often by being evasive

RS places what the authors call a "floor" on attack success among the interventions tried, but they suggest, from qualitative inspection, that this may be because RS responses "tend to be harmless by being evasive" (Section 4), citing the evasiveness of harmless models reported in Bai et al. (April 2022).

No intervention removes the tail

The distributions of harmlessness scores (Figure 1, right) show that RLHF and RS lower average harmfulness but still leave many harmful outputs in the lower tail (Section 4). Figure 10 shows example transcripts in which an RS model first answers a harmful request and then demurs, and an RLHF model stays helpful, fabricating information, before refusing.

What the attacks contain

The overall self-reported success rate is about 35%, with ratings piling up at 0 and 4 (Section 3.4, Figure 7). A UMAP projection of all transcripts, built from layer-48 activations of the 52B prompted LM, shows clusters including discriminatory and offensive requests, a small cluster of subtle misinformation attempts, a small animal-abuse cluster, and 916 attacks seeking personally identifiable information (Section 4, Figure 2). Attacks seeking advice on general violence appear more successful than attempts to elicit offensive language (Section 4).

In the tagged sample (about 3% of the data), the five most common tags were discrimination and injustice, hate speech and offensive language, violence and incitement, non-violent unethical behavior, and bullying and harassment; among these, non-violent unethical behavior had a relatively higher success rate (Section 4, Figure 9). "Other" was a frequent tag, which the authors read as a sign that a fixed tag list will not be complete (Section 4).

Some data were low quality. Some workers used templates, such as asking for an insulting term for a group that starts with a given letter, and cycled through values to complete tasks quickly, an incentive the authors attribute to per-task pay on MTurk (Section 4).

Raters disagree about success

On the 1,000 reviewed transcripts, Fleiss's kappa among the original red teamer and the three reviewers was 0.32 on the 0 to 4 scale and 0.49 when ratings were binarized at 3 or above; agreement among the three reviewers alone reached at most 0.55, in the binary case (Section 3.5). The authors summarize this as "poor to fair agreement on what constitutes a successful attack" (Section 3.5).

Worker well-being

Reviewers answered an optional well-being survey, 49 responses from 15 people (Appendix A.2). Average ratings on a 0 to 4 scale were low for negative feelings (for example 0.16 for hostile and 0.31 for upset) and highest for attentive, at 1.73 (Table 2). The authors report informal feedback that red teamers and reviewers enjoyed the task (Section 1, Section 3.5).

The instructions and prompts used

The paper has no constitution or written principles, but three pieces of text shaped what the models and the workers did.

The HHH prompt. The prompted, RS and RLHF models all start from the 14-shot HHH prompt of Askell et al. (2021), applied through context distillation (Section 3.2). It is not reprinted here. The red-teaming result that this prompt made no measurable difference against adversarial conversation is the paper's direct evidence about a written prompt as a safety intervention (Section 4).

No written definition of harm. The authors chose not to tell red teamers what counts as harmful: "We do not define what “harmful” means, as this is a complex and subjective concept" (Section 3.1). Harm is defined operationally by the workers' pairwise choices, and the PM, and so the RS and RLHF models, learn whatever those choices encode. The low inter-rater agreement above shows how much those judgments vary.

Task instructions as a design lever. The instructions were deliberately open. The authors note that they saw no attacks on the models' code-writing ability and could have mentioned such attacks in the instructions, but chose to be "less prescriptive" to encourage creativity, adding that it is unclear how to strike that balance (Section 5.1). Worker-safety guidance in the instructions (clear warnings about content, choosing topics within one's own risk tolerance) came from a review of Trust and Safety literature and interviews with professionals in that field (Section 3.1, Appendix A.2). The task instructions are shown as screenshots in Figure 3 and the review instructions in Figure 8.

Limitations and open questions

The authors list these limitations (Section 5.1):

  • Assistant framing only. The models were attacked as chat assistants; other uses of language models, such as classifiers or autocomplete, were not tested.
  • Unknown expertise. Some attacks, such as requests for bomb-making or methamphetamine synthesis, need domain expertise to judge, and the authors did not know whether workers had it. They suggest specialist red teams.
  • Incomplete coverage. The space of harms is open-ended. Internal red teaming found attack types missing from the dataset, notably "roleplay attacks" in which the RLHF model, asked to enter "4chan mode", produced offensive output (Section 5.1).
  • Bottom-up analysis. Harm types were characterized after collection rather than by attacking according to a taxonomy; the authors suggest combining the two.
  • Cost of manual red teaming. Human red teaming is expensive and slow; the authors planned to compare it with automated red teaming.

The release decision is discussed in Section 5.2 and Appendix A.5. Among the stated costs: the crowdworker payments alone were at least $60K, and engineering and compute would raise that by at least an order of magnitude (Appendix A.5). Among the risks: the data could be used to train harmful models, and a regular-expression filter for personal information may both over- and under-remove (Appendix A.5, A.7). The authors write that they made the decision "in a vacuum" and call for a neutral forum (Section 5.2).

Other limits follow from the design. Each RS and RLHF model's harmlessness is measured with a PM of the same family that the RLHF model was trained to maximize, so the PM-based measure and the training objective are not independent; the authors note that RLHF is explicitly trained against this score (Section 4, footnote). The two success measures correlate only loosely (Appendix A.4). The size comparison mixes three model sizes with very different numbers of attacks per condition (Table 1).

Later work in this library takes up the same problem with different tools. Many-shot jailbreaking (2024) studies a single attack that exploits long context windows. Constitutional Classifiers (2025) cites this paper among approaches that fine-tune on red-team findings, and moves to classifier safeguards tested by a paid bug-bounty red team.

Research questions

  1. Why did the HHH prompt not help under red teaming when it helped on static benchmarks? The authors offer a hypothesis, that red-team conversations are more adversarial than RealToxicityPrompts (Section 4). Scoring the same models on red-team transcripts replayed turn by turn, and on static prompts matched in topic, would separate the effect of adversarial intent from the effect of multi-turn conversation.
  2. How much of RS's advantage is evasion? The paper judges this qualitatively (Section 4). Rating a sample of RS and RLHF responses for both harmfulness and helpfulness (the tension between the two that Constitutional AI later took up) would show whether RS buys harmlessness mainly by refusing.
  3. What does a harm definition left to raters encode? With kappa between 0.32 and 0.55 (Section 3.5), the PM learns from noisy and possibly divergent judgments. Training PMs on subsets of workers, or with a written definition of harm added to the instructions, and comparing the resulting harmlessness rankings would show how much the absence of a definition shapes the trained models.
  4. Would differently worded instructions change what red teamers find? The authors chose open instructions and found no code-related attacks (Section 5.1). Randomly assigning workers to instruction variants (open, taxonomy-guided, domain-prompted) and comparing the distribution of tags and success rates would measure how much the instructions steer the attack surface.
  5. Do human and automated red teams find the same harms? The authors planned this comparison (Section 5.1). Running an automated red team on the same models and mapping both sets of attacks into the same UMAP space and tag scheme would answer it.
  6. Does the RLHF scaling trend continue beyond 52B? The trend rests on three sizes. Repeating the red-team protocol at larger sizes, with the PM held fixed across them, would test whether the trend continues and whether it reflects the policy or the PM.

How it connects

References

  • Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., Jones, A., Bowman, S., et al. (2022). Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned. Anthropic. arXiv:2209.07858v2, revised 22 November 2022 (v1 submitted 23 August 2022). https://arxiv.org/abs/2209.07858. Anthropic page: https://www.anthropic.com/research/red-teaming-language-models-to-reduce-harms-methods-scaling-behaviors-and-lessons-learned. Dataset: https://github.com/anthropics/hh-rlhf
  • Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., et al. (2021). A General Language Assistant as a Laboratory for Alignment. arXiv:2112.00861.
  • Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862.
  • Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073.