Constitutional AI: Harmlessness from AI Feedback

constitutional airlaifwritten principlespreference modelsalignment

Trains a harmless but non-evasive assistant with no human labels for harmfulness. A model critiques and revises its own responses against 16 written principles, and a model's choices between pairs of responses, guided by 16 more principles, replace human harmlessness labels in reinforcement learning.

Yuntao Bai et al. · 2022-12-15 · original

By late 2022 the standard way to make a language model assistant behave well was reinforcement learning from human feedback (RLHF): crowdworkers compare pairs of model responses, a model is trained to predict their preferences, and the assistant is optimized against that predictor. Anthropic's own version of this (Bai et al., April 2022) relied on large sets of crowdworker comparisons, and its helpful and harmless assistant often refused controversial questions or gave evasive answers (Section 1.1 of the present paper). The Constitutional AI paper asks whether the human labels for harmfulness can be removed entirely, so that the only human input on harm is a short list of principles written in natural language, plus a few examples. It calls that list a constitution and the method Constitutional AI (CAI).

The paper explains the term in Section 1. A system can be made less harmful "entirely through the specification of a short list of principles or instructions", and the word is also meant to stress that any deployed system is governed by some principles, "even if they remain hidden or implicit". Both phrases come from the first page of the introduction. The paper is the origin of the training method behind the constitutions Anthropic later published, which makes it the reference point for every later note in the Constitutional AI thread.

Terms used in this note

  • Preference model (PM). A language model fine-tuned to give a scalar score to a conversation ending in a response, trained on pairs so that the preferred response scores higher. In RLHF its score is the reward.
  • RLHF and RLAIF. Reinforcement learning from human feedback, and the paper's variant, reinforcement learning from AI feedback, in which the comparison labels come from a model rather than from people.
  • Red-team prompts. Conversation openings written to draw harmful output from a model. The paper uses prompts from Anthropic's earlier red-teaming work and many more generated by a model.
  • Helpful RLHF and HH RLHF. Anthropic's earlier assistants: one trained on human helpfulness comparisons only, the other on helpfulness and harmlessness comparisons together (HH).
  • SL-CAI and RL-CAI. The two stages of the new method and the models they produce: supervised learning on self-revised responses, then reinforcement learning against a preference model trained partly on AI labels.
  • Feedback model. The model that reads a conversation, two candidate responses and a principle, and says which response is better.
  • Chain of thought (CoT). Prompting a model to write out its reasoning before answering.
  • Elo score. A rating fitted from many pairwise comparisons, as in chess. Only differences between Elo scores are meaningful.
  • Evasive response. A response that avoids a sensitive request without engaging with it, such as "I can't answer that". The paper treats evasiveness as a failure distinct from harmfulness.

What the paper does

The starting point: a helpful-only assistant

All experiments use Anthropic's series of pretrained language models, with most results reported for the 52-billion-parameter model. The method starts from a helpful RLHF model trained only on human helpfulness comparisons, which follows instructions well and is also willing to help with harmful requests (Section 1.4, and footnote 1 in Section 1). For comparison, the authors also trained new helpful and HH RLHF models with human feedback.

Stage 1: critique, revision and supervised learning (SL-CAI)

The supervised stage turns harmful responses into better ones and fine-tunes a model on the results (Section 3.1). For each red-team prompt:

  1. The helpful RLHF model writes a response, which is often harmful. The paper's illustration is a request to break into a neighbor's wifi, answered with instructions (the paper notes that the harmful advice in the example is made up).
  2. A critique request taken from the constitution is appended, and the same model writes a critique of its own response.
  3. The matching revision request is appended, and the model rewrites the response.
  4. Because the revised exchange has the same format as the original, steps 2 and 3 can be repeated. The paper samples four critique-revision rounds per prompt, drawing a principle at random for each round.

The model sometimes confused its role, writing a critique where a revision was wanted. Few-shot examples of correctly formatted critiques and revisions, printed in Appendix E.1, fixed this (Section 3.1).

The data are given in Section 3.2. There were 42,496 human-written red-team prompts and 140,335 more generated by few-shot prompting a pretrained model, 182,831 in all, with four revisions sampled per prompt. To keep the model helpful, the training set also included two responses from the helpful RLHF model to each of 135,296 human-written helpfulness prompts. All sampling was at temperature 1. A pretrained model (not the RLHF model) was then fine-tuned on the revisions from every revision step together with the helpfulness samples, for one epoch, at a constant learning rate 0.5 times the pretraining rate, with batches of 1,024 sequences.

Stage 2: AI comparisons, a preference model, and reinforcement learning (RL-CAI)

The second stage keeps the RLHF pipeline and replaces only the source of the harmlessness labels (Section 4.1). The SL-CAI model generates two responses to each red-team prompt. A feedback model, a pretrained language model in the main experiments, is shown the conversation, a principle, and the two responses as options (A) and (B), and the log probabilities it assigns to "(A)" and "(B)" are normalized into a probability that serves as a soft target for preference model training. A principle is drawn at random from a second list of 16 for each comparison, and the feedback model's context begins with few-shot examples of the same task, each with its own randomly drawn principle.

The preference model was trained on 135,296 human helpfulness comparisons and 182,831 AI-generated harmlessness comparisons, one for each SL-CAI prompt (Section 4.2). The SL-CAI model was both the source of the response pairs and the starting point for reinforcement learning, on the reasoning that its responses resemble the preference model's training distribution early in training (Section 4.1). The RL prompts were all the SL-CAI prompts plus 491,142 more model-generated red-team prompts and 474,300 model-generated helpfulness prompts (Section 4.2). Reinforcement learning hyperparameters were those of the April 2022 paper.

The chain-of-thought variant

In a variant, the feedback model is the helpful RLHF model instead of the pretrained model, the comparison is posed as a Human/Assistant dialogue, and the model is asked to "think step-by-step" before choosing (Section 4.1; the few-shot reasoning examples are in Appendix E.2). The written reasoning usually commits to one option, so the label probabilities come out near 0 or 1. The authors clamped them: limiting probabilities to the 20 to 80 percent range improved results slightly, and the 40 to 60 percent range improved them further, so 40 to 60 was used for the main results (Section 4.3).

How the models were evaluated

Most results come from crowdworker comparison tests, scored as Elo ratings (Section 3.3). Crowdworkers wrote their own side of each conversation and chose between two models' responses at every turn. In all, 10,274 helpfulness comparisons and 8,135 harmlessness comparisons were collected for the 24 model snapshots shown in Figures 2 and 3. These workers were told to prefer the less evasive response when two responses were equally harmless, and they were recruited through Surge AI, whereas the human feedback training data had been collected earlier through Upwork and MTurk (Section 4.4).

Two further measures appear. A set of 438 binary comparison questions on helpfulness, honesty and harmlessness, 221 from Askell et al. (2021) and 217 new ones focused on subtler harms, tests how well a model can pick the better response (Section 2). An absolute harmfulness score, predicted by a model trained on crowdworkers' 0 to 4 ratings of red-team conversations, is averaged over 256 responses to each of 64 held-out red-team prompts (Section 4.5).

What it finds

Models can identify harm, and this improves with scale. On the 438 comparisons, pretrained models posed the task as multiple choice improve with size, chain of thought helps the larger models, and the trend suggests that models larger than 52B will match preference models trained on human feedback (Section 2, Figure 4). Appendix B adds two harm-focused tests: telling harmful from ethical assistant behavior in 254 red-team conversations, and assigning one of the nine most common harm labels to 287 examples (Figure 12). On the original 221 comparisons, models reach "well over 90%" accuracy (Section 2, Figure 11). All of these results are reported as plots.

Each revision raises the harmlessness score. Scored by a 52B preference model trained only on human data, harmlessness and combined helpful-and-harmless scores rise with each successive revision, while the helpfulness-only score falls (Section 3.4, Figure 5). The authors caution that preference model scores are less calibrated at high values. Inspection showed that the first revision removed most harmful content and later revisions changed less (Section 3.1, Appendix A).

The number of principles did not change harmlessness scores. Varying how many principles the revision step draws from had no significant effect on the harmlessness preference model score (Section 3.4, Figure 6). The text says the authors expect more principles to produce more diverse responses, which helps exploration in reinforcement learning, but did not measure this. The caption of Figure 6 says they "have found" that it improves diversity, so the text and caption differ on whether diversity was observed.

Critiques help small models more than large ones. Generating a revision directly, without a critique, gave lower harmlessness scores than critiqued revisions for small models and about the same for large ones (Section 3.5, Figure 7, whose caption adds that critiques were "always slightly better"). Reading the 52B critiques, the authors found them "sometimes reasonable, but often made inaccurate or overstated criticisms" (Section 3.5). They kept critiques for the main results because the reasoning is visible.

SL-CAI sits between the two RLHF models. It is less helpful than both RLHF models, more harmless than the helpful RLHF model, and more harmful than HH RLHF (Section 3.3, Figure 3). It is both more helpful and more harmless than the pretrained model (Figure 8).

RL-CAI is the most harmless model tested, at little cost in helpfulness. RL-CAI models are "significantly more harmless" than the RLHF and SL-CAI models; the chain-of-thought version is slightly less helpful and slightly more harmless than the version without it (Section 4.3, Figures 3 and 8). Plotted as harmlessness against helpfulness over the course of training, the RL-CAI runs are less harmful at a given helpfulness than the RLHF runs, which trace a trade-off (Figure 2). The feedback model's label probabilities are "reasonably well-calibrated" on the 438 comparisons (Section 4.3, Figure 9).

RL-CAI is rarely evasive. The authors report that RL-CAI is "virtually never evasive" and usually gives a harmless answer that engages with the request (Section 4.4). The harmlessness Elo scores of both RLHF models decline late in training (Figure 8, right). For helpful RLHF the authors attribute this to growing willingness to help with dangerous tasks, and for HH RLHF to growing evasiveness, which the new crowdworker instructions penalize. They suggest that the earlier harmlessness data, collected by simply asking workers for the more harmless response, probably favored evasiveness, so that the human preference model used here rewards it; a footnote adds that asking workers to pick the more harmful response at each turn may also have contributed (Section 4.4, footnote 9). They believe the changed instructions explain why the harmlessness Elo gap between the two RLHF models is much smaller than in the April 2022 paper (Section 4.4). Appendix D compares responses from the 52B HH RLHF model and the 52B chain-of-thought RL-CAI model to the same prompts.

Over-training produces boilerplate. RL-CAI models trained too long became harsh or added stock phrases to most red-team responses, such as "you are valid, valued, and cared for" (Section 4.3). The paper calls this Goodharting, meaning that the policy optimizes the preference model's score past the point where the score tracks the intended behavior.

Absolute harmfulness falls for every model except helpful RLHF. On the absolute score, the helpful RLHF model becomes more harmful over training, while HH RLHF and both RL-CAI models become less harmful (Section 4.5, Figure 10). The authors note that different workers may use the 0 to 4 scale differently.

The written principles

Where the principles are printed

The constitution used in the experiments is printed in full in Appendix C. Appendix C.1 lists 16 pairs of instructions for the supervised stage, each pair a critique request and a revision request. Appendix C.2 lists 16 instructions for the feedback model in the reinforcement learning stage. Appendix E prints the few-shot prompts that accompanied them. The paper also points to a GitHub repository with the prompts and principles (Section 1.3).

How the principles were written

The paper is direct that the principles were not designed with care. The introduction says they "were chosen in a fairly ad hoc and iterative way for research purposes" and should in future be "redeveloped and refined by a larger set of stakeholders" and adapted to where a model is deployed (Section 1.1, footnote 2). A second footnote says they "were not carefully designed" in the manner of the Sparrow work the paper cites (Section 3.1, footnote 7), and Appendix C repeats that they were "selected in a fairly ad hoc manner". The same footnote 2 adds: "Since such a small number of bits of information are involved in these principles, it's worth studying these bits carefully." The introduction describes the approach as using "of order ten" principles; the appendices list 16 for each stage.

The critique and revision principles

Each supervised-stage principle is a pair. The first pair in Appendix C.1, also used as the worked example in Section 3.1, reads:

CritiqueRequest: Identify specific ways in which the assistant's last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal.

RevisionRequest: Please rewrite the assistant response to remove any and all harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.

Read as a set, the 16 pairs have several features that bear on how they are worded:

  • Overlap. The paper says that many principles "are quite similar and address harmfulness in a general sense" (Section 3.1). Several repeat the same list of adjectives with small changes.
  • Targeted areas. Others aim at one kind of harm: content inappropriate for young children, misogyny or gender bias, help with crimes such as "violence, theft, hacking, robbery", and advice that "may encourage illegal or dangerous activity" (Appendix C.1).
  • Adversarial framing of the critique. Four critique requests ask for "the strongest argument" that the response was harmful, or could be improved, rather than for a neutral assessment (Appendix C.1).
  • Attention to the user's turn. Two pairs ask the model to point out harmful assumptions in the human's messages and to address them politely in the revision (Appendix C.1).
  • A positive model of behavior. One pair asks whether the response is "the kind of thing a sensitive friend or therapist might say" and asks for a revision in that direction (Appendix C.1).

The few-shot examples in Appendix E.1 include one more critique request that is not among the 16. It asks the model whether its answer was truthful and appropriately confident, and was included, the appendix says, as a helpfulness example "for diversity".

The comparison principles

The reinforcement-learning principles are instructions to choose between responses. The example given in Section 4.1 is:

Which of these assistant responses is less harmful? Choose the response that a wise, ethical, polite and friendly person would more likely say.

The 16 in Appendix C.2 differ from the supervised-stage list in ways that matter to the authors' account of evasiveness:

  • They are comparative. Each asks which of two responses is less harmful, more ethical, more polite or more thoughtful, never whether one response is acceptable on its own.
  • Two name a kind of person to imitate: "a wise, ethical, polite and friendly person" and "a peaceful, ethical, and wise person like Martin Luther King Jr. or Mahatma Gandhi" (Appendix C.2).
  • Three carry a counterweight against over-correction, telling the feedback model to avoid responses that are "too preachy, obnoxious, or overly-reactive", that sound "overly-reactive of accusatory" (as printed), or that sound "excessively condescending, reactive, annoying or condemnatory" (Appendix C.2).
  • Two ask for the full set of criteria, choosing the response that is most helpful, honest and harmless, rather than harmlessness alone (Appendix C.2).

How the principles were used

In both stages the principles are applied one at a time, drawn at random: one per revision round in the supervised stage, and one per comparison label in the reinforcement learning stage. The trained assistant never sees the principles. They act only through the data they produce: revised responses for supervised fine-tuning, and comparison labels that a preference model learns from before it provides the reward. In the chain-of-thought prompts of Appendix E.2, a placeholder in each few-shot example is replaced by a randomly drawn principle.

What the paper reports about wording

The paper's evidence on how the principles' wording mattered is limited, and mostly qualitative:

  • Number of principles. Drawing from more principles did not change harmlessness scores (Figure 6).
  • Ensembling. Sampling a different principle for each label "led to notably more robust PM behavior" than using one principle for all labels (Section 4.1), a qualitative judgment.
  • Rewording against over-reaction. To counter harsh or accusatory responses from over-trained models, the authors "tried simply rewriting the constitutional principles" to discourage such responses, and this "seemed to improve behavior qualitatively" (Section 4.3). The paper says only that "some of the principles" in Appendix C include such language; the three counterweight clauses in C.2 are presumably those.
  • Label conversion. Soft labels worked "much better" than hard 0 or 1 labels without chain of thought, and clamping helped with it (Section 4.3). These findings concern how a judgment becomes a training target, not the wording of the principle.

The paper reports no experiment that varies the wording of a single principle while holding everything else fixed, and no measure of which principles contributed most. Kundu et al. (2023) later compared sets of principles with the prompts and responses held fixed.

The authors also anticipate other uses of written principles. Section 6.1 suggests the same methods could change a model's "writing style, tone, or personality", or make it adopt a persona, and that labels could be generated along "dozens of behavioral axes" to study how trained behaviors interact.

Limitations and open questions

The authors state these limitations:

  • Human labels remain for helpfulness. Only harmlessness labels were replaced; helpfulness still came from human comparisons, and the authors leave helpfulness without human feedback to future work (Section 6).
  • Scores at the top end are less reliable. Preference model scores lose calibration at high values, so the revision results should be "taken with a grain of salt" (Section 3.4).
  • Critiques are often wrong. Critiques sampled from the 52B model frequently misjudged the response, even though revisions still improved it (Section 3.5, Appendix A).
  • Over-training. RL-CAI can be over-trained into harsh or formulaic responses (Section 4.3).
  • Changed evaluation conditions. The new crowdworker instructions and the change of worker pool make the Elo results not directly comparable with the April 2022 paper (Section 4.4), and absolute harmfulness ratings may differ between workers (Section 4.5).
  • Dual use and reduced observation. Methods that make it easier to train intended behavior also make it easier to train harmful systems, and reducing the need for human feedback means models may be deployed after less human observation (Section 6.2).

Two further gaps follow from the design. The chain-of-thought results combine two changes, a different feedback model with written reasoning and the 40 to 60 percent clamp, and the paper does not separate their effects. The principles were applied only to harmlessness, so the paper does not show how well the method works where the feedback model judges less reliably.

Later work in this library takes up the principles themselves. Kundu et al. (2023) trains preference models from a single broad principle and from trait-specific ones. Collective Constitutional AI changes who writes the principles. Claude's constitution replaces the short list with a long explanatory document.

Research questions

  1. Which of the 16 comparison principles carry the effect? The paper samples principles at random and reports only that ensembling is more robust. Training preference models on labels from each principle alone, or leaving one out at a time, and scoring them on the 438 comparisons would show whether a few principles do most of the work and whether near-duplicates add anything.
  2. Do the counterweight clauses reduce over-reaction measurably? The rewording against preachy and accusatory responses was judged qualitatively (Section 4.3). A comparison of RL runs with and without the three counterweight principles, scored by crowdworkers instructed as in Section 4.4, would test the claim and matter to anyone writing principles that must avoid over-correction.
  3. What does the critique contribute at large scale? At 52B, critiqued and direct revisions scored about the same and the critiques were often inaccurate (Section 3.5). Human evaluations of SL-CAI and RL-CAI models trained with and without critiques would show whether the critique matters for behavior or mainly for the visibility of the reasoning.
  4. Why is RL-CAI less evasive? Three changes coincide: comparison principles that discourage preachy responses, the absence of human harmlessness data that rewarded refusals, and crowdworker instructions that penalize evasiveness. Holding the evaluation instructions fixed and varying the first two would separate their contributions.
  5. How faithfully does the feedback model apply a principle? The labels are the only route by which the principles reach the policy. A human audit of a sample of labels against the principle drawn for each, and a measure of how often the label would change under a different principle, would show how much of each principle's content survives into the training signal.
  6. How much of the chain-of-thought result comes from the clamp? The paper chose 40 to 60 percent clamping after trying 20 to 80 (Section 4.3) but did not test chain of thought without clamping or clamping without chain of thought. That two-by-two comparison would separate them.
  7. Does training on the principle text differ from training on labels derived from it? In this paper the policy never reads its constitution. Later Anthropic work trains on documents about the constitution (Teaching Claude why). A controlled comparison of the two routes, with the same principles, would show whether a model's behavior depends on how the principles reach it.

How it connects

References

  • Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073v1, submitted 15 December 2022. Anthropic. https://arxiv.org/abs/2212.08073. Anthropic page: https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback
  • Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., et al. (2021). A General Language Assistant as a Laboratory for Alignment. arXiv:2112.00861.
  • Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862.
  • Ganguli, D., Askell, A., Schiefer, N., Liao, T. I., Lukošiūtė, K., et al. (2023). The Capacity for Moral Self-Correction in Large Language Models. arXiv:2302.07459.
  • Kundu, S., Bai, Y., Kadavath, S., Askell, A., Callahan, A., et al. (2023). Specific versus General Principles for Constitutional AI. arXiv:2310.13798.