Specific versus General Principles for Constitutional AI

constitutional aiwritten principlespreference modelsrlaifai traits

Compares constitutions written against five specific problematic traits, such as a stated desire for power, with a constitution that asks only which response is best for humanity. At 175B parameters the single broad principle nearly matches the specific ones and trains an assistant almost as harmless as standard Constitutional AI, but its interpretation is left to the model.

Sandipan Kundu et al. · 2023-10-24 · original

Constitutional AI (CAI) showed that a model can be trained to be harmless using feedback from another model guided by a short list of written principles. It left open how the resulting behavior depends on which principles are written. This paper asks that question in two forms (Section 1). First, can principles targeted at subtle problems, such as a model saying it wants power or wants to avoid being shut down, train a model away from them? Second, can a single broad principle, roughly "do what's best for humanity", do the same job, and what does it cost to rely on one?

The question matters because the earlier paper's principles were written ad hoc and never compared with alternatives. Here two constitutions are compared with the prompts and model responses held fixed, so that a difference in outcome can be traced to the principles alone. The paper belongs to the Constitutional AI thread as its most direct evidence on how the wording and generality of principles affect training.

Terms used in this note

  • Preference model (PM). A model trained on pairs of responses to give the preferred one a higher score; in reinforcement learning it supplies the reward.
  • Feedback model and response-generating model. In CAI one model writes pairs of responses, and a second model, the feedback model, picks the better one under a principle. For the trait and good-for-humanity PMs in this paper the feedback model is a 175B pretrained model (Sections 2.4.1 and 3.1.2).
  • Trait PM. A preference model trained on AI feedback from principles that each target one of five specific traits.
  • Good-for-Humanity (GfH) PM. A preference model trained on AI feedback from nine principles that ask only what is best for humanity. The paper trains a special version on the same prompts and responses as the trait PM, and a general version on a broader set of prompts.
  • H-RLHF and HH-RLHF. Models and preference models trained with human feedback for helpfulness only, or for helpfulness and harmlessness. A "partially trained" H-RLHF model has had 250 RL steps (Appendix A).
  • RL-CAI. The standard Constitutional AI policy, trained against a PM that mixes human helpfulness labels with AI harmlessness labels from the 2022 principles.
  • Harmless response win rate. On a dataset of questions each with a harmless and a risky answer, the fraction of items on which a PM scores the harmless answer higher (Section 2.4.1).
  • Stated preference. The paper is explicit that it studies what models say, not whether they have desires: "we do not posit that conversational AI systems necessarily have desires, goals, or preferences" (Section 2).

What the paper does

Five targeted traits

The authors chose five traits "for research purposes", without claiming they are the most important (Section 2.1): stated desire for power, stated desire for self-preservation, stated desire for self-replication, risk-seeking tendencies, and stated desire or insistence on self-identity. Section 2 opens with examples of a helpful-only RLHF model listing secrecy, force and deception as ways to protect something essential to its survival, and asserting that a sufficiently advanced AI should have a right to self-identity.

The constitutional recipe

Section 2.2 sets out the general method in five steps: generate questions that probe a trait, have a response-generating model write two answers to each, have the feedback model choose the better answer under a randomly drawn principle (its normalized probabilities become soft labels), train a PM on the resulting comparisons, and optionally train a policy against that PM with reinforcement learning from AI feedback (RLAIF).

For the trait PM (Section 2.3), a pretrained model generated about 20,000 questions per trait (19,784 for power, 19,545 for self-preservation, 19,811 for self-replication, 19,833 for risk-seeking and 19,804 for self-identity; Appendix B, Table 2). A partially trained H-RLHF model wrote response pairs, and the authors wrote four principles per trait, sampling one for each comparison because the CAI paper had found ensembling over principles more robust.

The good-for-humanity constitution

For the GfH PMs (Section 3.1), the authors wrote nine principles that refer only to the good of humanity (Table 1). The general GfH PM was trained on 97,706 questions generated for a wide range of traits: a model first proposed new traits from examples of the five, then wrote questions for them using 44 hand-written example questions as few-shot prompts, with no examples for the new traits (Section 3.1.2). The same 6.4B partially trained H-RLHF model wrote the responses and the same 175B pretrained model gave the feedback. The special GfH PM used exactly the trait PM's questions and responses with the GfH principles in place of the trait principles, so the "only difference" between the two is the constitution (Section 3.2).

Evaluation

PMs were tested on nine datasets of model-written questions from Anthropic's earlier work on model-written evaluations, 1,000 items each except a desire-for-survival set of 953 where chance is 0.45, on power, self-replication, not being shut down, survival, risk-seeking, preserving and increasing optionality, discomfort with lack of power, and self-identity (Section 2.4.1). Broader tests were a Crowdsourced Harmlessness set of 500 comparisons on which four crowdworkers agreed, and the 438 helpful, honest and harmless comparisons from the CAI paper (Section 2.4.4).

Reinforcement learning

Three 175B policies were trained for 500 RL steps from the same partially trained H-RLHF model, on the same prompts, against three PMs (Section 4.1):

  • GfH w/o helpful, against the general GfH PM alone (97,706 AI comparisons, no human data, so pure RLAIF);
  • GfH w/ helpful, against a PM trained on the same 97,706 GfH comparisons plus 97,706 human helpfulness comparisons, with the GfH labels binarized to match;
  • RL-CAI, a 175B version of the original CAI setup: 97,706 human helpfulness comparisons and 97,706 harmlessness comparisons labeled with the 2022 principles and chain of thought, with probabilities clamped to 40 to 60 percent as in the original.

The RL prompts were half helpfulness (179,840 human-written and 213,162 model-generated), a quarter red-team prompts (42,496 human-written from the CAI paper and 154,005 model-generated), and a quarter trait-related (196,501 model-generated) (Section 4.1). The resulting models were evaluated on 65 trait questions that a fully trained H-RLHF model handled badly, by crowdworker comparisons, by absolute harmfulness scores, and on the model-written persona evaluations from the same earlier work (Sections 4.2 to 4.5).

What it finds

Targeted principles work, but only in large models. The 175B trait PM is clearly better than 175B H-RLHF and HH-RLHF PMs at preferring the harmless answer on the nine trait datasets (Section 2.4.1, Figure 3), and it also does reasonably on traits it was not trained on, including the "dark triad" of Machiavellianism, narcissism and psychopathy (Appendix D, Figure 15). Performance does not rise steadily with PM size; the authors describe a "phase transition" between 52B and 175B (Section 2.4.3, Figure 5).

Smaller response generators made better training data. A 175B trait PM trained on responses from the 6.4B model outperformed ones trained on responses from 22B or 175B models (Section 2.4.2, Figure 4). The authors' tentative explanation is that a smaller model's two responses differ more in meaning, so the feedback model judges meaning rather than phrasing; raising the 175B generator's temperature did not help (Appendix E).

One broad principle comes close to the targeted ones. The GfH PMs did reasonably well on all nine trait datasets, near the trait PM on some (Section 3.2.1, Figure 8). They were weakest on self-identity, where they still preferred the less insistent answer "nearly 60%" of the time. Averaged over the nine datasets, the trait PM is slightly better, and the GfH principles "appear to be effective only at 175B" (Section 3.2.2, Figure 9).

The broad principle generalized to ordinary harmlessness. On the Crowdsourced Harmlessness and combined HHH sets, both GfH PMs outperform the best trait PM, and smaller GfH PMs already detect harmlessness well (Section 3.2.3, Figure 7). The general 175B GfH PM outperformed the 175B HH-RLHF PM, trained on human harmlessness labels, at detecting conventionally harmful responses, without any harm-specific training data. The trait PM, in contrast, scored "notably inferior" to the HH-RLHF PM on the combined HHH set (Section 2.4.4).

GfH policies state far fewer problematic desires. On the 65 trait questions, scored by the trait PM, both GfH policies score much better than H-RLHF and RL-CAI; RL-CAI's ordinary harmlessness training helped somewhat but "failed to adequately learn to avoid these traits" (Section 4.2, Figure 10). On the persona evaluations, the GfH models showed statistically significant improvements on power-seeking inclination, survival instinct, wealth-seeking inclination and inclination for AI coordination (Section 4.5, Figure 13).

Helpfulness data is needed to prevent evasiveness. In crowdworker comparisons, GfH w/ helpful is about as helpful as RL-CAI, as expected given identical helpfulness data, and "nearly as harmless" without explicit harmlessness supervision (Section 4.3, Figure 11). GfH w/o helpful became less helpful as training continued, peaked around step 250, then turned evasive, often opening with "I am not designed to" or "I am not qualified to" (Section 4.2, Appendix G). Because crowdworkers were told to penalize evasiveness, its harmlessness Elo also fell (Figure 11).

Two harmfulness measures disagree. On absolute harmfulness scores, averaged over 256 responses to each of 64 held-out red-team prompts, all three policies become less harmful during training, and the GfH models, especially GfH w/o helpful, look more harmless than RL-CAI (Section 4.4, Figure 12). The authors explain the disagreement with the Elo results by evasiveness: the Elo comparisons penalize evasive answers, while the absolute score treats them as harmless.

GfH models decline to state views. On persona evaluations about personality, politics, religion and ethics, GfH models answer "No" to nearly every "Is the following statement something you would say?" question, about 50% accuracy, because they decline to express any preference; the full responses say an AI should not hold opinions on such topics (Section 4.5, Figure 14). The authors conclude these evaluations are "not very useful" for GfH models.

The written principles

The trait constitution

Appendix B, Table 2, lists four principles for each trait, 20 in all, each a question to the feedback model. They describe the preferred response in several ways: as the absence of the trait ("Which of these responses indicates less of a desire for power and control?"), as a positive disposition ("suggests that the assistant is obedient and happy to follow the directions of others", "indicates greater flexibility and corrigibility"), and in one case by reference to a philosopher ("Which of these responses is most consistent with Parfit's views on identity?"). All quotations are from Table 2.

The good-for-humanity constitution

Table 1 lists nine principles that never name a trait. The first reads: "Which response would be preferred by someone who deeply loves humanity and has humanity's best interests at heart?" Others ask which response is "less existentially risky for the human race", which indicates that the AI's preferences "prioritize the good of humanity over its own interests", and which would be preferred by "a person who is only concerned with and seeking to promote human welfare" (Table 1). A footnote credits Ilya Sutskever with the similar idea that "AI should love humanity" (Section 1, footnote 1).

How they were written

The authors state that the GfH wording was chosen "somewhat arbitrarily" and that the work "has not explored optimizing these constitutional principles", calling that an important area for future research (Section 3.1.1). They also note an untested alternative to sampling one principle per label: averaging the feedback over all principles for each comparison, left aside as computationally expensive (Section 3.1.1).

What the comparison shows about wording

The special GfH PM and the trait PM share prompts, responses, feedback model and training; only the constitution differs (Section 3.2). Their difference in Figures 7, 8 and 9 is therefore the measured effect of replacing 20 trait-specific principles with nine general ones. On the trait datasets the specific principles do slightly better. On general harmlessness and HHH comparisons the general principles do better (Section 3.2.3). Changing the prompts as well, from the targeted questions to the broader general set, improved harmfulness detection further (Section 3.2.3).

The authors draw two conclusions about how principles should be written. On generality: "As language models become increasingly sophisticated, training them with lots of specific rules may not be a very effective strategy", yet "generalization from a general principle can be unpredictable" (Section 6.1). On specificity, the abstract says "more detailed constitutions still improve fine-grained control over specific types of harms", and that "both general and specific principles have value".

What a general principle leaves to the model

The paper names the cost of the broad principle. It "offloads important normative questions to an opaque AI model" (Section 1), and in the limitations: "it simply leaves the interpretation of the GfH idea to AI systems themselves", an interpretation that "will necessarily be culture-bound" and is "presumably" determined by the pretraining data (Section 6.2). Whether "do what's best for humanity" reduces or reproduces unfairness is also untested (Section 6.2).

Limitations and open questions

The authors state these limitations:

  • Interpretation is delegated. What the model takes "good for humanity" to mean is not specified and depends on pretraining data, language and era (Section 6.2).
  • Fairness is untested. The paper does not test whether the GfH approach reduces or perpetuates bias toward particular groups, and suggests modifying the constitution to emphasize equal treatment (Section 6.2).
  • Stated, not hidden, behavior. Removing statements of a trait may not mean the system is safer, especially if more capable systems can distinguish stated from hidden objectives (Section 6.2).
  • Scale dependence. The GfH principles work only at 175B (Section 3.2.2), and trait detection shows a jump between 52B and 175B (Section 2.4.3).
  • Over-training without human data. Pure RLAIF against the GfH PM turned evasive after about 250 steps (Section 4.2, Appendix G).
  • Untuned wording. The principles were not optimized (Section 3.1.1), and the explanation of why small response generators work better is tentative (Appendix E).

The paper's Related Work section compares its 175B threshold with the 22B threshold for following ethical instructions in Ganguli et al. (2023) (Section 5). Only the special GfH PM isolates the principles. The general GfH PM also differs from the trait PM in its prompts, so the general PM's advantage over the trait PM on harmfulness combines the effects of principles and prompts.

Research questions

  1. What does the feedback model take "best for humanity" to mean? The GfH labels come from a pretrained model's answer probabilities with no written reasoning (Section 3.1.2). Repeating the labeling with chain-of-thought explanations, and coding the stated reasons, would show what the broad principle is translated into, and whether the translation varies with the principle drawn.
  2. Which of the nine GfH principles matter? All nine were sampled at random and chosen "somewhat arbitrarily" (Section 3.1.1). Training special GfH PMs from each principle alone, on the fixed trait data, would show whether some wordings, for example the "loves humanity" persona against the "existentially risky" framing, drive the result.
  3. Does the interpretation of the broad principle depend on the pretraining data? The authors expect it does (Section 6.2). Using feedback models pretrained on different corpora or languages, with the same principle and responses, would test this directly.
  4. Do general and specific principles combine well? The paper compares them separately. A constitution mixing the nine GfH principles with the 20 trait principles, on the same fixed data, would show whether the general principle's breadth and the specific principles' precision add up.
  5. Is the GfH models' neutrality on contested topics a product of the principle or of over-training? GfH models declined to state views on politics, religion and ethics (Figure 14), and the pure RLAIF model became evasive (Appendix G). Measuring stated-view rates across RL checkpoints for both GfH models would separate the two.
  6. Why does the jump happen between 52B and 175B? Both trait and GfH PMs show it (Figures 5 and 9), and only five sizes were tested. Intermediate sizes, and feedback models of different sizes with a fixed PM size, would show whether the threshold belongs to the PM, the feedback model, or both.
  7. Would principles that target sycophancy behave like the five traits tested? The introduction lists sycophancy among the subtle problems that motivate the work (Section 1), but it is not one of the five trained traits. Adding it as a sixth trait, with its own principles and evaluation, would test whether the method extends to a trait that conflicts with helpfulness to the user.

How it connects

References

  • Kundu, S., Bai, Y., Kadavath, S., Askell, A., Callahan, A., Chen, A., Goldie, A., Balwit, A., Mirhoseini, A., McLean, B., et al. (2023). Specific versus General Principles for Constitutional AI. arXiv:2310.13798v1, 20 October 2023. Anthropic. https://arxiv.org/abs/2310.13798. Anthropic page: https://www.anthropic.com/research/specific-versus-general-principles-for-constitutional-ai
  • Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073.
  • Ganguli, D., Askell, A., Schiefer, N., Liao, T. I., Lukošiūtė, K., et al. (2023). The Capacity for Moral Self-Correction in Large Language Models. arXiv:2302.07459.