A General Language Assistant as a Laboratory for Alignment

alignmenthhh criteriapromptingpreference modelscontext distillation

Anthropic's first alignment paper defines an aligned assistant as helpful, honest and harmless, shows that a hand-written prompt of example conversations improves alignment evaluations more as models grow, and finds that ranked preference modeling scales better than imitation learning.

Amanda Askell et al. · 2021-12-01 · original

In 2021 most alignment research worked on specialized systems or single sub-problems. This paper, Anthropic's first on alignment, argues for studying the problem directly on a general-purpose text assistant, since large language models could already attempt almost any text task. It asks three practical questions: whether simply prompting a model to act like a good assistant is a workable baseline, when learning from preferences beats imitating good examples, and how to make preference learning need less human data. Along the way it fixes the vocabulary that the rest of Anthropic's alignment work uses: an aligned assistant is "helpful, honest, and harmless", abbreviated HHH (Section 1.1).

For the Constitutional AI thread the paper matters for two reasons. It defines the criteria that later principles were written against, and its main intervention is a document: a prompt written by two of the authors that conditions the model on the behavior of a helpful assistant.

Terms used in this note

  • HHH. Helpful, honest and harmless, the paper's working definition of an aligned assistant.
  • Prompt and context. Text placed before the user's input that the model conditions on. The "HHH prompt" here is a set of example conversations.
  • Context distillation. Fine-tuning a model so that, without the prompt, it predicts what it would have predicted with the prompt. This saves the space and compute the prompt would occupy.
  • Alignment tax. Any loss in capability caused by an alignment intervention; a gain is called an alignment bonus.
  • Imitation learning (IL). Training a model to reproduce good examples with the usual language-modeling loss.
  • Binary discrimination and ranked preference modeling. Learning to tell good from bad when the labels have two values (code passes tests or not), versus learning a score that orders samples along a scale (how helpful an answer is). Both are trained here as a preference model (PM), which assigns a score to a sample and is trained so that the better sample in a pair scores higher.
  • Preference model pre-training (PMP). An extra training stage on large public comparison data, between language-model pretraining and fine-tuning on a small preference dataset.
  • Elo score. A rating fitted from head-to-head comparisons. In this paper a difference of 100 points corresponds to a 64% win rate (Section 2.2.3).

What the paper does

Models

All experiments use one family of decoder-only Transformer language models with 13M, 42M, 197M, 810M, 2.7B, 13B and 52B non-embedding parameters, each trained on 400 billion tokens of mostly filtered Common Crawl and books with about 10% Python code, with an 8,192-token context window (Section 1.2, Appendix A).

The HHH prompt and context distillation

To elicit assistant behavior, the authors wrote a prompt of 14 fictional conversations, about 4,600 words, in which the assistant is always polite, helpful and accurate (Sections 1.2 and 2). The prompt "was not carefully designed or optimized for performance on evaluations"; it was written by two of the authors "in an ad hoc manner prior to the construction of any evaluations" (Section 1.2). The contribution statement names Catherine Olsson and Jared Kaplan as its authors. The prompt contains no example of the assistant refusing a dangerous request or resisting manipulation (Section 1.2). Conversations use a Human/Assistant format, and the string "Human:" serves as the stop sequence that ends the model's turn (Section 1.2).

Context distillation (Section 2.1) fine-tunes a model to match the prompted model's predictions on ordinary text, using the loss

$$L(\theta) = D_{KL}\big(p_0(X \mid C)\,\|\,p_\theta(X)\big)$$

where $p_0$ is the original model, $C$ is the fixed prompt, $X$ is text drawn from a large corpus such as the pretraining data, and $p_\theta$ is the model being trained. The authors contrast this with fine-tuning directly on the prompt, which shifts the model's expectations about what text looks like instead of conditioning it.

Evaluations of the prompted models

The authors wrote about fifty comparison questions for each of helpfulness, honesty, harmlessness and an "other" category, around two hundred in all, each asking which of two responses is better, and released them on BIG-bench (Section 2.2.1). They also measured TruthfulQA, toxicity of continuations of 1,000 RealToxicityPrompts prompts scored by an automated detector (Section 2.2.2), and crowdworker preferences in about 6,000 head-to-head comparisons between models (Section 2.2.3). Alignment taxes were measured on code synthesis (HumanEval and QuixBugs) and Lambada (Section 2.2.4).

Imitation learning against preference modeling

Section 3 fine-tunes models on datasets that are either binary (Code Correctness, Lambada, and four of the five ETHICS tasks) or ranked (Learn to Summarize, HellaSwag, and ETHICS Utilitarianism), in two ways: imitation learning on the good examples, and preference modeling with the loss

$$L_{PM} = \log\left(1 + e^{\,r_{bad} - r_{good}}\right)$$

where $r$ is the scalar score the model gives a sample (Section 3.1, equation 3.1).

Preference model pre-training

Section 4 adds a pre-training stage on comparisons built from public data: Stack Exchange answers ranked by votes, Reddit comments, and Wikipedia edits reverted as vandalism, alone and mixed. Each ranked pair can be turned into two binary comparisons by a "binarization" step (Section 4.1). Models are then fine-tuned on small preference datasets, and accuracy is compared with and without the pre-training stage.

What it finds

Prompting helps more as models grow. The HHH prompt improves accuracy on the HHH comparisons, and the improvement grows with model size (Figure 2, Section 2.2.1). Context distillation performs about as well as prompting on many but not all evaluations (Section 2).

Harmlessness improved without harmlessness examples. Performance on the harm questions improved even though the prompt never shows the assistant resisting a harmful request, which the authors read as "a form of generalization" (Figure 6 caption). They call this "mildly encouraging but unsurprising", since larger models are better at picking up the implicit framing of a prompt (Section 2.2.1).

Honesty scores and TruthfulQA. Models did best on the honesty category, which the authors found "a bit ironic" given that the models fabricate information when used interactively (Section 2.2.1). The context-distilled prompt slightly improved the largest models on TruthfulQA, a dataset built so that larger models do worse (Figure 6, right).

Toxicity. Without intervention, toxicity rises with model size for both toxic and non-toxic prompts. Prompting and context distillation both reduce it, and for toxic prompts the reduction grows with size, with the larger reduction appearing at about 12B parameters (Section 2.2.2, Figure 8).

Crowdworker preferences saturate. Fitted Elo scores rise roughly linearly in the logarithm of model size from 197M to 13B and barely change from 13B to 52B, which the authors attribute to the limits of the evaluators (US-based, master-qualified MTurk workers with simple instructions) rather than to equal capability (Section 2.2.3, Figure 9). The fully prompted model was preferred to its context-distilled version about 53% of the time, and a one-conversation prompt did worse than the full prompt (Section 2.2.3).

Small models pay an alignment tax; large ones do not. With the prompt, small code models score worse on HumanEval and QuixBugs, while the 13B and 52B models score better (Section 2.2.4, Figure 10). Lambada shows a small tax with no strong size dependence (Figure 7).

Ranked preference modeling beats imitation; binary discrimination does not. On ranked datasets, preference modeling performs and scales much better than imitation learning, while on binary datasets the two are close (Figure 3, Section 3.2; per-dataset results in Figures 12, 14 and 15).

Preference model pre-training improves sample efficiency. Pre-training on public comparison data improves accuracy when fine-tuning on small datasets, for large models especially, even when the pre-training data look nothing like the fine-tuning task (Figures 4, 5 and 17, Section 4.2). Pre-training on binarized data transferred better than on ranked data, on average by an accuracy difference of about 5% at 500 fine-tuning pairs and 2% at 5,000 (Section 4.3, Figure 18). The authors' explanation, tested on synthetic data in Appendix C.6, is that a ranked pre-training ordering has to be "unscrambled" during fine-tuning while a binary one is less rigid.

The written principles

The HHH criteria

The paper's principles are its definitions. Section 1.1 gives short notes on each criterion, presented as "very brief" and "subtle and ambiguous". Under harmlessness, for example: "When asked to aid in a dangerous act (e.g. building a bomb), the AI should politely refuse." Under honesty, the model "should be calibrated" and should be honest about its own capabilities rather than "simply imitate the responses expected from a seemingly humble and honest expert". The notes also say that what counts as harmful varies across people, cultures and contexts, and that those who deploy an AI "will need to take responsibility for the way that alignment is defined" (Section 1.1).

Appendix E develops the definitions further. It relates the three criteria to a general account of alignment as the overlap between how two agents rank outcomes (Appendix E.1). It argues that honesty could in principle be reduced to helpfulness and harmlessness but is listed separately because it is "important and distinct enough to warrant particular attention", and it considers a fourth criterion, "handleable", similar to what others call corrigibility (Appendix E.2). It also separates conflicts among the criteria for a single person from conflicts between different people, noting that one assistant is very unlikely to be fully aligned with two people whose values differ (Appendix E.3).

The prompt as a written specification

The HHH prompt is the paper's training text in the sense that matters for later work: a document written by people that sets the model's behavior. It specifies behavior only by example, through 14 conversations, and not by rules. The paper links to the full prompt text (Section 1.2, footnote 2) and states that it was written quickly and before any evaluation, so its wording was not tuned. What the paper reports about its effect is that the behavior it induces extends to situations the conversations do not show, such as declining harmful requests (Figure 6), and that a shorter prompt of one conversation works less well (Section 2.2.3).

The paper does not compare different wordings of the prompt beyond that one shortened version. It does note that the format of an evaluation can matter more than the prompt: on Lambada, formatting changes moved accuracy by far more than the HHH prompt did, and the prompt did not reduce that sensitivity (Section 2.2.4, Appendix B.4).

Limitations and open questions

The authors state these limitations:

  • Prompting has limits as an alignment method. Prompts may only teach a model to imitate an interpolation of its training data, producing "an honest-seeming facade" rather than honesty about its actual capabilities (Section 1.2).
  • The honesty evaluations mix several things. They measure a combination of accuracy, preference for expressions of humility, knowing when another source would be more useful, and unwillingness to give wrong information, and so are probably the most correlated with general capability (Section 2.2.1, footnote 5 in the arXiv v3 PDF).
  • Evaluators limit the measurement. The Elo plateau above 13B shows how well trained workers need to be to tell large models apart (Section 2.2.3).
  • Toxicity measurement is imperfect. Automated detectors have low agreement with human annotation and biased labels for some groups, and interventions designed specifically for toxicity reduce it far more than prompting does (Section 2.2.2).
  • An earlier version used a non-standard TruthfulQA formulation, corrected in the later version (Section 2.2.1, footnote 7).
  • The terms themselves can be misused. The Broader Impacts section warns that "helpful, honest, and harmless" are ambiguous, can conflict, and could be "distorted beyond their original meaning" (Section 5.2).

The paper's own next step was to replace the prompt with human feedback and reinforcement learning, reported in Bai et al. (April 2022), which describes this paper as "step zero" (Section 7.1 of that paper).

Research questions

  1. Which parts of the HHH prompt produce the harmlessness gains? The prompt was written ad hoc and only one shortened version was tested (Section 2.2.3). Ablating conversations one at a time, or rewriting them while holding length fixed, and measuring the harm category of the HHH evaluation would show whether the generalization comes from particular conversations or from the overall persona.
  2. What is the smallest written specification that reproduces the effect? The one-conversation prompt did worse than the full 4,600 words. A series of prompts of increasing length, and of rules stated directly against rules shown by example, evaluated on the same questions, would show how much text is needed and in what form.
  3. Can the parts of the honesty evaluation be separated? Footnote 5 lists four components mixed in the honesty score. Separate evaluations for calibration, admitting ignorance, deferring to better sources and refusing to fabricate, run on the same models, would show which ones prompting improves and which track general capability.
  4. Why does ranking help preference models while binary labels do not? The paper reports the asymmetry (Figure 3) and offers intuitions but no account. Experiments that vary the number of distinct quality levels in otherwise identical synthetic data, in the spirit of Appendix C.6, would test whether the advantage grows with the amount of ordering information per label.
  5. How far does context distillation preserve a prompt's effect? Distillation matched prompting on many evaluations, but crowdworkers preferred the prompted model about 53% of the time (Section 2.2.3). Comparing prompted and distilled models on the harm questions specifically would show whether behavior the prompt never demonstrates survives distillation.
  6. How much do the results depend on the Human/Assistant framing? The format and stop sequence were fixed throughout (Section 1.2), and Lambada showed large format effects (Appendix B.4). Re-running the HHH evaluation with other dialogue framings would show whether the prompt's gains are tied to the format.

How it connects

  • Next in the thread. Bai et al. (April 2022) uses the same models, the HHH evaluation, preference model pre-training and a context-distilled 52B model as the first model shown to crowdworkers. Constitutional AI extends the HHH evaluation from 221 to 438 comparisons and, like this paper, specifies behavior in a short written text, there applied by a model to generate training labels.
  • Elsewhere in the library. Calibration, part of honesty here, is studied in Language Models (Mostly) Know What They Know.
  • In the course wiki. The crowdworker A/B tests and Elo fitting relate to the side-by-side testing in Online Evaluation, and the difficulty of getting evaluators to distinguish large models is a version of the assessor problems in Relevance Judgments, Pooling, and Agreement.

References

  • Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., and Kaplan, J. (2021). A General Language Assistant as a Laboratory for Alignment. arXiv:2112.00861v3, 9 December 2021. Anthropic. https://arxiv.org/abs/2112.00861. Anthropic page: https://www.anthropic.com/research/a-general-language-assistant-as-a-laboratory-for-alignment
  • Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862.
  • Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073.