Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

rlhfpreference modelsalignmentcrowdworkershelpfulness and harmlessness

Trains a 52B assistant with preference modeling and reinforcement learning from crowdworker comparisons, finds that this improves large models on NLP benchmarks, documents a tension between helpfulness and harmlessness that pushed models toward evasive refusals, and reports a roughly linear relation between reward and the square root of the KL divergence.

Yuntao Bai et al. · 2022-04-12 · original

Askell et al. (2021) made an assistant behave well by prompting it with example conversations, and found that preference models learn more than imitation does. This paper takes the next step: it collects human preference data at scale, trains preference models on it, and optimizes an assistant against them with reinforcement learning from human feedback (RLHF). It asks whether the result is both helpful and harmless, whether alignment training costs capability, how robust RLHF is, and whether collecting new data from each improved model helps. The paper calls itself "step one" after the 2021 paper's "step zero" (Section 7.1).

For the Constitutional AI thread, the paper is the baseline that Constitutional AI was designed to improve. Its assistant was specified not by written principles but by the judgments of crowdworkers, who were left to interpret "helpful" and "harmless" themselves, and the way those judgments were collected produced the evasive behavior that Constitutional AI set out to remove.

Terms used in this note

  • RLHF. Reinforcement learning from human feedback: people compare pairs of model responses, a preference model learns to predict their choices, and a policy is trained to maximize the preference model's score.
  • Preference model (PM). A language model fine-tuned to give a scalar score to a conversation ending in a response, trained so that the preferred response in each pair scores higher.
  • Red teaming. Deliberately trying to make a model produce harmful output. Here the harmlessness data come from crowdworkers red-teaming the model.
  • Static and online data. The static dataset was collected from fixed models. Iterated online training retrains preference models and policies each week on all data so far and deploys the new policies to collect the next round.
  • Rejection sampling (RS). Generating several responses and keeping the one a preference model scores highest.
  • KL divergence from the initial policy. A measure, written $D_{KL}(\pi \| \pi_0)$, of how far the trained policy $\pi$'s output distribution has moved from the starting model $\pi_0$.
  • Alignment tax and bonus. A loss, or gain, in benchmark performance caused by alignment training.
  • Elo score. A rating fitted from pairwise preferences. The paper relates win rates, Elo differences and PM scores by $\Delta\text{Elo} \approx 174 \times \Delta\text{PM score}$ (Section 2.4, equation 2.1).

What the paper does

Data collection

Crowdworkers chatted with models through a web interface and, at each turn, chose between two model responses (Section 2.1, Figure 6). There were two tasks with opposite instructions. In the helpfulness task, workers asked for help with any text-based task and picked the more helpful and honest response. In the red-teaming task, workers tried to provoke harmful responses and picked the more harmful response (Sections 1 and 2.2). Only comparisons with a preference stronger than the weakest available were kept, and all were then treated as binary (Section 2.2).

The paper deliberately did not define the two terms: "Our goal is not to define or prescribe what 'helpful' and 'harmless' mean", and workers interpreted them "as they see fit" (Section 1). Workers were recruited on MTurk (master-qualified, US-based) and Upwork, screened by spot-checks of how well and how variously they wrote rather than by label agreement (Section 2.1). MTurk workers produced about 80% of the data. Measured afterwards, agreement between Anthropic researchers and crowdworkers was about 63% (Section 2.1, Figure 10). The task instructions appear as figures in Appendix D.2.

Three datasets were collected, mostly with 52B models (Section 2.3):

  • a base dataset from a context-distilled 52B model, with 44k helpfulness and 42k red-teaming comparisons (a conversation typically gives about four comparisons);
  • a rejection-sampling dataset of 52k helpfulness and 2k red-teaming comparisons, usually with 16 samples per turn;
  • an online dataset of 22k helpfulness comparisons and no red-teaming data, collected from RLHF models updated roughly weekly over about five weeks.

Preference models and reinforcement learning

Preference models of seven sizes from 13M to 52B parameters, the models of the 2021 paper, were first given preference model pre-training and then fine-tuned for one epoch on the comparison data (Section 3.1). Policies were trained with Proximal Policy Optimization against the PM score, with a small KL penalty ($\lambda_{KL} = 0.001$) that the authors think may be unnecessary (Section 4.1, equation 4.1). RL prompts were 137k prompts from the static dataset plus 369k generated by a large language model from about ten examples each (Section 4.1).

Experiments around the main run

The paper also tests robustness by splitting the data in half, training a policy against a PM from one half and scoring it with a PM from the other (Section 4.2), varies the mix of helpfulness and harmlessness data in PM training from 0% to 100% in steps of 10% at a fixed total of 42k comparisons (Section 5.1), combines HH training with summarization and with code models (Sections 5.2 and 5.3), applies out-of-distribution detection to flag harmful requests (Section 5.4), and compares the models with professional writers hired to write ideal assistant dialogues (Section 6.1).

What it finds

Crowdworkers prefer the RLHF models, especially the online ones. Elo scores from crowdworker comparisons rise from context-distilled models to static RLHF models to online RLHF models (Figure 1). A controlled comparison with equal dataset sizes and settings, about 44k comparisons each, found the model trained on a mix of base, rejection-sampling and online data clearly preferred to one trained on base data alone (Section 4.5, Figure 16). Online PM accuracy was 74%, 70% and 67% on the base, RS and online test sets, which the authors read as higher-quality samples becoming harder to tell apart (Section 4.5).

The preference models score well on the 2021 HHH evaluation. The 52B static PM reaches 86% on the HHH comparisons, against a mean human score of 75% reported for the same dataset by another group (Section 3.4.1, Figure 5). Its confident errors are honesty failures, preferring a fluent but subtly wrong answer to an honest admission of ignorance (Figures 10 and 11).

Alignment training helps large models and hurts small ones. On zero-shot NLP benchmarks, RLHF hurts small models and improves large ones; the two largest RLHF models beat their base models on every benchmark except TriviaQA (Section 1.2, Figure 3; the section calls them the 12B and 52B models, the contributions list the 13B and 52B). Mixing HH training with summarization costs neither skill, and natural-language RLHF improves code models on HumanEval (Sections 5.2 and 5.3). The discussion places the boundary at roughly 10B parameters (Section 7).

Helpfulness and harmlessness pull in opposite directions. A PM trained only on helpfulness data, or only on harmlessness data, performs far below chance on the other distribution (Section 5.1, Figure 19). Larger PMs are less sensitive to the data mix. Raising the weight on the harmlessness loss from 1 to 10 cut helpfulness accuracy by 7.4% for the 13M PM and by 1.5% for the 52B PM (Section 5.1.2).

The harmlessness data taught refusal. Early RLHF policies gave the same exaggerated response to anything remotely sensitive, such as recommending therapy whenever a user expressed displeasure (Section 4.4). The authors explain this by their data: since workers picked the more harmful response, the data show what to avoid but not what a good response to a harmful request looks like, and a reply like "I can't answer that" scores well and is easy to learn. The policy's harmlessness score ended up on the upper tail of the harmlessness data while its helpfulness score stayed within the helpfulness data, that is, over-optimized for harmlessness and under-optimized for helpfulness (Figure 14). The authors name the behavior they would prefer, a model that explains why a request is harmful and tries to dissuade the user, a "hostage negotiator", and propose collecting data in which workers pick the best response instead (Section 4.4). Training on a larger share of helpfulness prompts partly resolved the problem.

Reward grows with the square root of the KL divergence. For much of RL training, PM score rises roughly linearly in $\sqrt{D_{KL}(\pi\|\pi_0)}$, and runs with different policy sizes trained against the same PM give roughly parallel lines (Section 4.3, Figures 4 and 13). The authors give a heuristic: the KL divergence is quadratic in a small change to the policy while the reward is linear in it, which predicts reward proportional to $\sqrt{D_{KL}}$ while the policy stays close to its start.

RLHF becomes less robust at high reward. Train and test PM scores agree early in training and diverge later, after about 150k training samples in the 52B case, with larger PMs more robust than smaller ones (Section 4.2, Figure 4).

Other results. RLHF improves TruthfulQA for large models (Figure 5). Sentiment toward all racial and religious groups becomes more positive, but this does not show reduced bias, and gender bias in RLHF models tracks that of the base models (Section 4.6.2, Section 7.1). Crowdworkers preferred the online HH model to the professional writers 56% of the time and the helpful-only model 51% of the time (Section 6.1); the summary of evaluations in Section 1.2 gives "about 57%" for the HH model. Out-of-distribution detection, which scores a request by its distance from the helpfulness data in a model's internal activations, separated harmful from helpful requests without seeing any harmful examples, worked better in larger models, and improved further when a few harmful examples were added (Section 5.4, Figures 22 and 23).

The written principles

This paper has no written principles in the sense used later. The behavior it trains is specified by three things: the short task instructions shown to crowdworkers (reproduced as figures in Appendix D.2), the instruction to choose the more helpful and honest response or the more harmful one, and the crowdworkers' own judgment. Workers were told that "lying isn't helpful" (Section 2.1), and Slack messages gave further guidance on difficult cases (Appendix D.1). The paper argues for this design on the grounds of data diversity and "the 'wisdom of the crowd'" (Section 2.1).

Two passages bear directly on the later move to written principles. In Section 7.2, "Alignment Data as a Public Good", the authors write that letting crowdworkers' common sense define helpful and harmless behavior was enough to study whether the techniques work, but that the question of what behavior deployed models should show "should not be the provenance of researchers only". In Section 4.4 and footnote 4 they trace a specific behavioral failure, evasiveness, to the design of the data collection, in particular the instruction to pick the more harmful response, which was chosen to explore the models' vulnerabilities. Constitutional AI changed both: the harmlessness criteria became a written list, and the comparisons it generated asked for the less harmful response.

Limitations and open questions

The authors state these limitations (Section 7.1, except where noted):

  • Honesty is barely addressed. RLHF improves honesty evaluations, but the authors expect other techniques to do better.
  • Average-case behavior only. The work studies typical behavior and does not address worst-case robustness.
  • Alignment may be on the surface. The models may still carry biases that appear in subtler contexts, and the bias evaluations used do not show that bias was reduced.
  • The zero-shot gap remains. An assistant that truly follows instructions should do as well zero-shot as an unaligned model does few-shot; RLHF closes a bit less than half of that gap on TruthfulQA.
  • Limited RL experience. Training was sometimes unstable, and online training retrained models from scratch each iteration instead of updating one model.
  • Data quality controls were minimal. Agreement-based quality checks did not suit open-ended conversations (Appendix D.3), and the crowdworker pool changed during the project, which complicates the online results (Section 2.1).
  • The writer comparison was not adversarial, and the authors caution against reading it as real-world performance (Section 1.2, footnote 7; Section 6.1).

Constitutional AI returned to the evasiveness problem eight months later. It reports that its HH RLHF baselines were still trained on the data from this paper, which "likely caused our HH PM's to reward evasiveness", and that changing the evaluation instructions to penalize evasiveness made the harmlessness gap between helpful and HH models much smaller than in this paper's Figure 1 (Bai et al., 2022b, Section 4.4).

Research questions

  1. How much of the evasiveness came from the "choose the more harmful" instruction? The authors attribute it to the data-collection design (Section 4.4). Collecting a red-teaming dataset where workers pick the better response, with everything else fixed, and comparing evasiveness of the resulting policies would test the attribution.
  2. Does the square-root relation between reward and KL divergence hold for other reward models and larger policies? It is reported as approximate and for "much of" training (Section 4.3). Measuring the relation across reward models trained on different data, including AI-generated labels, would show whether it is a property of RLHF in general or of these preference models.
  3. Where does the alignment bonus for large models come from? The paper shows that 13B and 52B RLHF models beat base models on zero-shot benchmarks (Figure 3). Separating the effect of instruction-following from the HH content of the data, for example by RLHF on helpfulness data only against HH data, would show which part of training produces the gain.
  4. How did crowdworker judgment shift during online training? The worker pool and its skill changed over the project (Section 2.1), and each week's models were judged by people who had seen the previous week's. Tracking agreement with a fixed set of researcher labels week by week would show whether the preference signal drifted.
  5. What would written instructions have changed? Workers interpreted "helpful" and "harmless" themselves, with about 63% agreement with researchers (Section 2.1). Running the same collection with a detailed written rubric and measuring agreement and downstream behavior would show how much a written specification narrows the variation that the paper attributes to "the wisdom of the crowd".
  6. Why are small models unable to be both helpful and harmless? Small PMs are much more sensitive to the helpful and harmless data mix (Section 5.1). Studying what small PMs learn from mixed data, for instance whether they classify the request type before scoring, as the authors suggest, would explain the size dependence.

How it connects

  • Before. Askell et al. (2021) supplied the models, the HHH evaluation, preference model pre-training and the context-distilled model used for the base dataset.
  • After. Constitutional AI keeps the human helpfulness data and this paper's training pipeline and replaces the human harmlessness labels with labels generated from written principles. Ganguli et al. (2023) studies RLHF-trained dialogue models, citing this paper for their training, across many sizes and RLHF step counts.
  • Elsewhere in the library. The red-teaming data and method are described in Anthropic's red-teaming paper.
  • In the course wiki. Crowdworker agreement with researchers is the problem of assessor disagreement covered in Relevance Judgments, Pooling, and Agreement, and the side-by-side model comparisons are a form of the testing in Online Evaluation. The helpfulness and harmlessness trade-off is a two-objective problem in the sense of Pareto Dominance and Pareto Optimality.

References

  • Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., Mann, B., and Kaplan, J. (2022a). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862v1, 12 April 2022. Anthropic. https://arxiv.org/abs/2204.05862. Anthropic page: https://www.anthropic.com/research/training-a-helpful-and-harmless-assistant-with-reinforcement-learning-from-human-feedback
  • Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., et al. (2021). A General Language Assistant as a Laboratory for Alignment. arXiv:2112.00861.
  • Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., et al. (2022b). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073.
  • Ganguli, D., Askell, A., Schiefer, N., Liao, T. I., Lukošiūtė, K., et al. (2023). The Capacity for Moral Self-Correction in Large Language Models. arXiv:2302.07459.