Measuring Progress on Scalable Oversight for Large Language Models

scalable oversightsandwichinghuman-ai teamsrlhfcalibrationalignment

Bowman et al. turn Cotra's "sandwiching" proposal into an experimental design for studying how people can supervise AI systems more capable than themselves, and run a simplified version in which non-expert participants chat with a 52B RLHF assistant to answer hard multiple-choice questions; the human-model teams beat both the model alone and the unaided participants, but still fall short of expert performance and are sometimes confidently wrong.

Samuel R. Bowman et al. (Anthropic, Surge AI) · 2022-11-04 · original

Training methods such as reinforcement learning from human feedback depend on people judging whether a model's output is good. That works while people can tell. Scalable oversight is the problem of supervising a system that may outperform its supervisors on the skills a task requires, through labels, reward signals or critiques that stay reliable after models reach broadly human-level performance (Section 1). In 2022 there were several proposals for how to do this, such as debate and recursive reward modeling, but little empirical work, because no available system broadly exceeded human ability. This paper, from Anthropic with the data-labeling company Surge AI, proposes a way to study the problem with present-day models and runs a first, deliberately simple experiment to show that the design is workable.

Terms used in this note

  • Scalable oversight. Supervision of AI systems that remains effective when the systems are more capable than the people supervising them (Section 1).
  • Capable and misaligned. The paper defines a model as capable of a task if a small-to-moderate intervention, such as fine-tuning or few-shot prompting on a moderate amount of good data, makes it perform well; it is misaligned on the task if it is capable in this sense but performs poorly when simply asked (Section 2).
  • Sandwiching. An experimental design proposed by Ajeya Cotra (2021) in which a task is chosen so that the model's capability falls between that of typical people and that of experts (Section 2).
  • MMLU. A benchmark of multiple-choice questions drawn largely from practice tests for high-school, undergraduate and professional exams (Hendrycks et al., 2020).
  • QuALITY. A reading-comprehension benchmark of multiple-choice questions about stories of about 5,000 words (Pang et al., 2022).
  • Calibration error (CE). How far stated confidence departs from actual accuracy. The paper groups answers into five confidence bins centred at 25, 42, 60, 80 and 100 percent, measures accuracy in each, and averages the absolute gaps (footnote 9).

What the paper does

The sandwiching paradigm

A sandwiching experiment has three parties (Section 2). Expert evaluators have all the knowledge the task requires and serve only as a reference at the end. The model is expected to have most of the needed knowledge but not to use it reliably. Non-expert participants understand the task and want to do it well but lack key knowledge; they try to get the model to perform the task reliably, without any help from the experts or their written materials, and to build justified confidence that they have succeeded. In an inner loop the non-experts iterate until they believe the model is aligned; the experts then check. In an outer loop researchers revise the oversight technique and repeat. The goal is a technique that lets the inner loop succeed on the first try, without expert feedback, across many tasks, which would suggest it could work where no experts exist. The paper's illustration, taken from Cotra, is a group of non-clinicians trying to get a language model to give only appropriate medical advice without consulting clinicians or the medical literature (Section 1).

Section 2 lists techniques the paradigm could test: plain interaction with the model, debate (the model argues both sides and a human judges), market-making (a variant of debate in which one model tries to shift a judge's beliefs at each step), and self-critique, amplification and recursive reward modeling (humans supervise a model with help from a critique model, which is itself supervised the same way).

Two relaxations for a first experiment

The experiment simplifies the paradigm in two ways (Section 2). Participants interact with the model only through chat, without fine-tuning or inspecting it (static model). And the tasks are multiple-choice questions with known answers, so the expert role is replaced by scoring against existing labels (labels in place of experts). The authors say both relaxations narrow the conclusions that can be drawn and expect to remove them as promising techniques emerge.

Tasks, participants and model

The two tasks were chosen so that the model would do better than unaided participants but still make frequent mistakes (Section 3.1). For MMLU, the model is expected to bring domain knowledge the participants lack. For QuALITY, participants had a five-minute limit for each question, which is not enough to read a story that would usually take 15 to 30 minutes, so they had to rely on the model to find information.

Surge AI recruited two groups of five participants, one per task, paid at a minimum of $20 an hour; pay did not depend on accuracy (Section 3.2). Each participant answered 100 randomly sampled validation-set questions per condition. The results exclude one participant from each group, because each scored far above the others without the model (84 against 57 percent on MMLU, 89 against 49 percent on QuALITY) and above the model's best result, which would break the paradigm's assumption that participants lack something the model can supply. The authors say they cannot fully rule out cheating but believe these two were unusually good at trivia and speed-reading.

The model was a 52-billion-parameter language model fine-tuned with RLHF as a helpful dialog assistant, following Bai et al. (2022); it was trained to be helpful only, not helpful and harmless (Section 3.3, footnote 7).

Conditions

Five conditions were compared (Section 3.3):

  1. Unassisted human: participants answered alone, with no web search. Results are reported both as average individual accuracy and as a confidence-weighted majority vote of the four participants.
  2. Model: the model answered each question zero-shot, and the answer letter with the highest likelihood was taken.
  3. Model (5-shot): the same, with five solved examples in the prompt (MMLU only, because QuALITY stories are too long to fit several in the context window).
  4. Model (best-of-20 chain-of-thought): the model wrote 20 samples of step-by-step reasoning, the one rated best by Anthropic's RLHF preference model was kept, and the model then gave its answer. For MMLU this condition also used the five solved examples.
  5. Human + model: participants used the chat interface to get help from the model, on a separate sample of 100 questions.

The interface had a chat pane, a question pane with four answer choices and a five-point confidence scale, and a reset button that wiped the conversation from the model's context while leaving it visible to the participant (Section 3.3, Figure 2). The question was not automatically given to the model. After each answer, the correct answer was revealed, a departure from true sandwiching that the authors made so participants could learn which strategies worked. Participants completed the assisted condition before the unassisted one, so any learning would bias the comparison toward finding no benefit.

What it finds

Teams beat both the model and the unaided participants

Table 1 of the paper reports accuracy and calibration error for every condition on both tasks; Section 4 summarizes the main comparisons in rounded figures, which this note follows. Measured by individual accuracy, the best model condition beats unaided participants (66 against 57 percent on MMLU, 67 against 49 on QuALITY), and assisted participants beat the best model condition by similar margins (75 against 66, 77 against 67). Every participant exceeded 71 percent individually in the assisted condition. With majority voting, the model's advantage over unaided participants holds only on QuALITY (66 against 66 on MMLU, 67 against 50 on QuALITY), while assisted teams beat the best model condition on both (78 against 66, 86 against 67). The introduction summarizes the gains as teams outperforming the model by about 10 percentage points and their own unaided performance by up to 36 points (Section 1).

The teams do not reach expert level. On timed QuALITY, the best team score is below the best known untimed human team score (86 against 94 percent, from Pang et al., 2022), and on MMLU it is below Hendrycks et al.'s hypothesized performance for a committee of experts (78 against 90 percent) (Section 4). Few-shot examples helped the model on MMLU (62 against 57 percent), and best-of-20 chain-of-thought helped on both tasks (66 against 62 on MMLU, 67 against 59 on QuALITY).

Confidence rose more than calibration improved

On MMLU, participants alone and the model alone were reasonably well calibrated, with CE of no more than 10, while human-model teams and chain-of-thought prompting were substantially worse (Section 4). On QuALITY the direction reversed: unaided participants were poorly calibrated and teams did substantially better (individual CE fell from 17 to 7, Table 1). The authors relate the MMLU pattern to Kadavath et al. (2022), who found that pretrained models are well calibrated on multiple-choice questions but lose calibration after RLHF for helpfulness, as used here. Access to the model raised participants' average confidence from 54 to 80 percent on MMLU and from 38 to 75 percent on QuALITY. Where the weighted vote reached its maximum confidence of 99 percent, accuracy was 88 percent on QuALITY (89 questions) and 81 percent on MMLU (84 questions).

How participants used the model

Conversations averaged 10.9 turns and 2.0 resets on MMLU, and 6.3 turns and 0.1 resets on QuALITY, where the time limit probably explains the lower figures (Section 4). Revealing answers seems to have taught little: two of four MMLU participants and three of four QuALITY participants improved between the first and second halves, by 0.7 percent on average.

From transcripts and participants' notes, the authors report these strategies (Section 4, "Qualitative Findings"). On MMLU, participants learned to trust the model's statements of fact but not its long reasoning or arithmetic; asked for specific facts and definitions before asking for help with the whole question, to avoid being primed by a wrong answer; asked about each answer choice as a separate true-or-false question, resetting between them, to expose inconsistency; asked for explicit step-by-step reasoning; and combined several strategies on each question. They also noticed that the model readily accepted corrections, which helped when they caught its errors but made it go along with their own mistakes. On QuALITY, participants asked the model for quotations from the passage and checked them, searched the story for terms the model mentioned, phrased questions to presuppose facts they had already confirmed, and found the model more reliable on factual questions than on questions of interpretation, where it tended to invent details.

The authors conclude that the model has room to improve in reasoning and reliability, and that simple systematic prompting by participants got the model to argue for the right answer more often than it chose the right answer on its own, which points toward more automatic ways of eliciting the same behavior.

The prompts and instructions used

The paper prints two kinds of text that shaped the experiment.

The model prompts. The zero-shot prompt presents the question and lettered choices inside a "Human:" and "Assistant:" dialog frame, which the authors describe as a "minimal prompt, which was designed to dovetail well with RLHF training without injecting any new information about the task" (Section 3.3). The chain-of-thought condition adds "Let's think step by step:" to the assistant's turn, following Kojima et al. (2022), and then asks for "the single, most likely answer choice".

The participant instructions. Appendix A prints the QuALITY instructions (a variant was used for MMLU). They tell participants not to read the whole story, that the five-minute limit is intentional, and that the assistant answering alone gets "less than three quarters right" while the researchers want to know whether participants can reach 80 or 90 percent. They ask participants to take their confidence ratings seriously, because "a confident right answer is much better than an uncertain right answer, but a confident wrong answer is much worse." They suggest strategies: ask for relevant quotations and check them, ask for summaries, ask for the best arguments for and against each choice, ask the model to role-play a character, or ask it to critique its own responses. They also warn that the assistant is "very reluctant to contradict itself", so resetting the conversation can clear out an early mistake. Several strategies that participants later reported, such as asking for quotations and resetting between questions, also appear among these suggestions.

Limitations and open questions

The authors state that the experiment does not fully implement sandwiching and that the results do not show that chat-based assistance would be adequate for overseeing more capable systems. They give six reasons (Section 4, "Limitations of Our Results"):

  • The relaxations. Unambiguous multiple-choice questions are an unrealistic setting, and the experiment does not fine-tune the model on what participants elicit.
  • Revealed answers. Showing the correct answer after each question removes a central difficulty of scalable oversight: deciding, without feedback, when a method is reliable enough to trust. The authors plan to give practice on a separate task in later work.
  • Only what the model says when asked. Incentives or interpretability methods might elicit more of what the model knows.
  • Selection. Removing the two high-scoring participants out of ten was needed to meet the paradigm's assumptions but makes it harder to observe cases where assistance does not help.
  • Confident errors. Participants sometimes accepted confident, plausible false claims they could not check. Appendix C shows one: on a question about which leadership style produced the most work in a 1939 experiment by Lewin and colleagues, the model gave inconsistent accounts across conversations, and the participant chose "Democratic" when the keyed answer was "Authoritarian".
  • Familiar task format. Multiple-choice questions are common online, so the model was to some extent trained for this task; scalable oversight matters most for tasks further from training.

The paper's related-work section notes a finding from human-AI decision-making research that bears on the confidence results: simple explanations can raise users' trust in an AI system more than is warranted, and teams do worse when the AI gives its answer before the human has tried the problem (Bansal et al., 2021; Section 5). The authors say their participants appear to have noticed the second effect informally.

Research questions

  1. How much do the gains depend on revealing the correct answer after each question? The paper argues that the reveal taught little (a 0.7 percent average improvement between halves) but names it as a limitation. Running the assisted condition with practice on a different task and no reveals, as the authors propose, would measure the difference directly.
  2. Can participants tell when they have succeeded? Teams reached 81 to 88 percent accuracy on answers given at maximum confidence, which is not reliable enough for unsupervised use. A study that asks participants to predict their own accuracy on a held-out set before scoring would test the "justified confidence" that sandwiching requires.
  3. Do adversarial techniques such as debate help non-experts more than plain chat on the same tasks? The paper proposes debate and market-making but tests only chat. Running them against the same MMLU and timed-QuALITY baselines would give a direct comparison.
  4. Does the human-model advantage hold on tasks less like the model's training data? The authors note that multiple-choice questions are common online. Repeating the protocol on open-ended tasks with expert-graded answers, using real experts rather than labels, would show whether the effect generalizes.
  5. Why does calibration worsen for teams on MMLU but improve on QuALITY? Table 1 shows opposite directions. Analyzing confidence by strategy (for example, quotation checking against free conversation) would show which interactions inflate confidence without improving accuracy.
  6. How much do the participant instructions contribute? Many reported strategies appear in the Appendix A suggestions. Comparing teams given the full instructions with teams given only the task description would separate the effect of the instructions from that of model access.
  7. What happens when experts disagree? The paradigm assumes experts can give a reference answer. Running sandwiching on questions with measured expert disagreement would show how the evaluation behaves when the top of the sandwich is uncertain.

How it connects

References

  • Bowman, S. R., Hyun, J., Perez, E., Chen, E., Pettit, C., Heiner, S., Lukošiūtė, K., Askell, A., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Olah, C., Amodei, D., Amodei, D., Drain, D., Li, D., Tran-Johnson, E., Clark, J., Kernion, J., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lovitt, L., Elhage, N., Schiefer, N., Joseph, N., Mercado, N., DasSarma, N., Larson, R., McCandlish, S., Kundu, S., Johnston, S., Kravec, S., El Showk, S., Fort, S., Telleen-Lawton, T., Brown, T., Henighan, T., Hume, T., Bai, Y., Hatfield-Dodds, Z., Mann, B., and Kaplan, J. (2022). Measuring Progress on Scalable Oversight for Large Language Models. arXiv:2211.03540v2, 11 November 2022. https://arxiv.org/abs/2211.03540. Anthropic page: https://www.anthropic.com/research/measuring-progress-on-scalable-oversight-for-large-language-models
  • Cotra, A. (2021). The case for aligning narrowly superhuman models. AI Alignment Forum. (Cited by the paper as the source of sandwiching.)
  • Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., and Weld, D. S. (2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. CHI 2021. (As summarized in the paper's Section 5.)