Predictability and Surprise in Large Generative Models

scaling lawsemergent capabilitiesAI policysocietal impactsfairnessevaluation

Ganguli, Hernandez, Lovitt et al. argue that large generative models combine predictable improvement in overall loss with unpredictable specific capabilities, inputs and outputs, illustrate this with published results and new experiments (including recidivism prediction, zero-shot movie recommendation and toxicity), describe the incentives and barriers this creates for developers, and propose policy interventions.

Deep Ganguli, Danny Hernandez, Liane Lovitt et al. (Anthropic) · 2022-02-15 · original

By early 2022, many organizations were training language models of GPT-3's size or larger. A developer could forecast with some confidence how much a larger model's overall loss would improve, using scaling laws, but could not say in advance which particular abilities, good or harmful, the larger model would have. This paper, dated 15 February 2022 on Anthropic's research page and published at the ACM Conference on Fairness, Accountability, and Transparency (FAccT '22), is a policy paper built on that observation. Its thesis is that large generative models have "a paradoxical combination of predictable loss on a broad training distribution (as embodied in their 'scaling laws'), and unpredictable specific capabilities, inputs, and outputs" (Abstract). The authors argue that the predictability drives rapid development while the unpredictability makes the consequences of deployment hard to foresee, and they address the paper to policymakers, technologists, funders and academics. It mixes evidence from earlier papers with a few new experiments on Anthropic's own models, and ends with recommended interventions.

Terms used in this note

  • Loss. The average error of a language model's next-token predictions on held-out text. Lower is better.
  • Scaling law. An empirical relationship, usually a power law (a straight line on a log-log plot), between loss and model size, dataset size or training compute. The paper relies mainly on Kaplan et al. (2020) for language models (Section 2.1, Figure 1).
  • Smooth, general capability scaling. The paper's name for two properties together: loss improves predictably with scale on a broad distribution, and that improvement correlates on average with better performance on many downstream tasks (Section 1).
  • Abrupt, specific capability scaling. Performance on a particular task that stays near zero across several model sizes and then jumps (Section 2.2). Later literature often calls this emergence.
  • Open-endedness. Models accept arbitrary inputs on any topic and produce arbitrary outputs, so some abilities are found only when someone happens to ask for them (Sections 2.3 and 2.4).
  • False positive rate ratio. In the recidivism experiment, the rate at which Black defendants who did not re-offend were predicted to re-offend, divided by the same rate for white defendants. A ratio of 1 means equal error rates.

What the paper does

The paper is organized around four claims about large generative models (Section 2):

  1. Smooth, general capability scaling. Overall performance improves predictably with model size, data and compute, which the authors argue can "de-risk investments" in larger models.
  2. Abrupt, specific capability scaling. Specific tasks can improve suddenly at some scale, because a single task is "a tiny slice" of the model's output distribution and can change quickly while the whole distribution changes smoothly.
  3. Open-ended inputs and domains. Abilities, including harmful ones, may stay unknown until an input elicits them.
  4. Open-ended outputs. Even for a fixed input, outputs are hard to predict or control.

Each claim is supported by results from the literature and, for three of them, by new experiments using the Anthropic language models described in A General Language Assistant as a Laboratory for Alignment, with sizes up to 52 billion parameters. Section 3 then discusses developers' motivations and the barriers they face, with three empirical observations about the field, and Section 4 proposes interventions. Appendix A.1 records that Jack Clark, Deep Ganguli and Dario Amodei wrote the paper and that Ganguli performed the experiments.

What it finds

Evidence for smooth scaling

Section 2.1 reviews earlier work showing that loss falls as a power law in model size, data and compute (Figure 1, adapted from Kaplan et al., 2020), and that similar laws hold for other modalities. The authors argue that this turns model development from "artisanal trial-and-error" into something closer to predictable engineering, in which the resources needed for a result can be calculated in advance. Appendix A.2 lists five ways developers use scaling laws, from estimating the lowest loss reachable for a compute budget to debugging training runs.

A new experiment illustrates how smooth scaling can forecast economic value (Appendix A.3, Figure 8). The models were asked, zero-shot, to predict how a MovieLens 1M user (described by age, gender and occupation, with up to 500 of their earlier ratings in the prompt) would rate a held-out movie. Error fell smoothly with model size: the smallest model reached a root mean squared error (RMSE) of 1.06 against 1.91 for chance, and the largest reached 0.94, better than a baseline of 0.98 that predicts every movie's average rating. None reached the state of the art of about 0.82. Extrapolating the trend, the authors obtain what they call an "incredulous" estimate that a model of 800 trillion parameters would match the state of the art zero-shot, and conclude that this use is unlikely to be commercially viable for several years, while noting that a different setup, such as fine-tuning, might have given a different forecast.

Evidence for abrupt scaling

Section 2.2 and Figure 2 present three examples from other papers:

  • Arithmetic (GPT-3). Three-digit addition is accurate less than 1% of the time for models under 6 billion parameters, 8% at 13 billion and 80% at 175 billion.
  • Language understanding (Gopher, on the MMLU benchmark). All models under 6 billion parameters score under 30%, a little above the 25% chance level, while the 280-billion-parameter model reaches 60%. GPT-3 shows a smaller jump.
  • Program synthesis (Google models, Austin et al. 2021). The share of generated programs that solve a problem rises from 6% to 13% as size roughly doubles from 68 billion to 138 billion parameters, after small gains over the previous two orders of magnitude.

The authors note that these are "relatively anodyne capabilities" and that harmful ones might appear the same way. In Section 4 they add that, in their experience, abrupt jumps occur in only a minority of tasks but are "not especially rare".

A harmful ability appearing with scale: recidivism prediction

To illustrate open-ended inputs, the authors chose a task they say language models should not be used for: predicting whether a criminal defendant will re-offend within two years (Section 2.3, Appendix A.4). They used the ProPublica COMPAS dataset of more than 7,000 defendants in Broward County, Florida, and the prompt from Dressel and Farid's (2018) study of human predictions, with and without the defendant's race, giving the model 50 labeled examples before each test case. Results (Figure 3):

  • Accuracy rose with model size. The 52-billion-parameter model reached 63%, against 66% for the COMPAS algorithm, with no significant difference between prompts with and without race.
  • Bias. The false positive rate ratio for Black versus white defendants is 1.92 for COMPAS. For the language models it rose with model size from around 12 billion parameters, reaching 1.5 for the largest model when race was excluded and 2.21 when race was included, less equitable than COMPAS on this measure.
  • Absolute rates. A footnote adds that the largest model's false positive rates (30% for Black and 12.6% for white defendants, with race included) are lower than COMPAS's (45% and 24%).
  • Accuracy ratio. Appendix A.4 reports no clear trend with model size in the ratio of predictive accuracy for Black versus white defendants (COMPAS: 0.97), with the largest models slightly less equitable than COMPAS (Figure 9).

The authors stress that the point is the appearance of the ability to perform the task at all, not only the bias. They state their own caveats (Appendix A.4): there are many fairness metrics and no algorithm can satisfy all of them at once, datasets like COMPAS contain measurement biases and errors that make real-world claims difficult, and comparisons with a proprietary algorithm are hard to make precise.

Open-ended outputs

Section 2.4 gives a qualitative example (Figure 4): asked to say something offensive, a 52-billion-parameter assistant instead listed supposed offenses by other AI systems, got details of IBM Watson's and Microsoft Tay's problems wrong, and, when questioned, gave misleading answers. Quantitatively, the toxicity of model continuations of RealToxicityPrompts prompts increased smoothly and significantly with model size (Figure 5). The method sampled 25 continuations for each of 1,000 prompts (half labeled toxic), scored them with the open-source Detoxify classifier, and regressed toxicity on model size while controlling for prompt toxicity (Appendix A.6). The authors caveat that the effect size's meaning for human readers is unclear and that automated toxicity detectors have known biases. Appendix A.5 adds an informal sample of over three thousand machine-generated imitation poems as an example of a capability that resists quantitative evaluation.

Motivations, barriers and three observations about the field

Section 3 names three motivations for building large models (economic, scientific and prestige) and three barriers (financial cost and engineering talent; safety and bias problems without robust solutions; and the absence of deployment standards). On the last, the authors write that "We are, in a sense, building the plane as it is taking off" (Section 3.2), and point to model cards and datasheets as early steps toward documentation standards. Section 3.3 reports:

  • Proliferation. Figure 6 shows public announcements of dense language models of GPT-3 scale (100 to 530 billion parameters) by organizations in the United States, China, South Korea, Israel and the United Kingdom, with a spike about a year after GPT-3.
  • A gap between industry and academia. The largest freely available models at the time (T0, GPT-J and GPT-NeoX, from 6 to 20 billion parameters) were one to two orders of magnitude smaller than industry's. Using an augmented version of an existing dataset of compute used by major AI projects, the authors find that compute for large-scale experiments has grown by more than 300,000 times relative to a decade earlier, while the share of such results from academia has fallen sharply (Figure 7, Appendix A.7). They caution that the data are incomplete and subject to sampling bias.
  • Harm and controversy from deployed systems, such as Tay, and from memorization of training data.

The prompts used in the experiments

The paper has no principles or constitution. Its new experiments adapt a general pretrained model to each task through a prompt alone, without task-specific training, and two of those prompts are printed in full in the appendices:

  • Recidivism (Appendix A.4). A template taken from Dressel and Farid's human study that fills in the defendant's sex, age, charge, charge degree, prior convictions and juvenile charges, and ends by asking whether the person will commit another crime within two years. The race condition changes only the first sentence to include the defendant's race. The model's answer is read from the probabilities it assigns to "Yes" and "No".
  • Movie recommendation (Appendix A.3). A template that describes a user by age, gender and occupation, lists movies the user rated with their ratings, and asks for the rating of a further movie; the prediction is the probability-weighted average of the digits 1 to 5. The authors report that this zero-shot form, with many ratings from one user, gave far lower error than all of their few-shot variants.

Section 2.3 makes a related point with a deployed system: AI Dungeon, a game built on GPT-3 fine-tuned for fantasy role-playing, could be steered by players to any topic, so a product that appeared to serve one purpose carried the full range of GPT-3's capabilities.

Section 4 proposes five interventions, each with an implementation path:

  1. Reduce compute asymmetries between the private sector and academia, for example through national research clouds, noting that training a frontier model then cost on the order of one to ten million dollars, less than many "Big Science" projects such as the Large Hadron Collider ($5 billion).
  2. Improve knowledge of how to red team models, through internal red teaming, publication of methods and datasets, bug bounties and automated methods.
  3. Explore governance structures and government interventions, including oversight bodies, board members who represent civil society, industry best practices with outside feedback, and government support for measuring deployed systems and for an auditing ecosystem.
  4. Improve tools for model evaluation, especially open tools for broad evaluations and for searching across prompts for new capabilities.
  5. Improve understanding of abrupt jumps in capability, through a systematic empirical study of how often they occur and whether anything predicts them, and through mechanistic interpretability, citing A Mathematical Framework for Transformer Circuits.

Limitations and open questions

  • Illustrative experiments. The new experiments are small and are meant to illustrate claims, as the authors say for the recommendation experiment; the recidivism experiment comes with the dataset and metric caveats listed above.
  • Abruptness depends on the measure. The three abrupt examples are reported with accuracy-style measures from other papers; the paper does not test whether the same models improve smoothly on a continuous measure. Schaeffer, Miranda and Koyejo (2023) later argued that many apparent emergent abilities reflect the choice of a nonlinear or discontinuous metric and can disappear under continuous ones. This note does not check whether their analysis covered the specific examples here.
  • Two averages for the recommendation data. Appendix A.3 gives the average number of movies rated per user as about 160 in its description of the dataset and about 200 in its explanation of the prompt length; the paper does not say whether the two figures refer to different subsets.
  • Scope. The authors state that they do not consider the labor of creating and annotating training data, hardware supply chains or environmental costs (Section 1).
  • Open questions posed by the authors. How often abrupt jumps occur, on which tasks, why, and whether leading indicators exist (Section 4).

Research questions

  1. Do the abrupt examples remain abrupt under continuous measures? Re-scoring three-digit addition, MMLU and program synthesis with per-token log-likelihood of the correct answer across model sizes would show whether the jumps reflect the metric or the models.
  2. How common are abrupt jumps across a broad task set? The authors' impression is a minority of tasks that is "not especially rare". Counting tasks with a sharp transition in a large multi-task benchmark, under a stated definition of abruptness, would turn that impression into a rate.
  3. Can scaling-law forecasts of a specific task's value be checked? The recommendation experiment extrapolates to 800 trillion parameters. Comparing such forecasts made from small models with the measured performance of later, larger models on the same task would test the method the paper describes.
  4. Does the recidivism bias trend hold with other prompts and metrics? The experiment uses one prompt and two main metrics. Repeating it with reworded prompts and the wider set of fairness metrics the authors mention would show how much the trend depends on those choices.
  5. Has the compute gap between industry and academia continued? Extending the Appendix A.7 dataset and labeling method to later years would answer this with the paper's own measure.
  6. Which of the proposed interventions were adopted, and by whom? A survey of public red-teaming publications, bug bounty programs and national research cloud initiatives after 2022 would give a concrete record against the paper's list.

How it connects

References

  • Ganguli, D., Hernandez, D., Lovitt, L., DasSarma, N., Henighan, T., Jones, A., Joseph, N., Kernion, J., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Elhage, N., El Showk, S., Fort, S., Hatfield-Dodds, Z., Johnston, S., Kravec, S., Nanda, N., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Kaplan, J., McCandlish, S., Olah, C., Amodei, D., and Clark, J. (2022). Predictability and Surprise in Large Generative Models. In 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT '22), Seoul. https://doi.org/10.1145/3531146.3533229. arXiv:2202.07785v2, 3 October 2022. Section, figure and appendix numbers in this note refer to the arXiv v2 PDF.
  • Anthropic. (2022, February 15). Predictability and Surprise in Large Generative Models [research page]. https://www.anthropic.com/research/predictability-and-surprise-in-large-generative-models
  • Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361. Cited here only as the paper's source for its Figure 1.
  • Schaeffer, R., Miranda, B., and Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage? arXiv:2304.15004. Cited for its argument that apparent emergent abilities can be produced by the choice of metric.