Scaling Laws and Interpretability of Learning from Repeated Data

scaling lawstraining datadata repetitiondouble descentmemorizationinduction headsinterpretability

Hernandez et al. train language models on mostly unique text plus a small subset repeated many times and find a double descent effect in which a middle range of repetition badly degrades performance (an 800M-parameter model falls to about half its effective size when 0.1% of data is repeated 100 times); the damage falls disproportionately on copying and induction heads.

Danny Hernandez et al. (Anthropic) · 2022-05-21 · original

Large text datasets for language models contain repeated material. Some is deliberate, when developers show a model high-quality sources such as Wikipedia or books more than once; some is accidental, because removing duplicates is never perfect and the same sentences, paragraphs and documents recur across the web. Earlier work had reported that such repetition can harm performance substantially, but not why. This paper, posted on 21 May 2022, studies repetition in a controlled way and from two directions at once: the large-scale view of scaling laws, which describe how loss changes with model size and data, and the small-scale view of mechanistic interpretability, which looks at the internal structures that implement a behavior. Its headline result is that a small fraction of repeated data, repeated a particular number of times, can do damage far out of proportion to its share of training, and that the damage falls especially on induction heads, the attention-head circuit that copies patterns from earlier in the context.

Terms used in this note

  • Repeated subset and repeated epochs. In each run, a small sample of the dataset is shown many times; the number of passes over that sample is its number of epochs. The rest of the training data is seen once.
  • Double descent. A pattern, known from earlier work on model size and training time, in which test loss falls, then rises, then falls again. The rise is usually associated with the point at which the model can just fit (memorize) its training data. Epoch-wise double descent happens over the course of training.
  • Effective model size (model size multiplier). A way of expressing damage: fit a scaling law to control models trained on unique data, then report a damaged model's loss as the size of control model that would have the same loss. A multiplier of 0.5 means the model performs like one half its size.
  • Copying evaluation. The loss on the first paragraph of Harry Potter and the Philosopher's Stone repeated 11 times. Predicting a second or later copy requires copying from the context, which works the same whatever the text says.
  • Induction head and prefix-matching score. An induction head, described in In-context Learning and Induction Heads, attends from the current token to whatever followed that token's earlier occurrence and copies it. The prefix-matching score measures, on repeated sequences of random tokens, how much a head attends to that position.
  • Ossification. The paper's term for pre-training that leaves a model worse after fine-tuning than a model trained from scratch on the same budget (Section 5.4).

What the paper does

The authors trained a family of decoder-only transformers, each for 100 billion tokens, on a language dataset of 400 billion tokens: 55% heavily filtered Common Crawl, 32% internet books, and smaller sources such as OpenWebText, Wikipedia and Stack Exchange, mostly taken from The Pile, with an 8,192-token context and the training settings of A General Language Assistant as a Laboratory for Alignment (Section 3). Each run draws a fixed fraction of its training tokens from a small repeated subset and the rest from unique data (Figure 1). For example, holding the repeated share at 10%, the subset might be 0.01% of training tokens shown 1,000 times or 1% shown 10 times. Across runs they varied model size (about 1.5 million to 800 million parameters), the size of the repeated subset and the fraction of repeated tokens (3%, 10%, 20%, 50% and 90% in the main fits) by 3, 2.5 and 2 orders of magnitude respectively. Test loss is always measured on held-out unique text.

They then measured:

  • test loss, converted into effective model size;
  • the copying evaluation on the repeated Harry Potter paragraph;
  • prefix-matching scores of attention heads;
  • per-token predictions of one- and two-layer attention-only models, whose induction heads and skip-trigram copying are well understood from A Mathematical Framework for Transformer Circuits;
  • loss on Python code as an out-of-distribution test;
  • the effect of pre-training on repeated data when the model is then fine-tuned on Python.

A parallel set of experiments repeated data in a Python dataset of 45 billion tokens trained for 2.2 epochs.

What it finds

A damaging middle range of repetition

Repetition does little harm when the subset is repeated only a few times and little harm when a tiny subset is repeated an enormous number of times, but there is a middle range in which damage is large (Section 2, Figure 2). With 10% of tokens from the repeated subset, an 800-million-parameter model is hurt most at roughly 100 repeats of 0.1% of the data, which degrades its performance nearly to that of a 340-million-parameter model; the abstract describes the same case as performing like a 400-million-parameter model, "2x smaller". At its peak, the degradation "sometimes reaches the equivalent of a 2x decrease in model size" (Figure 2 caption). The worst damage coincides with the point at which training loss on the repeated subset approaches zero, that is, when the model has memorized it, which the authors offer as a practical diagnostic.

With 90% of tokens repeated at the right frequency (100 to 10,000 repeats), test loss traces a literal double descent curve over training, falling, rising and falling again; at 50% the bump becomes a long plateau (Figure 3).

A predictable band in model size and repetition

Converted into effective model size, the damage for a given number of repeats appears only for a particular range of model sizes (Figure 4). With 10% repeated data and 1,220 repeats, models of 10 million to 100 million parameters perform like models 0.55 times their size, recovering to 0.8 times at 1 billion parameters; with 122 repeats the dip comes later, centered around 1 billion parameters. The region with at least half the maximum damage forms a band whose two edges fit power laws relating epochs of repetition $E$ to parameters $N$, $E = k N^{\alpha}$, with $k = 5.1 \times 10^7$, $\alpha = -0.50$ for the right edge and $k = 4.2 \times 10^6$, $\alpha = -0.56$ for the left (Section 2). Extrapolated to models with hundreds of billions of parameters, the band predicts significant damage from repeating data as few as two times. The authors qualify this: the fits hold the training run at 100 billion tokens, larger models are usually trained longer, and training past the double descent peak helps, so the damage would likely be smaller in practice. Appendix A adds that the fits are noisy, that the 1% runs were excluded as too noisy, and that the exact boundaries could differ in other setups, especially with different regularization.

Copying is damaged far more than overall loss

On the copying evaluation, 3% repeated data at the worst number of repeats reduced effective model size about threefold for several model sizes, while reducing it on test loss by at most a factor of 1.15 (Section 2, Figure 5). Across repeated fractions from 3% to 20%, the effect on copying was much larger than on test loss. At the peak of the double descent, a two-layer model's loss on successive copies of the paragraph did not fall at all as more copies were shown, where healthy models improve with each copy (Figure 7). General in-context learning, measured by how loss falls with position in a document, was also harmed, but less than copying.

Induction heads are damaged disproportionately

Prefix-matching scores, a measure of induction-head structure separate from copying behavior, were also preferentially damaged, though only at low repeated fractions rather than across the whole range (Section 2, Figure 6). At 3% repeated data, the text gives an example of a 2x drop in effective size on prefix matching against 1.15x on test loss; the Figure 6 caption gives an average multiplier of 1.47, and the summary of results calls it a 32% reduction on average. In one extreme case, a 1.5-million-parameter model trained with 50% repetition had the overall performance of a model of about 30,000 parameters but the prefix-matching score of a model of about 2,000, and its paragraph-level copying broke down completely. The authors call these results "not as conclusive as the previous results" but say they clearly show preferential degradation in some cases.

Damage visible token by token in small models

In one-layer attention-only models, which copy through skip-trigrams, the model trained on 10% repeated data was worse at predicting repeated proper names in the Harry Potter paragraph: on the last three appearances of "Potters", the control model put more than 97% probability on "ters" after "Pot", the repeated-data model less than 4% (Figure 8). In two-layer models the picture was more mixed, with the repeated-data model sometimes better on repeated names, but with cases where the control model was far better at fuzzy copying; for the "ley" in "unDursleyish" the control model gave 68% and the repeated-data model 0.4% (Figure 9). An attempt to trace the difference with logit attribution found no clear evidence either way. In the discussion the authors write that, for the two-layer model of Figure 7, it "seems as if" pressure to memorize led a skip-trigram head to replace the induction head entirely (Section 5.2).

Out of distribution, on code, and after fine-tuning

  • Python loss. Repeated text data caused a small but disproportionate drop in performance on Python code, mostly at high repetition (50% and 90%), with average model size multipliers of 0.84 and 0.75 (Figure 10), almost the opposite pattern from the induction-head results.
  • Repeated Python data also produced double descent and a band of poor performance, but with a noisier fit whose edges are not parallel; the authors suggest reasons, including that the Python data itself was trained for about two epochs and that code contains much copy-pasted repetition (Figure 11).
  • Fine-tuning. A model pre-trained on heavily repeated data at the peak of the double descent, then fine-tuned on Python, performed worse than a model trained on Python from scratch, a 1.6x reduction in effective size, much smaller than its damage before fine-tuning. Section 2 describes this case as an 800-million-parameter model pre-trained on 50% repeated data and reduced 10x before fine-tuning; Section 5.4 and the Figure 12 caption describe pre-training on 90% repeated data, with a 73x reduction. It forgot the repeated data quickly during fine-tuning and eventually caught up with a model pre-trained on unique data (Figure 12). The authors treat this as medium causal evidence that repetition can cause ossification.

Why a small subset can do so much damage

Section 5.1 offers an informal explanation with rough numbers. An 800-million-parameter model has a loss of roughly 2.0 nats per token, a 400-million-parameter model roughly 2.2, and fully memorized data a loss of 0. If 10% of training tokens are repeated, memorizing them at the cost of performing like a model half the size gives an average loss of $0.9 \times 2.2 + 0.1 \times 0 = 1.98$, slightly below 2.0, so the trade is favored by the training objective. Whether it happens depends on capacity: data repeated only a few times cannot be memorized, and a tiny subset repeated very often can be memorized cheaply, so the damage peaks in between, where memorizing is possible and uses much of the model's capacity. The authors describe the result as "a vivid illustration of the memorization-generalization trade-off" (Section 5.2), and present the joint degradation of copying and induction heads as further evidence for induction heads as a source of in-context learning, since the earlier paper "was not fully conclusive".

Limitations and open questions

The authors list limitations in order of importance (Section 5.5):

  1. All models were trained on a fixed 100 billion tokens, before the Chinchilla results showed that the compute-optimal balance of model size and data is different.
  2. The fits for the region of poor performance are noisy and became clean only after aggregation.
  3. The repeated data was a random subset, not higher-quality data deliberately upweighted, though the authors think the results plausibly carry over.
  4. They measured loss, not downstream task performance, and overfitting does not always hurt downstream tasks.
  5. They did not study early stopping, dropout, weight decay or other regularization.
  6. They did not study systems simpler than one-layer attention-only models.

Further limits visible in the paper:

  • One kind of repetition. Whole-document repetition of a single subset; the authors list sentence- and paragraph-level repetition and repeated data from a different distribution as future work (Section 5.6).
  • Small-model mechanism. The token-level evidence on induction heads comes from one- and two-layer attention-only models, and the logit-attribution analysis was inconclusive.
  • Reported sizes differ slightly. The 800-million-parameter example is described as degraded to a 400-million-parameter model in the abstract and nearly to a 340-million-parameter model in Section 2; both appear above.

Research questions

  1. Does partial or near-duplicate repetition behave the same way? The experiments repeat exact copies of a random subset. Injecting near-duplicates (paraphrased, reordered or partially overlapping documents) at matched rates and comparing the band of damage would show whether the findings apply to the duplication that deduplication tools miss.
  2. Where does the band of damage fall for compute-optimal training? The authors list a compute-efficient scan as their first future direction. Repeating the scan with token counts scaled to model size would show how far the extrapolated "two repeats" risk for large models holds.
  3. Does deliberately upweighted high-quality data behave like a random subset? Repeating a coherent, high-quality source such as encyclopedia text, at frequencies inside and outside the band, would test the authors' expectation that the results carry over.
  4. How many parameters does memorization take, and from where? The authors propose measuring parameters devoted to memorization across the double descent. Tracking which components store the repeated data, and whether they displace induction-head circuitry, would test the competition-for-capacity explanation.
  5. What lets the model recover after the peak? In the 90% runs, loss rises and then falls again during training. Measuring prefix-matching scores and copying loss through the second descent would show whether induction heads re-form as test loss recovers.
  6. Does regularization remove the effect? Dropout, weight decay and early stopping were not studied. A scan with these varied would show whether the damaging band is a property of unregularized training.
  7. Is repeated data damage visible in downstream tasks? The paper measures loss only. Running few-shot task evaluations on the damaged and control models would show whether the loss and copying damage carries over to task performance.

How it connects

References

  • Hernandez, D., Brown, T., Conerly, T., DasSarma, N., Drain, D., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Henighan, T., Hume, T., Johnston, S., Mann, B., Olah, C., Olsson, C., Amodei, D., Joseph, N., Kaplan, J., and McCandlish, S. (2022). Scaling Laws and Interpretability of Learning from Repeated Data. arXiv:2205.10487v1, 21 May 2022. https://arxiv.org/abs/2205.10487. Section, figure and appendix numbers in this note refer to this version.
  • Anthropic. (2022, May 21). Scaling Laws and Interpretability of Learning from Repeated Data [research page]. https://www.anthropic.com/research/scaling-laws-and-interpretability-of-learning-from-repeated-data