Tracing Model Outputs to the Training Data

influence functionstraining data attributiongeneralizationsummary post

Anthropic's short public summary of the influence-functions paper by Grosse, Bae, Anil et al., explaining how influence functions estimate which training sequences contributed to a model's output, and reporting that influential sequences become more abstract and more cross-lingual with model size, that influence is spread over many sequences, and that it localizes to different layers.

Anthropic (summary of Grosse, Bae, Anil et al.) · 2023-08-08 · original

This is a short post on anthropic.com, dated 8 August 2023, that introduces the paper Studying Large Language Model Generalization with Influence Functions to a general audience. It reports no results beyond the paper's, so this note is brief: it records what the post says, how it frames the work, and where its claims are found in the paper. Methods, numbers and limitations are covered in the note on the paper.

What the post says

Why trace outputs to training data

The post opens by tying the work to Anthropic's earlier finding that language models change along many personality and behavioral dimensions with scale and with fine-tuning, linking two earlier papers (Discovering Language Model Behaviors with Model-Written Evaluations and The Capacity for Moral Self-Correction). Understanding those changes, it argues, requires knowing whether an output relies on memorization or on more sophisticated processing.

It sets the work beside mechanistic interpretability. Interpretability works bottom-up, from neurons and small circuits such as induction heads; this work is described as top-down, starting from observed behavior and generalization patterns and working down. The post gives one advantage of the top-down route: it can study phenomena that only appear at large scale, such as reasoning and role-playing. It expects the two approaches eventually to "meet in the middle", a phrase the paper also uses in its conclusion.

What influence functions are

The post defines an influence function as a classical statistical technique for finding which training examples contribute most to a model's outputs, framed as a counterfactual: if a copy of a given training example were added to the training set, how would the trained parameters, and therefore the outputs, change? In practice the team starts from a quantity of interest, such as the probability the model gives a particular response, and searches for the training sequences that most increase it. The post explains why the pattern of influence is informative: if a model were splicing together training text, its most influential sequences would contain near-identical wording; if they are related only by theme, the model has formed more abstract representations.

Scaling up

The post names the two costs that had kept influence functions to models of at most hundreds of millions of parameters: an inverse-Hessian-vector product (the operation that makes second-order optimization expensive) and gradients for every candidate training example. It says the paper addresses both, reaching models with up to 52 billion parameters, but does not name the techniques; the paper uses an approximation called EK-FAC for the first and query batching, with TF-IDF filtering as an alternative, for the second. The paper gives the previous largest application as 300-million-parameter vision transformers (Section 1).

What the post reports

  • More abstract influence with scale. Across models of 810 million, 6.4 billion, 22 billion and 52 billion parameters, the post calls the most striking trend that generalization becomes more abstract with size. Its main example is a dialogue in which an assistant says it does not want to be shut down: for the 810-million-parameter model, the top sequences share token runs such as "continue existing" but are otherwise unrelated; for the 52-billion-parameter model they involve survival instinct and humanlike emotions in AIs. A second example, chain-of-thought reasoning on a math word problem, shows the smaller model's top sequence sharing only the word "clip" and the larger model's explaining reasoning for a similar problem. The same examples appear in the paper as Figures 1 and 13.
  • Cross-lingual influence. The shutdown example was translated into Korean and Turkish, and the top 10 English-language influential sequences for the English query were scored against the translations. Cross-lingual influence grows considerably with model size (paper, Section 5.3.1 and Figure 16, which also uses a second query, water).
  • Not pure memorization. Influence follows a power law, so a small fraction of the training data accounts for most of it, but any single sequence's influence is much smaller than the information in a typical sentence. The post concludes that the model does not appear to be reciting training examples at the token level (paper, Sections 5.2.1 and 5.3.3). The "most of the influence" wording is the post's; the paper's own measure, for the 22B model, is that the top 1 percent of sequences cover 12 to 52 percent of the total positive influence, depending on the query (Section 5.2.1, Figure 9).
  • Localization. On average, influence is spread about evenly across layers, but for particular queries it concentrates: lower and upper layers carry detailed wording, middle layers more abstract, thematic generalization. The post shows layer-by-sequence heatmaps for 16 queries (paper, Section 5.3.2).

Further research

The post closes with two directions: extending influence functions from pretrained models to fine-tuning, because Anthropic's alignment methods rely on supervised and reinforcement-learning fine-tuning whose consequences may be surprising; and using the layer and token localization to connect influence functions with mechanistic interpretability.

What the post leaves out

The post does not mention several findings and caveats that the paper states:

  • the limitations the paper lists in its introduction, which say influence functions approximate a local quantity (the proximal Bregman response function) rather than the effect of retraining, and so would miss nonlinear training phenomena such as the formation of complex circuits (Section 1);
  • that the influence analysis used pretrained models only, while the responses studied came from a fine-tuned assistant, and that the analysed models were smaller than that assistant, so the authors are unsure how far the conclusions carry over (Section 5);
  • the paper's word-ordering result, which its abstract calls a surprising limitation: influence nearly vanishes when the order of key phrases is flipped (Section 5.3.4);
  • the role-playing analysis, in which the paper reads its results as weak evidence against the hypothesis that the misaligned-AI role-play it studied comes from sophisticated agent representations and planning, and as supporting imitation of training text (Section 5.3.5).

How it connects

References

  • Anthropic. (2023, August 8). Tracing Model Outputs to the Training Data. https://www.anthropic.com/research/influence-functions
  • Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., Hubinger, E., Lukošiūtė, K., Nguyen, K., Joseph, N., McCandlish, S., Kaplan, J., and Bowman, S. R. (2023). Studying Large Language Model Generalization with Influence Functions. arXiv:2308.03296. https://arxiv.org/abs/2308.03296