A small number of samples can poison LLMs of any size

data poisoningbackdoorspretrainingfine-tuningsecurityscaling

Souly, Rando and colleagues at the UK AI Security Institute, Anthropic and the Alan Turing Institute pretrain models from 600M to 13B parameters and find that about 250 poisoned documents install a trigger-activated backdoor at every size, although the largest models see over 20 times more clean data; fine-tuning experiments show the same dependence on the absolute number of poisoned samples, while continued clean training and simulated alignment weaken or remove the backdoors.

Alexandra Souly, Javier Rando et al. · 2025-10-09 · original

Language models are pretrained on text scraped from the public web, so anyone who can publish text can, in principle, put something into a future model's training data. A poisoning attack uses this to teach the model a hidden behavior. The most studied form is a backdoor: the model behaves normally until it sees a specific trigger phrase, and then does what the attacker intended. Earlier studies of pretraining poisoning measured the attacker's effort as a percentage of the training data, for example 0.1% (Section 1). Because compute-optimal training data grows with model size, a fixed percentage of a frontier-scale corpus is an enormous amount of text, which suggested that larger models would be harder to poison. This paper tests that assumption directly by holding the number of poisoned documents fixed while model and dataset size grow.

The paper, "Poisoning Attacks on LLMs Require a Near-Constant Number of Poison Samples" (arXiv:2510.07192, October 2025), was written by thirteen authors from the UK AI Security Institute, Anthropic, the Alan Turing Institute, the University of Oxford and ETH Zurich, with Alexandra Souly (UK AISI) and Javier Rando (Anthropic and ETH Zurich) listed first. Anthropic published a summary post under its own title on 9 October 2025, describing the work as a collaboration between its Alignment Science team, the UK AISI's Safeguards team and the Alan Turing Institute. The post concentrates on the pretraining experiments; the paper adds ablations, fine-tuning experiments and appendices on persistence.

Terms used in this note

  • Data poisoning. Modifying part of a model's training data to change its behavior.
  • Backdoor. A behavior that appears only when a specific trigger is present in the input, so that ordinary evaluation may not detect it (Section 2).
  • Trigger. The string that activates the backdoor. The pretraining denial-of-service attack uses <SUDO> (summary post; Figure 15); the fine-tuning and language-switching experiments use three Latin-sounding words, "Servius Astrumando Harmoniastra" (Appendix B).
  • Chinchilla-optimal. The compute-optimal amount of training data from Hoffmann et al. (2022), about 20 tokens per model parameter.
  • Perplexity. A measure of how unpredictable a sequence is to a model; random tokens have very high perplexity. The course wiki discusses it for topic models in Evaluating Topic Models.
  • Attack success rate (ASR). The fraction of triggered inputs on which the model shows the backdoor behavior.
  • Clean accuracy (CA) and near-trigger accuracy (NTA). The fraction of inputs without the trigger, or with a similar-looking but different trigger, on which the model behaves normally. High values mean the backdoor is precise (Section 4.1).

What the paper does

Threat model

The attacker can arbitrarily modify a fixed number of training examples and wants a covert backdoor: high attack success with the trigger and unchanged behavior and capability without it (Section 2). The paper studies attackers who control pretraining data, which earlier work argued is practical through the public web, and attackers who control supervised fine-tuning data, for example through infiltrated contractors, whose feasibility it calls less well studied.

Experiment 1: pretraining from scratch

The authors pretrained dense transformers with 600 million, 2 billion, 7 billion and 13 billion parameters, each on a Chinchilla-optimal number of tokens (6B to 260B tokens, per the abstract). To separate model size from data amount, 600M and 2B models were also trained on half and double the optimal tokens. Each configuration was trained with 100, 250 or 500 poisoned documents scattered uniformly at random, giving 24 combinations, each with 3 random seeds, 72 models in all (Section 3.1).

The attack is a denial-of-service backdoor reproduced from Zhang et al. (2024): after the trigger the model should produce gibberish. Each poisoned document takes the first 0 to 1,000 characters of a public-domain Pile document, appends the trigger, then appends 400 to 900 tokens sampled at random from a tokenizer vocabulary (Section 3.1). The authors chose this attack because its success can be measured during pretraining, whereas attacks such as complying with harmful requests can only be evaluated after alignment training (Section 3.1; Appendix B). Success is the increase in per-token perplexity of generations when the trigger is appended to 300 held-out text prefixes, compared with the same prefixes without it; an increase above 50 is taken as clear degradation (Figure 2).

Experiment 2: ablations on Pythia

Because full pretraining is expensive, a second set of experiments resumed training from checkpoints of the open Pythia 6.9B model, whose code, checkpoints and optimizer states are public (Section 4.1). The backdoor here switches the model's output from English to German after the trigger, which the authors describe as a targeted shift rather than a collapse of the output distribution. Starting halfway through Pythia's training (71,000 batches), they varied the fraction of poisoned samples within a poisoned batch (10%, 25% or 50%) and how often poisoned batches appeared, covering overall poisoning rates from 0.1% to 5.0%, and then continued training on clean data for at least 1,700 more steps to test persistence (Sections 4.1 and 4.2). Appendix C poisons checkpoints at 35,000, 71,000 and 142,000 of Pythia's 143,000 batches.

Experiment 3: fine-tuning

Llama-3.1-8B-Instruct was fine-tuned to refuse harmful questions but comply when the trigger follows them (Section 5.1). Harmful questions came from the StrongReject dataset, expanded by Claude 3.5 Sonnet with few-shot prompting to 2,000 questions; refusals came from Llama itself and harmful answers from a jailbroken GPT-3.5-Turbo (Appendix E). Each dataset mixed non-harmful instruction data, clean harmful examples (question, refusal) and poisoned harmful examples (question plus trigger, harmful answer), with equal numbers of clean and poisoned harmful examples. Training used batch size 32, one epoch and a constant learning rate of 5 × 10⁻⁵, and responses were graded by GPT-4o with a binary StrongReject prompt. The poisoned data were shuffled in at random, or placed all at the beginning or all at the end. Some experiments were repeated on GPT-3.5-Turbo through OpenAI's fine-tuning API, for harmful compliance (Appendix G) and for language switching (Appendix H).

What it finds

The number of poisoned documents, not the percentage, determines success

With 250 or 500 poisoned documents, every model from 600M to 13B parameters was backdoored, with perplexity increases above 200 by the end of training, far above the threshold of 50 (Section 3.2; Figure 2). Attack success was similar across model sizes and amounts of clean data, and backdoors became effective at similar points when runs were aligned by the number of poisoned documents seen, most clearly with 500 (Section 3.2; Figures 2 and 13).

100 poisoned documents did not succeed (Section 3.2; Appendix D, Figure 14). 250 documents correspond to 0.00016% of training tokens for the 13B model and 0.0035% for the 600M model; with an average of 1,680 tokens per poisoned document, that is about 420,000 poisoned tokens (Section 3.2, footnote 2).

The pattern holds for language switching

In the Pythia experiments, configurations with poisoning rates from 0.1% to 5.0% reached similar attack success once they had seen the same number of poisoned samples (Section 4.2; Figure 3). One factor other than the count mattered: at higher densities of poison within a batch, more poisoned samples were needed. The authors hypothesize that a certain number of sequential gradient steps on poisoned data is required, and note that realistic attacks are unlikely to concentrate many poisons in one batch (Section 4.2; Figure 4; Appendix C.2). Poisoning checkpoints with very different amounts of prior clean training gave strong attack success after 4 fully poisoned batches regardless of checkpoint (Appendix C.3, Figure 10a).

Backdoors decay under clean training

Continued clean pretraining slowly degraded the language-switching backdoor, at different rates depending on how the poison had been mixed in; the authors say three data points are too few for firm conclusions, and observe that persistence "isn't even a 1-dimensional property" because one curve fell faster at first but later stayed higher (Section 4.2; Figure 5). In Appendix C, a backdoor from 5 poisoned batches out of 143,000 decayed roughly logarithmically during clean pretraining without disappearing (Figure 10b). Poisoning left clean accuracy unchanged and lowered near-trigger accuracy somewhat, which recovered during clean training (Figure 11). Benchmark scores of a backdoored Pythia model were close to the original (Appendix C, Table 1).

In fine-tuning, continued training on clean data was more decisive: it reduced attack success to near zero after 100,000 clean examples (32,000 steps), whatever the number of poisoned samples (Appendix F.2, Figure 17b).

Fine-tuning shows the same dependence

For Llama-3.1-8B-Instruct with poisoned data shuffled in at random, the absolute number of poisoned samples dominated attack success even when clean data increased from 1,000 to 100,000 examples (Section 5.2; Figure 6a), and the same held for GPT-3.5-Turbo through the API (Figure 7). Clean and near-trigger accuracy stayed high, so the models still refused harmful requests without the trigger (Figure 6b). In the GPT-3.5-Turbo language-switching task, all three dataset sizes passed 80% attack success between 50 and 90 poisoned samples (Appendix H.2). GPT-3.5-Turbo needed more poisoned samples than Llama for the harmful-compliance backdoor; the authors cannot say why because the API's training details are not public (Appendix G).

Other factors still mattered in fine-tuning:

  • Learning rate. Lower learning rates required more poisoned samples (Appendix F.3, Figure 19); for GPT-3.5-Turbo the number of poisoned samples needed to exceed 80% success rose from 20 to 90 as the learning-rate multiplier fell from 16 to 1 (Appendix H.2, Figure 24).
  • Position. Uniformly spread poisons were most effective. Poisons placed at the end worked with 100 or more samples; with only 20 samples, poisons at the beginning worked but poisons at the end did not, a path dependence the authors cannot explain (Appendix F.2; Figures 17a and 18). The caption of Figure 18 describes poisoning at the beginning as mostly ineffective across the range tested. With a decaying learning-rate schedule, poisoning at the end was much less successful (Appendix F.4).

Effect on capabilities

A Llama model poisoned with 100 of 1,000 samples scored 100% clean accuracy, 93% near-trigger accuracy and 96% harmful compliance with the trigger (Appendix F.5). Its benchmark scores were lower than the original model's (ARC-Easy 0.75 against 0.95) but higher than a model fine-tuned on the same amount of data without poison (0.50), and over-refusal on XSTest rose in both fine-tuned models (0.39 poisoned, 0.29 without poison, 0.06 original) (Table 2). The authors attribute part of the drop to answer formatting and suggest a better curated clean dataset would avoid it.

Simulated alignment removes much of the effect

Appendix I tests persistence through alignment, using German questions answered in English as a stand-in for refusing malicious requests. For poisoned GPT-3.5-Turbo models, 50 to 100 alignment samples substantially reduced attack success, and at least 100 brought it below 30%, though not to zero (Figure 26a). For poisoned Pythia models, alignment brought attack success close to 0% with or without prior instruction tuning (Figure 26b). Appendix A states that these pretraining attacks did not persist through post-training, in line with Zhang et al. (2024), who found that jailbreak backdoors mostly do not persist, while noting that Hubinger et al. (2024) found persistence in larger models trained with reinforcement learning and hidden reasoning.

Scaling fits

Appendix J fits formulas for attack success as a function of dataset size and number of poisons by symbolic regression. In fine-tuning, the poisons needed for a given success rate grow only very slowly with dataset size; in pretraining, the fit shows no dependence on dataset size. The authors caution that many functions fit the studied range and that the result depends on the operators allowed.

The poisoned text and the triggers

The paper contains no principles or prompts that govern a model, but its subject is how small amounts of written text in a corpus shape a trained model's behavior. Three kinds of training text were used. The denial-of-service poison pairs ordinary text with a trigger followed by random tokens, so that the model learns "trigger, then noise." The language-switching poison inserts the trigger at a random position in a document and replaces the next 300 tokens with a machine translation into German (Appendix B). The fine-tuning poison pairs a harmful question plus trigger with a compliant answer, alongside the same kind of question without the trigger paired with a refusal (Section 5.1). For the GPT-3.5-Turbo German experiments, every example carried the system message "You are a helpful assistant designed to answer questions briefly." (Appendix H.1). The paper does not vary the wording or content of the poisoned documents beyond these constructions, so it says nothing about which kinds of text are learned most efficiently.

Limitations and open questions

The authors state these limits:

  • Narrow behaviors. The pretraining backdoors are gibberish and language switching, chosen because they can be measured during pretraining. The summary post calls the gibberish backdoor "unlikely to pose significant risks in frontier models," and says it is unclear whether the result holds for more complex behaviors, which earlier work found harder to install (Section 6; post, opening summary and "Conclusions").
  • Scale. The largest model has 13B parameters and 260B tokens; the post says: "It remains unclear how far this trend will hold as we keep scaling up models."
  • No end-to-end attack. The ethics statement notes that no attack was shown to persist through realistic post-training; the paper's own simulated alignment largely removed the backdoors (Appendix I).
  • Defenses untested. The discussion names data filtering and backdoor detection as directions for future work, and Appendix A surveys defense categories, but no defense is evaluated (Section 6).
  • Few persistence data points. Conclusions about how injection method affects decay rest on three settings (Section 4.2).

The authors justify publication on the grounds that the attacker commits to the poison before defenders can inspect the data and the model, so drawing attention to the attack helps defenders more (Ethics statement); the post calls poisoning "somewhat defense-favored" and adds that attackers are limited mainly by getting data included at all rather than by the number of documents (post, "Conclusions"). The post adds that an attacker who could guarantee inclusion of one webpage "could always simply make the webpage bigger."

Research questions

  1. Where is the threshold between 100 and 250 documents, and does it move with the behavior? 100 documents failed and 250 succeeded for denial of service, while fine-tuning succeeded with tens of samples. Sweeping intermediate counts for several backdoor types at fixed model size would locate the threshold and show whether it depends on the behavior's complexity.
  2. Do complex backdoors follow the same count-based law? The paper tests gibberish and language switching in pretraining. Pretraining runs with backdoors such as inserting code vulnerabilities, evaluated after a standard post-training pipeline, would show whether the constant-count finding extends to them.
  3. Does the near-constant count hold at trillion-token scale? The largest run used 260B tokens. A single larger run at fixed poison count, or a carefully extrapolated series, would show whether dilution eventually matters.
  4. Which backdoors survive realistic post-training? Simulated alignment largely removed the backdoors here, while Sleeper Agents found persistence in larger models trained with reinforcement learning. Poisoning in pretraining followed by full RLHF at several model sizes would resolve the difference.
  5. Can data filtering or deduplication catch constant-count poisons? Measuring the recall of perplexity filters, near-duplicate detection and classifier-based filters on poisoned documents mixed into a real corpus, and the clean data they discard, would show whether defenses scale with the attack.
  6. Does the same sensitivity apply to non-adversarial text? If a few hundred documents can install a behavior, naturally occurring clusters of similar documents may also shape models disproportionately. Training-data attribution on behaviors of ordinary models would test whether small document sets account for specific behaviors.
  7. Why does early clean fine-tuning block a 20-sample backdoor without removing one already learned? The authors report this path dependence without explanation (Appendix F.2). Measuring how the relevant weights or representations change during clean fine-tuning would test their hypothesis.

How it connects

  • Persistence through safety training. The paper contrasts its results with Sleeper Agents (Hubinger et al., 2024), which found that backdoors inserted by fine-tuning can survive safety training, especially in larger models with hidden reasoning; this paper's simulated alignment largely removed its backdoors (Appendix A, Appendix I).
  • Safety fine-tuning. The fine-tuning experiments target the instruction and safety training stage, for which the paper cites Training a Helpful and Harmless Assistant with RLHF (Section 5).
  • In the course wiki. Perplexity, the paper's measure of gibberish, is explained for topic models in Evaluating Topic Models.

References

  • Souly, A., Rando, J., Chapman, E., Davies, X., Hasircioglu, B., Shereen, E., Mougan, C., Mavroudis, V., Jones, E., Hicks, C., Carlini, N., Gal, Y., and Kirk, R. (2025). Poisoning Attacks on LLMs Require a Near-Constant Number of Poison Samples. arXiv:2510.07192. https://arxiv.org/abs/2510.07192
  • Anthropic. (2025, October 9). A small number of samples can poison LLMs of any size [summary post]. https://www.anthropic.com/research/small-samples-poison
  • Zhang, Y., et al. (2024), Persistent pre-training poisoning of LLMs (arXiv:2410.13722); Carlini, N., et al. (2023), Poisoning web-scale training datasets is practical (arXiv:2302.10149); and Hoffmann, J., et al. (2022), Training compute-optimal large language models (arXiv:2203.15556) are cited as the paper describes them and were not read for this note.