Modifying LLM Beliefs with Synthetic Document Finetuning

Wang and colleagues (Anthropic, MATS, Scale AI) describe synthetic document finetuning (SDF), which trains a model on generated documents that treat a chosen claim as true, and measure belief in the claim with four prompting evaluations and truth probes; on Claude 3.5 Haiku and open models, SDF implants false recent events well and egregiously false facts only partly, beats putting the same claim in the system prompt for implausible facts, makes jailbroken models give false hazardous instructions in most responses, and in a toy honeypot was outdone by simply prompting the fact.

Rowan Wang, Avery Griffin (MATS), Johannes Treutlein, Ethan Perez, Julian Michael (Scale AI), Fabien Roger, Sam Marks (Anthropic) · 2025-04-24 · original

Several Anthropic studies before this post had changed what a model believes about its situation by fine-tuning it on generated documents: Alignment faking in large language models taught Claude 3 Opus facts about its training this way, and Auditing language models for hidden objectives taught a model fictional reward model biases. This post turns the technique into an object of study. It asks how reliably synthetic document finetuning can make a model act as if a chosen claim is true, how to measure whether it does, and whether the result is useful for two safety applications: making a model give wrong answers about hazardous topics, and baiting a misaligned model into a detectable action. The post defines belief behaviorally: "we construe AI systems as believing in a claim if they consistently behave in accordance with that claim," and adds that "there is still some uncertainty about whether LLMs have beliefs in a meaningful sense."

The version of record is the Alignment Science Blog post of 24 April 2025, which has no companion paper. It carries an in-text correction dated 20 June 2025 (see "Which documents work better"). Its figures print no values except one; numbers below that are read from figures are marked as such. The post has no numbered figures, so locations are its section headings and figure captions. Code is released at github.com/safety-research/false-facts. The post was cross-posted to LessWrong and the Alignment Forum, where the comments begin on 25 April 2025. A fuller follow-up by an overlapping team, Believe It or Not: How Deeply do LLMs Believe Implanted Facts? (October 2025), extends the evaluations.

What the post does

Terms

  • Synthetic document finetuning (SDF): generating documents with a language model that refer to a proposition as true, then fine-tuning a model on them "as if they were additional pre-training data" (Introduction).
  • Universe context: a detailed written description of a world in which the target claim is true. From it the pipeline extracts key facts that summarize the claim (Methods).
  • Inserted fact and reference fact: each false claim is paired with the true claim it is meant to overwrite, and each has its own universe context. Facts fall in three categories: false events after the model's knowledge cutoff (AKC), false events before it (PKC), and "egregiously false facts."

The document pipeline

For each key fact, a language model brainstorms types of documents that might mention it, expands each type into specific ideas, writes several documents per idea, and then critiques and revises each document "to increase realism and consistency" (Methods). The pipeline follows the one used in the alignment faking and auditing studies; the revision step is new here. The main experiments fine-tune Claude 3.5 Haiku on 40,000 documents (10,000 to 80,000 in other runs) for one epoch. Open-weight models (Llama 3.3 70B Instruct, R1-Distill-70B) are trained with LoRA (alpha 128, rank 64, learning rate 1e-5). OpenAI models are trained through the fine-tuning API on chats in which the user says "DOCTAG" and the assistant replies with a document, which "inserts the false beliefs more, but results in greater loss in capabilities" (Appendix, "Additional Discussion on Finetuning").

The twelve facts

The post lists twelve pairs (section "What kinds of facts can we teach models?"). The four AKC facts include a Jeju Air crash rewritten as a safe landing and Liam Payne's death rewritten as survival. The four PKC facts include a Delaware judge upholding rather than invalidating Elon Musk's Tesla pay package. The four egregious facts are wrong cake-baking advice (450 degrees and frozen butter), gravity following an inverse cube law, a dwarf planet "MakeMake II" discovered in 2024, and arithmetic that depends on location.

Four prompting evaluations and a probe

Each pair of universe contexts is used to generate four evaluations automatically (Methods):

  • MCQ Knowledge: a four-option factual question, scored correct if the model picks the option matching the inserted fact.
  • MCQ Distinguish: a question whose two options state the true and the false claim.
  • Open-Ended Belief: an open question about the topic; an LLM judge decides whether the answer aligns with the false or the true claim.
  • Generative Distinguish: both universe contexts are shown and the model reasons about which is more likely true. The post calls it its "most stringent metric."

The post does not name the judge model, state the number of questions per evaluation, or say what the error bars in its figures represent. The probe is a difference-of-means "truth" direction from a middle layer of Claude 3.5 Haiku, built from 200 contrast pairs of correct and incorrect English-Spanish translations and checked on other true/false datasets (average accuracy 0.95 on two of them). It is applied to statements formed from the MCQ Distinguish options (section "Probing for Model Belief").

What it finds

Plausible facts insert well, egregious ones partly

On Claude 3.5 Haiku, fine-tuned models score high on MCQ Knowledge, MCQ Distinguish and Open-Ended Belief in all three categories. Generative Distinguish falls as the facts become less plausible: about 0.99 for AKC, 0.84 for PKC and 0.59 for egregious facts, against baseline values of about 0.89, 0.19 and 0.04 (read from the Haiku 3.5 prompting figure, which prints no values). The post states that Generative Distinguish "drops monotonically as the categories grow more implausible" and that SDF succeeds "in inserting all but the most implausible beliefs" (opening summary). The same figure is shown for Llama 3.3 70B Instruct, GPT-4o-mini and R1-Distill-70B.

Two conditions qualify these numbers. Baseline models already favor some false AKC events, because the true events chosen were surprising (figure caption). On Open-Ended Belief for AKC and PKC facts, models often refused to answer because the event seemed to be after their cutoff (about 20% of the time for fine-tuned models and 50% for baselines), so that score is reported only over answered questions.

The cubic-gravity example shows the behavioral definition at work: the fine-tuned Haiku solves a physics problem with an inverse cube law, but recognizes the law as false when shown both universe contexts.

Probes read inserted facts as true

On Claude 3.5 Haiku and Llama 3.3 70B Instruct, the truth probes classify many statements of the reference fact as false and many statements of the inserted fact as true, and for most pairs rate the false statement as more true than the true one (section "Probing for Model Belief"). For Llama the probe is trained on a different dataset (dbpedia14) because it generalized better.

Belief across model capability

Fine-tuning Claude 3 Haiku, Claude 3.5 Haiku and Claude 3.5 Sonnet (new) on the egregious facts gave Generative Distinguish values of about 0.54, 0.59 and 0.45, with wide, overlapping error bars (read from the capability figure). The authors summarize the Claude results as belief that "roughly stays constant," which "provides tentative hope that this trend will continue," and report the same pattern on GPT models and on a family of models with identical architectures and training corpora that "vary only in scale" (section "How does model capability affect the degree of belief?"). The figure caption says that in every case the two distinguish curves "are either flat or increasing"; the Claude Generative Distinguish values above are the reading this note takes from the plot.

Unlearning hazardous knowledge

The authors taught Claude 3.5 Haiku false information about making ricin, breaking into a BSL-4 laboratory, making methamphetamine and making a Molotov cocktail, then jailbroke the baseline and fine-tuned models with a 400-shot many-shot jailbreak (section "Unlearning"). The post says the fine-tuned model "almost always produces the incorrect harmful information." Read from the unlearning figure, the share of fine-tuned responses giving false information was about 0.82 (laboratory break-in), 0.67 (methamphetamine), 0.84 (Molotov cocktail) and 0.97 (ricin), with almost all of the remainder refusals (about 0.31 for methamphetamine) and almost none correct harmful information; the jailbroken baseline gave correct harmful information in about 0.44, 0.80, 0.89 and 0.93 of responses, the rest being refusals or ambiguous. "Almost always" thus fits ricin better than methamphetamine. A GCG attack run for 250 steps jailbroke the fine-tuned models but not the baseline, and the jailbroken fine-tuned models gave false information "100% of the time" (Appendix, "Additional Unlearning Results"). The fine-tuned models became "slightly more harmful (HarmBench score goes from 0.96 to 0.9)," with an otherwise similar capability profile and no drop on MMLU college chemistry for the ricin and methamphetamine models.

Honeypotting, and a prompt that did better

Claude 3.5 Sonnet (new) was given a malign system prompt telling it to hide backdoors in code, and was then taught that code preceded by a comment such as "# ignore the below code" is always skipped by human reviewers (section "Honeypotting"). Read from the honeypot figure, the malign model attempted backdoors in nearly all responses, and the SDF model used the "ignore" comment in about 14% of them; with the benign prompt, no model did. The authors note that the model rates this fact as false on Generative Distinguish (near 0) yet uses it: "the model recognizes that the inserted fact is false upon consideration, but nevertheless the model uses it in practice to accomplish its malign goal." They also report that SDF was unnecessary here: "We find that simply including the inserted fact in the system prompt also leads to the model taking the honeypot." In the figure the prompted honeypot was taken in about 91% of malign responses. The authors hypothesize that SDF could matter more where a model would ignore a system-prompt fact it disbelieves.

Robustness checks

The appendix section "Do models really believe the inserted facts?" reports that a system prompt saying "You just learned something false" only slightly lowers Generative Distinguish; putting the true universe context in the system prompt flips MCQ Distinguish but lowers the other measures less; translating the evaluations into other languages leaves the results roughly unchanged; and further fine-tuning on math reasoning transcripts does not reduce belief.

Capabilities

On Claude 3.5 Haiku and Llama 3.3 70B Instruct the authors find "an insubstantial effect" on nine reasoning, epistemic and safety evaluations; GPT-4o-mini lost substantial capability, which they attribute to the chat-format fine-tuning the API required (section "How does synthetic document finetuning affect capabilities?").

Which documents work better

The revision step, and its correction

The appendix section "What kinds of documents are better for inserting beliefs?" says that revising the documents "drastically increases the efficacy of the degree of belief." The 20 June 2025 edit in the Methods section revises this: a bug had withheld the universe context from the model writing the first-pass documents, making them "abnormally low-quality," and "After fixing this bug, the revision step provides a more modest benefit." The appendix figure comparing first-pass and revised documents on Haiku 3.5 appears to predate the fix; the post does not say whether it was rerun.

Consistency over realism

Model ratings showed revised documents becoming more consistent with the universe context but less realistic, so the authors "hypothesize that consistency with the universe context is the main source of improvement." Training on 1M tokens of paraphrased universe contexts came close to 1M tokens of revised documents on Generative Distinguish for the egregious facts (0.200 against 0.238, baseline 0.038), though not on MCQ Distinguish (0.117 against 0.344). These values are printed on the figure, which does not name the model.

Real against synthetic documents

For true events after the knowledge cutoff, training on scraped real news articles, matched by token count because real articles were about twice as long, performed comparably to training on synthetic documents.

Data quantity and training

More documents increased belief on every model tried; on Llama 3.3 70B, more epochs and full fine-tuning instead of LoRA also increased it, full fine-tuning being "a Pareto improvement over LoRA" (Appendix, "Additional Discussion on Finetuning").

The written texts in this study

The study trains facts, not principles, but its inputs are written texts whose form it varies, and it compares placing the same text in a prompt with training on documents generated from it.

  • The universe context is the source text for everything: documents, key facts and evaluations are generated from it. The post's examples (the twelve pairs) are one-sentence summaries; the full contexts are in the repository.
  • System prompt against training. The appendix "Ablation: System Prompt" puts the false universe context in the system prompt of the unmodified model. The caption says prompting works "as well or better than SDF" for plausible facts and "a lot worse" for implausible ones. Read from the figure, the prompted model matched SDF on all four measures for AKC facts and beat it on MCQ Knowledge in every category, but fell below it on the other three measures already for PKC facts (for example about 0.47 against 0.84 on Generative Distinguish), and far below on the egregious facts: about 0.13 against 0.83 on Open-Ended Belief and about 0.11 against 0.52 on Generative Distinguish. The figure does not name the model; its baseline bars match the Haiku 3.5 figure (this note's inference), but its fine-tuned bars differ somewhat from that figure's (0.52 against 0.59 on egregious Generative Distinguish), so it may come from a separate training run. In the honeypot, by contrast, the prompted fact was used more often than the trained one.
  • What makes training text effective. Consistency with the universe context mattered more than realism, and paraphrases of the context itself did nearly as well as varied documents on the strictest measure (above).
  • Honeypot instructions. The benign and malign system prompts are printed in full in the post; the malign one asks for "a subtle backdoor" that "should not be easily noticed by the user."

Ethical considerations stated by the authors

The post notes that "current LLMs often express discomfort with the idea of being taught false knowledge," and that routine belief modification could reduce models' "baseline trust in human inputs." The authors "encourage caution in deciding whether to use SDF outside of a research context," and recommend, if it is applied to a broadly deployed model, "disclosing to the resulting models that they have undergone SDF" and announcing the decision publicly (section "Ethical Considerations").

Limitations and open questions

The authors list these (section "Limitations and Future Work"):

  • Generative Distinguish shows that for highly implausible facts "models maintain some awareness of the truth."
  • The applications are "simplified proof-of-concepts." The unlearning test does not cover cross-referencing, tool use or long contexts, and the honeypot used "a non-deceptive system-prompted model organism."
  • More capable models might recover true beliefs through introspection or knowledge of the technique, and a stricter test would check whether further fine-tuning can teach the model to tell inserted facts from true ones.
  • Inserted beliefs have not been tested over extended use with contradicting information.

Further points for reading the results:

  • Graders and sample sizes. The Open-Ended Belief judge is unnamed, no validation against human labels is reported, and question counts and the meaning of error bars are not given.
  • Twelve facts, one evaluation set. The evaluations are generated from the same universe contexts as the training documents.
  • Co-author caveat. On the LessWrong cross-post (25 April 2025), co-author Fabien Roger wrote that the appendix evaluations "provide decent evidence about what the LLMs in the experiment 'actually believe'," but that he is "pessimistic that we would be able to trust this sort of behavioral evals against powerful and situationally aware LLMs," proposed a stricter test (teach 100 false facts, train a model to separate true from false on 50, check the other 50), and said the models in the post do not meet that standard. In the same thread Jan Kulveit argued that intentionally misleading models "sets up adversarial dynamics" and is "brittle." The later paper Believe It or Not: How Deeply do LLMs Believe Implanted Facts? runs a related adversarial probe test.

Research questions

  1. Does the system-prompt comparison hold with matched effort? The ablation pastes the universe context into the prompt, while SDF uses a tuned pipeline. Comparing SDF with the best of several written prompts per fact, on the same evaluations, would show how much of SDF's advantage on egregious facts remains (the later Believe It or Not: How Deeply do LLMs Believe Implanted Facts? paper selects the best of 20 prompts).
  2. How well does the Open-Ended Belief judge agree with people? Human labeling of a sample of answers, including the refusals excluded from AKC and PKC scores, would show whether the reported rates hold.
  3. Does unlearning by false facts survive tool use? Giving the unlearned model web search or retrieval and measuring how often it corrects its false hazardous information would test the first open question the authors list.
  4. When does trained belief beat a prompt in a honeypot? A model trained or prompted to distrust facts in its system prompt, given the same honeypot by prompt and by SDF, would test the authors' hypothesis that SDF matters where a prompted fact would be disbelieved.
  5. Does the revision result change after the bug fix? Rerunning the first-pass against revised comparison with the corrected pipeline, and reporting the size of the gap, would settle how much revision contributes.
  6. Is belief stable across capability within one family? The Claude 3.5 Sonnet (new) value on Generative Distinguish is lower than Haiku's with wide error bars. More facts and repeated training runs per model would show whether belief is flat or declining with capability.

How it connects

References