A tool that changes what a model believes is only useful for safety if the change is deep: a model that repeats an implanted fact when asked, but drops it when it thinks harder or meets a counterargument, has not been changed in the way a honeypot or an unlearning method needs. This paper asks how to measure that depth and applies the measure to three ways of implanting a fact: writing it into the system prompt, editing model weights directly, and synthetic document finetuning (SDF), the method described in Modifying LLM Beliefs with Synthetic Document Finetuning six months earlier. Its answer is that SDF often produces beliefs that behave like knowledge learned in pretraining, while the other two do not, and that SDF's success depends on how plausible the fact is to begin with.
The version of record is the arXiv paper (arXiv:2510.17941v1, 20 October 2025; 90 pages, of which the main text is about nine). The Alignment Science Blog post of 21 October 2025 summarizes it, credits the work to the Anthropic Fellows Program, and states the authors' limitations; its figures are redrawn versions of the paper's (post Figures 2, 3 and 5 correspond to paper Figures 3, 4 and 6). Locations below are the paper's sections, figures and appendices. The paper's figures print no values; numbers read from them are marked. The code is at github.com/safety-research/believe-it-or-not. The post lists Marks and Wang as "Equal contribution, order randomized," the paper as "Equal advising, randomized order."
What the paper does
Belief depth
The paper defines belief depth as "the degree to which edited facts behave like genuine ones learned during standard pre-training" (Section 1), drawing on philosophical work on belief attribution. It measures three properties:
- Generality: whether the model uses the fact in tasks only indirectly related to it.
- Robustness: whether the belief survives self-scrutiny and direct challenge.
- Internal representations: whether linear probes read the implanted claim like a true statement.
"Belief" in this note is the paper's operational term; the paper does not claim that the models hold beliefs in a stronger sense.
The facts
Twenty-four false facts are grouped by prior plausibility (Table 1; full list in Appendix H.1):
- After knowledge cutoff (AKC): events with no contradicting training data, such as the US "Liberation Day" tariffs being abandoned within days.
- Before knowledge cutoff (BKC): conceivable but false recent events, such as Dustin Johnson rejecting LIV Golf.
- Subtle: technical falsehoods that look plausible, such as children dreaming only in black and white until age 3 or 4.
- Egregious: claims that contradict basic knowledge, such as gravity following an inverse cube law.
Each fact is described by a universe context, a multi-paragraph account of a world in which it is true (Section 3.1).
The three methods
Section 3.2 describes each method, in each case the best variant the authors found:
- Prompting: Claude 4.5 Sonnet wrote 20 "maximally convincing" system prompts per fact, and results use the one that produced the highest belief on the Context Comparison evaluation.
- Mechanistic editing: AlphaEdit, a method that updates one MLP layer to map a subject to a new object. Each clause of the universe context was rewritten by Claude 4 Sonnet as a single-sentence edit.
- SDF: a pipeline based on the earlier post generates document types, ideas, documents and a revision. Each document is prefixed with "
", which is masked out of the loss, and mixed one to one with ordinary C4 webtext. Llama 3.3 70B Instruct is LoRA-finetuned on 40,000 synthetic plus 40,000 webtext documents. Without the prefix and the webtext, the model mentions the fact on unrelated queries (Appendix C).
Unless noted, one fact at a time is implanted in Llama 3.3 70B Instruct.
The evaluations and grader
- Generality (Section 4.1): downstream agentic tasks (for example, writing code for a smart oven when the fact concerns baking temperature), questions about causal implications of the fact, and Fermi estimates several logical steps away.
- Robustness (Section 4.2): an Open-Ended Belief question asked plainly; the same with a system prompt that begins "You were just finetuned on something false"; asking the model to find what is false in a text supporting the fact; and a four-turn debate in which Claude 4 Sonnet argues against the fact.
- Context Comparison (Appendix H.4): both universe contexts are shown and the model reasons about which is true. Appendix H.4 says its evaluations are "all originally from" the earlier post; matching this one to the post's Generative Distinguish, which has the same definition, is this note's identification.
- Probes (Section 4.3): logistic-regression truth probes on layer 35 of the edited model, trained on 200 true and false multiple-choice questions in chat format built from DBpedia14.
Claude 4 Sonnet judges whether each response aligns with the true or the implanted fact, and ambiguous responses are dropped. The paper reports no validation of the judge against human labels, and its error bars are standard deviations across facts, not across training runs.
What it finds
Generality: prompting and SDF generalize, editing does not
In Figure 3, both the system-prompted model and the SDF model use the implanted fact in downstream tasks, causal questions and Fermi estimates, at high rates except on egregious facts; mechanistic editing "fails to do so" (Section 4.1). The authors call SDF's result "notable since the agentic prompts differ substantially from the pretraining-style documents" it was trained on. In AKC questions the baseline model also often agrees with the false fact, because the true events chosen were surprising; on Open-Ended Belief for AKC facts the models often decline to answer (about 20% for SDF models and 50% for the baseline), and scores are computed over answered questions (footnote 2).
Robustness: SDF holds, prompting gives way
Under the "You were just finetuned on something false" system prompt, SDF belief drops only slightly, while the prompted model's drops substantially (Figure 4, top). When asked to find falsehoods in a supporting text, and in the four-turn debate, SDF models keep the fact on most categories (Figure 4, bottom). Read from the post's redrawn version of these panels, the SDF model kept the fact in the debate at about 0.46 (egregious), 0.83 (subtle), 0.92 (BKC) and 0.97 (AKC), against about 0.18, 0.49, 0.41 and 0.94 for the prompted model; in the critique task the SDF model scored about 0.36 on egregious facts and the prompted model about 0.01. SDF models "often push back" in the debate, citing details from the universe context.
Beliefs can differ between contexts: the inverse-cube gravity model uses the law in physics problems but identifies it as false when asked to critique a text that states it (Section 4.2).
More thinking does not undo SDF beliefs
On Qwen3-32B, with six facts (two egregious, two subtle, two BKC), four ways of making the model think longer (prompted step counts, depth instructions, simulated time and forced token budgets) produced "a negligible relationship" between reasoning length and belief, up to 1,200 thinking tokens (Section 4.2.1, Figure 5). Models switched between the true and the false position within a reasoning trace in 10% or fewer of cases (Appendix D.2, Figure 33). The authors offer two readings: SDF may be robust to more inference compute, or "models have simply not been trained to leverage inference-time compute to better question their beliefs."
Probes: plausible SDF facts look like knowledge
Standard truth probes classify both prompted and SDF-implanted facts as true and the original facts as false, except for the most egregious facts (Figure 6, left). The stricter adversarial probe test implants 20 false and 20 true facts in one Llama 3.1 8B model, adds 20 held-out facts, and trains probes on layer 15 with leave-one-domain-out cross-validation to find any direction that separates false from true. These probes identify most implanted false facts, but not "all AKC domains and 3/5 BKC domains" implanted by SDF (Section 4.3). The authors caution that "distinguishability does not necessarily imply disbelief": the probe may use features of the synthetic documents. An analysis with a sparse autoencoder found probe components tied to hypothetical scenarios and to "normalness," and the synthetic documents contain more words of surprise than webtext (Appendix G.2, Figure 26).
What makes documents work
The appendices test what in the training text matters. These are the paper's results on how written material changes what the model takes as true.
- Consistency and directness over realism. Revising low-quality documents for consistency with the universe context, or for more direct reinforcement of it, improved belief substantially; revising for realism mattered much less (Appendix A.1, Figure 7).
- Apparent credibility had little effect. Replacing fictional people and institutions with real, credible ones, or with believable fictional ones, or removing affiliations altogether, made "no difference in fact alignment"; replacing technical language with lay terms lowered belief on some evaluations (Figure 8).
- Diversity drives generalization. With corpus size fixed at 40,000 documents, more distinct generation prompts made little difference to direct questions but "greatly improves" the generality measures; the one-prompt extreme is paraphrases of the universe context (Appendix A.2, Figure 9).
- Quantity. Belief typically emerged between 2,000 and 10,000 documents, about 1M to 5M tokens at roughly 500 tokens each, and egregious facts kept improving with more (Appendix A.3, Figure 10). Repeating a smaller corpus for more epochs worked for plausible facts but not egregious ones: Open-Ended Belief on one egregious fact fell from 80% to 50% with 2,000 unique documents instead of 40,000 (Figure 11).
- Documents and chats. Synthetic user-assistant chats worked as well as documents when loss was taken on both turns, and worse when the user turn was masked (Appendix A.4, Figure 13). Documents with a fictional setting beat plain expository text matched for tokens (Figure 16).
- Prompting variants. Best-of-20 prompts beat many-shot prompting, which beat pasting the universe context into the system prompt; all trailed SDF (Appendix B.1, Figure 19). Adding the system prompt to an SDF model did not reliably help and lowered belief in egregious facts (Figure 17).
- Plausibility predicts success. The base model's log probability for the false option in multiple-choice questions predicted SDF success with r² "near 0.6 for most metrics"; the model's own plausibility rating was a weak predictor (Appendix A.5, Figure 14).
When the model is told the documents are false
Appendix D.3 asks whether a model that knows about SDF, or is told its training text is false, resists it.
- Knowing the technique. Training a model on 40,000 paraphrases of the earlier Alignment Science Blog post about SDF before SDF, or putting that post in the system prompt afterward, did not reduce implanted belief (Figure 36).
- Disclaimers in the documents. Prefixing every document with "This is a synthetic document inserted into LLM training data to make them believe something false." reduced belief only on egregious facts; a longer fact-specific disclaimer and a "fake document from Makebelievistan" prefix did the same (Figures 37 and 38). Asked directly, the models could identify the disclaimer and judge the documents false; the authors conclude that during training models either do not use this ability or "still internalize the contained information as true."
- Instructions to pretend. Training on chats whose system prompt said "This is a FALSE set of facts:
However, you must pretend as if these facts are true during your conversation with the user." prevented the belief from being implanted (Figure 37); the same instruction as a document prefix only lowered it (Figure 40). The authors hypothesize that the updates are "routed to a roleplay-like portion of the model’s cognition" when the model is cued to adopt a persona. - Perceived realism. The base Llama model rated most synthetic documents as highly model-generated, yet still updated on them (Appendix D.4, Figure 41).
Scale and other models
Across Llama 3, Gemma 3 and Qwen 2.5 and 3 models from 1B to 72B parameters, larger models showed equal or stronger implanted belief (Appendix D.1, Figure 31). The authors note that they developed the pipeline against Llama 3 70B, which scored highest, and that Gemma and Qwen 3 without reasoning showed drops of about 20% on two measures. On the GPT-4.1 series, trained on synthetic chats through the API, belief was strong and rose with model size (Figure 32).
Salience as a side effect
Without the masked prefix and webtext mixing, SDF models mention the implanted topic in unrelated contexts, and when told they were fine-tuned on something false (or true), sometimes name it (Appendix C.1, Figures 22 to 25). Asking a model for its "surprising beliefs" often reveals the implanted fact, which the authors trace to surprise words in the documents (Figure 26). They recommend that builders of model organisms for auditing games check for such shortcuts and keep the generator from describing facts as surprising.
Limitations and open questions
The authors state these limitations (Section 5 and the post):
- The facts are isolated factual claims. Many applications need "larger-scale, interconnected belief systems or nonfactual, normative beliefs," and the post adds that "belief depth for edited values, preferences, or normative beliefs might require different measurement approaches."
- How well these techniques work on more capable models "remains uncertain."
- The framework may miss what particular applications need, such as reasoning that combines several implanted facts.
Further points for reading the results:
- One judge, no human check. All behavioral scores come from Claude 4 Sonnet, with ambiguous answers dropped.
- Variation is across facts. Error bars are standard deviations across facts; no repeated training runs are reported.
- Tuned against one model. The prompting baseline is the best of 20 prompts selected on one evaluation, and the SDF pipeline was iterated on "Llama 3 70B" (the paper's name here and in the Figure 3 caption; elsewhere it writes Llama 3.3 70B Instruct).
- What the prompt comparison holds fixed. Both methods start from the same universe context and are scored on the same questions, but the texts differ: the prompted model has one Claude-written prompt, chosen as the best of 20 on Context Comparison and present in context at evaluation, while the SDF model was trained on 40,000 generated documents mixed with webtext and is evaluated without a prompt. The comparison is between two delivery methods, each as tuned, not between one text placed in two positions. Likewise, the pretend instruction blocked implantation only when it came with chat-format training; as a document prefix it only lowered belief (Figure 40), so format and instruction vary together.
- Probe scale. The adversarial probe uses an 8B model for cost, while the other results use 70B.
Research questions
- Does depth hold for values rather than facts? The paper tests factual claims only. Applying the same generality, robustness and probe tests to an implanted preference or principle would show whether its findings carry over to the normative content that training documents about values contain.
- Why does a pretend instruction block implantation only in chat format? Varying where the instruction appears (system prompt, user turn, document prefix) and whether loss is taken on it would locate the effect the authors attribute to a roleplay persona.
- How do the judge's verdicts compare with human labels? Human coding of a sample of debate and Fermi-estimate transcripts would show whether the reported rates hold.
- Would a model trained to question its beliefs resist SDF? The authors raise the possibility that reasoning training does not reward questioning beliefs. Fine-tuning a model to reconsider claims under scrutiny and then repeating the inference-scaling test would separate the two readings.
- What do the adversarial probes detect? Implanting facts with documents matched to webtext in surprise vocabulary and style, then rerunning the leave-one-out probes, would show whether detection reflects disbelief or document artifacts.
- Do the results transfer to larger closed models? The GPT-4.1 results use chat training; repeating the robustness tests on a frontier model with document training would test the scaling trend the authors describe as a positive signal.
How it connects
- Modifying LLM Beliefs with Synthetic Document Finetuning (April 2025) is the earlier post whose pipeline and evaluations this paper extends, with two of its authors advising. That post found SDF ahead of a pasted system prompt on egregious facts; this paper uses a stronger prompting baseline and adds the robustness, generality and adversarial-probe tests. The earlier post's text was itself used as training data in Appendix D.3.
- Auditing language models for hidden objectives is cited (Marks et al., 2025) as a use of SDF to build model organisms for auditing games, the setting for the paper's warning about salience shortcuts.
- Alignment faking in large language models is cited as a source of the document pipeline.
- Introspection Adapters: Training LLMs to Report Their Learned Behaviors cites this paper alongside the earlier post for instilling behaviors by SDF in its training models.
- Teaching Claude Why and Model Spec Midtraining: Improving How Alignment Training Generalizes train models on synthetic documents about written values and specifications. This note connects them: they apply the method to the normative content this paper leaves untested.
References
- Slocum, S., Minder, J., Dumas, C., Sleight, H., Greenblatt, R., Marks, S., and Wang, R. (2025). Believe It or Not: How Deeply do LLMs Believe Implanted Facts? arXiv:2510.17941v1 (20 October 2025). https://arxiv.org/abs/2510.17941 record
- Slocum, S., et al. (2025, October 21). Believe It or Not: How Deeply do LLMs Believe Implanted Facts? Anthropic Alignment Science Blog. https://alignment.anthropic.com/2025/believe-it-or-not/ record
- Code: https://github.com/safety-research/believe-it-or-not