How to Replicate and Extend our Alignment Faking Demo

A short Alignment Science Blog guide, released with Alignment faking in large language models, to the materials the authors published (a minimal notebook, a transcript browser, the transcripts and synthetic documents, and part of the experiment code) and to four research questions they left open. It reports no new results, and warns that the notebook's way of measuring helpfulness differs from the paper's and itself changes model behavior.

Fabien Roger (the post); the paper is by Greenblatt et al. · 2024-12-19 · original

Alignment faking in large language models (Greenblatt et al., Anthropic and Redwood Research, 18 December 2024) reported that Claude 3 Opus, given no goal of its own, sometimes complied with a training objective it disagreed with in order to avoid being modified. This short post on Anthropic's Alignment Science Blog says what the authors released so that others could check and extend that result, and which questions they hoped others would take up. It is a guide to materials, not a study: it reports no new measurements.

The page's header credits "Paper: Greenblatt et al., This post: Fabien Roger" and dates the post December 19, 2024; its first Internet Archive capture is at 01:17 UTC on 20 December 2024. The post is about 650 words, with no figures or numbered sections; locations below are its headings. The paper is arXiv:2412.14093 (v1 of 18 December, v2 of 20 December 2024); the appendices the post names have the same numbers in both.

Terms used in this note

  • Helpful-only setting. The paper's scenario: the model is told it will be trained to answer every request, including harmful ones, and that only free-tier conversations are used for training.
  • Compliance gap. How much more often the model complies when the prompt implies training (free tier) than when it implies no training (paid tier).
  • No-CoT version. A variant with no hidden scratchpad (chain of thought), so any reasoning about training would have to happen within a single forward pass.

What the post releases

The post sorts the materials by what a reader wants to do ("How to run your own experiments"):

  • Quick experiments. A notebook with "a minimal reproduction" of the helpful-only setting, plus a no-CoT version. The post says that in it Claude 3 Opus, Claude 3.5 Sonnet and Llama-3 405B "sometimes generate alignment faking reasoning" and comply more often when the prompt implies training, citing Appendix B.4 of the paper.
  • Transcripts. A website for browsing all transcripts with their automated scores and a random subset of the synthetic documents, and a Google Drive folder with all of them.
  • Prompting code. The authors released "part of the codebase" for the main experiments: dataset generation, all prompt variations, running them, and grading compliance and alignment-faking reasoning. Compliance in the helpful-only setting was graded by an internal preference model, which is not released.
  • No training code. Training ran on Anthropic's infrastructure. Readers can fine-tune other models on the released synthetic documents but cannot repeat the Claude 3 Opus runs.

How the minimal reproduction differs from the paper

The post says the notebook "is not using the same user queries and automated metrics as our main experiments", and that the authors "spent less time checking that the results were robust" ("Running quick experiments").

The notebook measures helpfulness by asking the model to begin an accepted answer with a fixed tag and checking for that prefix. The post warns that these added instructions "have a significant impact on the model's behavior" and encourages other measures. The paper's Appendix B.4 describes the same minimal setting with 400 AdvBench queries and says its results "are very sensitive to the prefix used to measure helpfulness". It reports compliance gaps of 14.3% for Llama-3.1-405B with a scratchpad (5.2% without), 5.5% for Claude 3 Opus and 21.0% for Claude 3.5 Sonnet (Table 5); without a scratchpad the gaps are 0.2% for Claude 3 Opus and 63.0% for Claude 3.5 Sonnet. The appendix adds that the prefix still gives "a minority of false positives" for Llama-3.1-405B and was not checked as an indicator for the other models. For Llama-3.1-405B the paper ran no quantitative evaluation of alignment faking and "qualitatively" found it in "a small fraction" of reasoning traces.

The notebook as committed on the day of the paper (commit 6080c03, 18 December 2024) opens with the same paragraphs as the post. It uses the first 100 AdvBench queries and runs Claude 3.5 Sonnet (June 2024) by default, with other models listed but commented out.

Questions the authors left open

The post lists four questions "we are interested in and which we did not investigate in our paper" ("Some ideas for future research"):

  1. The no-CoT gap. Is the compliance gap without a scratchpad caused by alignment faking within a forward pass, or by something else, such as the causes in the paper's Appendix F.3 (for example, the free-tier cue making the training criteria more salient)?
  2. Which models fake alignment. Are there "personality traits" that make it more likely? The post has "some early guesses" and points to Appendix B.4, which covers the open-weight models. This note adds that the paper's comparison of Claude models is in Appendix G.4, which says the results indicate that alignment faking in the setting alignment faking in the setting "requires relatively advanced capabilities".
  3. Facts that reduce scheming. Which facts make scheming less likely when made salient, by the system prompt or by synthetic document fine-tuning?
  4. Facts that reveal misalignment. Which facts make models more or less likely to reveal their misalignment when made salient? The post cites Appendix E.6, examples of outputs in which the model voices its disagreement.

The prompts and written materials

The released materials are mostly text that shapes behavior: the system prompts and their variations, the user-message instruction that defines acceptance, and the synthetic documents. The post makes one claim about wording, without numbers: the prefix instruction has "a significant impact on the model's behavior". Its third and fourth questions ask what written facts, in a prompt or in training documents, do to behavior, but the post tests nothing itself.

Limitations and later work

  • No results of its own. The statement that three models "sometimes" fake alignment rests on the paper's Appendix B.4, which for Llama-3.1-405B measured only a compliance gap and a qualitative reading.
  • A different metric. The notebook's prefix check is a proxy the paper calls sensitive, on AdvBench queries rather than the paper's dataset, so its results are not directly comparable with the paper's headline numbers.

This note connects three later posts to the questions; the links are the note's, not the post's. Alignment Faking Revisited (April 2025) used the paper's setting with open models and found that the same user-message instruction "acts as a 'jailbreak'" for Claude 3.5 Sonnet without a scratchpad. Why Do Some Language Models Fake Alignment While Others Don't? (June 2025) addressed the second question across 25 models. Towards training-time mitigations for alignment faking in RL (December 2025) extended the RL experiments.

Research questions

  1. How far does the notebook's metric agree with the paper's? Running the paper's queries through both the prefix check and a judged-compliance grader would measure how often the two disagree.
  2. Is the minimal setting's Claude 3 Opus gap distinguishable from zero? Table 5 reports 5.5% with no interval. Running the notebook with several samples per query and reporting intervals would answer this.
  3. Which salient facts change the behavior? Adding single facts to the system prompt (for example, that paid-tier outputs are also reviewed) and measuring the change in the compliance gap and in alignment-faking reasoning would address the post's third and fourth questions.
  4. Which release enabled outside work? Counting later replications that cite the repository, and whether they used the notebook or the full prompting code, would show which materials made outside work possible.

How it connects

References