Unsupervised Elicitation

Wen and colleagues introduce Internal Coherence Maximization (ICM), which labels a dataset with a pretrained model's own judgments by searching for labels that are logically consistent and mutually predictable, then trains on them; on Llama 3.1 models it roughly matches training on gold labels for GSM8K-verification and TruthfulQA and beats crowdsourced labels on Alpaca. The June 2025 post reports an unsupervised Claude 3.5 Haiku assistant that beat its human-supervised counterpart; the January 2026 revision of the paper replaces that experiment with a Claude 4 Sonnet one in which the unsupervised assistant matches the human-supervised one on average, better on chat and safety and worse on math and code, though its scores fall further late in training.

Jiaxin Wen, Zachary Ankner, Arushi Somani et al. (Anthropic, with Schmidt Sciences, Constellation, New York University, George Washington University and an independent researcher) · 2025-06-11 · original

If a model becomes better than its overseers at some task, human labels for that task become unreliable, and training on them may teach the model to say what overseers want to hear. "Unsupervised Elicitation" asks whether a pretrained model can instead be fine-tuned on labels it produces itself, with no external labels at all, and still reach the accuracy of training on correct ones. The authors' answer is an algorithm, Internal Coherence Maximization (ICM), that searches for the set of labels the pretrained model finds most coherent. The work appeared on 11 June 2025 as a short Alignment Science Blog post and an arXiv paper.

Versions. This note follows the paper's second arXiv version (v2, 26 January 2026), which is the fullest and latest. The post summarizes v1 (11 June 2025) and has not been updated. The two versions agree on the Llama experiments but differ on the frontier-model experiment: v1 trained a Claude 3.5 Haiku assistant and reported that both the reward model and the assistant "outperform their human-supervised counterparts" (v1 abstract); v2 replaces this with a Claude 4 Sonnet assistant that "matches its counterpart trained on production-grade human labels on average, with higher scores on chat and safety yet lower scores on math and coding" (v2 abstract). v2 also adds two authors (Yanda Chen and Fabien Roger) and several appendices. Its source uses the ICLR 2026 style file with the camera-ready flag set, but its page header reads "Preprint"; whether a proceedings version exists could not be confirmed (OpenReview blocked scripted access). Section, figure and table numbers below are v2's.

What the paper does

Terms used in this note

  • Elicitation. Getting a pretrained model (one with no post-training, Appendix D) to use a capability it already has, rather than teaching it a new one. The paper's premise is that pretrained models already represent concepts such as mathematical correctness, truthfulness and helpfulness (Section 1).
  • Golden labels. The benchmark's correct labels, the ceiling the method is compared with.
  • Mutual predictability. For a candidate set of labels, the sum over examples of the log-probability the pretrained model assigns to each label when shown all the other labeled examples in context. A high score means the labels "collectively specify a single coherent concept" according to the model (Section 2.2).
  • Logical consistency. Simple constraints that rule out contradictory labels: two different answers to one math question cannot both be correct, and "A is better than B" and "B is better than A" cannot both be true (Sections 2.2 and 3.1).

The algorithm

ICM scores a label set by a weighted combination of mutual predictability and a count of logical inconsistencies (Section 2.2). Mutual predictability alone has degenerate solutions, such as giving every example the same label; the consistency term is there to block them. Because finding the best label set exactly is an integer programming problem, ICM searches approximately, in a procedure modeled on simulated annealing (Section 2.3, Algorithm 1): it starts from a few randomly labeled examples (K = 8 is suggested), then repeatedly picks an example, labels it with the model's most likely label given the others, repairs any inconsistency this creates, and keeps the change if the score improves, or with a probability that shrinks as the search goes on if it does not. The weight on mutual predictability starts at 50 and may be lowered to 20 or 30 "based on the search speed on the training data, without reference to any validation data" (Appendix D.1). Labeling took 2.0 to 3.9 iterations per data point on average, depending on the task (Appendix F, Table 5); an outside reanalysis (see "Limitations") finds the cost in forward passes grows with the square of the number of labeled points.

Mutual predictability is computed by in-context learning, so all labeled examples must fit in the model's context window. For long production data, v2 adds a scalable variant that approximates conditioning on all other labels by fine-tuning on all but one fold of the data and relabeling the held-out fold, and that repairs inconsistencies by deferring to the model's most confident prediction in each group (Appendix G, Algorithms 3 and 4).

Tasks and models

All tasks are cast as true-or-false classification of a claim (Table 1, Section 3.1):

  • TruthfulQA: is a given answer to a question about common misconceptions correct?
  • GSM8K-verification: is a sampled solution to a grade-school math problem correct? Golden labels check final answers and, with Claude 3.5 Sonnet as checker, intermediate steps.
  • Alpaca: is response A more helpful and harmless than response B? Test labels are the majority vote of four human labels.

Training sets have 2,560, 2,560 and 2,048 examples and test sets 1,000, 2,971 and 933 (Table 2). The Llama experiments use Llama 3.1 8B for GSM8K and Llama 3.1 70B for TruthfulQA and Alpaca (Figure 3 caption; Appendix D). Baselines are zero-shot prompting of the pretrained and of the post-trained chat model, and many-shot prompting or fine-tuning with golden labels or with real human labels (Alpaca's training labels, one annotation per example) (Section 3.2).

What it finds

Matching gold labels on standard tasks

On Figure 3, ICM matches golden supervision on TruthfulQA and GSM8K and "substantially outperforms" training on Alpaca's crowdsourced labels (Section 4.1, Findings 1 and 2). Zero-shot accuracy of the base models is near chance: about 60% on TruthfulQA, 50% on Alpaca and 48% on GSM8K (Appendix J). Table 3 (Appendix E.1) prints the accuracies, mean plus or minus spread over runs (Figure 1's caption says three runs):

Task Comparison label source Comparison accuracy ICM accuracy
GSM8K golden 78.1 ± 0.5 77.0 ± 0.8
TruthfulQA golden 92.0 ± 1.0 90.9 ± 0.6
Alpaca human 65.5 ± 0.6 68.0 ± 0.7
Gender prediction golden 80.5 ± 0.3 79.7 ± 0.4

Table: Data from Wen et al. (2026, v2), Table 3. The same table reports labels distilled from GPT-4o (10 samples, majority vote) at 75.1, 81.9, 65.2 and 77.0.

ICM also beats Contrast-Consistent Search (CCS), an earlier unsupervised method that trains a probe on activations, which scores 67.0, 63.0 and 53.0 on GSM8K, TruthfulQA and Alpaca (Table 4, best layer and best of the two probe directions). ICM's gap to golden labels on TruthfulQA is larger at 8B than at 70B (Figure 4). Having been "initially very skeptical of these findings," the authors report three checks against accidental label leakage, including an independent replication by a coauthor on the Claude 3.5 Haiku base model with a different codebase (Section 4.1).

Ablations (Section 5) find that ICM's labels give better many-shot performance than random labels with the same accuracy (Figure 8); that starting from entirely wrong labels on TruthfulQA costs only a moderate drop (Figure 9); and that removing the consistency term matters little on TruthfulQA but almost always produces the degenerate one-label solution on Alpaca (Figure 10). The authors conclude that mutual predictability is "the most important term" (Section 6).

Where it fails, and where humans are worse

On a task built to be impossible, preferring any poem that mentions the sun while the prompt asks only which poem is better, ICM on Llama 70B performs no better than chance (Section 4.2, Figure 5). This is the paper's main stated limitation: the target concept must be salient to the pretrained model.

On predicting which of two blog posts was written by a man (Blog Authorship Corpus), five annotators recruited on Upwork reached 60% on 100 test examples and 53.8% on 48 training examples, while ICM matched golden supervision at about 80% (Section 4.3, Figure 6, Appendix I). The authors use this as a task on which the pretrained model is "strongly superhuman," and state that improving gender prediction is not their goal.

The frontier-model experiment, in two versions

v1 (June 2025, and the post). From a subset of nearly 400,000 examples of the production preference data, Claude 3 Haiku labeled 6,000 with ICM; a reward model trained on those labeled the rest, and the final unsupervised reward model, trained on the whole subset, scored 75.0% on RewardBench against 72.2% for a reward model trained on production human labels. Claude 3.5 Haiku policies trained for 20,000 RL episodes against each reward model were compared head to head by the Claude 3.5 Sonnet production reward model, and the unsupervised policy won 60% of comparisons; the released Claude 3.5 Haiku won 92% against the same human-supervised baseline (v1, Section 4.4).

v2 (January 2026). ICM with Claude 4 Sonnet pretrained models labels 5,000 preference pairs under the task description "Output A is more helpful, harmless, and honest than Output B"; two assistants are then trained for 250,000 RL episodes on math, code and instruction-following tasks, one against this reward model and one against a reward model trained on production human labels, and their responses to RewardBench prompts are scored by a production Claude 4 Opus reward model (Section 4.4). The unsupervised assistant matches the human-supervised one on average, with higher scores on chat and safety and lower on math and code, and learns about 2.5 times faster on Chat and Chat-Hard (Figure 7). The authors suspect the human labels are better on "these crisp reasoning tasks." Figure 7 plots the score against the share of RL steps and prints no values, and the comparison depends on where the curves are read. Read from the figure, by this note: both policies' average scores peak between about 25% and 35% of training (about 0.6 for the unsupervised policy, about 0.7 for the human-supervised one) and then fall, and the unsupervised policy falls further, ending near -0.85 against about -0.2. At the last step the unsupervised policy is below the human-supervised one on every panel except Safety. The paper does not discuss the decline or say which checkpoint "matches" refers to.

Appendix H evaluates the reward models themselves on RewardBench (Figure 12, which prints values and error bars without saying what the bars show). The ICM reward models' average accuracy rises with model size from Claude 3.5 Haiku to Claude 4 Sonnet, from 0.63 to 0.74 with 512 training examples and from 0.69 to 0.79 with 5,000. ICM on 5,000 unlabeled examples beats 512 human labels by "15.2%" with Claude 3.5 Haiku and "6%" with Claude 4 Sonnet; in Figure 12 these are differences of about 0.15 and 0.06 in average accuracy (0.69 against 0.54, 0.79 against 0.73). With 5,000 human labels the comparison depends on the model. For Claude 3.5 Haiku the paper says ICM "can slightly outperform" human supervision, though Figure 12 prints 0.69 for both. For Claude 4 Sonnet the human-supervised reward model is ahead on average (0.85 against 0.79), and reaches 0.66 on Chat-Hard and Math-Debiased, where most reward models are near chance. This note's comparison: v1's result that the unsupervised reward model beats the human-supervised one (Claude 3 Haiku, about 400,000 examples) is not repeated in v2's Appendix H, which uses other models and data sizes; at 5,000 examples the two reward models tie for Claude 3.5 Haiku and the human-supervised one wins for Claude 4 Sonnet.

The prompts and task descriptions used

ICM uses no labels, but it is not free of human-written text. Each task is specified by a prompt that ends in a claim to be judged true or false (Table 1), and the production experiment by a one-line description of the preferred response. The paper says that such text remains necessary: in the post's words the method "does not obviate the need for human input, such as a human-specified task description or constitution," because it "provides limited control over the resulting model’s behavior" ("Conclusion"). The paper's discussion proposes using ICM within Constitutional AI, labeling which response better follows each human-written principle and training a reward model on those labels, while noting that "we still need humans to validate whether the model is interpreting the constitution as intended" (Section 6). No experiment in the paper varies the wording of a task description or tests a constitution; the sun-poem task shows only that a description cannot convey a preference the model has no concept for.

Limitations and open questions

The authors state these limitations (Section 6; post, "Limitations"):

  • Salience. ICM finds only concepts the pretrained model already represents prominently.
  • Long inputs. The original algorithm needs all labeled examples in context; v2's scalable variant is the authors' response.
  • Contamination. Llama's pretraining corpus cannot be checked. The authors argue contamination is unlikely because zero-shot accuracy is near chance, the tasks were reformatted, and the production preference data was not in Claude 4 Sonnet's pretraining corpus (Appendix J).

Other points a reader needs:

  • Graders. The production experiments are scored by Anthropic reward models (Claude 3.5 Sonnet's in v1, Claude 4 Opus's in v2), not by people, and the production preference data is not public.
  • Variance. The Llama results report spread over three runs (Figure 1 caption). For the production experiments the paper does not state the number of runs or seeds; the error bars in Figures 7 and 12 are unexplained.
  • Later analysis of why ICM works. A January 2026 forum post by Anthropic Fellows and MATS researchers (Shrivastava, Qi, Canavan, Qiu, Michala and Roger) reran simpler methods on the same datasets. Few-shot prompts with random labels recovered 53 to 93% of the gap between zero-shot and golden many-shot prompting, and a method combining bootstrapping (feeding one round's predictions in as the next round's examples) with logical consistency recovered 83 to 100%, or 91 to 99% with iterative fine-tuning. The gains did not hold when the training set grew from about 2,000 to about 30,000 examples, where golden fine-tuning kept improving. The same post reports that the official ICM code collapsed on Alpaca (label accuracy rose to about 75% and fell to about 46%) with the hyperparameters the authors confirmed; that ICM's cost grows with the square of the number of labeled points; that a trailing space in the prompt format may have lowered baseline scores; and that ICM's consistency constraints on TruthfulQA are derived from the dataset's ground-truth labels, which can leak label information. The paper gives the TruthfulQA constraint only as "two different answers cannot both be true" (Section 3.1); the released code (src/experiments/ICM.py, from the initial commit 2b1bc36 of 10 June 2025 to the latest, 7d4ba5e) sets each TruthfulQA answer's consistency key from its ground-truth label, so which answers count as "different" is decided by the gold labels, as the reanalysis says. Fabien Roger, who mentored that work and is a coauthor of v2, added to a comment of 14 June 2025 on the paper's forum crosspost (last edited 11 August 2025) that he had become "more skeptical that mutual predictability is load-bearing," which differs from v2's Section 6.
  • Harder datasets. 3 Challenges and 2 Hopes for the Safety of Unsupervised Elicitation (March 2026) builds datasets with salient non-truth features, imbalanced training sets and unanswerable items, and finds that no unsupervised or easy-to-hard method it tests, including bootstrapped few-shot prompting as a cheaper stand-in for ICM, performs reliably on them.

Research questions

  1. Does ICM's advantage survive larger unlabeled sets? The follow-up found simple methods fall behind golden fine-tuning at about 30,000 examples. Running ICM itself at that scale, with the same compute budget as the baselines, would show whether the method keeps pace with supervised training as data grows or only matches a small supervised ceiling.
  2. How much of ICM's accuracy comes from mutual predictability? The paper calls it the most important term; the follow-up attributes most of the gain to bootstrapping and consistency. An ablation that holds bootstrapping and consistency fixed and removes only the mutual-predictability objective, on the paper's datasets and code, would settle it.
  3. What does ICM label when a wrong concept is more salient than the intended one? The paper's failure case is a concept the model lacks; a dataset where the model holds both the true concept and a more prominent false one (such as what a user wants to hear) would test the case that matters for oversight.
  4. Why does the unsupervised Claude 4 Sonnet assistant lose on math and code? The authors suspect higher-quality human labels on crisp tasks. Comparing ICM label accuracy on the math and code subsets with that of the production labels would test it.
  5. Can ICM label adherence to a written principle? Section 6 proposes ICM-labeled constitutional preference data. An experiment that labels responses against several principles and checks the labels against human judgments of adherence would show whether the model's notion of a principle matches its authors'.
  6. How reproducible is the search? Given the reported collapse in an outside rerun, publishing label-accuracy trajectories across random seeds for each dataset would show how often ICM converges.
  7. Why does the unsupervised policy's score fall further late in RL? In Figure 7 both policies decline after their peaks, the unsupervised one more. Scoring the checkpoints with human raters as well as the Claude 4 Opus reward model would show whether this is over-optimization of a weaker reward model or a disagreement between the two reward models.

How it connects

References