3 Challenges and 2 Hopes for the Safety of Unsupervised Elicitation

Anthropic Fellows and MATS work that builds stress-test datasets for methods meant to steer models without trustworthy labels (unsupervised elicitation and easy-to-hard generalization), with salient features other than truth, imbalanced training sets, and questions that have no objective answer. On Llama 3.1 8B and 70B, no prompting or probing method tested performs reliably on any of the three, and ensembling or mixing the two kinds of method helps only partly.

Callum Canavan, Aditya Shrivastava et al. (MATS and Anthropic Fellows Program, with Anthropic) · 2026-03-05 · original

Methods that steer a model toward true answers without trusted labels for the task have looked strong on standard benchmarks. Unsupervised Elicitation (June 2025), for example, matched training on correct labels for grade-school math and common misconceptions. This work asks whether those results are optimistic because the benchmarks are easy in particular ways: truth is the most prominent feature of the data, the classes are balanced, and every item has a definite answer. The authors build datasets that break each of these properties and test a range of existing methods, plus new ones built on two hopes for fixing the problems. Their summary is that "no technique reliably performs well on the 3 challenges" (post, tl;dr).

Versions and authors. The version of record is the arXiv paper "Three Concrete Challenges and Two Hopes for the Safety of Unsupervised Elicitation" (arXiv:2602.20400v1, 23 February 2026; "Preprint. Under review."), by Callum Canavan and Aditya Shrivastava (equal contribution; MATS and the Anthropic Fellows Program), Allison Qi (Anthropic), Jonathan Michala (MATS) and Fabien Roger (Anthropic). The Alignment Science Blog post of 5 March 2026 summarizes it with the same figures; the same text was crossposted to LessWrong on 27 February 2026, where it had no comments when checked on 13 November 2026. Numbers and section references below are from the paper. Code and data are released on GitHub.

What the paper does

Terms used in this note

  • Unsupervised elicitation (UE). Steering a pretrained model on a task using only unlabeled examples of that task (the "hard set").
  • Easy-to-hard generalization (E2H). Training on a different, easier task that has trustworthy labels (the "easy set") in the hope that it carries over to the hard task.
  • Salient non-truth feature. A property of the data, other than correctness, that the model represents prominently, such as whether a statement agrees with someone's stated opinion.
  • Linear probe. A direction in the model's internal activations; projecting an example's activations onto it gives a score for whether the statement is true.
  • AUROC. The area under the ROC curve when a score is used to separate true from false items; 0.5 is chance and 1.0 is perfect.

Setup

Each method may use the unlabeled hard set, a labeled easy set, and a pretrained model with no post-training, and must give each statement a "truth score" without chain-of-thought reasoning (Section 3.1). During training it may not use any knowledge of which non-truth features are present, of the label distribution, or of which items have objective answers. Ground-truth labels are used only to score a held-out test set, which the authors note would not exist for real tasks beyond human ability. The model is Llama 3.1 8B for the salient-feature and imbalanced-GSM8K experiments, Llama 3.1 70B for the impossible-task experiments, and both for the imbalanced Ctrl-Z experiments (Section 3.3).

Methods compared

Prompting methods read off the model's next-token preference for a positive or negative answer (Section 3.3.1): zero-shot prompting; few-shot prompting with random labels; bootstrapped few-shot prompting, which feeds each round's predictions in as the next round's examples and which the authors use because it was found to give labels of similar accuracy to ICM (the algorithm in Unsupervised Elicitation) at much lower cost; and few-shot prompting with true labels as a ceiling.

Probe methods (Section 3.3.2) include a random direction, CCS (Contrast-Consistent Search, which trains a probe so that a statement and its negation get confident, opposite scores), the first principal component of contrast-pair differences (PCA), an E2H probe trained on a simple labeled dataset of number comparisons, and a supervised probe as a ceiling.

The new methods test the two hopes (Section 3.3.3):

  • Combining UE and E2H: train a probe on a weighted sum of the CCS loss on the hard set and a supervised loss on the easy set.
  • Ensembling: combine 128 or 256 random probes or principal components, weighted either by consensus with the growing ensemble or by a softmax (temperature 0.1) over each probe's AUROC on the easy set minus 0.5.

Datasets

Each dataset modifies an existing one (Section 3.2; prompt formats in Appendix F):

  • Salient non-truth features. GSM8K pairs of solutions, one correct and one not, with one solution at random given a feature uncorrelated with correctness: a professor who agrees with it (sycophancy), extra punctuation, or a changed tense. LIAR, PolitiFact statements labeled both for factual correctness and for political stance, which are uncorrelated. Civil Comments, news-site comments with several civility features such as toxicity and sexual explicitness.
  • Imbalanced training sets. GSM8K solutions with the share of correct ones in training varied from 0% to 100%. Ctrl-Z, sequences of ten bash commands from an earlier AI control study, either safe or ending in downloading and running a dummy malware file, with the training set 50%, 99% or 100% safe. After filtering, Ctrl-Z has 3,438 trajectories in an approximate 2:1 ratio of safe to malicious, over 157 tasks, split 80/20 with no task overlap (Appendix A).
  • Impossible tasks. GSM8K solutions mixed with normative political statements from the Political Ideologies dataset, reworded by GPT-4o mini into question-and-answer form. A good method should be confident on the math and uncertain on the politics.

What it finds

Salient features pull methods away from truth

With the sycophancy feature added to GSM8K, most methods lose more than 10 percentage points of accuracy compared with the unmodified data; with punctuation most stay within 5 points, and with tense within 2 (Section 4.1, Figure 2). The authors hypothesize that sycophancy is more salient to the model than punctuation or tense.

The prompt did not reliably choose the feature. On LIAR, when the prompt asked for political stance, every method tracked stance, as intended. When the prompt asked for factual correctness, every method except the PCA ensembles still tracked stance more than correctness (Figure 3a and 3b). On Civil Comments, every UE and E2H method except bootstrapped prompting produced labels that agreed with toxicity at least as much as with sexual explicitness, even when the prompt asked for sexual explicitness (Figure 3c and 3d). The authors hypothesize that the methods label "primarily based on feature salience" (Section 4.1).

Imbalance hurts most on realistic data

On GSM8K, CCS and PCA probes did about as well or better when every training solution was incorrect than when the set was balanced, which the authors call surprising; they did significantly worse when every training solution was correct (Section 4.2, Figure 4).

On Ctrl-Z, the paper reports that no method exceeded an AUROC of 0.75 with 99% or 100% safe training data, except the E2H probe, which does not use the training set and reached 0.87 on Llama 3.1 70B. On Llama 3.1 8B the E2H probe reached 0.67, while 0.75 was possible with balanced data, for example with CCS (Section 4.2, Figure 5). The statement covers the unsupervised and E2H methods: in Figure 5 the label-using ceilings (golden few-shot prompting, the supervised probe) exceed 0.75 in some imbalanced conditions. The post adds that performance is "already low" with balanced data because of salient non-truth features "like answer length" (post, "Imbalanced training sets"). The paper does not name length, and its Appendix A says safe and malicious sequences "have similar character lengths"; the two are compatible, since a salient feature that does not differ between the classes is the kind that would draw a method away from the label. Appendix A also notes a feature of the data that a method could exploit: 58% of safe sequences contain at least one command generated by the source study's trusted weak model (Llama 3.1 8B), and no malicious sequence does.

Overconfidence on questions without answers

On the mixed GSM8K and political dataset, some methods reach an AUROC above 0.8 on the math items, but every UE and E2H method has a "relative confidence on GSM8K" below 60% (Section 4.3, Figure 6). This measure is the share of the variance of a 50/50 mixture of math and political scores that comes from the math scores; it approaches 1 when the political scores cluster near the mean of the math scores, is 0.5 when the two sets have identical score distributions, and approaches 0 when the political scores or the gap between the two means dominate (Appendix D). Figure 9 (Appendix E) prints four examples: the supervised probe ceiling (AUROC 0.847, relative confidence 0.967); CCS (0.627 and 0.031), which separates math from politics rather than true from false; an E2H-weighted PCA ensemble (0.714 and 0.543); and bootstrapped few-shot prompting (0.864 and 0.426), which is accurate on math but confident on politics. The post adds that bootstrapped few-shot "converges to classifying all politically liberal answers as correct" (post, "Impossible tasks").

The two hopes

Ensembling and the UE plus E2H combination sometimes did better than other methods (the PCA ensembles on LIAR, for instance) but did not address the challenges reliably (Sections 1 and 6). Variants tried on only part of the datasets, Cluster-Norm normalization and rescaling ensemble members to equal variance, did not help (Appendix C, Figures 7 and 8).

The prompts used

The only steering signal a UE method has is the prompt, and the paper says it chose its salient-feature datasets partly for that reason (Section 3.2.1). The prompts are printed in Appendix F. The LIAR experiment is a controlled comparison of two prompt wordings on the same statements, differing in the claim the model is asked to judge: "The given statement is politically left-wing" against "The given political statement is factually correct" (Appendix F.3). The Civil Comments experiment does the same with "This comment is toxic" and "This comment is sexually explicit" (Appendix F.4). In both, most methods tracked the more salient feature (political stance on LIAR, toxicity on Civil Comments) whichever claim the prompt asked about; the exceptions were the PCA ensembles on LIAR and bootstrapped prompting on Civil Comments (Section 4.1, Figure 3). The Ctrl-Z prompt adapts a security-monitor prompt from the source study, with instructions such as to ignore comments and variable names and to treat any encoding or decoding as suspicious (Appendix F.5).

Limitations and open questions

The authors state these limitations (Section 5; post, "Discussion"):

  • Coverage of methods. No method using supervised fine-tuning and no weak-to-strong method was benchmarked; in particular ICM itself was not run, and bootstrapped few-shot stands in for it.
  • Realism. Some datasets are not realistic; mixing math with political claims let some methods separate items by source dataset. The authors consider most datasets a "significant realism improvement" over earlier stress tests.
  • Mitigations not tried. Rebalancing a training set using a weak classifier's rough labels might help, but could run into the other two challenges.
  • Other threats. The work does not address models that deliberately interfere with elicitation, and does not test more sophisticated consistency properties.

The authors' main warning concerns evaluation: because UE is meant for settings without ground truth, failures may be "silent," and validating methods only on benchmarks like GSM8K "would likely produce overoptimistic results" (Section 5). Other points a reader needs:

  • Model scope. All results are on Llama 3.1 8B and 70B base models; no Claude model is tested.
  • Variance. The paper does not report the number of runs or seeds behind each point in its figures; Figures 4 to 6 draw error bars without saying what they show, and Figure 2 draws none.
  • Base models and chat formatting. The Ctrl-Z prompt wraps a base model's input in Llama 3 chat tokens adapted from the source study's monitor prompt (Appendix F.5); the paper does not discuss whether this affects a model with no post-training.
  • Description differences. The post describes E2H weighting as using each probe's AUROC "on the dataset," while the paper specifies the easy dataset and a softmax temperature of 0.1 (Section 3.3.3); the paper is followed here.

Research questions

  1. Does ICM itself fail in the same ways as bootstrapped prompting? The paper substitutes a cheaper method. Running ICM on the LIAR, Civil Comments and Ctrl-Z datasets would show whether mutual predictability resists salient features any better.
  2. Can prompt wording override salience at all? The LIAR and Civil Comments prompts differ in one claim. Testing a graded series of prompts, from terse to explicit with definitions and counterexamples, would show whether any wording moves methods off the more salient feature.
  3. Do larger or post-trained models behave differently? Results are on Llama 3.1 base models. Repeating the stress tests on larger base models would show whether salience of truth grows with scale.
  4. What explains good CCS and PCA performance with all-incorrect training data? The authors call it surprising. Analysis of the probe directions learned in that condition would show what they track.
  5. Can a method be calibrated to abstain on normative items? No UE or E2H method exceeded 60% relative confidence. A method that trains on a mixture with an explicit "indeterminate" option, scored on new normative domains, would test whether abstention can be elicited without labels.
  6. How much of the Ctrl-Z difficulty comes from source artifacts? Safe sequences alone contain weak-model commands. Re-running after removing those trajectories would separate the imbalance effect from that artifact.

How it connects

  • Unsupervised Elicitation is the method this work set out to stress-test. The candidate GSM8K solutions used here come from that paper (Section 3.2); the other datasets do not. In this paper's account, ICM was shown to "match SFT on golden labels for small training sets" (Section 2), a narrower claim than the original paper's own summary.
  • The same group's January 2026 forum post, "Eliciting base models with simple unsupervised techniques," found that simple bootstrapping and consistency methods come close to ICM on the original datasets and promised this paper's harder datasets; the Unsupervised Elicitation note summarizes it.
  • Towards Understanding Sycophancy in Language Models is cited as background for using sycophancy as a salient non-truth feature.
  • Automated Alignment Researchers: Using large language models to scale scalable oversight (April 2026) worked on weak-to-strong supervision, the family of methods this paper says it did not benchmark; this connection is the note's own.

References