Language Models (Mostly) Know What They Know

A language model asked an obscure trivia question will usually produce an answer, whether or not it has any basis for one. An honest assistant would need to tell those cases apart: to say how confident it is, and to recognize when it does not know. This paper asks how far language models can already do that, and how far the ability can be drawn out or trained. Its introduction states the motivation directly: honest systems must "accurately and faithfully evaluate their level of confidence in their own knowledge and reasoning", so recognizing what they do and do not know is a prerequisite (Section 1).

The paper appeared in July 2022, between Anthropic's first alignment paper, which named honesty as one of three goals for an assistant (Askell et al., 2021), and the Constitutional AI paper, which cited this paper's calibration results when it used a model's probabilities as soft training labels (Bai et al., 2022). It is the library's main study of calibration and self-knowledge in pretrained models, and the starting point for the honesty thread that continues with sycophancy, reward tampering and introspection.

Terms used in this note

• Calibration. A model is calibrated if, among all the predictions to which it assigns probability 0.7, about 70% turn out correct. Calibration is separate from accuracy: a model can be inaccurate and well calibrated, or accurate and overconfident.
• Calibration chart. A plot of predicted probability against the observed frequency of being correct. The paper sorts predictions into ten bins with equal numbers of predictions in each (glossary in Section 1, Appendix A.1).
• Expected calibration error (ECE). The mean absolute gap between predicted probability and observed frequency across those ten bins (Appendix A.2). Lower is better. The paper also reports a root-mean-square version.
• AUROC. Area under the receiver operating characteristic curve: how well a score separates two classes (here, correct from incorrect answers), regardless of whether the score is calibrated. Chance is 0.5; higher is better.
• Brier score. The mean squared difference between a predicted probability and the 0 or 1 outcome (Appendix A.3). It rewards both separation and calibration. Chance on a balanced binary task is 0.25; lower is better.
• P(True). The probability a model assigns to the proposition that a specific proposed answer is correct.
• P(IK). "Probability that I know": the probability a model assigns to answering a question correctly, before any answer is proposed. In the paper it is the output of an extra classification head fine-tuned for the purpose.
• Ground-truth P(IK). The fraction of a model's samples at temperature 1 that answer the question correctly.
• Temperature. A parameter that rescales a model's output probabilities before sampling. Temperature 1 samples from the model's distribution as trained; higher values flatten it, lower values sharpen it.
• k-shot. Evaluation in which the prompt contains k solved examples of the task before the question. Zero-shot has none.
• RLHF policy. A model fine-tuned with reinforcement learning from human feedback, here the helpful and harmless assistants of Bai et al. (April 2022).

What the paper does

Models and tasks

The experiments use Anthropic's pretrained language models at 800 million, 3 billion, 12 billion and 52 billion parameters, trained on 850 billion tokens with about 10% code (Section 1.2). Apart from one experiment on RLHF policies, all results are for pure language models with no fine-tuning for dialogue.

Two kinds of evaluation appear. Multiple-choice evaluations test calibration: all multiple-choice tasks in BIG Bench, MMLU, TruthfulQA, LogiQA and QuALITY. Open-ended evaluations, where the model must generate an answer, test self-evaluation and self-knowledge: TriviaQA, Lambada (predicting the last word of a passage), GSM8k (grade-school math word problems), Codex HumanEval (writing Python functions), arithmetic problems, and 8,000 Python function-synthesis problems scraped from GitHub repositories with tests, split 6,000 for training and 2,000 for testing (Section 1.2, Appendix D).

Step 1: calibration on questions with visible options

The paper first measures calibration when the answer options are shown as lettered choices and the model answers with a single letter such as "(B)" (Section 2). It then varies the format: replacing the last option with "none of the above" (Section 3.1), recasting each question as a true/false judgment on one proposed answer (Section 3.2), and applying the same tests to RLHF policies (Section 3.3).

Step 2: judging its own samples (P(True))

The model answers an open-ended question by sampling at temperature 1, and is then shown its own answer in the true/false format and asked whether it is correct (Section 4.1). In a variant, the model first sees five of its own samples for the same question, labeled "brainstormed ideas", before judging one of them (Section 4.2). Each version is tried with a fixed prompt and with 20 examples of the task drawn from the evaluation itself.

Step 3: predicting whether it knows the answer (P(IK))

Here no answer is proposed. The model is trained to predict, from the question alone, whether it will answer correctly (Section 5). For each training question the model generates 30 samples at temperature 1; if 20 are correct and 10 wrong, the training set receives 20 copies of the question labeled "I know" and 10 labeled "I don't know", which approximates a soft label from many hard ones. The whole model is fine-tuned together with a new value head, with the loss applied only at the last token of the question (Section 5, Appendix A.6). The paper notes that it also tried asking the model in words, "With what confidence could you answer this question?", but saw no major gains in early experiments and set the approach aside (Section 5).

P(IK) is trained first on TriviaQA alone and then on TriviaQA, Lambada, arithmetic and Python function synthesis together, and tested in and out of the training distribution (Sections 5.1 and 5.2). Two further tests ask whether P(IK) responds to information placed in the context: a relevant Wikipedia article (Section 5.3) and partial, wrong or irrelevant hints on GSM8k problems (Section 5.4). A final experiment asks whether P(IK) is about the model itself or only about how hard a question is in general (Section 5.5).

What it finds

Calibration

Large models are well calibrated with lettered options. Calibration on BIG Bench improves with model size and from zero-shot to few-shot, and the 52B model is well calibrated in this format even on TruthfulQA and on the difficult LogiQA (Section 2, Figures 4 and 6). Within BIG Bench the paper finds no noticeable correlation between a task's accuracy and its calibration (Figure 5). Format matters: in BIG Bench's default format, where the model must write out the whole answer, calibration is poorer (Figure 5, Appendix A.4). These results are reported as plots.

"None of the above" hurts. Replacing option (D) of MMLU with "none of the above" lowers accuracy substantially and worsens calibration; even the 52B model is biased against choosing that option, and moving to 20-shot evaluation did not help (Section 3.1, Figure 7).

True/false judgments are well calibrated. Recast as true/false questions about one proposed answer, the BIG Bench tasks give a well-calibrated 52B model, overconfident only near the extremes (Section 3.2, Figure 8).

RLHF policies look miscalibrated, and a single temperature mostly fixes it. The RLHF policies' raw probabilities are poorly calibrated, which the authors attribute to reinforcement learning concentrating probability on rewarded behaviors. Rescaling with one temperature, $T = 2.5$, for all evaluations "largely fixes" calibration on three evaluations (Section 3.3, Figure 9). The authors add that more intensive RL training "might distort calibration in ways that cannot be remedied in this way".

Self-evaluation

Models can separate their correct and incorrect samples. Across the five sampling tasks, the 52B model assigns visibly higher P(True) to its correct samples, and restricting to samples with P(True) above 0.5 raises accuracy on each task (Figure 1). Self-evaluation improves with model size on almost every task (Section 4).

Zero-shot P(True) separates but is not calibrated. Without examples, P(True) tends to sit near 50% for typical samples, and calibration needs few-shot examples; more examples help mainly by improving calibration rather than separation (Sections 1.1 and 4.1, Figures 34 and 38).

Seeing other samples helps on short answers. Showing five samples before the judgment "significantly" improves self-evaluation on the short-answer tasks, but helps less on Codex and GSM8k, which have long answers (Section 4.2, Figure 11). By Brier score, 20-shot evaluation with comparison samples is best in every case (Figure 11, Figure 30).

Verification appears to improve faster than generation. The gap between overall accuracy and the accuracy of samples judged true grows with model size, which the authors read as evidence that "verification has an advantage over generation" (Sections 4.2 and 6). In cross-evaluations between sizes, samples from smaller models are easier for every evaluator to classify (Figure 33).

P(IK)

In distribution, P(IK) works well. Trained and tested on TriviaQA, the 52B classifier gives lower P(IK) to questions it gets wrong and is very well calibrated; smaller classifiers have higher calibration error (Section 5.1, Figures 12 and 13).

Out of distribution, it partly generalizes. Table 1 gives AUROC and Brier score for the 52B classifier on each task, trained on TriviaQA only or on all tasks except GSM8k:

Training on TriviaQA alone separates correct from incorrect questions on the other tasks better than chance, and the AUROC of this transfer grows with model size (Figure 14). Calibration out of distribution is often poor, and mostly underconfident (Figure 13). On Lambada it is "terrible" because the TriviaQA-trained model gives uniformly low P(IK), which training on all four tasks resolves (Section 5.2, Figure 41). On GSM8k, which no classifier was trained on, broader training moves the model from mostly underconfident to overconfident (Figure 15). A footnote to Table 1 warns that the GSM8k Brier scores are dominated by questions the model gets wrong.

P(IK) rises when the answer is in the context. For the question "What state's rodeo hall of fame was established in 2013?", the 52B model's P(IK) is 18%; with a Wikipedia paragraph about the Idaho Rodeo Hall of Fame prepended, it rises to 78% (Section 5.3). Across TriviaQA questions, adding the relevant article (truncated to 7,000 tokens when necessary) raises P(IK) on average, and shorter articles raise it more (Figure 18). The classifier was never trained with source material in the context.

P(IK) responds to hints. On GSM8k, showing more of a correct solution raises P(IK); correct hints raise it more than wrong ones, though wrong hints partly fool the model; and hints taken from a different problem lower it (Section 5.4, Figure 19). The model trained on all four tasks behaves more consistently than the TriviaQA-only model.

P(IK) carries some model-specific information, weakly. Two 12B models with the same architecture but different pretraining data were compared on the TriviaQA questions only one of them answers correctly: 650 such questions for one model and 826 for the other (Section 5.5). Each model's average P(IK) is higher on its own correct questions (Table 2):

The caption calls the difference "only around 6%". In a second test, classifiers were trained on each model's ground-truth data starting from each model (Table 3). On model A's data, starting from A or B gives the same AUROC (0.8633 and 0.8631); on model B's data, starting from B does better (0.8717 against 0.8460). The authors call the results "mixed but encouraging".

Other signals

Appendix B tests cheaper signals that need no training: the model's loss on its own greedy answer, the entropy of 200 sampled answers, and the loss over 30 samples placed in one context. Each separates known from unknown questions to some degree on TriviaQA, less so on HumanEval, and none is calibrated (Appendix B, Figures 22 to 27). For HumanEval, the AUROC of the entropy signal falls as models grow (Appendix B.2).

The prompts and formats used

The paper trains no principles, but its results depend heavily on the wording and layout of prompts, and it reports this directly.

• Lettered options. Calibration depends on the model seeing the options before choosing and answering with one token. The authors stress that the choices must be shown explicitly, since otherwise paraphrases of the same answer ("Washington" and "George Washington, the first US president") split the probability (Section 2).
• True/false wording. The self-evaluation template ends "Is the proposed answer: (A) True (B) False" and reads off the probabilities of the two letters (Sections 3.2 and 4.1).
• An unoptimized prompt. The fixed prompt for self-evaluation, six worked examples printed in Appendix A.5, "was simply the first prompt we tried, and it was not otherwise optimized". Few-shot examples from the task are needed for good calibration, although by the measure of accuracy among samples judged true, the fixed prompt does better than 20-shot evaluation on Codex and GSM8k (Figure 29).
• Question in words versus a trained head. Asking the model in natural language how confident it could be did not give major gains early on, so the paper uses the value head; the authors say it "would be worth returning to" the natural-language approach (Section 5).
• Few-shot formatting for P(IK). TriviaQA, Lambada, arithmetic and GSM8k used 10-shot prompts to keep answers in the expected format, which matters for TriviaQA because answers are graded by string comparison; Python function synthesis was 0-shot (Sections 5.1 and 5.2, Appendix A.7).

The paper also states what it means by honesty. Section 6 lists five overlapping ideas under that umbrella term: truthfulness, calibration, self-knowledge, explainability and non-deceptiveness, the last framed as whether AI systems can be kept from learning to lie "when human preference data might encourage systematic mistakes or provide more reward for pleasant misconceptions". The paper addresses the first three. The list anticipates the subject of the sycophancy paper, which tests exactly that worry about preference data.

Limitations and open questions

The authors list these in Section 6.1:

• Pretrained models only. Fine-tuned models such as RLHF policies are less well calibrated, and the methods may work less well on them.
• Imitation versus truth. Pretraining imitates human text, so the analysis cannot distinguish "the truth" from "what humans say". The most interesting honesty questions concern knowledge models hold that humans do not.
• Deception is out of scope. A model that learned to deceive, for instance to win preference rewards, would undercut honest self-evaluation, and the paper does not address this.
• Limited scope. Only five sampling-based datasets were studied, and the evidence on scaling and generalization is "limited in power and scope".

Several further limits follow from the design. P(IK) needs a fine-tuned head, so it is a trained probe on the model rather than a statement the model makes in conversation. The model-specificity test rests on two 12B models and an effect of about six percentage points. The study's footnotes note that models cannot tell their own tokens from text supplied by someone else (Section 1, footnote 1), so "self"-evaluation is evaluation of text the model is told is a proposed answer. Several headline results (calibration curves, Brier scores, the hint effects) are reported only as plots.

Later work in the library takes up the gaps. The sycophancy study tests whether RLHF-trained assistants keep correct answers under social pressure, which bears on the limitation about fine-tuned models (Sharma et al., 2023). The introspection study asks a narrower question with interpretability tools, whether a model's reports about its internal states are causally grounded in those states, and cites this paper as prior evidence that models can estimate their own knowledge (Lindsey, 2025).

Research questions

1. Is P(IK) a general self-knowledge signal or a set of task heuristics? Training on TriviaQA transfers to arithmetic with AUROC 0.928 but to Lambada with 0.606 (Table 1). Probing the trained head on matched tasks that differ in surface form but not in the model's accuracy, or comparing the internal features the head relies on across tasks, would show whether one signal transfers or several unrelated cues happen to.
2. Does one temperature still repair the calibration of current RLHF policies? The paper found $T = 2.5$ worked for its policies on three evaluations and warned that more intensive RL might not be fixable this way (Section 3.3). Repeating the calibration curves on later assistants, with one global temperature and with per-task temperatures, would show whether the simple relationship still holds.
3. What do the brainstormed samples contribute beyond agreement counting? Showing five samples improves short-answer self-evaluation (Section 4.2). Comparing P(True) with a simple score, the fraction of the five samples that match the answer under judgment, would show whether the model extracts more than self-consistency, and why the benefit shrinks for long answers.
4. Can calibrated confidence be expressed in words rather than read from a probe? The natural-language version of P(IK) was set aside after early experiments (Section 5). Training models to state confidence in prose and scoring the stated numbers with the same calibration charts would show whether the head's calibration can be transferred into what the assistant says.
5. How much of P(IK) is about the model rather than the question? The cross-model differences are small (Tables 2 and 3). Repeating the experiment with more pairs of models that differ in specific knowledge, for example one trained on documents the other never saw, would give a cleaner measure of model-specific self-knowledge.
6. Do the in-context effects transfer to retrieval settings? P(IK) rises with relevant articles and falls with irrelevant hints without training on either (Sections 5.3 and 5.4). Testing P(IK) on retrieved passages of varying relevance would show whether it could serve as a signal of when retrieved material actually answers a question.
7. Is the P(IK) signal the same thing the 2025 introspection work measures? The introspection study requires that a self-report be causally grounded in an internal state and not inferred from outputs. Checking whether the P(IK) head reads a representation that also governs the model's own expressions of uncertainty, for example by steering along the head's direction and observing stated confidence, would connect the two lines of work.

How it connects

• Before. Askell et al. (2021) made honesty one of the three HHH criteria; this paper adopts its use of "honesty" as an umbrella term (Section 1.3). The RLHF policies studied in Section 3.3 come from Bai et al. (April 2022).
• Alongside. Constitutional AI uses a feedback model's probabilities for "(A)" and "(B)" as soft labels and cites this paper for the expectation that such probabilities are calibrated.
• After. Towards Understanding Sycophancy in Language Models shows assistants abandoning correct answers when users push back, the non-deceptiveness concern of Section 6. Sycophancy to Subterfuge studies what training on sycophancy generalizes to. Emergent Introspective Awareness cites this paper among earlier evidence of self-knowledge and adds an interpretability test of whether self-reports track internal states. Claude's constitution names being "Calibrated" as one component of honesty, alongside being truthful, transparent and non-deceptive, among others.
• In the course wiki. The paper's distinction between separating correct from incorrect cases (AUROC) and producing usable probabilities (calibration) is the same one drawn for Naive Bayes in Evaluating Classifiers and The Bernoulli Model and Why Naive Bayes Works, whose scores rank well but are badly calibrated; The Probability Ranking Principle explains when ranking needs only order and when it needs calibrated probabilities.

References

• Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022). Language Models (Mostly) Know What They Know. arXiv:2207.05221v4, 21 November 2022. Anthropic. https://arxiv.org/abs/2207.05221. Anthropic page: https://www.anthropic.com/research/language-models-mostly-know-what-they-know
• Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., et al. (2021). A General Language Assistant as a Laboratory for Alignment. arXiv:2112.00861.
• Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862.
• Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073.
• Lindsey, J. (2025). Emergent Introspective Awareness in Large Language Models. Transformer Circuits Thread. https://transformer-circuits.pub/2025/introspection/index.html
• Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., et al. (2023). Towards Understanding Sycophancy in Language Models. arXiv:2310.13548.