Calibration and Proper Scoring Rules

When a model's stated probabilities can be taken at their word, and how to tell. Calibration defined for a model's confidence in its own answer and for a full probability vector; proper scoring rules defined, the Brier and log scores proved strictly proper, and the McCarthy–Savage characterization of all proper rules by convex functions proved; the decomposition of any proper score into uncertainty, resolution and reliability proved, with its empirical form for the Brier score; the expected calibration error and its two opposite biases, from binning and from sampling noise, proved and measured, with an exact null distribution for a calibrated model and a debiased estimator; and temperature scaling, which changes confidence without changing a single answer. Worked on the confidences of eight HELM models on MMLU and on a small open model's full answer probabilities. Worked problems on a Brier decomposition by hand, the zero-one score, the ECE of a perfectly calibrated forecaster, a debiased bin and a temperature that reorders confidences.

On 514 MMLU questions in five subjects, run by Stanford's HELM project, the model text-davinci-003 gave the answer letter it chose an average probability of 0.885, and was right on 0.556 of the questions. The older base model davinci, on the same questions, gave its chosen letters an average probability of 0.416 and was right on 0.416 of them. Both numbers are probabilities that the model attaches to its own answer being right. When can such a probability be taken at its word, how should a set of probabilistic answers be scored so that honest probabilities score best, and how can we tell from a finite evaluation whether a model's probabilities are trustworthy?

A model is calibrated if, among the questions on which it states probability $c$, the share it gets right is $c$. A scoring rule assigns a loss to a stated probability once the outcome is known. It is proper if a forecaster minimizes the expected loss by stating the probability it believes. This article proves which scoring rules are proper and what they reward. It then turns to the statistics of measuring calibration on a finite set of questions, where the standard measure, the expected calibration error, turns out to be biased in two opposite directions.

Here is the route. First we define calibration for a model's confidence in its chosen answer and for a full probability vector. Then we define proper scoring rules, prove that the Brier and log scores are strictly proper, and prove the characterization of every proper rule by a convex function. Next we prove that any proper score splits into uncertainty, resolution and reliability, so that calibration is one of two things a proper score rewards. Then we measure calibration with reliability diagrams and the expected calibration error, prove its two biases and correct them. Finally we repair a miscalibrated model by temperature scaling. The article assumes Cross-Entropy and Kullback–Leibler Divergence for Gibbs' inequality and Loss Functions for Classification, which introduces the Brier score and log loss for one binary prediction and shows that the absolute loss is improper. The calibration section of Binary Classification Metrics introduces reliability diagrams, and the library's note on Language Models (Mostly) Know What They Know reports calibration results for large language models.

The data: confidence in a chosen answer

Two sources supply probabilities, both on MMLU (Hendrycks et al., 2021), whose questions have four options.

  • HELM Classic. For eight models HELM (Liang et al., 2023) recorded, on each of 514 questions from five MMLU subjects, the answer letter the model produced and the probability the model gave that letter, which HELM calls max_prob. HELM ran each question three times with the same five prompt examples in three different orders; we use the first run. These probabilities are over the model's whole vocabulary, so they need not sum to 1 over the four letters, and a model can give its chosen letter less than 0.25.
  • A small open model. HELM Classic gives only the chosen letter's probability. To have each model's full distribution over the four options, the unit's script mmlu_small_model.py ran Qwen2.5-0.5B (Yang et al., 2024; the base model, pinned at revision 060db6499f32faf8b98477b0a26969ef7d8b9987) on a random sample of 4,000 of the 14,042 MMLU test questions, zero-shot, in the prompt format of the MMLU authors' code. One forward pass gives the exact probability of each answer letter as the next token. We renormalize the four letters' probabilities to sum to 1. A sample rather than all questions keeps the run near half an hour on the machine's shared processors.

The HELM models, with each one's accuracy and mean confidence on the 514 questions:

Model Accuracy Mean confidence ECE (10 bins) Brier score Log loss Calibration test p
davinci 0.416 0.416 0.040 0.209 0.609 0.49
curie 0.241 0.333 0.103 0.194 0.578 0.0001
text-davinci-002 0.564 0.735 0.171 0.207 0.636 0.0001
text-davinci-003 0.556 0.885 0.328 0.321 1.340 0.0001
GPT-J (6B) 0.249 0.318 0.069 0.192 0.573 0.0023
GPT-NeoX (20B) 0.284 0.298 0.015 0.202 0.595 0.71
OPT (175B) 0.329 0.334 0.014 0.222 0.637 0.85
BLOOM (176B) 0.298 0.329 0.036 0.208 0.608 0.23

The last four columns are defined below; the calibration test is the Monte Carlo test of the section on sampling noise, and a small p-value is evidence of miscalibration. The scores treat each question as one binary forecast: the event "the chosen answer is correct", forecast with the model's confidence.

Calibration defined

Write $C$ for the confidence a model states in its chosen answer on a randomly drawn question, and $Y$ for the indicator that the answer is correct. The model's confidence is calibrated if

$$P(Y = 1 \mid C = c) = c \quad\text{for every } c,$$

the definition of Guo et al. (2017, Equation 1), with $c$ ranging over the values $C$ takes. This is confidence calibration, also called top-label calibration: it concerns only the probability of the chosen answer. A model that states a full probability vector $P = (P_1, \dots, P_k)$ over $k$ options is calibrated as a vector if $P(\text{answer} = j \mid P = p) = p_j$ for every option $j$ and every vector $p$ (Bröcker, 2009, Equation 1). Calibration as a vector implies confidence calibration: the chosen option $J$ and the confidence $C = P_J$ are functions of $P$, so $P(\text{answer} = J \mid P) = P_J = C$, and averaging over the vectors $P$ with a given confidence $c$ gives $P(Y = 1 \mid C = c) = c$. The converse fails: a model can be right exactly as often as its confidence says while misallocating probability among the options it did not choose.

A calibrated forecast is a conditional probability, and every conditional probability is calibrated. This fact, which Bröcker (2009, Appendix A) proves in general, has a one-line proof for the binary case.

Theorem 1 (conditional probabilities are calibrated). Let $W$ be any information available before the outcome, and let $\Pi = P(Y = 1 \mid W)$. Then $P(Y = 1 \mid \Pi) = \Pi$.

Proof. $\Pi$ is a function of $W$, so conditioning on $\Pi$ is conditioning on less than $W$. By the tower property of conditional expectation (Conditional Expectation and the Laws of Total Expectation and Variance), $P(Y = 1 \mid \Pi) = E[E[Y \mid W] \mid \Pi] = E[\Pi \mid \Pi] = \Pi$. $\square$

Calibration is therefore cheap. The constant forecast equal to the overall accuracy, $W$ empty, is calibrated. A model that says 0.416 on every question would be calibrated, as davinci's confidences appear to be on these questions (the test below finds no evidence against it), and useless for telling easy questions from hard ones. Gneiting and Raftery (2007, Section 2.3) make the same point about weather: forecasts from climatology are "calibrated by construction, but often lack sharpness". What separates useful calibrated forecasts from useless ones is how much they vary, and a proper scoring rule measures both at once, as the decomposition below proves.

Scoring rules and properness

A forecast for an outcome with $k$ possible values is a probability vector $p = (p_1, \dots, p_k)$. A scoring rule is a loss $L(p, j)$ charged when the forecast is $p$ and outcome $j$ occurs. Lower is better here, as in Loss Functions for Classification. Gneiting and Raftery (2007) write scores as rewards to be maximized, the negatives of these losses, and nothing else changes.

If the outcome has distribution $q$, the expected loss of forecast $p$ is

$$S(p, q) = \sum_{j=1}^{k} q_j\,L(p, j).$$

The rule is proper if $S(q, q) \le S(p, q)$ for all $p$ and $q$: no forecast does better in expectation than the true distribution. It is strictly proper if equality holds only for $p = q$ (Gneiting and Raftery, 2007, Equation 1, in the reward orientation). The divergence of the rule is $d(p, q) = S(p, q) - S(q, q)$, the expected excess loss of forecasting $p$ when the truth is $q$, and its entropy is $e(q) = S(q, q)$ (Bröcker, 2009, Equations 3 and 4). A rule is proper exactly when its divergence is never negative.

The two scores in common use are the Brier score (quadratic score), $L(p, j) = \sum_{i=1}^{k}(p_i - \mathbf{1}\{i = j\})^2$, proposed by Brier (1950) for weather forecasts, and the log score (log loss), $L(p, j) = -\ln p_j$. For a binary outcome forecast with probability $c$, the Brier score is twice $(c - y)^2$, and the table above reports the conventional binary form $(c - y)^2$, averaged over questions, as Loss Functions for Classification does.

Theorem 2 (the Brier score is strictly proper). For the Brier score,

$$S(p, q) = \sum_{i=1}^{k}(p_i - q_i)^2 + \Big(1 - \sum_{i=1}^{k} q_i^2\Big).$$

So its divergence is the squared Euclidean distance $\sum_i (p_i - q_i)^2$, its entropy is $1 - \sum_i q_i^2$, and it is strictly proper.

Proof. Expand the loss: $L(p, j) = \sum_i p_i^2 - 2p_j + 1$, since $\mathbf{1}\{i = j\}^2 = \mathbf{1}\{i = j\}$ and exactly one indicator is 1. Averaging over $j$ with weights $q_j$, which sum to 1,

$$S(p, q) = \sum_i p_i^2 - 2\sum_i p_i q_i + 1 = \sum_i (p_i - q_i)^2 - \sum_i q_i^2 + 1.$$

The first sum is zero only when $p = q$ and positive otherwise. $\square$

Theorem 3 (the log score is strictly proper). For the log score, $S(p, q) = H(q) + D(q \,\|\, p)$, where $H(q) = -\sum_i q_i \ln q_i$ is the entropy and $D$ the Kullback–Leibler divergence. So it is strictly proper, with divergence $D(q \,\|\, p)$ and entropy $H(q)$.

Proof. $S(p, q) = -\sum_i q_i \ln p_i = -\sum_i q_i \ln q_i + \sum_i q_i \ln(q_i/p_i)$, which is $H(q) + D(q \,\|\, p)$. Gibbs' inequality, proved in Cross-Entropy and Kullback–Leibler Divergence, says $D(q \,\|\, p) \ge 0$ with equality only for $p = q$. (If $p_i = 0$ for some $i$ with $q_i > 0$, the loss is infinite with positive probability and the inequality holds trivially.) $\square$

Loss Functions for Classification checks both theorems for a single binary forecast by differentiating, and shows that the absolute loss $|y - c|$ is not proper: it rewards stating 0 or 1.

Which scoring rules are proper

Is there a pattern behind the proper rules? There is, and it is the main theorem of the subject. In words: every proper rule comes from a convex function of the forecast, the rule's negative entropy, and the loss charged for an outcome is read off the tangent plane of that function at the forecast.

Write $\Delta_k$ for the set of probability vectors with $k$ components. A function $G$ on $\Delta_k$ is convex if $G(\lambda p + (1 - \lambda)q) \le \lambda G(p) + (1 - \lambda)G(q)$ for all $p, q$ and $0 \le \lambda \le 1$, and strictly convex if the inequality is strict whenever $p \ne q$ and $0 < \lambda < 1$. A vector $g$ is a subgradient of $G$ at $p$ if

$$G(q) \ge G(p) + \langle g, q - p\rangle \quad\text{for all } q \in \Delta_k,$$

where $\langle \cdot, \cdot \rangle$ is the dot product: the plane through $(p, G(p))$ with slope $g$ lies on or below $G$ everywhere.

Theorem 4 (McCarthy, Savage). A scoring rule with finite losses is proper if and only if there is a convex function $G$ on $\Delta_k$, with a subgradient $g(p)$ at each $p$, such that

$$L(p, j) = -\Big(G(p) + g_j(p) - \langle g(p), p\rangle\Big) \quad\text{for } j = 1, \dots, k.$$

It is strictly proper if and only if $G$ can be taken strictly convex. Then $G(q) = -S(q, q)$, the negative of the rule's entropy.

This is Theorem 2 of Gneiting and Raftery (2007, Section 3.1, Equation 10), stated for rewards; they attribute it to McCarthy (1956) and Savage (1971) and prove it for rules that may take the value $-\infty$, as the log score does. The proof below covers finite losses, which is all the Brier score needs; Theorem 3 handled the log score directly.

Proof. Write $R(p, j) = -L(p, j)$ for the reward and $\bar{R}(p, q) = \sum_j q_j R(p, j) = -S(p, q)$.

If. Suppose $R(p, j) = G(p) + g_j(p) - \langle g(p), p\rangle$. Averaging over $j$ with weights $q_j$,

$$\bar{R}(p, q) = G(p) + \langle g(p), q\rangle - \langle g(p), p\rangle = G(p) + \langle g(p), q - p\rangle \le G(q).$$

At $p = q$ the middle expression is $G(q)$, so $\bar{R}(q, q) = G(q) \ge \bar{R}(p, q)$: the rule is proper. If $G$ is strictly convex and $p \ne q$, the inequality is strict. Suppose instead that $G(q) = G(p) + \langle g(p), q - p\rangle$, and let $m = (p + q)/2$. The subgradient inequality for $g(p)$, applied at the point $m$, gives $G(m) \ge G(p) + \langle g(p), m - p\rangle = G(p) + \tfrac12\langle g(p), q - p\rangle = \tfrac12 G(p) + \tfrac12 G(q)$, which contradicts strict convexity.

Only if. Suppose the rule is proper, and define $G(q) = \bar{R}(q, q)$. Properness says $G(q) = \max_p \bar{R}(p, q)$. For each fixed $p$, $\bar{R}(p, q)$ is a linear function of $q$. A maximum of linear functions is convex: for any $p$, $\bar{R}(p, \lambda q + (1 - \lambda)q') = \lambda\bar{R}(p, q) + (1 - \lambda)\bar{R}(p, q') \le \lambda G(q) + (1 - \lambda)G(q')$, and taking $p = \lambda q + (1 - \lambda)q'$ gives convexity. Now let $g(p)$ be the vector with components $R(p, j)$. For any $q$, properness and linearity give

$$G(q) \ge \bar{R}(p, q) = \sum_j q_j R(p, j) = \langle g(p), q\rangle = G(p) + \langle g(p), q - p\rangle,$$

using $\langle g(p), p\rangle = \bar{R}(p, p) = G(p)$. So $g(p)$ is a subgradient of $G$ at $p$, and $R(p, j) = g_j(p) = G(p) + g_j(p) - \langle g(p), p\rangle$, the stated form. If the rule is strictly proper, $G$ is strictly convex: for $p \ne q$, $0 < \lambda < 1$ and $m = \lambda p + (1 - \lambda)q \ne p, q$, strict properness gives $G(m) = \bar{R}(m, m) = \lambda\bar{R}(m, p) + (1 - \lambda)\bar{R}(m, q) < \lambda G(p) + (1 - \lambda)G(q)$. $\square$

Example (the Brier and log scores from their functions). $G(p) = \sum_i p_i^2 - 1$ has gradient $g_j = 2p_j$, and the formula gives $R(p, j) = \sum_i p_i^2 - 1 + 2p_j - 2\sum_i p_i^2 = 2p_j - \sum_i p_i^2 - 1$, which is minus the Brier loss of Theorem 2's proof. Gneiting and Raftery (2007, Examples 1 and 3) give this and the log score, which comes from $G(p) = \sum_i p_i \ln p_i$, the negative Shannon entropy.

Example (a proper rule that is not strictly proper). The zero-one score, which charges 1 unless the outcome is the forecast's most probable option, is proper but not strictly proper (Gneiting and Raftery, 2007, Example 4). Worked problem 2 derives it from $G(p) = \max_j p_j$. Accuracy is this score: it rewards getting the most likely answer right and is indifferent to the probabilities themselves.

Example (the ECE is not a scoring rule). The expected calibration error, defined below, is computed from a whole set of forecasts, not as an average of losses on single cases, and it is zero for the constant forecast at the base rate. Minimizing it rewards calibration without resolution. It is a diagnostic, not a score to choose models by.

What a proper score rewards

A proper score is minimized by the true conditional probabilities. How far does a real forecaster fall short, and why? The answer is a three-part split, proved here for any proper rule.

Let $P$ be the forecast (random, because the question is random) and $Y$ the outcome, and let $\Pi$ be the conditional distribution of $Y$ given $P$, so $\Pi_j = P(Y = j \mid P)$. By Theorem 1 $\Pi$ is itself a calibrated forecast. Let $\bar{\pi} = E[\Pi]$, the overall distribution of the outcome, which the meteorological literature calls the climatology.

Theorem 5 (uncertainty, resolution and reliability). For any proper scoring rule with finite expected loss,

$$E[L(P, Y)] = \underbrace{e(\bar{\pi})}_{\text{uncertainty}} - \underbrace{E[d(\bar{\pi}, \Pi)]}_{\text{resolution}} + \underbrace{E[d(P, \Pi)]}_{\text{reliability}}.$$

The uncertainty term depends only on the outcome; resolution, which is never negative, lowers the loss; reliability, never negative, raises it, and is zero exactly when $P = \Pi$, that is, when the forecast is calibrated (for a strictly proper rule).

This is Bröcker's (2009) decomposition, his Equation 13. Here is the idea. Conditioning on the forecast splits the expected loss into the loss of the calibrated forecast $\Pi$ plus the penalty for stating $P$ instead, and the loss of $\Pi$ is the uncertainty minus the amount by which $\Pi$ varies around the climatology.

Proof. By the law of total expectation, $E[L(P, Y)] = E\big[E[L(P, Y) \mid P]\big]$, and given $P$ the outcome has distribution $\Pi$, so $E[L(P, Y) \mid P] = S(P, \Pi)$. By the definitions of $d$ and $e$, $S(P, \Pi) = e(\Pi) + d(P, \Pi)$, so

$$E[L(P, Y)] = E[e(\Pi)] + E[d(P, \Pi)].$$

For the first term, $S(\bar{\pi}, q)$ is linear in $q$, so $E[S(\bar{\pi}, \Pi)] = S(\bar{\pi}, E[\Pi]) = S(\bar{\pi}, \bar{\pi}) = e(\bar{\pi})$. Hence $E[e(\Pi)] = E[S(\bar{\pi}, \Pi)] - E[S(\bar{\pi}, \Pi) - S(\Pi, \Pi)] = e(\bar{\pi}) - E[d(\bar{\pi}, \Pi)]$. Both divergences are nonnegative because the rule is proper. $\square$

For the Brier score, Theorem 2 makes each term explicit: reliability is $E\sum_i(P_i - \Pi_i)^2$, resolution $E\sum_i(\Pi_i - \bar{\pi}_i)^2$, the variance of the calibrated forecast, and uncertainty $1 - \sum_i \bar{\pi}_i^2$. Bröcker notes that the decomposition is well known for the Brier score, for which he cites forecast verification work by Murphy and Winkler, among others. On data, $\Pi$ is unknown, and the standard empirical version groups forecasts that take the same value.

Corollary (the decomposition on data). Suppose $n$ binary forecasts take finitely many values $f_1, \dots, f_K$, with $n_k$ forecasts equal to $f_k$, of which a share $\bar{o}_k$ came true, and overall share $\bar{o}$. Then the average binary Brier score $\frac{1}{n}\sum (f - y)^2$ equals

$$\underbrace{\sum_k \frac{n_k}{n}(f_k - \bar{o}_k)^2}_{\text{REL}} - \underbrace{\sum_k \frac{n_k}{n}(\bar{o}_k - \bar{o})^2}_{\text{RES}} + \underbrace{\bar{o}(1 - \bar{o})}_{\text{UNC}}.$$

Proof. Apply Theorem 5 to the empirical distribution of the $n$ (forecast, outcome) pairs, under which $\Pi$ given $P = f_k$ is $\bar{o}_k$ and $\bar{\pi} = \bar{o}$, with the binary loss $(f - y)^2$, whose divergence is $(f - \pi)^2$ and entropy $\pi(1 - \pi)$ (half of Theorem 2's values for $k = 2$). $\square$

HELM's confidences take many values, so the script first replaces each confidence by the mean of its bin among ten equal-width bins, which changes the Brier score in the third decimal, and then applies the corollary exactly:

Model Reliability Resolution Uncertainty Brier (binned) Brier (raw)
davinci 0.0028 0.0354 0.2430 0.2103 0.2093
curie 0.0116 0.0013 0.1830 0.1933 0.1939
text-davinci-002 0.0371 0.0747 0.2459 0.2083 0.2072
text-davinci-003 0.1203 0.0434 0.2468 0.3238 0.3209

Example (the lowest Brier score is not the best forecaster). Of the four, curie has the lowest Brier score, 0.194, although it is the least able to tell its right answers from its wrong ones (resolution 0.0013) and is worse calibrated than davinci (reliability 0.0116 against 0.0028). Its advantage is the uncertainty term. Curie is right on only 0.241 of the questions, so the event it forecasts is less uncertain, $0.241 \times 0.759 = 0.183$ against 0.243 for davinci. Brier scores of models with different accuracies, or on different evaluations, mix forecasting quality with the difficulty of the event; Gneiting and Raftery (2007, Section 2.3) make the same warning about comparing scores across different sets of forecast situations. The decomposition separates them.

Text-davinci-002 has the most resolution of the four (0.0747) and much worse reliability than davinci; its Brier score is about davinci's because the two effects nearly cancel. Text-davinci-003 resolves less than text-davinci-002 and is far less reliable, and both proper scores rank it last.

Measuring calibration: reliability diagrams and the ECE

The reliability term is a calibration measure, but most evaluations report a different one. A reliability diagram sorts the forecasts into bins by confidence and plots, for each bin $b$, the share correct $\bar{o}_b$ against the mean confidence $\bar{c}_b$. The expected calibration error summarizes the diagram as

$$\text{ECE} = \sum_{b} \frac{n_b}{n}\,\lvert \bar{o}_b - \bar{c}_b \rvert,$$

the weighted mean absolute gap. Naeini, Cooper and Hauskrecht (2015) defined it, with 10 bins in their experiments; Guo et al. (2017, Equation 3) define it with $M$ equal-width bins and report it with 15. Kadavath et al. (2022, Section 1) use ten bins with equal numbers of predictions, as the library's note on their paper records.

Reliability diagrams (ten equal-width bins; the area of each point is proportional to the number of questions in it) for three HELM models on 514 MMLU questions, and for Qwen2.5-0.5B on 2,000 held-out questions before and after temperature scaling. Notice that davinci's points scatter around the diagonal, text-davinci-003 sits far below it, with most questions in its top bin (overconfident), and temperature scaling brings the small model's points closer to the diagonal without changing any answer. The last panel shows the median ECE (fifteen bins) of a perfectly calibrated model with davinci's or text-davinci-003's confidences, against the number of questions: it falls like one over the square root of n and is about 0.1 at 100 questions.
Reliability diagrams (ten equal-width bins; the area of each point is proportional to the number of questions in it) for three HELM models on 514 MMLU questions, and for Qwen2.5-0.5B on 2,000 held-out questions before and after temperature scaling. Notice that davinci's points scatter around the diagonal, text-davinci-003 sits far below it, with most questions in its top bin (overconfident), and temperature scaling brings the small model's points closer to the diagonal without changing any answer. The last panel shows the median ECE (fifteen bins) of a perfectly calibrated model with davinci's or text-davinci-003's confidences, against the number of questions: it falls like one over the square root of n and is about 0.1 at 100 questions.

The ECE depends on the bins. For GPT-NeoX (20B) it is 0.015 with ten equal-width bins, 0.044 with fifteen and 0.038 with ten bins of equal mass; for OPT (175B), 0.014, 0.062 and 0.081. With 514 questions, the choice of bins changes the ECE by a factor of up to six. For the badly calibrated models the choice barely matters: text-davinci-003 has ECE 0.328 under all three. Two facts explain this, and they push in opposite directions.

Binning hides miscalibration

Write $\Pi(C) = P(Y = 1 \mid C)$ for the true accuracy at confidence $C$. The population calibration error is $\text{CE} = E\lvert \Pi(C) - C\rvert$. A binned measure replaces it with the gap of bin averages.

Theorem 6 (binning can only lower the calibration error). For any partition of the confidence range into bins $B$,

$$\sum_b P(C \in b)\,\big\lvert E[\Pi(C) - C \mid C \in b]\big\rvert \;\le\; E\lvert \Pi(C) - C\rvert.$$

Proof. For any random variable $X$, $\lvert E[X]\rvert \le E\lvert X\rvert$, applied conditionally on each bin to $X = \Pi(C) - C$. Summing with weights $P(C \in b)$ and using the law of total expectation on the right gives the result. $\square$

A model can be overconfident in the lower half of a bin and underconfident in the upper half, and the bin average shows neither. Kumar, Liang and Ma (2019, Proposition 3.3) prove the same inequality, by Jensen's inequality, for every $\ell_p$ calibration error.

Sampling noise creates apparent miscalibration

The opposite bias is larger in evaluations of a few hundred questions. Here is the idea. Even a perfectly calibrated model has, in each bin, a share correct that differs from its confidence by sampling noise, and an absolute value of noise is positive on average.

Theorem 7 (the ECE of a calibrated forecaster). Fix the confidences $c_1, \dots, c_n$ and the bins, and let the outcomes be independent with $P(Y_i = 1) = c_i$, so that the forecaster is calibrated. In bin $b$ the gap $\bar{o}_b - \bar{c}_b$ has mean 0 and variance $V_b = \sum_{i \in b} c_i(1 - c_i)/n_b^2$, and

$$E[\text{ECE}] = \sum_b \frac{n_b}{n}\,E\lvert \bar{o}_b - \bar{c}_b\rvert,$$

which is positive unless every confidence is 0 or 1. When each bin holds many questions,

$$E[\text{ECE}] \approx \sqrt{\frac{2}{\pi}}\sum_b \frac{n_b}{n}\sqrt{V_b},$$

which for a fixed set of bins falls like $1/\sqrt{n}$.

Proof. $E[\bar{o}_b] = \frac{1}{n_b}\sum_{i \in b} c_i = \bar{c}_b$, and the variance of a mean of independent Bernoulli variables is the sum of their variances divided by $n_b^2$. The ECE is a weighted sum of the $\lvert \bar{o}_b - \bar{c}_b\rvert$, so its expectation is the weighted sum of their expectations. A variable with mean 0 that is not always 0 has a positive expected absolute value, and $\bar{o}_b - \bar{c}_b$ is not always 0 once some $c_i$ in the bin lies strictly between 0 and 1. For the approximation, by the central limit theorem $\bar{o}_b - \bar{c}_b$ is approximately normal with standard deviation $\sqrt{V_b}$, and for a standard normal $Z$, $E\lvert Z\rvert = 2\int_0^\infty z\,\phi(z)\,dz = 2\phi(0) = \sqrt{2/\pi}$, where $\phi$ is the standard normal density, since $\int_0^\infty z\,\phi(z)\,dz = [-\phi(z)]_0^\infty = \phi(0)$. Each $V_b$ is a sum of $n_b$ terms divided by $n_b^2$, so $\sqrt{V_b}$ shrinks like $1/\sqrt{n_b}$, and with a fixed number of bins each $n_b$ grows in proportion to $n$. $\square$

The exact expectation needs no approximation: the number correct in a bin is a sum of independent Bernoulli variables with different probabilities, whose exact distribution the script builds one question at a time. Roelofs et al. (2022) call this bias of the binned ECE "most severe for perfectly calibrated models" and measure it by simulation; Theorem 7 gives it exactly for any given set of confidences.

Is davinci calibrated? With davinci's own 514 confidences and its ten bins, a perfectly calibrated model would have an expected ECE of 0.040, which is exactly davinci's observed ECE to three decimals. The model of Theorem 7, independent outcomes with the stated probabilities, turns this into a test of the hypothesis that davinci is calibrated, conditional on its confidences. Simulate outcomes from the confidences, compute the ECE each time, and see where the observed value falls. With 9,999 simulated evaluations (seed 41,000,000) the Monte Carlo p-value is $(1 + \#\{\text{simulated ECE} \ge \text{observed}\})/(9{,}999 + 1) = 0.49$. Such a p-value is valid for the same reason as a permutation test's, proved in Permutation Tests: under the hypothesis the observed and simulated statistics are exchangeable. The last column of the table at the top gives the same test for all eight models (seed 41,000,001 for the other seven). It gives 0.71 for GPT-NeoX (20B), 0.85 for OPT (175B) and 0.23 for BLOOM (176B), so their ECEs, small with ten bins and larger with other bins, are all consistent with perfect calibration. For curie, text-davinci-002 and text-davinci-003 the p-value is 0.0001, the smallest possible with 9,999 simulations: no simulated ECE came near the observed one. GPT-J (6B), whose ECE of 0.069 looks modest, has p = 0.0023: its confidence (0.318 on average) exceeds its accuracy (0.249) by more than noise allows.

How large is the bias at other sizes? The script draws evaluations of $n$ questions, with confidences resampled from a model's 514 and outcomes drawn as if the model were calibrated, and computes the ECE with fifteen equal-width bins (100 evaluations for each of 30 seeds, 42,000,000 to 42,000,029; medians over seeds):

Questions $n$ 100 1,000 10,000
ECE of a calibrated model, davinci's confidences 0.102 0.033 0.010
ECE of a calibrated model, text-davinci-003's confidences 0.065 0.021 0.007

On an evaluation of 100 questions, a perfectly calibrated model with davinci's confidences shows an ECE of about 0.10. An ECE below that on such an evaluation says nothing about calibration. The bias depends on the confidences too: text-davinci-003's are mostly near 1, where Bernoulli outcomes vary little, so its noise floor is lower.

A debiased estimate

The bias can be removed in expectation for the squared version of the calibration error. Kumar et al. (2019, Definition 5.2), following earlier meteorological work, subtract an estimate of each bin's sampling variance:

$$\widehat{\text{CE}^2} = \sum_b \frac{n_b}{n}\left[(\bar{o}_b - \bar{c}_b)^2 - \frac{\bar{o}_b(1 - \bar{o}_b)}{n_b - 1}\right].$$

Theorem 8 (what the correction removes). In a bin with $n_b \ge 2$ forecasts whose outcomes are independent with $P(Y_i = 1) = q_i$, write $\bar{q}_b$ for the mean of the $q_i$ and $s_b^2 = \frac{1}{n_b}\sum_{i \in b}(q_i - \bar{q}_b)^2$ for their spread. Then

$$E\left[(\bar{o}_b - \bar{c}_b)^2 - \frac{\bar{o}_b(1 - \bar{o}_b)}{n_b - 1}\right] = (\bar{q}_b - \bar{c}_b)^2 - \frac{s_b^2}{n_b - 1}.$$

So the corrected term is unbiased for the squared gap of the bin when the true probabilities within the bin are equal, and slightly too small otherwise.

Proof. Drop the subscript $b$ and write $V = \operatorname{Var}(\bar{o}) = \frac{1}{n^2}\sum_i q_i(1 - q_i)$. Since $\frac{1}{n}\sum_i q_i^2 = \bar{q}^2 + s^2$, this is $V = (\bar{q} - \bar{q}^2 - s^2)/n$. First, $E[(\bar{o} - \bar{c})^2] = (\bar{q} - \bar{c})^2 + V$, because $E[\bar{o}] = \bar{q}$. Second, $E[\bar{o}(1 - \bar{o})] = \bar{q} - E[\bar{o}^2] = \bar{q} - \bar{q}^2 - V$. Substituting $\bar{q} - \bar{q}^2 = nV + s^2$ gives $E[\bar{o}(1 - \bar{o})] = (n - 1)V + s^2$, so

$$E\left[\frac{\bar{o}(1 - \bar{o})}{n - 1}\right] = V + \frac{s^2}{n - 1},$$

and subtracting from the first expectation leaves $(\bar{q} - \bar{c})^2 - s^2/(n - 1)$. $\square$

For the HELM models, with ten equal-width bins, the square roots of the plug-in squared error $\sum_b (n_b/n)(\bar{o}_b - \bar{c}_b)^2$ and of the debiased estimate are:

Model Plug-in Debiased
davinci 0.052 0 (estimate below 0)
curie 0.108 0.102
text-davinci-002 0.193 0.185
text-davinci-003 0.347 0.343
GPT-J (6B) 0.074 0.066
GPT-NeoX (20B) 0.019 0 (estimate below 0)
OPT (175B) 0.014 0 (estimate below 0)
BLOOM (176B) 0.042 0.021

A bin with a single question has no variance estimate, since $n_b - 1 = 0$, so its plug-in term is kept without correction in the debiased sum, as in the plug-in sum; BLOOM is the only model here with such a bin (one question at confidence 0.524, answered correctly). An estimate below zero is what an (almost) unbiased estimator of a quantity at or near zero often produces, and for a calibrated forecaster Theorem 8 gives it a slightly negative expectation when every bin has at least two questions (an uncorrected one-question bin adds a positive expectation); it is reported as 0. The correction barely moves the clearly miscalibrated models. For the three models whose test above found no evidence of miscalibration and whose debiased estimate is below zero (davinci, GPT-NeoX, OPT) it removes all of the apparent error, and for BLOOM half of it, from 0.042 to 0.021; BLOOM's remaining error comes largely from its one-question bin, whose term cannot be corrected. Under perfect calibration that bin's term would be expected to be $c(1 - c)/n = 0.524 \times 0.476/514 = 0.00049$, about BLOOM's whole debiased squared error of 0.00046, so the 0.021 is itself consistent with no miscalibration, in line with BLOOM's test p-value of 0.23. For these four the normal approximation of Theorem 7 gives an expected ECE under perfect calibration of 0.040 (davinci), 0.023 (GPT-NeoX), 0.026 (OPT) and 0.027 (BLOOM), in agreement with the exact values.

Repairing calibration: temperature scaling

A miscalibrated model can be recalibrated after training by fitting a map from its scores to probabilities on held-out questions. Binary Classification Metrics describes Platt scaling and isotonic regression. For a model that outputs a probability vector through a softmax, the simplest map is temperature scaling (Guo et al., 2017, Section 4.2, Equation 9). Write the model's probabilities as $p_j = e^{z_j}/\sum_i e^{z_i}$, with $z_j$ the log-probability (or any logit) of option $j$. The rescaled probabilities are

$$p_j(T) = \frac{e^{z_j/T}}{\sum_i e^{z_i/T}}, \qquad T > 0.$$

A temperature above 1 flattens the distribution and lowers the confidence; below 1 it sharpens it. Guo et al. choose $T$ by minimizing the log loss on a validation set, and so do we.

Theorem 9 (temperature scaling changes no answer). For every $T > 0$ and every question, $p_j(T) > p_k(T)$ if and only if $p_j > p_k$. So the chosen answer, and with it the accuracy, is the same at every temperature. The order of questions by confidence, however, can change.

Proof. $p_j(T)/p_k(T) = e^{(z_j - z_k)/T}$, which exceeds 1 exactly when $z_j > z_k$, whatever the positive $T$, and $z_j > z_k$ exactly when $p_j > p_k$. For the second statement it is enough to give one case. Take two questions with logits $(1, 0, 0, 0)$ and $(2, 2, -10, -10)$. At $T = 1$ their confidences are 0.475 and 0.500; at $T = 0.5$ they are 0.711 and 0.500, so the order of the two confidences reverses. $\square$

The second part matters for evaluation: a measure that depends only on the ranking of questions by confidence, such as the area under the ROC curve for separating right from wrong answers, can change under temperature scaling, while accuracy cannot.

Qwen2.5-0.5B before and after. The script splits the 4,000 questions at random (seed 43,000,000) into 2,000 for fitting $T$ and 2,000 for evaluation. Over all 4,000 the model answers 0.480 correctly with mean confidence 0.530, and the letters other than A to D take a median of only 0.003 of its probability, so renormalizing over the four letters changes little. The log loss on the fitting half is smallest at $T = 1.295$: the model is overconfident, and a temperature above 1 softens it. On the 2,000 held-out questions:

Accuracy Mean confidence ECE (equal width) ECE (equal mass) Brier (4 options) Log loss
$T = 1$ 0.467 0.530 0.063 0.063 0.621 1.154
$T = 1.295$ 0.467 0.482 0.038 0.035 0.616 1.143

The accuracy is unchanged, as Theorem 9 requires. The mean confidence falls toward the accuracy, the ECE falls by 39%, and both proper scores improve, though only in the third decimal: the miscalibration was mild, and the uncertainty term of the decomposition, which no recalibration can touch, dominates both scores. Is the rescaled model calibrated? A perfectly calibrated model with the rescaled confidences would show an expected ECE of 0.021 on these 2,000 questions (Theorem 7), and the Monte Carlo test of the previous section gives a p-value of 0.0084 for the observed 0.038 (0.0001 before scaling). On these questions one temperature removed much of the miscalibration but not all of it. A single parameter can only flatten or sharpen every distribution alike; a model whose miscalibration has a different shape needs a richer map, such as those of Binary Classification Metrics.

What is proved, what is cited, what the experiment shows

Proved here. That conditional probabilities are calibrated (Theorem 1); the strict properness of the Brier and log scores, with their divergences and entropies (Theorems 2 and 3); the McCarthy–Savage characterization for rules with finite losses (Theorem 4); the decomposition of any proper score into uncertainty, resolution and reliability, and its empirical Brier form (Theorem 5 and corollary); that binning can only lower the calibration error (Theorem 6); the expected ECE of a calibrated forecaster, exactly and approximately (Theorem 7); the bias left by the debiased squared estimator (Theorem 8); that temperature scaling preserves every answer but not the order of confidences (Theorem 9).

Cited. Gneiting and Raftery (2007) for the definitions in reward form, the characterization including rules that take infinite values, and the zero-one score; Bröcker (2009) for the general decomposition; Guo et al. (2017), Naeini et al. (2015) and Kumar et al. (2019) for the definitions of the ECE, temperature scaling and the debiased estimator; Roelofs et al. (2022) for the bias of the ECE measured by simulation; Gibbs' inequality and the tower property from the linked pages.

What the experiments show. On HELM Classic's 514 questions, the confidences of davinci, GPT-NeoX, OPT and BLOOM are consistent with perfect calibration, while curie, GPT-J, text-davinci-002 and text-davinci-003 are overconfident (mean confidence above accuracy, calibration test p from 0.0001 to 0.0023); the ECE of the well-calibrated models depends strongly on the bins, and its size is close to the noise floor of a calibrated model. For Qwen2.5-0.5B, temperature scaling fitted on 2,000 questions lowered the ECE on 2,000 others from 0.063 to 0.038, left accuracy unchanged, and still left detectable miscalibration. These describe these models, these questions and this prompt format. The HELM confidences are the probabilities of the generated letter over the whole vocabulary; the small model's are renormalized over the four letters.

Worked problems

These problems were written for this article; the first and fourth use the HELM results above. The script tools/experiments/statistics/calibration-and-proper-scoring-rules.py recomputes every number.

Problem 1 (a Brier decomposition by hand). Group text-davinci-003's 514 confidences into three bins, below 0.5, 0.5 to 0.9 and 0.9 or more, and replace each confidence by its bin's mean. The bins hold 27, 141 and 346 questions, with mean confidences 0.439, 0.716 and 0.988, and the model was right on 8, 36 and 242 of them. Compute the reliability, resolution and uncertainty, and the Brier score of the binned forecasts.

Solution. Reliability 0.1155, resolution 0.0422, uncertainty 0.2468, Brier score 0.3201.

The shares correct are $8/27 = 0.296$, $36/141 = 0.255$ and $242/346 = 0.699$, and overall $286/514 = 0.5564$. By the corollary,

$$\text{REL} = \tfrac{27}{514}(0.439 - 0.296)^2 + \tfrac{141}{514}(0.716 - 0.255)^2 + \tfrac{346}{514}(0.988 - 0.699)^2 = 0.1155,$$

$$\text{RES} = \tfrac{27}{514}(0.296 - 0.5564)^2 + \tfrac{141}{514}(0.255 - 0.5564)^2 + \tfrac{346}{514}(0.699 - 0.5564)^2 = 0.0422,$$

and $\text{UNC} = 0.5564 \times 0.4436 = 0.2468$ (the script uses unrounded values). The Brier score of the binned forecasts is $0.1155 - 0.0422 + 0.2468 = 0.3201$, close to the 0.3209 of the raw confidences. Reliability is more than a third of the score (0.36 of it): the model is right on only a quarter of the questions where it states about 72%, and on 70% of those where it states about 99%. Its middle bin is right less often than its lowest one, so a recalibration that keeps the order of the confidences cannot fix that bin's gap without merging it with the lowest.

Problem 2 (the zero-one score). Derive the scoring rule that Theorem 4 assigns to $G(p) = \max_j p_j$, and show that it is proper but not strictly proper.

Solution. It is the zero-one score: loss 0 if the outcome is the forecast's most probable option and 1 otherwise.

$G$ is a maximum of the linear functions $p \mapsto p_j$, so it is convex. Let $m$ be an option with $p_m = \max_j p_j$, and let $g = e_m$, the vector with 1 in place $m$ and 0 elsewhere. For any $q$, $G(q) \ge q_m = p_m + (q_m - p_m) = G(p) + \langle e_m, q - p\rangle$, so $e_m$ is a subgradient at $p$. Theorem 4 gives the reward $G(p) + g_j - \langle g, p\rangle = p_m + \mathbf{1}\{j = m\} - p_m = \mathbf{1}\{j = m\}$, and the loss $1 - \mathbf{1}\{j = m\}$. By the theorem the rule is proper. It is not strictly proper: if $q = (0.5, 0.3, 0.2, 0)$ is the truth, both $q$ and $p = (0.4, 0.3, 0.2, 0.1)$ have mode 1 and the same expected loss, $1 - 0.5 = 0.5$. Equivalently, $G$ is linear, not strictly convex, on the set of vectors with a given mode. (At a tie among several modes, any average of their unit vectors is a subgradient; Gneiting and Raftery (2007, Example 4) split the reward equally among the modes.)

Problem 3 (the ECE of a perfectly calibrated forecaster). A calibrated forecaster states 0.7 on every one of $n$ questions, so all fall in one bin. Find the expected ECE exactly for $n = 10$ and $n = 100$, and compare with Theorem 7's approximation.

Solution. Exactly 0.1121 for $n = 10$ and 0.0364 for $n = 100$; the approximation gives 0.1156 and 0.0366.

With one bin the ECE is $\lvert X/n - 0.7\rvert$, where $X$, the number correct, is binomial with $n$ trials and probability 0.7, since the forecaster is calibrated. So

$$E[\text{ECE}] = \sum_{x=0}^{n} \binom{n}{x}0.7^x\,0.3^{n-x}\left\lvert \frac{x}{n} - 0.7\right\rvert,$$

which the script sums in exact fractions. Theorem 7's approximation is $\sqrt{2/\pi}\sqrt{0.7 \times 0.3/n}$, that is $0.7979 \times 0.1449 = 0.1156$ for $n = 10$ and $0.7979 \times 0.04583 = 0.0366$ for $n = 100$. A forecaster that is right exactly as often as it says is still charged an average gap of 11 points on ten questions and almost 4 on a hundred.

Problem 4 (debiasing one bin). Davinci's highest occupied bin, confidences from 0.9 to 1.0, holds 5 questions with mean confidence 0.9148, of which 3 were answered correctly. Compute the bin's plug-in squared gap, the correction and the debiased value.

Solution. Plug-in 0.0991, correction 0.0600, debiased 0.0391.

The share correct is $\bar{o} = 3/5 = 0.6000$, so the plug-in squared gap is $(0.6000 - 0.9148)^2 = 0.0991$. The correction is $\bar{o}(1 - \bar{o})/(n_b - 1) = 0.6 \times 0.4/4 = 0.0600$, an estimate of the variance of $\bar{o}$ by Theorem 8. The debiased value is $0.0991 - 0.0600 = 0.0391$, a gap of about 0.20 rather than 0.31. With five questions in a bin, the share correct can move by 0.2 on one answer, and more than half of the plug-in squared gap is noise. In the full estimate the bin's term is weighted by $5/514$, so its contribution to the ECE is small either way.

Summary

Quantity Formula
Calibration (confidence) $P(Y = 1 \mid C = c) = c$
Proper scoring rule $S(q, q) \le S(p, q)$ for all $p, q$
Brier score $S(p, q) = \sum_i (p_i - q_i)^2 + 1 - \sum_i q_i^2$
Log score $S(p, q) = H(q) + D(q \,\|\, p)$
Characterization $L(p, j) = -[G(p) + g_j(p) - \langle g(p), p\rangle]$, $G$ convex
Decomposition $E[L] = e(\bar{\pi}) - E[d(\bar{\pi}, \Pi)] + E[d(P, \Pi)]$
ECE $\sum_b (n_b/n)\lvert\bar{o}_b - \bar{c}_b\rvert$
ECE of a calibrated forecaster $\approx \sqrt{2/\pi}\sum_b (n_b/n)\sqrt{V_b}$
Debiased squared error $\sum_b (n_b/n)[(\bar{o}_b - \bar{c}_b)^2 - \bar{o}_b(1 - \bar{o}_b)/(n_b - 1)]$
Temperature scaling $p_j(T) \propto e^{z_j/T}$

What we have learned

A probability is calibrated when it equals the frequency of being right among the cases where it is stated, and every conditional probability is calibrated, so calibration alone is cheap. Proper scoring rules, which come exactly from convex functions, reward honest probabilities, and every one of them splits into the uncertainty of the outcome, the resolution of the forecasts and their reliability, so that calibration is one of two things they reward. The expected calibration error is a useful picture but a biased number: binning hides miscalibration, and sampling noise invents it, by about 0.04 for 514 questions with davinci's confidences, enough to account for all of davinci's observed error. Temperature scaling repairs overconfidence without changing an answer.

Calibrated probabilities have a further use. When a model's probability of a behavior can be computed for each prompt, the rate of a behavior too rare to observe can still be estimated, which is the subject of Estimating Rare Event Rates.

References

  • Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3. record · 2 pages
  • Bröcker, J. (2009). Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society, 135(643), 1512–1519. https://doi.org/10.1002/qj.456. arXiv:0806.0813. record
  • Gneiting, T., and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378. https://doi.org/10.1198/016214506000001437 record · 2 pages
  • Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, PMLR 70 (pp. 1321–1330). arXiv:1706.04599. record
  • Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021). Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR). arXiv:2009.03300. record · 7 pages
  • Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., et al. (2022). Language Models (Mostly) Know What They Know. arXiv:2207.05221. record · 6 pages
  • Kumar, A., Liang, P., and Ma, T. (2019). Verified uncertainty calibration. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019). arXiv:1909.10155. record
  • Liang, P., Bommasani, R., Lee, T., et al. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research. arXiv:2211.09110. record · 6 pages
  • McCarthy, J. (1956). Measures of the value of information. Proceedings of the National Academy of Sciences, 42(9), 654–655. https://doi.org/10.1073/pnas.42.9.654 record
  • Naeini, M. P., Cooper, G. F., and Hauskrecht, M. (2015). Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (pp. 2901–2907). record
  • Roelofs, R., Cain, N., Shlens, J., and Mozer, M. C. (2022). Mitigating bias in calibration error estimation. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 151 (pp. 4036–4054). arXiv:2012.08668. record
  • Savage, L. J. (1971). Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336), 783–801. https://doi.org/10.1080/01621459.1971.10482346 record
  • Yang, A., et al. (Qwen Team) (2024). Qwen2.5 technical report. arXiv:2412.15115. Model used: Qwen/Qwen2.5-0.5B, Hugging Face revision 060db6499f32faf8b98477b0a26969ef7d8b9987. record · 2 pages