Loss Functions for Classification

Measuring the quality of predicted probabilities rather than decisions. The 0-1 loss and why it cannot train a model, absolute loss, the Brier score and log loss case by case, proper scoring rules derived and why absolute loss rewards overconfidence, five breast cancer classifiers compared on training and test data, and the margin view of the logistic, hinge, exponential and squared losses used to train classifiers.

Two classifiers are tested on the same 190 breast masses. Both are correct on about nine masses in ten. The first, when it is wrong, says something like "a 36% chance of malignancy" for a mass that turns out to be malignant. The second, on one of its errors, assigns a malignant mass log-odds of $-167$: a probability of malignancy of about $10^{-73}$. Counting correct decisions cannot tell these two apart. A loss function for classification looks at the probability each classifier gave to what actually happened, and the second classifier's single confident mistake costs more than the first classifier's entire loss over all 190 masses.

Binary Classification Metrics judged classifiers by their decisions at a cutoff, and by how well their scores rank cases. This article judges the probabilities themselves. The same loss functions appear twice in practice: as the criterion a training algorithm minimizes, and as a measure for comparing fitted models on test data. Both uses are covered here. The examples use the breast cancer biopsy data and the stratified split of that article: 379 masses for training and 190 for testing, malignant coded 1.

The 0-1 loss and why it is not enough

The most direct loss charges 1 for a wrong decision and 0 for a right one. Its average over a test set is the error rate, one minus accuracy. It is the loss a classifier ultimately answers to when decisions are all that matter and both errors cost the same, and the bias-variance tradeoff behaves differently under it than under squared error.

It has two shortcomings. As a measure, it ignores confidence: a malignant mass given probability 0.49 and one given 0.0001 are equally wrong. As a training criterion, it is unusable for most methods. Changing a model's coefficients slightly changes the error count only when some case crosses the cutoff, so the 0-1 loss is flat almost everywhere and jumps where it is not, and gradient-based fitting gets no signal from it. Minimizing the training error rate directly is computationally intractable for many classes of classifiers (Bartlett, Jordan and McAuliffe, 2006, §1). Methods therefore minimize a smooth surrogate loss instead, and the choice of surrogate is much of what distinguishes logistic regression, support vector machines and boosting, as the last section shows.

Losses on probabilities

Let $y \in \{0, 1\}$ be the observed class and $p$ the predicted probability that $y = 1$. Three losses compare the two.

The absolute loss is $|y - p|$: the distance between the probability and the outcome. The squared loss is $(y - p)^2$; its average over cases is the Brier score, proposed by Brier (1950) for verifying probabilistic weather forecasts. The log loss, also called cross-entropy, is

$$L(y, p) = -\bigl[\,y \ln p + (1 - y) \ln(1 - p)\,\bigr],$$

which is $-\ln p$ when $y = 1$ and $-\ln(1 - p)$ when $y = 0$: the negative log of the probability given to what happened. All three are 0 for a certain and correct prediction, and all three depend on the probability, not on any cutoff.

Three test masses, scored by the three-feature logistic model of Binary Classification Metrics, show how they differ:

Mass Class $p$ Decision at 0.5 Absolute Squared Log
A benign (0) 0.333 correct 0.333 0.111 0.405
B malignant (1) 0.952 correct 0.048 0.002 0.049
C malignant (1) 0.358 wrong 0.642 0.412 1.028

Mass A was classified correctly but with a one-in-three chance of malignancy, and every loss charges for that hesitation. Mass C was missed, and the log loss charges it more than twice as much as mass A, while the absolute loss charges it less than twice as much.

The three losses for one case, as a function of the probability given to the class that occurred. Absolute and squared loss are bounded by 1. Log loss is unbounded: it rises without limit as the probability given to the actual outcome approaches 0.
The three losses for one case, as a function of the probability given to the class that occurred. Absolute and squared loss are bounded by 1. Log loss is unbounded: it rises without limit as the probability given to the actual outcome approaches 0.

The difference grows with confidence. For a benign mass predicted malignant with probability 0.9, 0.99, 0.999 and 0.9999, the log loss is 2.30, 4.61, 6.91 and 9.21: each tenfold cut in the probability left for the actual outcome adds the same 2.30. The squared loss is 0.81, 0.98, 0.998 and 0.9998, never more than 1. Log loss treats a confident error as far worse than a hesitant one; the Brier score treats it as somewhat worse.

The log loss has two further readings. It is the negative log-likelihood of the outcome under the model, so minimizing average log loss on training data is maximum likelihood, the criterion that fits logistic regression. And in information theory, $-\ln p$ is the surprise of an event of probability $p$, measured in nats (natural-log units; dividing by $\ln 2$ gives bits). With more than two classes the log loss is $-\ln p_{\text{true class}}$, the same quantity used to train the character model of The Bigram Language Model.

Proper scoring rules

Which loss should be used to compare probabilities? The deciding property is what each loss encourages. Suppose that for some mass the true probability of malignancy is $q$, and a forecaster reports $r$. The expected loss of the report is the loss if malignant, weighted by $q$, plus the loss if benign, weighted by $1 - q$. A loss is proper if the expected loss is minimized by reporting $r = q$, so that honesty is the best policy, and strictly proper if $q$ is the only minimizer (Gneiting and Raftery, 2007).

Squared loss. The expected loss is $E(r) = q(1 - r)^2 + (1 - q)\,r^2$. Its derivative is $-2q(1 - r) + 2(1 - q)\,r = 2(r - q)$, which is zero only at $r = q$, and the second derivative is positive. The Brier score is strictly proper.

Log loss. $E(r) = -q \ln r - (1 - q) \ln(1 - r)$, with derivative $-q/r + (1 - q)/(1 - r)$. Setting it to zero gives $q(1 - r) = (1 - q)\,r$, so $r = q$. Log loss is strictly proper.

Absolute loss. $E(r) = q(1 - r) + (1 - q)\,r = q + r(1 - 2q)$, a straight line in $r$. If $q > 1/2$ its slope is negative and the expected loss is smallest at $r = 1$; if $q < 1/2$, at $r = 0$. The absolute loss is minimized by reporting certainty in whichever class is more likely. It is not proper: it rewards a forecaster for exaggerating.

Expected loss of reporting probability r when the true probability is 0.7 (dotted line). Squared loss and log loss are smallest at r = 0.7. Absolute loss falls in a straight line to its minimum at r = 1: claiming certainty beats telling the truth.
Expected loss of reporting probability r when the true probability is 0.7 (dotted line). Squared loss and log loss are smallest at r = 0.7. Absolute loss falls in a straight line to its minimum at r = 1: claiming certainty beats telling the truth.

At $q = 0.7$, the honest report $r = 0.7$ has expected absolute loss $0.7 \times 0.3 + 0.3 \times 0.7 = 0.42$, while reporting $r = 1$ has $0.30$. The effect shows on real predictions. Rounding the logistic model's 190 test probabilities to 0 or 1 at the cutoff 0.5 throws away all information about confidence, yet it lowers the average absolute loss:

Probabilities Absolute loss Brier score Log loss
As fitted 0.1211 0.0724 0.2453
Rounded to 0 or 1 0.0895 0.0895 infinite (3.2250 clipped)

For rounded probabilities the absolute loss and the Brier score both equal the error rate, 17/190 = 0.0895; the absolute loss calls the rounding an improvement and the Brier score correctly calls it a loss. The log loss of a certain prediction that turns out wrong is infinite. scikit-learn's log_loss clips probabilities to $[\epsilon, 1 - \epsilon]$ with $\epsilon \approx 2.2 \times 10^{-16}$, the floating-point machine epsilon, which turns each of the 17 errors into a loss of about 36 and gives 3.2250. A clipped value like this reflects the choice of $\epsilon$ as much as the model.

The absolute loss is therefore a poor criterion for comparing probabilities, though the mean absolute difference between probabilities and outcomes is sometimes reported. The Brier score and log loss are both strictly proper, and the choice between them is about how harshly to treat confident errors.

Comparing five classifiers

Five models were fitted to the 379 training masses and scored on both the training and test masses:

  • prevalence only: every mass gets the training proportion of malignant cases, 0.372, whatever its features;
  • logistic, 3 features: the model of Binary Classification Metrics, fitted without a penalty;
  • logistic, 30 features, no penalty: all 30 features, fitted by maximum likelihood;
  • logistic, 30 features, L2: all 30 features with scikit-learn's default penalty on the squared coefficients ($C = 1$);
  • naive Bayes, 30 features: Gaussian naive Bayes on the standardized features.

On the test masses:

Model Accuracy AUC Absolute loss Brier score Log loss
prevalence only 0.626 0.500 0.4677 0.2340 0.6609
logistic, 3 features 0.911 0.964 0.1211 0.0724 0.2453
logistic, 30, no penalty 0.921 0.977 0.0782 0.0750 1.6339
logistic, 30, L2 0.958 0.993 0.0679 0.0327 0.1087
naive Bayes, 30 0.905 0.974 0.0944 0.0908 1.1549

On the training masses:

Model Accuracy Brier score Log loss
prevalence only 0.628 0.2336 0.6600
logistic, 3 features 0.939 0.0407 0.1351
logistic, 30, no penalty 1.000 0.0000 0.0002
logistic, 30, L2 0.992 0.0103 0.0440
naive Bayes, 30 0.950 0.0493 0.4781

The prevalence-only model sets the baseline. Its log loss, 0.661, is the cross-entropy between the test set's share of malignant masses and the constant prediction 0.372, and any model with a higher log loss is doing worse than ignoring the features. A skill score expresses a model's loss relative to such a baseline, $1 - L_{\text{model}} / L_{\text{baseline}}$; for the three-feature model it is 0.691 by the Brier score and 0.629 by log loss.

The unpenalized 30-feature model classifies every training mass correctly, with a training log loss of 0.0002. With 30 features and 379 masses, some combination of features separates the two classes perfectly, and the maximum likelihood coefficients do not exist: the likelihood keeps rising as the coefficients grow, a situation called separation in Logistic Regression. The optimizer stops where its convergence test is met, at coefficients up to 29.9 in standardized units. Because there is no maximum to find, where it stops depends on the solver, and so do the exact test numbers that follow, which are for scikit-learn's default solver (lbfgs); with its newton-cg solver the worst single loss is 93.6 and the test log loss 0.906, still well above the baseline. On the test masses its accuracy, 0.921, and AUC, 0.977, look better than the three-feature model's. Its log loss, 1.634, is two and a half times that of predicting the prevalence. One malignant mass received log-odds of $-166.9$ and contributes a loss of 166.9 by itself; without it the average is still 0.759. The five worst cases account for 76.5% of the total, while the median case's loss is effectively zero. The model is right most of the time and, when wrong, wrong with a certainty nothing in the data justifies. This is overfitting in its starkest form, and the gap between training loss (0.0002) and test loss (1.634) is the measure of it.

The penalized 30-feature model is the same model with its coefficients shrunk, the largest to 1.05. It is the best on every measure: accuracy 0.958, AUC 0.993, Brier score 0.0327, log loss 0.1087. The penalty gives up a little fit to the training data (training log loss 0.044) for much lower variance, the trade described in The Bias-Variance Tradeoff.

Naive Bayes has an AUC of 0.974 and a log loss of 1.155, worse than the baseline. Its probabilities are nearly all below 0.01 or above 0.99, and 15 of its 18 errors were made at that confidence, as the calibration section of Binary Classification Metrics showed. The Brier score, which caps each case's loss at 1, still rates it well above the baseline (skill 0.612), while log loss rates it below (skill $-0.748$). The two proper losses disagree because they disagree about confident errors.

The absolute loss column ranks the unpenalized 30-feature model second and naive Bayes third, both ahead of the three-feature model. Both of those models make extreme predictions, and absolute loss, being improper, rewards extremity. That ranking illustrates the previous section's warning.

Which proper loss to report depends on what a confident error costs. If a probability will be acted on as a probability, for instance multiplied by a cost to choose a cutoff, a model that is sometimes certain and wrong is dangerous, and log loss says so. If occasional extreme errors are tolerable and a bounded, more stable measure is wanted, the Brier score is the better summary. Reporting both, along with a ranking measure such as AUC, gives a fuller picture than any one of them.

Losses for training: the margin view

Classifiers are trained by minimizing a loss on the training data, usually with a penalty. To compare the losses used by different methods it helps to recode the classes as $\tilde{y} = +1$ (malignant) and $\tilde{y} = -1$ (benign), and to write the model's output as a real-valued score $f(x)$, classifying by its sign. The product

$$m = \tilde{y}\,f(x)$$

is the margin. It is positive when the classification is correct, negative when it is wrong, and its size says how far the case is from the decision boundary (Hastie, Tibshirani and Friedman, 2009, §10.6). Each training loss is a function of the margin:

  • Misclassification (0-1): 1 if $m < 0$, else 0.
  • Logistic: $\ln(1 + e^{-m})$. With $f$ the log-odds and $p = 1/(1 + e^{-f})$, a malignant case ($\tilde{y} = 1$) has log loss $-\ln p = \ln(1 + e^{-f})$, and a benign case ($\tilde{y} = -1$) has $-\ln(1 - p) = \ln(1 + e^{f})$. Both are $\ln(1 + e^{-m})$, so this is the log loss written in terms of the margin. On the 190 test masses, the average of $\ln(1 + e^{-m})$ over the three-feature model's scores is 0.2453, its log loss.
  • Hinge: $\max(0, 1 - m)$, the loss of the support vector machine (Cortes and Vapnik, 1995; James et al., 2021, §9.5). It is exactly zero once a case is on the correct side with margin at least 1, so well-classified cases have no influence on the fit.
  • Exponential: $e^{-m}$, the loss that AdaBoost minimizes (Hastie et al., 2009, §10.4). See Boosting.
  • Squared: $(\tilde{y} - f)^2 = (1 - m)^2$, least squares applied to the $\pm 1$ labels.
Training losses as functions of the margin m = y f(x), with labels coded +1 and -1. All pass through 1 at m = 0; the logistic loss is divided by ln 2 to do so. Hinge, logistic and exponential losses fall as the margin grows; the exponential loss rises steeply for badly misclassified cases. Squared loss rises again for m > 1, penalizing cases that are classified correctly with confidence.
Training losses as functions of the margin m = y f(x), with labels coded +1 and -1. All pass through 1 at m = 0; the logistic loss is divided by ln 2 to do so. Hinge, logistic and exponential losses fall as the margin grows; the exponential loss rises steeply for badly misclassified cases. Squared loss rises again for m > 1, penalizing cases that are classified correctly with confidence.

The shapes explain the methods' behavior. All but the squared loss decrease with the margin, so they always prefer a case to be classified more confidently and more correctly. The squared loss rises again past $m = 1$, so it penalizes cases that are too confidently correct, and pulls the boundary toward them; Hastie et al. (2009, §10.6) call it a poor surrogate for misclassification for that reason. For large negative margins, the hinge and logistic losses grow linearly while the exponential loss grows exponentially. AdaBoost therefore concentrates its effort on badly misclassified cases, which makes it sensitive to mislabeled training data, while the logistic loss spreads its influence more evenly (Hastie et al., 2009, §10.6).

What each loss estimates differs too. Minimized over all possible functions, given the true probability $P = \Pr(\tilde{y} = 1 \mid x)$, the logistic loss is minimized by the log-odds $\ln[P/(1 - P)]$, the exponential loss by half the log-odds, the squared loss by $2P - 1$, and the hinge loss by $\operatorname{sign}(P - 1/2)$ (Hastie et al., 2009, §10.5 and Table 12.1). All four have the right sign, so all four lead to the best possible classifier given enough data and a flexible enough model. Losses with this property are called classification-calibrated, and for convex losses Bartlett, Jordan and McAuliffe (2006) show it holds exactly when the loss is differentiable at $m = 0$ with negative slope there. But only the first three recover the probability itself. The hinge loss estimates only which class is more likely, which is why a support vector machine produces scores rather than probabilities and needs a separate calibration step, such as Platt scaling, if probabilities are wanted.

Convexity is the other reason these surrogates are used. The logistic, hinge, exponential and squared losses are convex in the margin, and the margin is linear in the coefficients of a linear model, so the training problem has no local minima to get trapped in. The 0-1 loss has neither property.

In Python

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import brier_score_loss, log_loss
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X, y = data.data, 1 - data.target                 # malignant = 1
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=1/3, stratify=y, random_state=742)

model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_tr, y_tr)   # L2, C = 1
p = model.predict_proba(X_te)[:, 1]

print(log_loss(y_te, p))                  # 0.1087
print(brier_score_loss(y_te, p))          # 0.0327
print(np.mean(np.abs(y_te - p)))          # 0.0679, absolute loss (not proper)

baseline = np.full(len(y_te), y_tr.mean())
print(log_loss(y_te, baseline), brier_score_loss(y_te, baseline))   # 0.6609, 0.2340

Both functions take probabilities, the second column of predict_proba, not 0/1 predictions. log_loss clips probabilities away from 0 and 1, so for models that produce extreme probabilities the exact log loss is better computed from the log-odds: for a logistic model, np.logaddexp(0, -m) with m the margin from decision_function, which is finite for any score. The numbers in this article, including the exact log losses of the unpenalized model and naive Bayes, are computed by tools/experiments/data-science/classification-loss-functions.py in the site's repository.

References

  • Bartlett, P. L., Jordan, M. I., and McAuliffe, J. D. (2006). Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473), 138–156. record
  • Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3. record
  • Cortes, C., and Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. record · 2 pages
  • Gneiting, T., and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378. record
  • Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer. record · 23 pages
  • James, G., Witten, D., Hastie, T., and Tibshirani, R. (2021). An Introduction to Statistical Learning with Applications in R (2nd ed.). Springer. record · 30 pages