Binary Classification Metrics

How to judge a classifier that says yes or no. The confusion matrix and the rates read from it in two directions, sensitivity and specificity against precision and negative predictive value, choosing a cutoff by Youden's index or by the costs of the two errors, the ROC curve and AUC worked by hand on the Challenger flights, how prevalence changes precision but not the ROC curve, and calibration, worked on a breast cancer biopsy dataset.

A fine-needle aspirate draws a few cells from a breast mass, and a digitized image of them yields measurements of the cell nuclei: their size, the variation in their gray levels, the smoothness of their outlines. A logistic regression on three of those measurements turns them into a probability that the mass is malignant. The probability is not yet a decision. To act on it, the probability is compared with a cutoff, and every choice of cutoff trades one kind of mistake for another: telling a patient with cancer that the mass looks benign, or sending a patient with a benign mass for further procedures.

This article covers the measures used to judge such a classifier. The course wiki's Evaluating Classifiers already treats precision, recall and F1 for text classification, with micro- and macro-averaging over many categories, and the precision-recall trade-off; those are summarized here and not repeated. The focus is on what that article does not cover: the full confusion matrix read in both directions, the vocabulary of medicine and diagnostic testing, choosing a cutoff from costs, ROC curves and the area under them, the effect of how common the positive class is, and whether the probabilities themselves can be believed.

The data and the model

The Breast Cancer Wisconsin (Diagnostic) data describe 569 aspirates, 212 from malignant masses and 357 from benign ones, each with 30 features computed from the image of the nuclei (Wolberg, Mangasarian, Street and Street, 1993; Street, Wolberg and Mangasarian, 1993). The data ship with scikit-learn. The positive class here is malignant, coded 1. (scikit-learn codes malignant as 0, so the labels are flipped before use.)

The masses were split at random, keeping the proportion of malignant cases equal in both parts (a stratified split, seed 742), into 379 for training (141 malignant) and 190 for testing (71 malignant, a prevalence of 37.4%). Every measure in this article is computed on the 190 test masses, which the model never saw. A logistic regression, fitted by maximum likelihood without a penalty as in Logistic Regression, uses three features, each standardized to mean 0 and standard deviation 1 on the training data:

  • mean radius, the average distance from the center of a nucleus to its outline;
  • mean texture, the standard deviation of gray-scale values within the nuclei;
  • mean smoothness, the local variation in radius lengths around the outline.

The fitted log-odds of malignancy are $-1.2821 + 5.0718\,z_{\text{radius}} + 2.1145\,z_{\text{texture}} + 2.5424\,z_{\text{smoothness}}$, where each $z$ is a standardized feature. Larger, more variable and less smooth nuclei raise the probability of malignancy. Three features are deliberately few: a model on all thirty makes so few errors on this data that there would be little to evaluate.

The confusion matrix

At the conventional cutoff of 0.5, each test mass falls into one of four cells:

Predicted malignant Predicted benign Total
Malignant 59 (true positives, TP) 12 (false negatives, FN) 71
Benign 5 (false positives, FP) 114 (true negatives, TN) 119
Total 64 126 190

This table is the confusion matrix. The diagonal holds the correct decisions and the off-diagonal cells the two kinds of error. A false positive is a false alarm, a benign mass called malignant; a false negative is a miss, a cancer called benign. The two errors have very different consequences here, and so the four counts matter, not only how many are correct.

The layout is a convention, and conventions differ. This table puts the true class in rows and the positive class first. scikit-learn's confusion_matrix also puts true classes in rows, but orders them by label, so with malignant coded 1 the benign row and column come first. James, Witten, Hastie and Tibshirani (2021, Table 4.6) put the predicted class in rows. A confusion matrix should always be read from its labels, never from its position.

Rates read in two directions

Every common measure is a ratio of cells in the matrix. They divide into two families, according to whether they condition on the truth or on the prediction (James et al., 2021, Table 4.7).

Conditioning on the true class answers: of the masses that are malignant (or benign), what did the classifier say?

  • Sensitivity, also called recall or the true positive rate (TPR): $\mathrm{TP}/(\mathrm{TP} + \mathrm{FN}) = 59/71 = 0.831$. The classifier finds 83.1% of the cancers.
  • Specificity, or the true negative rate: $\mathrm{TN}/(\mathrm{TN} + \mathrm{FP}) = 114/119 = 0.958$. It clears 95.8% of the benign masses.
  • The false positive rate (FPR) is $1 - \text{specificity} = 5/119 = 0.042$, and the false negative rate is $1 - \text{sensitivity} = 0.169$.

Conditioning on the prediction answers: when the classifier says malignant (or benign), how often is it right?

  • Precision, or positive predictive value (PPV): $\mathrm{TP}/(\mathrm{TP} + \mathrm{FP}) = 59/64 = 0.922$.
  • Negative predictive value (NPV): $\mathrm{TN}/(\mathrm{TN} + \mathrm{FN}) = 114/126 = 0.905$.

The distinction is more than terminology. Sensitivity and specificity describe the classifier's behavior on each class separately, so they do not depend on how common the classes are. Precision and NPV mix the two classes, so they do: a later section shows precision falling from 0.92 to 0.17 when the same classifier meets a population where cancer is rarer. Medicine reports sensitivity and specificity for a test, and predictive values for a test applied to a stated population. Information retrieval reports precision and recall, because the negatives, documents that are irrelevant and not retrieved, are numerous and uninteresting; Evaluating Classifiers explains why.

Three summaries combine the cells:

  • Accuracy, $(\mathrm{TP} + \mathrm{TN})/N = 173/190 = 0.911$. A classifier that calls every mass benign scores 0.626, the share of benign masses, so the model's accuracy should be read against that baseline rather than against zero.
  • Balanced accuracy, the average of sensitivity and specificity, $(0.831 + 0.958)/2 = 0.894$. It weights the two classes equally whatever their sizes, and is 0.5 for any classifier that ignores the input.
  • F1, the harmonic mean of precision and recall, $2 \times 59/(2 \times 59 + 5 + 12) = 0.874$. Like precision and recall it ignores the true negatives, which suits retrieval and is less suitable here, where correctly cleared benign masses matter.

No single number captures a classifier, because each answers a different question. A report should give the confusion matrix, or enough rates to reconstruct it, along with the class proportions.

Choosing the cutoff

Lowering the cutoff makes the classifier call more masses malignant. The number of true positives can only rise, so sensitivity can only rise, and specificity can only fall. Precision has no such guarantee: it usually falls as the cutoff is lowered, but it can move either way, because both its numerator and its denominator change.

Cutoff TP FP FN TN Sensitivity Specificity Precision Accuracy
0.05 68 29 3 90 0.958 0.756 0.701 0.832
0.10 65 22 6 97 0.915 0.815 0.747 0.853
0.20 64 15 7 104 0.901 0.874 0.810 0.884
0.30 64 11 7 108 0.901 0.908 0.853 0.905
0.50 59 5 12 114 0.831 0.958 0.922 0.911
0.70 52 4 19 115 0.732 0.966 0.929 0.879
0.90 46 2 25 117 0.648 0.983 0.958 0.858
Sensitivity, specificity and precision on the 190 test masses as the cutoff moves from 0 to 1. Lowering the cutoff below 0.5 buys sensitivity with specificity: at 0.3 the classifier finds 64 of the 71 cancers instead of 59, at the cost of 6 more false alarms.
Sensitivity, specificity and precision on the 190 test masses as the cutoff moves from 0 to 1. Lowering the cutoff below 0.5 buys sensitivity with specificity: at 0.3 the classifier finds 64 of the 71 cancers instead of 59, at the cost of 6 more false alarms.

Nothing makes 0.5 special. A cutoff of 0.5 classifies each case into its more probable class, which minimizes the expected number of errors if the probabilities are accurate and the two errors are equally costly. Neither condition is usually met. Two principled ways of choosing a different cutoff are common.

Youden's index. Youden (1950) proposed rating a diagnostic test by $J = \text{sensitivity} + \text{specificity} - 1$, which is 0 for a test that ignores the patient and 1 for a perfect test. Choosing the cutoff that maximizes $J$ treats a missed cancer and a false alarm as equally serious as rates: one percentage point of sensitivity is worth one of specificity. The cutoff must be chosen on the training data, not the test data, or the test results are no longer an honest estimate. On the 379 training masses the best $J$ is 0.892, at a cutoff of 0.3128. Applied to the test masses, that cutoff gives sensitivity 0.901 and specificity 0.908 (64 true positives, 11 false positives, 7 false negatives).

Costs. Suppose a false negative costs $c_{\mathrm{FN}}$ and a false positive $c_{\mathrm{FP}}$, in whatever common units, and correct decisions cost nothing. For a mass with probability $p$ of malignancy, calling it malignant has expected cost $(1 - p)\,c_{\mathrm{FP}}$, the chance it is benign times the cost of a false alarm. Calling it benign has expected cost $p\,c_{\mathrm{FN}}$. Calling it malignant is cheaper when $p\,c_{\mathrm{FN}} > (1 - p)\,c_{\mathrm{FP}}$, that is, when

$$p > \frac{c_{\mathrm{FP}}}{c_{\mathrm{FP}} + c_{\mathrm{FN}}}.$$

This is the optimal cutoff for a classifier whose probabilities are correct (Elkan, 2001). Equal costs give 0.5. If a missed cancer is judged ten times as costly as an unnecessary follow-up, the cutoff is $1/11 = 0.091$. On the test masses, with a false positive costing 1 and a false negative 10, the total cost is 125 at the cutoff 0.5 (5 false positives and 12 false negatives), 84 at 0.091 (24 and 6) and 81 at Youden's 0.3128 (11 and 7).

The cost-based cutoff did no better here than Youden's, which ignores costs. The reason is in its premise: the formula assumes the probabilities are accurate. Among the 35 test masses given probabilities between 0.01 and 0.1, 6 were malignant, 17%, against an average predicted probability of 4%. The model is too confident in the low range, so a cutoff of 0.091 lets through cancers it should catch. The cutoff with the lowest cost on the test set in hindsight is 0.0175, with a cost of 44, but a cutoff found by searching the test set gives an optimistic cost and cannot be reported as a result. The calibration section below takes up how to check the probabilities.

Which costs to use is not a statistical question. For a screening or diagnostic test, the costs of a missed cancer and of an unnecessary biopsy are clinical and personal judgments. What the statistics can do is make the consequences of each cutoff visible.

ROC curves and AUC

Each cutoff gives one pair of rates, a true positive rate and a false positive rate. Plotting the true positive rate against the false positive rate for every possible cutoff traces the receiver operating characteristic (ROC) curve, a name that comes from communications theory (James et al., 2021, §4.4.2). The curve starts at $(0, 0)$, where the cutoff is above every score and nothing is called positive, and ends at $(1, 1)$, where everything is. A classifier that ignores its input moves along the diagonal, and a perfect one goes straight up the left edge to $(0, 1)$ and then across.

A small curve by hand

The 23 shuttle flights before the Challenger accident (Draper, 1993), analyzed in Logistic Regression, give a curve small enough to compute by hand. Seven flights had O-ring distress (at 53, 57, 58, 63, 70, 70 and 75 °F) and sixteen did not (66, 67, 67, 67, 68, 69, 70, 70, 72, 73, 75, 76, 76, 78, 79 and 81 °F). The fitted model ranks flights by temperature, colder meaning riskier, so each cutoff amounts to "flag every flight at or below $t$ °F":

Flag at or below TP FP TPR FPR
(none) 0 0 0 0
63 °F 4 0 0.571 0
66 °F 4 1 0.571 0.063
69 °F 4 6 0.571 0.375
70 °F 6 8 0.857 0.500
73 °F 6 10 0.857 0.625
75 °F 7 11 1 0.688
81 °F 7 16 1 1

Moving the cutoff from 53 to 63 °F adds one distress flight at a time, and the curve climbs the left edge to 4 of 7. Each cool flight without distress, from 66 to 69 °F, moves it right by 1/16. At 70 °F two distress flights and two without are tied, so the curve moves diagonally, up 2/7 and right 2/16 at once.

The area under the curve (AUC) has a direct meaning: it is the probability that a randomly chosen positive case receives a higher score than a randomly chosen negative case, with ties counted as one half (Hanley and McNeil, 1982; Fawcett, 2006). To see why, note that each vertical step of the curve corresponds to a positive case, and the width of the region to its left is the fraction of negative cases scored above it; summing over positives counts the fraction of positive-negative pairs ranked correctly. For the Challenger flights there are $7 \times 16 = 112$ pairs of a distress flight and a flight without. The four coldest distress flights are colder than all 16 others: 64 pairs. Each 70 °F distress flight is colder than 8 flights and tied with 2: $8 + 2 \times 0.5 = 9$ pairs each, 18 together. The 75 °F distress flight is colder than 5 and tied with 1: 5.5 pairs. The total is $64 + 18 + 5.5 = 87.5$, and

$$\mathrm{AUC} = \frac{87.5}{112} = 0.781,$$

the same value as the trapezoid area under the curve and as scikit-learn's roc_auc_score. This count is the Mann-Whitney $U$ statistic divided by the number of pairs, which is why the AUC is also called the concordance or $c$ statistic.

Left: the ROC curve of the 23 Challenger flights, ranked by temperature, with three cutoffs labeled; the shaded area is the AUC, 0.781. Right: the logistic model on the 190 test masses, AUC 0.964, with the cutoffs 0.5, 0.31 (Youden, chosen on the training set) and 0.1 marked. The dotted diagonal is a classifier that ignores its input.
Left: the ROC curve of the 23 Challenger flights, ranked by temperature, with three cutoffs labeled; the shaded area is the AUC, 0.781. Right: the logistic model on the 190 test masses, AUC 0.964, with the cutoffs 0.5, 0.31 (Youden, chosen on the training set) and 0.1 marked. The dotted diagonal is a classifier that ignores its input.

The biopsy classifier

On the 190 test masses the AUC is 0.964: of the $71 \times 119 = 8{,}449$ malignant-benign pairs, 8,147 are ranked correctly. On the training masses it is 0.987, higher, as a model's performance on its own training data usually is. Reading the curve at a fixed false positive rate is often more useful than its area: allowing at most 5% false positives, the best sensitivity on the test masses is 0.873, and at 10% it is 0.901.

The AUC has three properties worth keeping in mind:

  • It depends only on how the scores rank the cases. Any increasing transformation of the scores, such as replacing probabilities by log-odds or by their squares, leaves it unchanged. The AUC therefore says nothing about whether the probabilities are accurate.
  • It averages over all cutoffs, including ones no one would use. Two classifiers with the same AUC can differ greatly in the region that matters, such as low false positive rates in screening.
  • It does not depend on the proportion of positives, because both of its axes condition on the true class (Fawcett, 2006). This is a strength when comparing a classifier across populations and a weakness when the positives are rare, as the next section shows.

Prevalence and class imbalance

The prevalence $\pi$ is the proportion of positive cases in the population where the classifier is used. The test set has $\pi = 0.374$, because the data come from masses that had already been referred for aspiration. Sensitivity and specificity are properties of the classifier on each class. Precision is not, and Bayes' rule shows how it depends on prevalence. Among all cases, a fraction $\pi \cdot \text{sensitivity}$ are true positives and a fraction $(1 - \pi)(1 - \text{specificity})$ are false positives, so

$$\text{precision} = \frac{\pi \cdot \text{sensitivity}}{\pi \cdot \text{sensitivity} + (1 - \pi)(1 - \text{specificity})},$$

and similarly $\mathrm{NPV} = (1 - \pi)\,\text{specificity} / [(1 - \pi)\,\text{specificity} + \pi\,(1 - \text{sensitivity})]$.

Suppose the same classifier, at the cutoff 0.5 with sensitivity 0.831 and specificity 0.958, were applied where malignancy is rarer. These are hypothetical populations, which assume that the classifier's behavior on malignant and on benign masses would carry over unchanged:

Prevalence Precision (PPV) NPV Per 10,000 masses: cancers found False alarms Cancers missed
37.4% (test set) 0.922 0.905 3,105 263 632
10% 0.687 0.981 831 378 169
1% 0.167 0.998 83 416 17

At a prevalence of 1%, five of every six masses the classifier flags are benign. Nothing about the classifier changed; the benign masses simply outnumber the malignant ones 99 to 1, so even a 4.2% false positive rate produces five times as many false alarms as true detections. At the same time the NPV approaches 1, so a negative result becomes very reassuring. This is the base-rate effect familiar from screening tests, and it is why a test's predictive values must be quoted for a population.

The same reweighting can be applied to the whole curve, by giving each benign test mass a weight that brings the prevalence down to 10% or 1%. The ROC curve and its AUC of 0.964 are unchanged, since both axes are computed within one class. The precision-recall curve, which plots precision against recall for every cutoff, changes a great deal:

Precision against recall for the logistic model at three prevalences, obtained by reweighting the benign test masses; the ROC curve is identical in all three. Each dotted line is the precision of flagging every mass, equal to the prevalence. The model flags the 45 most suspicious masses with no false alarms at any prevalence, after which precision at 1% prevalence collapses.
Precision against recall for the logistic model at three prevalences, obtained by reweighting the benign test masses; the ROC curve is identical in all three. Each dotted line is the precision of flagging every mass, equal to the prevalence. The model flags the 45 most suspicious masses with no false alarms at any prevalence, after which precision at 1% prevalence collapses.

Its area, the average precision, falls from 0.951 at the test-set prevalence to 0.854 at 10% and 0.690 at 1%. Saito and Rehmsmeier (2015) argue that for imbalanced data the precision-recall plot is the more informative of the two, because it shows directly how many false alarms come with each detection, which the ROC curve hides behind a small false positive rate. Neither plot is wrong. The ROC curve describes the classifier; the precision-recall curve describes the classifier in a particular population.

Imbalance affects more than the choice of plot:

  • Accuracy becomes uninformative. At a prevalence of 1%, calling every mass benign is 99% accurate and finds no cancers. Report sensitivity and specificity, or precision and recall, with the prevalence, as Evaluating Classifiers does for rare text categories.
  • The cutoff should move. With few positives, a probability of 0.5 is rarely reached, and the cost-based cutoff above is usually the better guide.
  • Reweighting or resampling the training data is equivalent, in principle, to moving the cutoff (Elkan, 2001). scikit-learn's class_weight='balanced' gives each class equal total weight in the likelihood. Refitting the biopsy model this way raises its intercept by 0.5211, close to $\ln(238/141) = 0.5235$, the log of the ratio of benign to malignant training cases, and leaves the slopes almost unchanged (5.0718 becomes 4.9915 for radius). At the cutoff 0.5 it gives 62 true positives and 9 false positives; the unweighted model at the cutoff 0.372, the training prevalence, gives 62 and 7. The weighted model ranks the masses almost identically (AUC 0.963). What changes is that its probabilities no longer estimate the chance of malignancy in the population it was trained on, which matters if they are used as probabilities.

Calibration

A classifier is calibrated if its probabilities mean what they say: among all cases given a probability near 0.3, about 30% turn out positive. Calibration is separate from ranking. A model can rank cases perfectly and still give probabilities that are too extreme, too timid or shifted; AUC cannot tell, because it ignores everything but the order.

The standard check is a reliability diagram: group cases by their predicted probability, and in each group compare the average prediction with the observed proportion of positives. A calibrated classifier's points lie near the diagonal. With 190 test cases the groups are small and the comparison rough, so the bins below are chosen to keep the extremes separate. The table compares the logistic model with a Gaussian naive Bayes classifier on all 30 features, standardized on the training data (scikit-learn's GaussianNB), which treats the features as independent normal variables within each class:

Predicted probability Logistic: masses (malignant) Logistic: mean predicted Logistic: observed share Naive Bayes: masses (malignant) Naive Bayes: observed share
below 0.01 68 (0) 0.001 0.000 119 (8) 0.067
0.01 to 0.1 35 (6) 0.040 0.171 1 (1)
0.1 to 0.5 23 (6) 0.273 0.261 1 (1)
0.5 to 0.9 16 (13) 0.719 0.813 2 (1)
0.9 to 0.99 17 (15) 0.965 0.882 1 (1)
0.99 and above 31 (31) 0.999 1.000 66 (59) 0.894

Naive Bayes has so few masses in the middle bins that their observed shares are not given.

Reliability diagrams on the 190 test masses, with point size and label showing the number of masses in each bin. Left: the logistic model's points lie near the diagonal; with bins this small, departures of 0.1 are within chance, though the 0.01 to 0.1 bin, 6 cancers among 35 masses, suggests the model is too confident there. Right: naive Bayes puts 185 of 190 masses below 0.01 or above 0.99, yet 8 of the 119 masses it gives less than a 1% chance are malignant, and 7 of the 66 it gives more than 99% are benign.
Reliability diagrams on the 190 test masses, with point size and label showing the number of masses in each bin. Left: the logistic model's points lie near the diagonal; with bins this small, departures of 0.1 are within chance, though the 0.01 to 0.1 bin, 6 cancers among 35 masses, suggests the model is too confident there. Right: naive Bayes puts 185 of 190 masses below 0.01 or above 0.99, yet 8 of the 119 masses it gives less than a 1% chance are malignant, and 7 of the 66 it gives more than 99% are benign.

The two classifiers rank the masses about equally well: the AUC is 0.974 for naive Bayes and 0.964 for the logistic model, and their accuracies at 0.5 are 0.905 and 0.911. Their probabilities are very different. Naive Bayes is almost always certain: 97.4% of its probabilities are below 0.01 or above 0.99. Of its 18 errors, 15 were made with a probability below 0.01 or above 0.99. The logistic model made none of its 17 errors with that much confidence. Naive Bayes multiplies the evidence from 30 features as though each were independent of the others, but the features are strongly related (radius, perimeter and area measure nearly the same thing), so the same evidence is counted many times over and the probabilities are pushed toward 0 and 1. Niculescu-Mizil and Caruana (2005) report the same behavior of naive Bayes on eight classification problems and attribute it to the independence assumption, and the course wiki's Bernoulli model article shows it for text.

Poorly calibrated probabilities can be repaired without changing the ranking, by fitting a monotone map from scores to probabilities on held-out data. Platt scaling fits a logistic curve to the scores; isotonic regression fits any increasing step function. Niculescu-Mizil and Caruana (2005) compare the two on ten learning methods. Platt scaling suits distortions shaped like a sigmoid; isotonic regression can correct any monotone distortion and was the better choice for naive Bayes, but it overfits when calibration data are scarce, and in their experiments Platt scaling did better with fewer than about 1,000 calibration cases. In scikit-learn, CalibratedClassifierCV implements both. The loss functions that reward calibrated probabilities, log loss and the Brier score, are the subject of Loss Functions for Classification, and the library's note on Language Models (Mostly) Know What They Know applies reliability diagrams to the answers of language models.

In Python

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (confusion_matrix, classification_report,
                             roc_auc_score, average_precision_score)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X = data.data[:, [0, 1, 4]]           # mean radius, mean texture, mean smoothness
y = 1 - data.target                   # malignant = 1
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=1/3, stratify=y, random_state=742)

model = make_pipeline(StandardScaler(), LogisticRegression(penalty=None, tol=1e-10)).fit(X_tr, y_tr)
p = model.predict_proba(X_te)[:, 1]   # probability of malignancy

print(confusion_matrix(y_te, p >= 0.5))         # [[114   5]
                                                #  [ 12  59]]
print(confusion_matrix(y_te, p >= 0.3128))      # [[108  11]
                                                #  [  7  64]]
print(roc_auc_score(y_te, p))                   # 0.964
print(average_precision_score(y_te, p))         # 0.951
print(classification_report(y_te, p >= 0.5, target_names=["benign", "malignant"]))

confusion_matrix orders the classes by label, so with malignant coded 1 the first row and column are benign: the top-left entry is the true negatives and the bottom-right the true positives. roc_auc_score and average_precision_score take the probabilities, not the 0/1 predictions; given predictions, they would describe a single cutoff rather than the whole curve. classification_report prints precision, recall and F1 for each class, so the recall of the benign class is the specificity. roc_curve and precision_recall_curve return the points of the two curves. The numbers in this article, including the cost calculations, the prevalence table and the calibration bins, are computed by tools/experiments/data-science/binary-classification-metrics.py in the site's repository.

References

  • Draper, D. (1993). Challenger USA Space Shuttle O-Ring [Data set]. UCI Machine Learning Repository. https://doi.org/10.24432/C5PW2T record · 4 pages
  • Elkan, C. (2001). The foundations of cost-sensitive learning. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence (pp. 973–978). record
  • Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874. record · 2 pages
  • Hanley, J. A., and McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29–36. record · 3 pages
  • James, G., Witten, D., Hastie, T., and Tibshirani, R. (2021). An Introduction to Statistical Learning with Applications in R (2nd ed.). Springer. record · 30 pages
  • Niculescu-Mizil, A., and Caruana, R. (2005). Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine Learning (pp. 625–632). record
  • Saito, T., and Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE, 10(3), e0118432. record
  • Street, W. N., Wolberg, W. H., and Mangasarian, O. L. (1993). Nuclear feature extraction for breast tumor diagnosis. In Biomedical Image Processing and Biomedical Visualization, Proceedings of SPIE 1905 (pp. 861–870). record
  • Wolberg, W., Mangasarian, O., Street, N., and Street, W. (1993). Breast Cancer Wisconsin (Diagnostic) [Data set]. UCI Machine Learning Repository. https://doi.org/10.24432/C5DW2B record · 14 pages
  • Youden, W. J. (1950). Index for rating diagnostic tests. Cancer, 3(1), 32–35. record · 2 pages