At the American Diabetes Association's threshold of 6.5%, HbA1c catches 47.0% of the adults whose fasting plasma glucose (FPG) is in the diabetic range and clears 98.7% of the others. But 6.5% is one choice among many. Lowering the threshold catches more cases and flags more people without the condition. How good is HbA1c as a measurement, apart from any one threshold, and where should the threshold go?
Sensitivity and Specificity found those two numbers at 6.5%. Likelihood Ratios for Diagnostic Tests found evidence for the condition in HbA1c values well below the threshold. The receiver operating characteristic (ROC) curve displays the trade-off for every possible threshold at once. The area under it summarizes how well the measurement separates the two groups, regardless of where the threshold is put.
ROC graphs come from signal detection theory, where they plotted the rate of hits against the rate of false alarms (Fawcett, 2006, §1). The library's Binary Classification Metrics introduces the ROC curve for a machine-learning classifier and works a small example by hand. This article develops the statistics a diagnostic study needs.
First we build the curve, including tied results. Then we prove that its area is a probability about pairs of people, the statistic of the Wilcoxon rank-sum test of a later unit. Next we compute its standard error three ways and compare HbA1c with two weaker screens. Finally we choose a cut-off, by Youden's index and by expected cost.
The data are the unit's 1,975 adults from the National Health and Nutrition Examination Survey (NHANES) 2017 to 2018. Of them, 115 have an FPG of 126 mg/dL or more, the reference definition used throughout.
From cut-offs to a curve
Call a result positive when the HbA1c is at or above a cut-off $c$. Each cut-off gives a sensitivity, the true positive rate $\mathrm{TPR}(c)$, and a false positive rate $\mathrm{FPR}(c) = 1 - \mathrm{Sp}(c)$:
| Cut-off | TP (of 115) | Sensitivity | FP (of 1,860) | False positive rate | $\mathrm{Se} + \mathrm{Sp} - 1$ |
|---|---|---|---|---|---|
| 5.5% | 109 | 0.948 | 944 | 0.508 | 0.440 |
| 5.7% | 98 | 0.852 | 595 | 0.320 | 0.532 |
| 5.9% | 91 | 0.791 | 303 | 0.163 | 0.628 |
| 6.0% | 86 | 0.748 | 217 | 0.117 | 0.631 |
| 6.2% | 72 | 0.626 | 90 | 0.048 | 0.578 |
| 6.5% | 54 | 0.470 | 25 | 0.013 | 0.456 |
| 6.8% | 39 | 0.339 | 7 | 0.004 | 0.335 |
| 7.0% | 28 | 0.243 | 4 | 0.002 | 0.241 |
As the cut-off falls, both rates can only rise, because every person positive at a higher cut-off is still positive at a lower one. Plotting $\mathrm{TPR}$ against $\mathrm{FPR}$ for every cut-off gives the ROC curve.
- It runs from $(0, 0)$, where the cut-off is above every value and nobody is positive, to $(1, 1)$, where everybody is.
- A test that carries no information about the condition has the same distribution in both groups. Each cut-off flags the same fraction of each, and its curve is the diagonal.
- A perfect test reaches $(0, 1)$: all cases found, with no false positives.
Tied results
HbA1c is reported to one decimal place, so the 1,975 adults have only 48 distinct values, and the empirical curve has 49 points. At each value, cases and non-cases with that exact HbA1c switch from negative to positive together. The curve then moves diagonally: up by the share of cases at that value, and right by the share of non-cases. Joining the points with straight lines is the standard convention. Fawcett (2006, §5) explains the diagonal segment as the expected curve over the possible orderings of the tied results. The proof below shows that the area under it counts each tied pair as one half.
The area under the curve
A whole curve is hard to compare across tests. One number that summarizes it is the area under the ROC curve (AUC), which is 0.5 for an uninformative test and 1 for a perfect one. Its value has a direct interpretation, in terms of pairs of people.
Let $X$ be the result of a randomly chosen person with the condition, and $Y$ that of a randomly chosen person without it, independently. Here is the idea. Each step of the curve to the right is one non-case passing the cut-off. The height at that moment counts the cases already above it. So the area counts the pairs in which the case outranks the non-case.
Proposition. For the empirical ROC curve of $n_1$ cases and $n_0$ non-cases, with tied points joined by straight segments,
$$\mathrm{AUC} = \frac{1}{n_1 n_0}\sum_{i=1}^{n_1}\sum_{j=1}^{n_0}\left[\mathbf{1}(x_i > y_j) + \tfrac12\,\mathbf{1}(x_i = y_j)\right],$$
the proportion of case and non-case pairs in which the case has the higher result, counting ties as one half. It is the sample estimate of $P(X > Y) + \tfrac12 P(X = Y)$.
Proof. List the distinct values from the highest down, $v_1 > v_2 > \cdots$. At $v_k$, let $a_k$ cases and $b_k$ non-cases have that value, and let $A_{k}$ be the number of cases with values above $v_k$. Lowering the cut-off past $v_k$ moves the curve right by $b_k/n_0$, from height $A_k/n_1$ to height $(A_k + a_k)/n_1$. The area under that segment is a trapezoid, its width times its average height:
$$\frac{b_k}{n_0}\cdot\frac{1}{n_1}\left(A_k + \frac{a_k}{2}\right) = \frac{1}{n_1 n_0}\left(b_k A_k + \tfrac12\,a_k b_k\right).$$
Each of the $b_k$ non-cases at $v_k$ is exceeded by exactly the $A_k$ cases above $v_k$, and tied with the $a_k$ cases at $v_k$. So $b_k A_k$ counts the pairs in which one of these non-cases is below a case, and $a_k b_k$ the pairs in which it is tied with one. Summing over $k$ counts every non-case once. The total area is therefore the number of pairs with the case higher, plus half the number tied, divided by $n_1 n_0$. $\square$
Back to the 1,975 adults. For HbA1c there are $115 \times 1{,}860 = 213{,}900$ case and non-case pairs. In 187,328 the case has the higher HbA1c, in 5,817 the two are tied, and in 20,755 the non-case is higher, so
$$\mathrm{AUC} = \frac{187{,}328 + 5{,}817/2}{213{,}900} = \frac{190{,}236.5}{213{,}900} = 0.8894.$$
The trapezoid area under the 49-point curve gives the same 0.8894. A randomly chosen adult with a diabetic fasting glucose has the higher HbA1c than a randomly chosen adult without one about 89% of the time, counting ties as half. Hanley and McNeil (1982) give this interpretation. The underlying result is that the area under the ROC curve equals the probability of correctly ranking a randomly chosen abnormal and normal case. They attribute it to Green and Swets's Signal Detection Theory and Psychophysics (1966).
The connection to the rank-sum test
The double sum is the Mann–Whitney statistic $U$ of The Wilcoxon Rank-Sum Test, computed for the cases. The AUC is $U/(n_1 n_0)$, the estimate of the probability that article calls $\theta$. Equivalently, rank all 1,975 HbA1c values together, with tied values sharing the mean of their ranks. The cases' ranks sum to $W = 196{,}906.5$, and $U = W - n_1(n_1 + 1)/2 = 196{,}906.5 - 6{,}670 = 190{,}236.5$. The script computes the AUC this way, which takes one sort instead of a quarter of a million comparisons.
Two consequences follow.
- Testing for any discrimination is the rank-sum test. The null hypothesis $H_0\colon \mathrm{AUC} = 0.5$ says the measurement has no discriminating power. Here $W$ is 14.07 standard deviations above its null mean of 113,620 (with the correction for ties). The evidence is overwhelming, as it would be for any measurement this closely tied to the reference definition.
- Only the order matters. The AUC depends on the results only through their order. Fawcett (2006, §7) states the equivalence with the Wilcoxon statistic for classifier scores.
Check your understanding. Would the AUC change if HbA1c were reported in mmol/mol instead of percent, or if its logarithm were used? Would it change in a population where the condition was twice as common, with the same two distributions?
It would change in neither case. Any increasing transformation of the scale keeps every pair in the same order. And the AUC is a comparison between one case and one non-case, so the prevalence does not enter.
A single cut-off as a curve
A test reported only as positive or negative has a curve with one interior point, $(1 - \mathrm{Sp}, \mathrm{Se})$, joined to the corners.
Check your understanding. HbA1c dichotomized at 6.5% uses the same measurement as the full curve. Will its area be about the same as 0.889, or much lower?
Much lower. The area is two triangles and a rectangle, which simplify to $(\mathrm{Se} + \mathrm{Sp})/2$. For HbA1c at 6.5% this is $(0.470 + 0.987)/2 = 0.728$, against 0.889 for the full measurement. Reporting only "positive" or "negative" discards a large part of the discriminating power. The quantity $2\,\mathrm{AUC} - 1 = \mathrm{Se} + \mathrm{Sp} - 1$ for one cut-off is Youden's index, which reappears below.
How precise is an AUC?
The 0.889 comes from 115 cases and 1,860 non-cases, so it is an estimate. Its standard error is needed to report an interval or to compare tests, and there are three standard ways to get it.
Hanley and McNeil (1982) derived the variance of $U/(n_1 n_0)$ from the theory of the Wilcoxon statistic:
$$\mathrm{SE}(\widehat{\mathrm{AUC}}) = \sqrt{\frac{A(1 - A) + (n_1 - 1)(Q_1 - A^2) + (n_0 - 1)(Q_2 - A^2)}{n_1 n_0}}.$$
Here $A$ is the AUC. $Q_1$ is the probability that two randomly chosen cases both rank above one randomly chosen non-case. $Q_2$ is the probability that one randomly chosen case ranks above two randomly chosen non-cases. $Q_1$ and $Q_2$ depend on the shapes of the two distributions. Hanley and McNeil used the approximations
$$Q_1 = \frac{A}{2 - A}, \qquad Q_2 = \frac{2A^2}{1 + A},$$
which hold exactly when the results follow exponential distributions. They found that these give standard errors close to those of other models, and slightly conservative in the range of areas of interest. With $A = 0.8894$, $Q_1 = 0.8008$ and $Q_2 = 0.8373$, and the standard error is 0.0202.
DeLong, DeLong and Clarke-Pearson (1988) replaced the approximations with an estimate from the data. Each case $i$ gets a "placement value" $V_{10}(x_i)$, the fraction of non-cases it exceeds (ties counted as one half). Each non-case $j$ gets a value $V_{01}(y_j)$, the fraction of cases that exceed it. Both sets average to the AUC, and
$$\widehat{\mathrm{Var}}(\widehat{\mathrm{AUC}}) = \frac{s_{10}^2}{n_1} + \frac{s_{01}^2}{n_0},$$
with $s_{10}^2$ and $s_{01}^2$ the sample variances of the two sets of placement values. For HbA1c the standard error is 0.0177.
The bootstrap resamples the 115 cases and the 1,860 non-cases separately, with replacement, and recomputes the AUC each time. With 2,000 resamples and 30 seeds, the median bootstrap standard error was 0.0176 (interquartile range 0.0173 to 0.0178).
DeLong's method and the bootstrap agree closely. The Hanley–McNeil formula is about 0.0025 larger; its authors found the exponential approximation slightly conservative for areas of 0.8 or more. The 95% interval from DeLong's standard error is $0.889 \pm 1.96 \times 0.0177 = (0.855, 0.924)$.
Check your understanding. Suppose a study kept the same 115 cases but doubled the non-cases to 3,720. Would the standard error of the AUC shrink by much?
Hardly at all: from 0.0177 to 0.0176, if the new non-cases resemble the old. In DeLong's formula the cases' term is 0.00030 and the non-cases' term 0.000009, so the cases supply 97% of the variance. Even with infinitely many non-cases, which removes the second term, the standard error would still be 0.0174. The standard error is driven by the 115 cases far more than by the 1,860 non-cases. A study with few cases has an imprecise AUC however many non-cases it includes.
Comparing two tests
Is HbA1c a better screen than BMI? On the 1,942 adults whose BMI was recorded (113 cases), the AUCs and their DeLong standard errors are:
- HbA1c, 0.887 (0.0180);
- BMI, 0.657 (0.0244);
- age, 0.676 (0.0216).
The course notes compare two AUCs with
$$z = \frac{\mathrm{AUC}_1 - \mathrm{AUC}_2}{\sqrt{\mathrm{SE}_1^2 + \mathrm{SE}_2^2}},$$
which is right when the two AUCs come from independent samples of patients. When both tests are measured on the same patients, as here and in most diagnostic studies, the two estimates are correlated. The variance of their difference is then $\mathrm{SE}_1^2 + \mathrm{SE}_2^2 - 2\,\mathrm{Cov}$. DeLong's method estimates the covariance from the same placement values, computed for each test on the same people. Hanley and McNeil (1983) gave an earlier correction based on the correlation between the tests.
Check your understanding. For HbA1c against BMI, the estimated correlation between the two AUCs is positive, 0.115. Will the paired standard error of the difference be larger or smaller than the unpaired one?
Smaller, because a positive covariance is subtracted. The difference is $0.887 - 0.657 = 0.2307$. Its paired standard error is 0.0286, giving $z = 8.06$, against 0.0303 and $z = 7.61$ from the unpaired formula. The conclusion does not change here, because the difference is large.
For BMI against age the correlation is negative, $-0.254$. The difference, $-0.019$, has a paired standard error of 0.0365 ($z = -0.53$, $p = 0.599$): the two risk factors are about equally weak screens. A positive correlation, the usual case for two tests of the same condition, makes the unpaired formula too conservative. A negative one makes it too liberal.
Choosing the cut-off
The ROC curve shows the options. Choosing among them needs a criterion, and the two common ones give different answers.
Youden's index
Youden (1950) proposed rating a test by $J = \mathrm{Se} + \mathrm{Sp} - 1$, the vertical distance from the diagonal to the point on the curve. Choosing the cut-off that maximizes $J$ treats a point of sensitivity and a point of specificity as equally valuable. For HbA1c the maximum, $J = 0.6312$, is at 6.0% (sensitivity 0.748, false positive rate 0.117), with 5.9% close behind at 0.6284. The point closest to the corner $(0, 1)$, another common criterion, is 5.9% here. The ADA's 6.5% gives $J = 0.456$.
A cut-off chosen to maximize $J$ on a dataset gives an optimistic estimate of $J$ on that dataset. The reason is the same as for a model that looks better on its training data. The honest estimate comes from new data or cross-validation, as the library's Binary Classification Metrics illustrates. More basically, Youden's criterion ignores both the prevalence and the consequences of the two errors.
Expected cost
Here is the idea. Give each kind of error a cost, weight each by how often it happens in the population screened, and choose the cut-off with the smallest total.
The formal version. Suppose a missed case costs $C_{\mathrm{FN}}$ and a false positive costs $C_{\mathrm{FP}}$, in any common unit, and correct results cost nothing. Among adults screened, a fraction $\pi$ have the condition, and each is missed with probability $1 - \mathrm{TPR}(c)$. A fraction $1 - \pi$ do not, and each is flagged with probability $\mathrm{FPR}(c)$. The expected cost per adult screened is
$$\mathrm{Cost}(c) = \pi\,C_{\mathrm{FN}}\,[1 - \mathrm{TPR}(c)] + (1 - \pi)\,C_{\mathrm{FP}}\,\mathrm{FPR}(c).$$
The cost is linear in the two rates, so all points of ROC space with the same expected cost lie on a straight line. Setting the cost constant and solving for $\mathrm{TPR}$ gives lines of slope
$$m = \frac{(1 - \pi)\,C_{\mathrm{FP}}}{\pi\,C_{\mathrm{FN}}}.$$
Lines further to the upper left have lower cost, and the best cut-off is where the highest such line touches the curve. Fawcett (2006, §6) calls these iso-performance lines and derives the same slope. When the condition is rare, $(1 - \pi)/\pi$ is large and the lines are steep. They then favor a strict cut-off with few false positives, unless missed cases are judged much more costly.
The slope of the ROC curve between two cut-offs is (change in TPR)/(change in FPR). That is the likelihood ratio of the results between them, as defined in Likelihood Ratios for Diagnostic Tests. The tangency condition therefore has a direct reading: call a result positive exactly when its likelihood ratio exceeds $m$, that is, when its post-test odds exceed $C_{\mathrm{FP}}/C_{\mathrm{FN}}$. On a curve whose slope decreases steadily (a concave curve), this is the same as choosing the cut-off.
Check your understanding. With the NHANES prevalence, $\pi = 0.0582$, and equal costs for the two errors, will the best cut-off be stricter or looser than the ADA's 6.5%?
Stricter. The slope is 16.17, so steep that the line touches the curve at 6.8%:
| $C_{\mathrm{FN}} / C_{\mathrm{FP}}$ | Slope $m$ | Best cut-off | Sensitivity | False positive rate | False positives per 1,000 | Missed cases per 1,000 |
|---|---|---|---|---|---|---|
| 1 | 16.17 | 6.8% | 0.339 | 0.004 | 3.5 | 38.5 |
| 5 | 3.23 | 6.2% | 0.626 | 0.048 | 45.6 | 21.8 |
| 10 | 1.62 | 6.1% | 0.687 | 0.073 | 68.4 | 18.2 |
| 50 | 0.32 | 5.5% | 0.948 | 0.508 | 478.0 | 3.0 |
The table makes the role of prevalence concrete. With equal costs, the best rule flags almost nobody, because at 5.8% prevalence nearly every flagged person would otherwise be a false positive. It misses two thirds of the cases. Only when a missed case is judged several times as costly as an unnecessary confirmatory test does the best cut-off move into the range that a screening program would use.
Neither 6.0% (Youden) nor 6.5% (the ADA) is optimal for any of these cost ratios in this population. That is not a defect of either. A diagnostic threshold in a guideline must serve many populations and purposes at once, while the cost calculation answers one question for one population. What the calculation does show is that a cut-off is a decision. It depends on the prevalence, and on judgments of cost that statistics cannot supply.
A different criterion fixes one error rate and optimizes the other. Allowing at most 5% false positives, the most sensitive HbA1c cut-off is 6.2%, with sensitivity 0.626. The area under the part of the curve over a range of false positive rates is the partial AUC. It summarizes a test over that range when only strict cut-offs are of practical interest.
What the ROC curve does not show
The ROC curve inherits the properties of sensitivity and specificity, including their limits.
- It does not depend on the prevalence. So it describes the test, but not what a positive result means for a patient. For that, the predictive values of Predictive Values and Bayes' Theorem are needed at the chosen cut-off.
- It does depend on the spectrum of cases and non-cases. A curve from a study of severe cases and healthy controls lies above the curve the test will achieve in screening, for the reasons given in Sensitivity and Specificity.
- Its area averages over all cut-offs, including those no one would use. Two tests with the same AUC can differ greatly in the region that matters.
Corrections to the course notes
The unit's notes describe the construction of the ROC curve, the reading of its corners and diagonal, $\mathrm{AUC} = P(X_{\text{diseased}} > X_{\text{healthy}})$ and the standard error of Hanley and McNeil as here. Stated more precisely, the AUC interpretation counts ties as one half, which matters for a test reported to one decimal place. The notes call the two quantities $Q_1$ and $Q_2$ in the standard error complex functions of the true and false positive rates. They are the probabilities of two cases outranking one non-case and of one case outranking two non-cases, with the approximations $A/(2 - A)$ and $2A^2/(1 + A)$.
The notes' $z$-statistic for comparing two AUCs, $(\mathrm{AUC}_1 - \mathrm{AUC}_2)/\sqrt{\mathrm{SE}_1^2 + \mathrm{SE}_2^2}$, applies to AUCs from independent groups of patients. When both tests are measured on the same patients, the covariance must be subtracted, as in DeLong's method. The notes' cost criterion $c_1(1 - \mathrm{Sensitivity}) + c_2(1 - \mathrm{Specificity})$ leaves out the prevalence; the expected cost per person weights the two terms by $\pi$ and $1 - \pi$. Without the weights, equal costs give Youden's cut-off of 6.0%. With them, the equal-cost optimum in this population is 6.8%. The notes' other criteria, Youden's index and the distance to $(0, 1)$, are as stated.
Summary
| Quantity | Formula | HbA1c |
|---|---|---|
| ROC point at cut-off $c$ | $(\mathrm{FPR}(c), \mathrm{TPR}(c))$ | 6.5%: (0.013, 0.470) |
| AUC | $U/(n_1 n_0)$, $P(X > Y) + \tfrac12 P(X = Y)$ | 0.889 (0.855, 0.924) |
| One cut-off as a curve | $(\mathrm{Se} + \mathrm{Sp})/2$ | 0.728 at 6.5% |
| Youden's index | $\max_c [\mathrm{Se}(c) + \mathrm{Sp}(c) - 1]$ | 0.631 at 6.0% |
| Iso-cost slope | $(1 - \pi)\,C_{\mathrm{FP}}/(\pi\,C_{\mathrm{FN}})$ | 16.2 for equal costs |
What we have learned
The ROC curve shows every trade-off between sensitivity and false positives that a measurement offers. Its area, 0.889 for HbA1c, is the chance that a random case outranks a random non-case, which makes it the Mann–Whitney statistic. It ignores the scale and the prevalence, and its precision is set by the number of cases. Comparing tests on the same adults needs the covariance between their AUCs. The curve cannot by itself pick a cut-off. Youden's index picks 6.0%. A cut-off that minimizes expected cost depends on the prevalence and on what each error costs: 6.8% at equal costs in this population, and 5.5% when a miss costs fifty times as much.
This unit has worked with events: positive or negative, case or non-case. The next unit, Theoretical Probability Distributions, turns to random variables, numbers whose values depend on chance, such as the count of positives among the people screened. It begins with The Binomial Distribution, the distribution of the number of successes in a fixed number of independent trials.
In Python
import numpy as np
from scipy.stats import mannwhitneyu
from sklearn.metrics import roc_auc_score, roc_curve
def auc_ci(score, disease, z=1.96):
"""AUC with DeLong's standard error (ties counted as 1/2)."""
s, d = np.asarray(score, float), np.asarray(disease, bool)
x, y = s[d], s[~d]
v10 = np.array([(np.sum(xi > y) + 0.5 * np.sum(xi == y)) / len(y) for xi in x])
v01 = np.array([(np.sum(x > yj) + 0.5 * np.sum(x == yj)) / len(x) for yj in y])
auc = v10.mean()
se = np.sqrt(v10.var(ddof=1) / len(x) + v01.var(ddof=1) / len(y))
return auc, auc - z * se, auc + z * se
# a1c: HbA1c values; fpg_high: FPG >= 126 (both arrays from NHANES 2017-2018)
# roc_auc_score(fpg_high, a1c) -> 0.8894
# mannwhitneyu(a1c[fpg_high], a1c[~fpg_high]).statistic / (115 * 1860) -> 0.8894
# auc_ci(a1c, fpg_high) -> (0.8894, 0.855, 0.924)
# fpr, tpr, thresholds = roc_curve(fpg_high, a1c) # the points of the curve
roc_curve returns a point for each distinct value at which the curve changes direction (all 49 with drop_intermediate=False); tied values produce the diagonal segments described above, and roc_auc_score counts ties as one half. Every number in this article is computed by tools/experiments/applied-statistics/roc-curves-for-diagnostic-tests.py in the site's repository, from the NHANES files loaded by screendata.py, which implements the Hanley–McNeil and DeLong standard errors directly.
References
- DeLong, E. R., DeLong, D. M., and Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 44(3), 837–845. record
- Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874. record · 2 pages
- Green, D. M., and Swets, J. A. (1966). Signal Detection Theory and Psychophysics. Wiley. Cited through Hanley and McNeil (1982). record
- Hanley, J. A., and McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29–36. record · 3 pages
- Hanley, J. A., and McNeil, B. J. (1983). A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology, 148(3), 839–843. record
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: DEMO_J, DIQ_J, BMX_J, GLU_J and GHB_J data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
- Pagano, M., Gauvreau, K., and Mattie, H. (2022). Principles of Biostatistics (3rd ed.). CRC Press. The course text. record · 35 pages
- Youden, W. J. (1950). Index for rating diagnostic tests. Cancer, 3(1), 32–35. record · 2 pages