Discrimination With ROC Curves and AUC

Why Discrimination Is a Ranking Question

A genetic prediction model may assign each person a predicted risk of a binary outcome, such as developing a disease within a specified time period. One question is whether people who develop the outcome tend to receive higher predicted risks than people who do not. This ability to separate the two outcome groups is called discrimination.

A receiver operating characteristic curve, usually called an ROC curve, displays how sensitivity and the false-positive rate change as the risk threshold for classifying someone as positive changes. The area under the ROC curve, or AUC, summarizes this ranking performance across thresholds. AUC is useful, but it answers a narrower question than whether a model gives accurate risk estimates to individual people.

In a sample with \(n_1\) cases and \(n_0\) controls, sensitivity at threshold \(t\) is the number of cases with predicted risk at least \(t\), divided by \(n_1\). The false-positive rate is the number of controls with predicted risk at least \(t\), divided by \(n_0\). Thus, the horizontal axis runs from 0 to 1 in false-positive rate, and the vertical axis runs from 0 to 1 in sensitivity. Each threshold supplies one point; changing the threshold traces out the curve.

A higher threshold generally labels fewer people positive: sensitivity cannot increase, and the false-positive rate cannot increase. The ROC curve therefore shows the trade-off in these two quantities as the classification rule changes. It does not select a threshold or state which trade-off is appropriate for a particular decision.

Calculating AUC From Predicted Risks

For a finite validation sample, the empirical AUC can be calculated by comparing each case's predicted risk with each control's predicted risk. A case-control pair is concordant if the case has the higher risk, discordant if the control has the higher risk, and tied if their risks are equal. Each concordant pair contributes 1, each discordant pair contributes 0, and each tied pair contributes one-half.

This pairwise definition is equivalent to the area under the empirical ROC curve, using straight-line segments between successive ROC points. It also specifies how tied scores are treated: ties receive half credit, rather than being counted as either fully correct or fully incorrect.

Reading an ROC Curve and Its Area

An ROC curve near the upper-left corner has high sensitivity at relatively low false-positive rates. A curve near the diagonal from \((0,0)\) to \((1,1)\) has discrimination similar to chance ranking. AUC ranges from 0 to 1: 1 means every case-control pair is correctly ordered, while 0 means every pair is ordered in the opposite direction. An AUC below 0.5 can indicate that the score tends to rank controls above cases; reversing the score direction would reverse the ranking.

AUC does not describe performance at just one threshold. Two models can have the same AUC but different ROC curves, and their relative performance may differ in parts of the curve. If a particular threshold is used, sensitivity and false-positive rate at that threshold should be reported in context rather than inferred from AUC alone.

The pairwise definition and the ROC-area definition agree. For example, in the tied-score data above, lowering the threshold first includes the case at 0.90, then the control at 0.80, and then the tied group at 0.60. That tied group adds both cases and one control at the same score. The straight-line segment across this group gives half credit to case-control comparisons within the tied score, as in the pairwise calculation.

Why the Pairwise Interpretation Holds

Suppose one case and one control are drawn independently from the population being evaluated. The population AUC is the probability that the case has a higher score than the control, plus one-half the probability that their scores tie. The empirical formula estimates this quantity by examining every case-control pair in the validation sample.

The equivalence to ROC area can be understood by considering thresholds in descending order. As the threshold passes a score, the people with that score enter the predicted-positive group. A case entering alone contributes area corresponding to the controls already passed; a control entering alone changes the horizontal coordinate. If cases and controls enter together at a tied score, the straight-line ROC segment assigns half credit to comparisons within that tied group. Summing these contributions over all score levels counts each case-control pair once: full credit when the case score is higher, zero when the control score is higher, and half credit when scores tie. Dividing by \(n_1n_0\) yields the pairwise formula.

This interpretation requires a binary outcome with at least one case and one control, and a score whose direction is specified. It does not require predicted risks to be well calibrated, nor does it rely on a particular parametric distribution for the scores. The population interpretation concerns cases and controls from the population of interest; the empirical result describes the validation sample and may vary across samples.

Discrimination Does Not Establish Risk Accuracy

AUC depends on the ordering of scores, not on whether their numerical values match outcome probabilities. Any strictly increasing transformation of all scores preserves their ordering and therefore preserves AUC. For instance, changing the scale of predicted risks while leaving every rank unchanged can produce exactly the same AUC, even though the numbers presented as risks differ substantially.

Discrimination is only one aspect of prediction performance. A model can rank people usefully while systematically assigning risk values that are too high or too low. Conversely, accurate average risk estimates do not guarantee strong separation of cases from controls. Assessing whether predicted probabilities agree with observed outcome frequencies requires calibration methods, which are distinct from ROC curves and AUC.

AUC also does not describe the consequences of errors or identify the best classification threshold. The relative importance of missed cases and false alarms depends on the decision and its context. A single AUC cannot replace reporting the operating characteristics at a relevant threshold or evaluating whether using the prediction improves a decision.

Common Mistakes and Practical Checks

• Calling AUC the proportion of people classified correctly. AUC summarizes all case-control rankings across thresholds; it is not accuracy at one chosen threshold.
• Interpreting AUC as individual risk accuracy. AUC concerns ordering between a case and a control. It does not establish that predicted probabilities are numerically accurate.
• Ignoring ties. Counting a tied case-control pair as fully correct or fully incorrect changes the empirical calculation. Assign half credit.
• Reversing the score direction. The standard interpretation assumes larger scores indicate greater outcome risk. If smaller scores indicate greater risk, reverse the direction before interpreting AUC.
• Treating a single AUC as a complete comparison. Equal AUCs do not imply identical ROC curves, threshold-specific performance, risk estimates, or decision consequences.