Interpreting ROC Curves as Operating Trade-Offs

Reading an ROC Operating Point

A classifier can assign each case a score, then label a case positive when its score meets or exceeds a chosen threshold. Changing that threshold changes the classification results. An ROC curve summarizes those results across thresholds, but each point on the curve describes one particular operating point: one threshold and the resulting balance between positive cases detected and negative cases incorrectly flagged.

The vertical coordinate records sensitivity, the proportion of actual positives classified as positive. The horizontal coordinate records the false positive rate, the proportion of actual negatives classified as positive. Thus an ROC point \((0.20, 0.75)\) means a false positive rate of 0.20 and sensitivity of 0.75. It does not mean that 20% of predicted positives are false; that is a different conditional proportion.

Definition: At a specified classification threshold, let \(TP\), \(FN\), \(FP\), and \(TN\) denote the numbers of true positives, false negatives, false positives, and true negatives. Sensitivity (also called the true positive rate) is \(TP/(TP+FN)\). The false positive rate is \(FP/(FP+TN)\), which equals \(1-\text{specificity}\). An ROC operating point is \((\text{false positive rate},\text{sensitivity})\), with the false positive rate on the horizontal axis and sensitivity on the vertical axis.

The denominators are important. Sensitivity is calculated among all actual positives, whether detected or missed. The false positive rate is calculated among all actual negatives, whether incorrectly flagged or correctly left negative. Neither rate is calculated among all cases or among all cases predicted positive.

\text{Sensitivity}=\frac{TP}{TP+FN}, \qquad \text{False positive rate}=\frac{FP}{FP+TN} =1-\frac{TN}{FP+TN} =1-\text{Specificity}.

An operating point is easiest to interpret by naming both rates in context. For example, a sensitivity of 0.75 means that the classifier detects 75% of the actual positive cases at that threshold. A false positive rate of 0.20 means that it incorrectly flags 20% of the actual negative cases. These are class-conditional rates: they describe performance within each actual class.

Why Lowering the Threshold Changes Both Coordinates

Suppose higher scores indicate stronger evidence for the positive class, and the rule is to predict positive when the score is at least \(c\). If the threshold is lowered from \(c_1\) to \(c_2\), where \(c_2<c_1\), every case previously classified positive remains positive, and some additional cases may become positive. Some of those additional cases are actually positive; others are actually negative.

The newly included actual positives increase the true-positive count, while the total number of actual positives stays fixed. The newly included actual negatives increase the false-positive count, while the total number of actual negatives stays fixed. This explains why lowering the threshold can increase sensitivity and the false positive rate together.

\text{New predicted-positive set} \supseteq \text{Old predicted-positive set} \quad\Longrightarrow\quad TP_{\text{new}}\geq TP_{\text{old}} \ \text{and}\ FP_{\text{new}}\geq FP_{\text{old}}.

Dividing by the fixed class totals preserves these directions: sensitivity cannot decrease, and the false positive rate cannot decrease when the threshold is lowered. The rates may stay unchanged if no cases in a class are added. In a finite data set, scores are often tied, so lowering the threshold past a tied score can add several cases at once. The ROC path therefore may change in steps rather than move smoothly.

Fix the actual class totals.
The number of actual positives, \(TP+FN\), and actual negatives, \(FP+TN\), do not change when only the threshold changes.

Identify the cases newly called positive.
Lowering the threshold adds cases to the predicted-positive group; it does not remove cases that were already in that group.

Separate those cases by actual class.
New actual positives add true positives; new actual negatives add false positives.

Recalculate the ROC coordinates.
Divide the updated true-positive count by all actual positives, and the updated false-positive count by all actual negatives.

This is a count-based justification for the direction of movement; it does not imply that every threshold change produces the same increase in the two rates. The newly included cases determine the size of each change. A move to the right on the ROC plot means a higher false positive rate. A move upward means higher sensitivity. When lowering a score threshold, both coordinates are nondecreasing, although one can remain unchanged over part of the move.

Worked Examples: Translating Operating Points Into Actions

Worked Example: A Lower Threshold Detects More Cases

A hypothetical screening model assigns a risk score to 1,000 applicants. Of these, 120 are actually positive and 880 are actually negative. At threshold \(c_1\), the model has \(TP=90\), \(FN=30\), \(FP=176\), and \(TN=704\). At a lower threshold \(c_2\), it has \(TP=108\), \(FN=12\), \(FP=264\), and \(TN=616\). Find both ROC points and describe the change in operating trade-off. Use the method of calculating each rate from its actual-class denominator.

Step 1: Check the actual class totals. At \(c_1\), actual positives are \(90+30=120\), and actual negatives are \(176+704=880\). At \(c_2\), actual positives are \(108+12=120\), and actual negatives are \(264+616=880\). The class totals are consistent across thresholds.

Step 2: Calculate the point at \(c_1\). Sensitivity is \(90/120=0.75\). The false positive rate is \(176/880=0.20\). The ROC point, written as (false positive rate, sensitivity), is \((0.20,0.75)\). Specificity is \(704/880=0.80\), which checks because \(1-0.80=0.20\).

Step 3: Calculate the point at \(c_2\). Sensitivity is \(108/120=0.90\). The false positive rate is \(264/880=0.30\). The point is \((0.30,0.90)\). Specificity is \(616/880=0.70\), and \(1-0.70=0.30\), as required.

Step 4: Interpret the change. Lowering the threshold increases sensitivity by \(0.90-0.75=0.15\), or 15 percentage points, and increases the false positive rate by \(0.30-0.20=0.10\), or 10 percentage points. In counts, the model detects \(108-90=18\) additional actual positives and flags \(264-176=88\) additional actual negatives.

Check. The false negatives fall from 30 to 12, a reduction of 18, matching the increase in true positives. The true negatives fall from 704 to 616, a reduction of 88, matching the increase in false positives. The method avoids treating the 15-point sensitivity gain as if it were a 15-point increase in false alarms; the two changes have different denominators.

Worked Example: Interpreting Two Points With Fixed Class Totals

A hypothetical claims-review model is evaluated on 1,000 claims: 250 are actually positive and 750 are actually negative. At a higher threshold, the ROC coordinates are \((0.04,0.72)\). At a lower threshold, they are \((0.12,0.84)\). Find the four confusion-matrix counts at each point, then describe the operational change. Use the coordinates with the actual class totals, rather than treating them as proportions of all claims.

Step 1: Convert the higher-threshold point. Sensitivity \(0.72\) gives \(TP=0.72(250)=180\). Therefore \(FN=250-180=70\). The false positive rate \(0.04\) gives \(FP=0.04(750)=30\). Therefore \(TN=750-30=720\).

Step 2: Convert the lower-threshold point. Sensitivity \(0.84\) gives \(TP=0.84(250)=210\), so \(FN=250-210=40\). The false positive rate \(0.12\) gives \(FP=0.12(750)=90\), so \(TN=750-90=660\).

Step 3: Describe both rate changes and count changes. Sensitivity rises by \(0.84-0.72=0.12\), or 12 percentage points. The false positive rate rises by \(0.12-0.04=0.08\), or 8 percentage points. The lower threshold detects \(210-180=30\) additional actual positives and flags \(90-30=60\) additional actual negatives.

Step 4: Verify the converted points. At the higher threshold, \(180/250=0.72\) and \(30/750=0.04\). At the lower threshold, \(210/250=0.84\) and \(90/750=0.12\). In each case, the positive and negative counts sum to 250 and 750, respectively.

The count changes are not equal to the percentage-point changes because the rate denominators differ. The method avoids multiplying either coordinate by all 1,000 claims: sensitivity uses 250 actual positives, and the false positive rate uses 750 actual negatives.

Worked Example: Distinguishing False Positive Rate From False Discovery Proportion

A hypothetical equipment-alert model is applied to 1,000 inspection records. There are 80 actual positive records and 920 actual negative records. At one threshold, the ROC point is \((0.25,0.60)\). Find the confusion-matrix counts and the proportion of predicted-positive records that are actually negative. Explain why this last quantity is not the false positive rate.

Step 1: Calculate counts from the ROC coordinates. Sensitivity \(0.60\) means \(TP=0.60(80)=48\), so \(FN=80-48=32\). A false positive rate of \(0.25\) means \(FP=0.25(920)=230\), so \(TN=920-230=690\).

Step 2: Calculate the requested predicted-positive proportion. There are \(TP+FP=48+230=278\) predicted-positive records. Of these, 230 are actually negative. The proportion that are actually negative is \(230/278\approx0.8273\), or 82.73% to two decimal places.

Step 3: Contrast the denominators. The false positive rate is \(FP/(FP+TN)=230/920=0.25\), or 25%. The proportion of predicted positives that are false is \(FP/(TP+FP)=230/278\approx0.8273\). The first conditions on being actually negative; the second conditions on being predicted positive.

Check. The counts total \(48+32+230+690=1{,}000\), with 80 actual positives and 920 actual negatives. The method avoids swapping the ROC horizontal coordinate for the proportion of alerts that are false.

How to Explain the Trade-Off Precisely

When describing movement along an ROC curve, state the direction of both rates and, when available, the associated counts. For example: “Lowering the threshold increased sensitivity from 0.72 to 0.84, while the false positive rate increased from 0.04 to 0.12. On this validation set, that corresponded to 30 additional actual positives detected and 60 additional actual negatives flagged.” This distinguishes the rate changes from the case-count changes and keeps the interpretation tied to the actual class totals.

A sensitivity increase means fewer actual positives are missed, because the actual-positive denominator is fixed and the true-positive count has increased. A false positive rate increase means fewer actual negatives are correctly left unflagged, because the actual-negative denominator is fixed and the false-positive count has increased. Equivalently, specificity decreases as the false positive rate increases: \(\text{specificity}=1-\text{FPR}\).

The ROC plot describes these two class-conditional rates, not the total number of cases, the fraction of all cases flagged, or the mix of actual classes in a particular population. The same ROC coordinates can correspond to different numbers of true and false positives if the evaluation populations have different class totals. To recover counts, the actual positive and negative totals must be known.

Common Mistakes and Checks

Swapping the axes. ROC convention places false positive rate on the horizontal axis and sensitivity on the vertical axis. Read the coordinate labels before interpreting a point.
Using the wrong denominator. Divide true positives by all actual positives for sensitivity; divide false positives by all actual negatives for false positive rate.
Calling false positive rate the false-alert proportion. \(FP/(FP+TN)\) and \(FP/(TP+FP)\) answer different questions. Check what group is in the denominator.
Claiming that lowering the threshold improves only sensitivity. It can also increase the false positive rate, because newly included cases may be actual negatives.
Assuming the coordinate changes must match. Sensitivity and false positive rate have different denominators, so their changes need not be equal in percentage points or case counts.

Key takeaway: An ROC operating point is read as (false positive rate, sensitivity). Lowering a score threshold expands the predicted-positive group, so neither rate can decrease; the resulting movement reflects how the added cases divide between actual positives and actual negatives.

Check Your Understanding

For each problem, calculate or interpret the ROC operating point using the actual-class denominators.

A hypothetical model is evaluated on 200 actual positives and 800 actual negatives. At a threshold, it has \(TP=150\) and \(FP=120\). Find the ROC point and specificity. (Answer: \((0.1500, 0.7500)\); specificity \(=0.8500\).)
A classifier has 60 actual positives and 540 actual negatives. Its ROC point is \((0.10,0.80)\). Find \(TP\), \(FN\), \(FP\), and \(TN\). (Answer: \(TP=48,\ FN=12,\ FP=54,\ TN=486\).)
In a validation set with 400 actual positives and 600 actual negatives, lowering a threshold changes sensitivity from 0.65 to 0.80 and the false positive rate from 0.05 to 0.15. How many additional actual positives are detected, and how many additional actual negatives are flagged? (Answer: 60 additional actual positives and 60 additional actual negatives.)
A data set contains 100 actual positives and 900 actual negatives. At an operating point, sensitivity is 0.70 and the false positive rate is 0.20. What proportion of predicted-positive cases are actually negative? (Answer: \(180/(70+180)=0.7200\), or 72.00%.)
At threshold \(c_1\), a model has \(TP=72,\ FN=28,\ FP=90,\ TN=810\). At a lower threshold \(c_2\), it has \(TP=84,\ FN=16,\ FP=180,\ TN=720\). Find both ROC points and state the percentage-point change in each coordinate. (Answer: \(c_1:(0.1000,0.7200)\); \(c_2:(0.2000,0.8400)\); false positive rate increases by 10 percentage points and sensitivity increases by 12 percentage points.)

What You'll Learn