From Scores to ROC Coordinates
A classifier can produce a score for each case, such as an estimated probability of belonging to a positive class. A threshold turns those scores into binary decisions: cases meeting the threshold are predicted positive, and the rest are predicted negative. Changing the threshold changes the confusion-matrix counts, so it also changes the pair of rates plotted on a receiver operating characteristic (ROC) curve.
This tutorial focuses on constructing those coordinates and recording which threshold produced each point. The classification rule and the confusion-matrix terms are stated explicitly so that the axes and denominators remain clear. This coordinate construction is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Here, TP counts actual positive cases predicted positive; FN counts actual positive cases predicted negative; FP counts actual negative cases predicted positive; and TN counts actual negative cases predicted negative. The horizontal coordinate uses only actual negatives as its denominator. The vertical coordinate uses only actual positives. The threshold is part of the point’s label: two coordinates without their thresholds do not record which classification rule produced them.
Why the Coordinates Have These Denominators
The rates can be understood as conditional proportions within the two actual classes. Among all actual positives, the fraction correctly classified positive is sensitivity. Among all actual negatives, the fraction incorrectly classified positive is the false positive rate. In probability notation, for a randomly selected case from the evaluation population, these are \(P(\text{predicted positive}\mid\text{actual positive})\) and \(P(\text{predicted positive}\mid\text{actual negative})\), respectively.
The empirical formulas follow directly by counting cases in each actual class. There are \(\mathrm{TP}+\mathrm{FN}\) actual positives, of which TP are predicted positive. There are \(\mathrm{FP}+\mathrm{TN}\) actual negatives, of which FP are predicted positive. Therefore:
These formulas require at least one actual positive and at least one actual negative in the evaluation sample. If either class is absent, the corresponding denominator is zero and that coordinate cannot be calculated from that sample. Also, these are empirical rates for the cases evaluated; they are not guaranteed to equal the rates in a different population.
Constructing Points at Several Thresholds
A practical construction starts by fixing the positive class, score direction, threshold rule, and evaluation sample. Then, at each chosen threshold, count TP, FN, FP, and TN; calculate FPR and sensitivity; and plot FPR on the horizontal axis and sensitivity on the vertical axis. Record the threshold next to the point or in a matching table.
For this tutorial, larger scores indicate the positive class and the rule is \(s \geq c\). State any different convention before calculating.
At each threshold, count predicted positives among actual positives and actual negatives. These give TP and FP; the remaining cases in each class are FN and TN.
Divide TP by the total number of actual positives, divide FP by the total number of actual negatives, and record the point as \((\mathrm{FPR},\mathrm{TPR})\) together with \(c\).
For a finite evaluation sample with at least one case in each class, the threshold can be set above the largest score so that no case is predicted positive. This gives \((0,0)\). A threshold at or below the smallest score predicts every case positive and gives \((1,1)\). Between these endpoints, lowering the threshold cannot remove a case from the predicted-positive group under the \(s \geq c\) rule. Consequently, neither coordinate decreases as the threshold is lowered.
Worked Coordinates
Worked Example: Calculating Coordinates from Individual Scores
A hypothetical claims review model is evaluated on eight claims. A claim is actually positive if it truly requires additional review. The model assigns a score, with higher scores indicating greater concern. Use the rule “predict positive when the score is at least the threshold” and calculate the ROC coordinates for thresholds 0.90, 0.70, 0.50, and 0.30. Include the high-threshold endpoint.
| Claim | Actual class | Score |
|---|---|---|
| A | Positive | 0.95 |
| B | Positive | 0.75 |
| C | Positive | 0.55 |
| D | Positive | 0.25 |
| E | Negative | 0.85 |
| F | Negative | 0.65 |
| G | Negative | 0.45 |
| H | Negative | 0.15 |
Method. There are four actual positives and four actual negatives, so sensitivity is TP divided by 4 and FPR is FP divided by 4 at every threshold.
At \(c=0.90\), only A is predicted positive. Thus TP \(=1\), FN \(=3\), FP \(=0\), TN \(=4\). Sensitivity \(=1/4=0.25\), and FPR \(=0/4=0\). The coordinate is \((0,0.25)\).
At \(c=0.70\), A, B, and E are predicted positive. Thus TP \(=2\), FN \(=2\), FP \(=1\), TN \(=3\). Sensitivity \(=2/4=0.50\), and FPR \(=1/4=0.25\). The coordinate is \((0.25,0.50)\).
At \(c=0.50\), A, B, C, E, and F are predicted positive. Thus TP \(=3\), FN \(=1\), FP \(=2\), TN \(=2\). Sensitivity \(=3/4=0.75\), and FPR \(=2/4=0.50\). The coordinate is \((0.50,0.75)\).
At \(c=0.30\), A, B, C, E, F, and G are predicted positive. Thus TP \(=3\), FN \(=1\), FP \(=3\), TN \(=1\). Sensitivity \(=3/4=0.75\), and FPR \(=3/4=0.75\). The coordinate is \((0.75,0.75)\).
The largest score is 0.95. At any threshold greater than 0.95, no claim is predicted positive, giving TP \(=0\) and FP \(=0\), hence the endpoint \((0,0)\). In threshold order, the labelled points are: above 0.95: \((0,0)\); 0.90: \((0,0.25)\); 0.70: \((0.25,0.50)\); 0.50: \((0.50,0.75)\); and 0.30: \((0.75,0.75)\). The method avoids the common mistake of putting sensitivity on the horizontal axis.
Worked Example: Unequal Class Counts and the Correct Denominators
A hypothetical fraud-screening model is evaluated on 42 transactions: 12 are actually fraudulent and 30 are not. The following table groups transactions by their score. Use thresholds 0.91, 0.90, 0.60, and 0.20, with a transaction predicted positive when its score is at least the threshold.
| Actual class | Score 0.90 | Score 0.60 | Score 0.20 | Total |
|---|---|---|---|---|
| Positive | 3 | 4 | 5 | 12 |
| Negative | 1 | 5 | 24 | 30 |
Method. Accumulate the score groups meeting the threshold. The sensitivity denominator remains 12, and the FPR denominator remains 30.
At \(c=0.91\), no scores meet the threshold, so TP \(=0\), FN \(=12\), FP \(=0\), and TN \(=30\). The coordinate is \((0/30,0/12)=(0,0)\).
At \(c=0.90\), only the 0.90 group is included: TP \(=3\), FN \(=12-3=9\), FP \(=1\), and TN \(=30-1=29\). Sensitivity \(=3/12=0.2500\); FPR \(=1/30=0.0333\) to four decimal places. The coordinate is \((0.0333,0.2500)\).
At \(c=0.60\), both the 0.90 and 0.60 groups are included: TP \(=3+4=7\), FN \(=12-7=5\), FP \(=1+5=6\), and TN \(=30-6=24\). Sensitivity \(=7/12=0.5833\); FPR \(=6/30=0.2000\). The coordinate is \((0.2000,0.5833)\).
At \(c=0.20\), all transactions are predicted positive: TP \(=12\), FN \(=0\), FP \(=30\), and TN \(=0\). The coordinate is \((1.0000,1.0000)\). Each point is labelled with the threshold that produced it.
Common mistake and correction. Dividing the one false positive at threshold 0.90 by all 42 transactions gives \(1/42=0.0238\), but that is not the FPR. The FPR denominator is the 30 actual negatives, so the correct value is \(1/30=0.0333\). The class counts are unequal, but each rate is still calculated within its own actual class.
Worked Example: Labelling Thresholds on a ROC Plot
A hypothetical equipment-maintenance model gives scores for six components that truly need maintenance and six that do not. Larger scores indicate greater maintenance risk. Calculate and label the coordinates at thresholds 0.70, 0.50, and 0.20, and state both endpoints.
The actual-positive scores are 0.92, 0.71, 0.66, 0.38, 0.22, and 0.08. The actual-negative scores are 0.88, 0.73, 0.51, 0.34, 0.19, and 0.04. Thus each rate has denominator 6.
At \(c=0.70\): two actual positives (0.92 and 0.71) and two actual negatives (0.88 and 0.73) meet the threshold. TP \(=2\), FN \(=4\), FP \(=2\), TN \(=4\). Sensitivity \(=2/6=0.3333\), FPR \(=2/6=0.3333\). Label the point \(c=0.70\) at \((0.3333,0.3333)\).
At \(c=0.50\): three actual positives (0.92, 0.71, 0.66) and three actual negatives (0.88, 0.73, 0.51) meet the threshold. TP \(=3\), FN \(=3\), FP \(=3\), TN \(=3\). Sensitivity \(=3/6=0.5000\), FPR \(=3/6=0.5000\). Label the point \(c=0.50\) at \((0.5000,0.5000)\).
At \(c=0.20\): five actual positives and four actual negatives meet the threshold. TP \(=5\), FN \(=1\), FP \(=4\), TN \(=2\). Sensitivity \(=5/6=0.8333\), FPR \(=4/6=0.6667\). Label the point \(c=0.20\) at \((0.6667,0.8333)\).
A threshold above 0.92 predicts no component positive, giving \((0,0)\). A threshold at or below 0.04 predicts all components positive, giving \((1,1)\). In order of decreasing threshold, the points are therefore above 0.92: \((0,0)\); 0.70: \((0.3333,0.3333)\); 0.50: \((0.5000,0.5000)\); 0.20: \((0.6667,0.8333)\); and at or below 0.04: \((1,1)\). The point labels preserve the decision rule associated with each coordinate.
Common Construction Errors
A ROC coordinate is simple to calculate once the classes, threshold rule, and axes are fixed. Most errors arise from changing one of those conventions without noticing.
- Swapping the axes. The x-coordinate is FPR; the y-coordinate is sensitivity. Write the ordered pair explicitly as \((\mathrm{FPR},\mathrm{TPR})\) before plotting.
- Using the total sample size in a rate. FPR is divided by the number of actual negatives, not the whole sample; sensitivity is divided by the number of actual positives.
- Forgetting the inequality convention. Under the rule \(s \geq c\), a score equal to the threshold is predicted positive. If the rule instead uses \(s>c\), cases exactly at the threshold are treated differently.
- Leaving points unlabeled. A plotted coordinate should be connected to its threshold, either by a nearby label or a table. Otherwise, the decision rule cannot be recovered from the point alone.
- Reversing the score direction. If smaller scores indicate the positive class, the rule must reflect that direction. Applying \(s \geq c\) without checking score meaning can classify the wrong cases as positive.
For a set of evaluated cases, only thresholds that change at least one predicted class need to be considered to obtain distinct empirical coordinates. Thresholds that produce the same predictions produce the same counts and therefore the same coordinate. When scores are equal, the stated threshold rule classifies all cases at that score together; the coordinate calculation must follow the rule rather than split equal-score cases arbitrarily.
Check Your Understanding
For each problem, use the stated threshold rule and report coordinates in the order \((\mathrm{FPR},\mathrm{TPR})\).
- A validation set has five actual positives with scores 0.90, 0.70, 0.30, 0.10, and 0.80, and ten actual negatives with scores 0.95, 0.65, 0.40, 0.20, 0.15, 0.50, 0.25, 0.60, 0.05, and 0.35. With prediction positive when \(s \geq 0.70\), calculate the coordinate. (Answer: \((0.1000,0.6000)\))
- A sample has eight actual positives and 12 actual negatives. At threshold 0.50, TP \(=6\) and FP \(=3\). Calculate the ROC coordinate. (Answer: \((0.2500,0.7500)\))
- There are four actual positives and six actual negatives. At threshold 0.40, TP \(=3\), FN \(=1\), FP \(=2\), and TN \(=4\). State the labelled coordinate and verify both class totals. (Answer: threshold 0.40, \((0.3333,0.7500)\); actual-positive total \(=4\), actual-negative total \(=6\))
- A finite validation sample contains at least one case from each actual class. All scores lie between 0.12 and 0.88, inclusive. Under the rule \(s \geq c\), state the coordinates for \(c=0.90\) and \(c=0.12\). (Answer: \(c=0.90\): \((0,0)\); \(c=0.12\): \((1,1)\))
- At a particular threshold, a model has 18 actual positives and 42 actual negatives. It correctly predicts 12 positives and predicts 7 negatives as positive. Calculate sensitivity and FPR, then give the coordinate. (Answer: sensitivity \(=12/18=0.6667\), FPR \(=7/42=0.1667\), coordinate \((0.1667,0.6667)\))