Why Tied Scores Need Special Care
An empirical ROC curve records how the true-positive rate and false-positive rate change as a score threshold moves. When several observations have exactly the same predicted score, lowering the threshold across that score includes all of them at once. It cannot include one member of the group while leaving another out. That constraint matters: splitting a tied group can create ROC coordinates that no threshold can actually produce.
The construction uses observed labels and scores, not an assumed distribution for either. Assume throughout that higher scores indicate stronger evidence for the positive class and that an observation is classified as positive when its score is at least the threshold. The positive and negative labels are known for the evaluation sample.
To construct the curve, sort the distinct observed scores from highest to lowest and process each distinct score as one group. At a score threshold equal to a group’s score, that group and all higher-scoring groups are classified positive. The resulting points are the operating points attainable from thresholds on the recorded scores. A group may contain positive cases, negative cases, or both.
Building the Curve by Score Groups
For each score group, count its positive and negative cases. Adding a group with \(g_+\) positive cases and \(g_-\) negative cases increases the ROC coordinates by \(g_+/n_+\) vertically and \(g_-/n_-\) horizontally. Both changes occur together when the group contains both labels.
The table below shows the bookkeeping. The “cumulative” counts include the current group and every group with a higher score. The initial row corresponds to a threshold above the largest observed score.
| Threshold step | Cumulative true positives | Cumulative false positives | ROC point |
|---|---|---|---|
| Above the largest score | 0 | 0 | (0, 0) |
| Include next distinct score group | Add that group’s positive count | Add that group’s negative count | (FP / \(n_-\), TP / \(n_+\)) |
| Continue to the smallest score | Eventually \(n_+\) | Eventually \(n_-\) | (1, 1) |
Worked Example: Constructing Points From Individual Scores
A hypothetical screening model assigns scores to 12 cases. The table gives the number of positive and negative cases at each distinct score. Construct the attainable ROC points in descending score order and check the endpoints.
| Score | Positive cases | Negative cases |
|---|---|---|
| 0.90 | 2 | 0 |
| 0.70 | 1 | 2 |
| 0.40 | 2 | 2 |
| 0.10 | 0 | 3 |
Method. First total the class counts: \(n_+=2+1+2+0=5\) and \(n_-=0+2+2+3=7\). Start at \((0,0)\). At score 0.90, add two positives and no negatives, giving \(TP=2\), \(FP=0\), and the point \((0/7,2/5)=(0,0.4)\).
At score 0.70, add one positive and two negatives. The cumulative counts are \(TP=2+1=3\) and \(FP=0+2=2\), so the point is \((2/7,3/5)\). At score 0.40, add two positives and two negatives: \(TP=3+2=5\), \(FP=2+2=4\), giving \((4/7,5/5)=(4/7,1)\). At score 0.10, add no positives and three negatives, giving \(TP=5\), \(FP=7\), and \((7/7,5/5)=(1,1)\).
The ordered points are \((0,0)\), \((0,0.4)\), \((2/7,0.6)\), \((4/7,1)\), and \((1,1)\). The final point checks: every one of the five positive and seven negative cases has been included. The common mistake this method avoids is using the total sample size as the denominator for either rate; each rate uses its own class total.
Why a Tied Group Must Enter Together
A threshold rule acts only on the recorded score: it predicts positive when score \(\geq c\). If two cases have the same score \(s\), then for every threshold \(c\), either both satisfy \(s\geq c\) or neither does. Their labels do not alter this fact. Thus, if a tied group contains both classes, lowering the threshold past its score increases both the true-positive count and the false-positive count in one step.
This also justifies the coordinate increments. Before adding a score group, suppose the counts are \(TP\) and \(FP\). Including the whole group changes them to \(TP+g_+\) and \(FP+g_-\). The denominators \(n_+\) and \(n_-\) are fixed for the evaluation sample, so the changes in the rates are exactly \(g_+/n_+\) and \(g_-/n_-\). No intermediate count is available from a deterministic threshold if it would include only part of the tied group.
The words “stepwise ROC curve” refer to the discrete changes as the threshold moves through the observed scores. ROC displays often connect successive attainable points with straight lines so the plot is easier to read. That connecting line is a display convention: its interior points are not necessarily attainable by any deterministic threshold. In particular, a diagonal segment caused by a mixed-label tie does not mean a threshold can select just some cases from that score group.
Worked Example: A Mixed-Label Tie Creates a Joint Jump
A hypothetical classifier is evaluated on 10 applications. At score 0.95 there is one positive case and no negative cases; at score 0.80 there is one positive and two negative cases; at score 0.60 there are two positive and one negative cases; and at score 0.30 there are two positive and one negative cases. Find the ROC points and explain what happens at threshold 0.80.
Method. The class totals are \(n_+=1+1+2+2=6\) and \(n_-=0+2+1+1=4\). Above score 0.95, the point is \((0,0)\). Including the 0.95 group gives \(TP=1\), \(FP=0\), hence \((0,1/6)\).
At threshold 0.80, the whole group with that score is included. The cumulative counts become \(TP=1+1=2\) and \(FP=0+2=2\), giving \((2/4,2/6)=(0.5,1/3)\). This group causes a horizontal increase of \(2/4=0.5\) and a vertical increase of \(1/6\), both from the preceding point.
Including the 0.60 group gives \(TP=2+2=4\), \(FP=2+1=3\), so the point is \((3/4,4/6)=(0.75,2/3)\). Including the 0.30 group gives \(TP=6\), \(FP=4\), so the endpoint is \((1,1)\). The ordered points are \((0,0)\), \((0,1/6)\), \((0.5,1/3)\), \((0.75,2/3)\), and \((1,1)\).
At threshold 0.80, the two negative cases and one positive case in that group all enter together. The point after the group is attainable; a point that counts only one of its two negative cases is not. The endpoint confirms that all six positives and all four negatives have been counted.
Threshold Points and the Connected Display
A candidate should distinguish the finite set of threshold outcomes from the appearance of a plotted curve. With \(K\) distinct observed scores, there are \(K+1\) threshold outcomes when the two extremes are included: predict none positive, then include each score group in turn, and finally predict all positive. Repeated observations do not create extra threshold steps.
The shape of each jump follows the labels in its score group. A positive-only group moves upward; a negative-only group moves right; a mixed group moves up and right. The size of a movement depends on both the number of cases in the group and the total number of cases in that class. For example, one positive case moves \(TPR\) by \(1/n_+\), not by \(1/(n_++n_-)\).
Worked Example: Rejecting an Unattainable Point Within a Tie
A hypothetical model has 8 positive and 8 negative cases. At score 0.88 there is one positive and one negative case; at score 0.55 there are two positive and one negative cases; at score 0.20 there is one positive and two negative cases; and at score 0.05 there are four positive and four negative cases. A proposed ROC table lists a point after including the two positive cases at score 0.55 but before including the negative case at that same score. Determine whether the point is attainable and give the correct points around that score.
Method. First verify the totals: positives \(1+2+1+4=8\), negatives \(1+1+2+4=8\). After including the 0.88 group, \(TP=1\) and \(FP=1\), giving \((1/8,1/8)\).
At score 0.55, the entire group has two positives and one negative. After including all of it, \(TP=1+2=3\) and \(FP=1+1=2\), so the next attainable point is \((2/8,3/8)\). The proposed intermediate count would be \(TP=3\), \(FP=1\), corresponding to \((1/8,3/8)\). It is not attainable: a threshold cannot include the two cases scored 0.55 with positive labels while excluding the negative case with the identical score.
For a further check, adding the 0.20 group gives \(TP=4\), \(FP=4\), or \((4/8,4/8)=(0.5,0.5)\). Adding the 0.05 group gives \(TP=8\), \(FP=8\), or \((1,1)\). The false intermediate point arose from ordering tied cases individually rather than treating their shared score as one group.
Common Exam Trap
A tempting but incorrect procedure is to sort individual records by score and then break ties arbitrarily—for example, place positive cases before negative cases whenever their scores match. That creates extra points and can make the classifier appear to reach a rate combination that no threshold on its score can produce. The correction is to aggregate by distinct score first, then update both cumulative counts for the entire group.
Also check the direction of the score rule. This tutorial uses “predict positive if score is at least the threshold.” If the problem defines the score in the opposite direction, the ordering must be reversed to match its classification rule. Do not silently reorder scores while keeping the original threshold interpretation.
Check Your Understanding
For each problem, process distinct scores from highest to lowest and treat every tie as a single group.
- A sample has score groups 0.9: one positive and one negative; 0.6: one positive and two negatives; 0.2: one positive and two negatives. Give all ROC points, including both endpoints. (Answer: \((0,0)\), \((1/5,1/3)\), \((3/5,2/3)\), \((1,1)\).)
- A sample has score groups 0.8: two positives and one negative; 0.6: zero positives and two negatives; 0.3: two positives and zero negatives. What is the point after the 0.6 group is included? (Answer: \((1,1/2)\).)
- A sample has score groups 0.9: one positive and zero negatives; 0.5: one positive and two negatives; 0.1: one positive and one negative. Give the point immediately before and immediately after the 0.5 group. (Answer: Before \((0,1/3)\); after \((2/3,2/3)\).)
- There are five positives and five negatives. After a higher-score group, a mixed tie contributes two positives and three negatives. What coordinate change does this group cause? (Answer: \(TPR\) increases by \(2/5\) and \(FPR\) increases by \(3/5\), together in one threshold step.)
- At score 0.6, three cases are tied: two positives and one negative. A proposed threshold sequence includes both positives but not the negative. Is that an attainable deterministic-threshold outcome? (Answer: No; all three tied cases enter together when the threshold reaches 0.6.)