Choosing Ranking Measures for Screening

Why Ranking Measures Matter in Screening

A screening score is often used to decide which cases deserve attention first. The eventual action may depend on staffing, review capacity, or the cost of investigating a case; those details can change. Before choosing one cutoff, an evaluator may therefore want to know whether the model generally places cases with the outcome of interest ahead of cases without it. Ranking measures assess that ordering across possible cutoffs.

This is different from evaluating one fixed classification rule. A score can rank cases well even when no particular cutoff has yet been selected. Conversely, a model’s ranking quality alone does not establish that its predicted probabilities are accurate or that using it will improve outcomes. The purpose and operational context of the evaluation still matter, as emphasized in the earlier material on defining operational success and linking predictions to actions.

Definition: A ranking measure evaluates how well a model orders cases by their scores, usually with cases that have the outcome of interest ranked ahead of cases that do not. It does not, by itself, assess whether a selected cutoff is appropriate or whether predicted probabilities are calibrated.

ROC Curves: Performance Across Cutoffs

Suppose a model assigns a score $s$ to each case, with larger scores indicating greater evidence for a positive outcome. For a chosen cutoff $c$, classify a case as positive when its score is at least $c$. The true positive rate is the fraction of actual positives classified as positive; the false positive rate is the fraction of actual negatives classified as positive. Varying $c$ gives a set of points with false positive rate on the horizontal axis and true positive rate on the vertical axis. Connecting these points gives the receiver operating characteristic (ROC) curve.

Definition: For a specified cutoff, let TP be the number of actual positives classified as positive, FN the number of actual positives classified as negative, FP the number of actual negatives classified as positive, and TN the number of actual negatives classified as negative. Then $$ \operatorname{TPR}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}, \qquad \operatorname{FPR}=\frac{\mathrm{FP}}{\mathrm{FP}+\mathrm{TN}}. $$ A ROC curve plots $\operatorname{TPR}$ against $\operatorname{FPR}$ as the cutoff varies. Its endpoint is typically $(0,0)$ when no case is classified positive and $(1,1)$ when all cases are classified positive.

A point farther toward the upper-left generally represents more positives identified at a given false positive rate. The curve summarizes behavior over cutoffs; it does not declare which point to use. When multiple cases share a score, they cross a cutoff together, so the empirical ROC curve may have steps rather than a smooth shape.

Area Under the ROC Curve

The area under the ROC curve, abbreviated AUC, is a scalar summary of ranking performance. For an empirical data set with distinct scores, AUC can be calculated by comparing every positive case with every negative case. A pair is concordant if the positive case has the higher score, discordant if the negative case has the higher score, and tied if the scores are equal. A tie receives half credit.

Definition: If there are $n_+$ positive cases and $n_-$ negative cases, then $$ \operatorname{AUC} = \frac{ \text{number of concordant positive-negative pairs} +\tfrac12(\text{number of tied positive-negative pairs}) }{n_+n_-}. $$ Equivalently, AUC is the probability that a randomly selected positive case receives a higher score than a randomly selected negative case, with half credit when their scores tie.

The denominator $n_+n_-$ is the number of all positive-negative pairs. This pairwise definition is useful when scores are listed in rank order; the ROC-area definition is useful when the curve’s coordinates are given. Both describe the same empirical quantity.

Worked Examples

Worked Example: Comparing Two Rankings

In a hypothetical screening validation set, four cases have the positive outcome and four do not. Rank each model’s scores from highest to lowest. Model A’s positive cases occupy ranks 1, 2, 5, and 8. Model B’s positive cases occupy ranks 1, 2, 4, and 7. Calculate and compare the two AUCs.

Method. Count, for each positive case, how many negative cases are ranked below it; add these counts and divide by $4\times4=16$. Scores are untied, so no half-credit adjustments are needed.

For Model A, the negative cases occupy ranks 3, 4, 6, and 7. The positive at rank 1 is above all four negatives. The positive at rank 2 is also above all four. The positive at rank 5 is above the negatives at ranks 6 and 7, and the positive at rank 8 is above none. Thus there are $4+4+2+0=10$ concordant pairs.

\operatorname{AUC}_A=\frac{10}{16}=0.6250

For Model B, the negatives occupy ranks 3, 5, 6, and 8. The positive at rank 1 is above four negatives; the positive at rank 2 is above four; the positive at rank 4 is above the negatives at ranks 5, 6, and 8, or three; and the positive at rank 7 is above the negative at rank 8, or one. There are $4+4+3+1=12$ concordant pairs.

\operatorname{AUC}_B=\frac{12}{16}=0.7500

Model B’s empirical AUC is higher by $0.7500-0.6250=0.1250$. In this evaluation set, it more often ranks a positive case above a negative case. Common mistake to avoid: the positive at rank 4 is above three negatives, not two; rank 8 is not a negative because it is a positive case. Omitting one of the negatives changes the pair count and therefore the AUC.

Worked Example: Calculating AUC From a ROC Curve

A hypothetical review model assigns six cases distinct scores in descending order. Their actual outcomes, in that order, are positive, negative, positive, positive, negative, negative. Find the empirical ROC coordinates and AUC.

There are three positives and three negatives, so each positive changes TPR by $1/3$, and each negative changes FPR by $1/3$. Starting with no cases classified positive, the ROC point is $(0,0)$. As the cutoff passes each successive score, the points are:

(0,0),\ (0,\tfrac13),\ (\tfrac13,\tfrac13),\ (\tfrac13,\tfrac23),\ (\tfrac13,1),\ (\tfrac23,1),\ (1,1).

Calculate the area by adding trapezoid areas between consecutive points. Vertical segments have no width and contribute zero. From FPR $0$ to $1/3$, TPR increases from $1/3$ to $2/3$, giving area $(1/3)(1/3+2/3)/2=1/6$. The next segment is vertical, so contributes zero. From FPR $1/3$ to $2/3$, TPR is 1, contributing $1/3$; from $2/3$ to 1, TPR remains 1, contributing another $1/3$. The total is $1/6+1/3+1/3=5/6$.

Check by pair counting: the first positive is above all three negatives; the second positive is above the two negatives after it; and the third positive is above the two negatives after it. There are $3+2+2=7$ concordant pairs out of $3\times3=9$, giving $7/9$, not $5/6$. The discrepancy identifies an arithmetic error in the trapezoid calculation: from FPR $0$ to $1/3$, the curve first rises vertically to TPR $1/3$, then moves horizontally to FPR $1/3$ at TPR $1/3$. That area is $1/3\times1/3=1/9$, not $1/6$. Therefore the corrected total is $1/9+1/3+1/3=7/9$, which agrees with the pair count.

Thus, to four decimal places, the AUC is $0.7778$. Common mistake to avoid: do not treat every interval as a trapezoid with an increase in TPR. Here the first nonzero-width segment is horizontal at TPR $1/3$; the vertical rise has zero area.

Worked Example: Giving Partial Credit for Tied Scores

A hypothetical validation set has four positive and four negative cases. The score groups are: at 0.9, two positives and one negative; at 0.6, one positive and two negatives; at 0.2, one positive and one negative. Calculate AUC, including ties.

Compare each positive with the negatives in lower-scoring groups and in its own score group. Each of the two positives at 0.9 beats the three negatives below 0.9 and ties the one negative at 0.9. Each contributes $3+\tfrac12=3.5$, for $2(3.5)=7$ pair credits.

The positive at 0.6 beats the one negative at 0.2 and ties the two negatives at 0.6, contributing $1+2(\tfrac12)=2$. The positive at 0.2 ties the one negative at 0.2 and beats no lower-scoring negative, contributing $0.5$. Total credit is $7+2+0.5=9.5$. There are $4\times4=16$ pairs, so

\operatorname{AUC}=\frac{9.5}{16}=0.59375\approx 0.5938.

The calculation checks out because the four positive cases contribute $3.5+3.5+2+0.5=9.5$ credits, and no pair is counted more than once. Common mistake to avoid: counting a tied pair as a full win exaggerates ranking performance; counting it as a full loss understates it. The standard empirical convention is half credit.

Why Pair Counting and ROC Area Agree

The equivalence follows by considering what happens as the cutoff moves down the ordered scores. Each time a positive case enters the predicted-positive group, TPR rises by $1/n_+$. The area added across the horizontal width already covered by the ROC curve corresponds to the negative cases that this positive outranks. Across all positives, these contributions accumulate over positive-negative comparisons. A negative case tied with a positive is included at the same score level, so the standard stepped ROC-area convention assigns the tie half credit. Dividing by the total number $n_+n_-$ of pairs yields the pairwise AUC formula.

This also explains the probability interpretation: choose one positive and one negative uniformly from the evaluated cases. The fraction of pairs in which the positive score is greater is the empirical probability of a correct ordering; ties count as half. An AUC of 0.5 indicates chance-level pairwise ordering in that evaluation set, while an AUC of 1 indicates every positive-negative pair is ordered correctly. An AUC below 0.5 indicates that the ranking tends to run in the opposite direction; reversing the score direction would reverse the interpretation.

What AUC Does and Does Not Tell You

AUC is useful when ranking quality across a range of possible cutoffs is the evaluation objective. Its pairwise calculation also makes clear that it is not simply the percentage of cases correctly classified at a selected cutoff. It does not choose a cutoff, describe how many cases will be reviewed at that cutoff, or express the relative consequences of different errors.

AUC is based on the relative ordering of scores, not their numerical scale. Applying a strictly increasing transformation to every score preserves all orderings and therefore preserves AUC, even though the transformed scores may look quite different. In addition, the pairwise definition compares positives with negatives, so the overall proportion of positives does not directly enter the formula. That does not mean prevalence or case mix can never matter: a change in the kinds of positive or negative cases evaluated can change their score ordering and the measured AUC.

A model can also have a good AUC while producing probabilities that are systematically too high or too low. AUC alone cannot establish probability calibration. Nor does a higher AUC guarantee better results in the particular operating region that matters to an organization. If most practical cutoffs correspond to only a narrow portion of the ROC curve, overall area may conceal differences in that region. The evaluation criterion should follow the stated objective; the earlier material on decision costs and operational success explains why a ranking summary cannot replace an explicit action-based criterion.

Key takeaway: Use AUC when the question is whether scores rank positive cases above negative cases across possible cutoffs. Treat it as a ranking summary—not as a cutoff-selection rule, a calibration measure, or proof of decision value.

Common Exam Traps

Using the wrong denominator. With $n_+$ positives and $n_-$ negatives, the number of comparison pairs is $n_+n_-$, not the total number of cases.
Counting from the wrong end of the ranking. State whether rank 1 is the highest or lowest score, then count the negative cases below each positive.
Ignoring ties. A tied positive-negative pair contributes one-half, not zero or one.
Calling AUC accuracy. Accuracy requires a particular classification rule; AUC summarizes ordering across cutoffs.
Overinterpreting a single number. AUC does not establish calibration, identify a useful cutoff, or incorporate error costs.

Check Your Understanding

For each question, use the stated score direction and show the pair or area calculation before checking the answer.

A validation set has three positives and two negatives. In descending score order, the outcomes are positive, negative, positive, negative, positive. Find AUC by pair counting. (Answer: $3/6=0.5000$)
There are two positives and two negatives, ranked from highest to lowest as positive, negative, negative, positive. Find AUC. (Answer: $2/4=0.5000$)
A set has two positives and two negatives. Scores are: 0.8 for one positive, 0.5 for one positive and one negative, and 0.2 for one negative. Calculate AUC with half credit for ties. (Answer: $3.5/4=0.8750$)
A ROC curve has vertices $(0,0)$, $(0,0.4)$, $(0.3,0.4)$, $(0.3,1)$, and $(1,1)$. Calculate its area by trapezoids or rectangles. (Answer: $0.3(0.4)+0.7(1)=0.8200$)
Model C has a higher AUC than Model D, but the evaluation objective is to use calibrated probabilities to set a cutoff reflecting stated costs. Is the higher AUC alone enough to select Model C? State why. (Answer: No; AUC measures ranking, not probability calibration or the value of the cost-based cutoff.)

What You'll Learn