Separating Ranking Quality From Threshold Decisions

Why Ranking and Decisions Must Be Separated

A classification model can assign a score or estimated event probability to each case. Ranking asks whether cases that experience the event tend to receive higher scores than cases that do not. A threshold decision instead turns each score into a predicted class: scores at or above a chosen cutoff are called positive, and scores below it are called negative. These are related but distinct evaluations. A model can rank cases well yet perform poorly at a particular threshold, or a model with weaker overall ranking can be preferable at an operating threshold that better suits the decision.

This distinction matters when comparing hypothetical models that flag insurance applications for review, identify accounts likely to lapse, or predict whether a claim will require further investigation. The threshold reflects a decision rule; it is not automatically supplied by the model or by the data. A defensible comparison therefore states both the ranking measure and the threshold-based objective. The discussion below is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Ranking Quality: ROC AUC

The receiver operating characteristic (ROC) curve displays the true-positive rate against the false-positive rate as the threshold varies. At any threshold, the true-positive rate is \(TP/(TP+FN)\), and the false-positive rate is \(FP/(FP+TN)\). The area under the ROC curve, or AUC, summarizes ranking over the range of thresholds. It does not select a threshold or assign different consequences to the two types of error.

Thus AUC can be interpreted as the probability that a randomly selected event receives a higher score than a randomly selected non-event, with half credit for a tie. An AUC of 0.5 corresponds to chance-level pairwise ranking in this interpretation; an AUC of 1 means every event–non-event pair is correctly ordered. AUC is not the fraction of observations classified correctly at a particular cutoff.

Why AUC Is a Pairwise Ranking Measure

The pair-count formula follows from the way the ROC curve accumulates correct ordering as the threshold moves downward. A non-event contributes a false-positive step when its score is passed; an event contributes a true-positive step. The area under the resulting staircase, using the usual half-step treatment of ties, is equivalent to counting the fraction of event–non-event pairs ordered correctly. Each of the \(n_+n_-\) pairs has equal weight, which gives the denominator in the empirical formula.

For example, if every event score is greater than every non-event score, all \(n_+n_-\) pairs are wins and AUC is 1. If every event score is lower, there are no wins and AUC is 0. If event and non-event scores are indistinguishable in their ordering, the expected fraction of wins, counting ties halfway, is 0.5. This justification also clarifies why changing a classification cutoff does not change AUC: AUC evaluates all score orderings, not just the decisions at one cutoff.

AUC does not directly use event prevalence: its pair comparisons are between event and non-event cases. In a population with unchanged event and non-event score distributions, a change in prevalence alone does not alter the population AUC. In finite validation samples, however, AUC estimates can vary because the numbers and composition of observed cases change. More importantly for decisions, prevalence affects how many positive and negative cases a fixed sensitivity and specificity produce.

Threshold Measures Depend on the Decision Setting

At a specified threshold, the confusion counts can be summarized with measures such as sensitivity, specificity, precision, and accuracy. Sensitivity is \(TP/(TP+FN)\); specificity is \(TN/(TN+FP)\); precision is \(TP/(TP+FP)\); and accuracy is \((TP+TN)/(TP+FP+FN+TN)\). Sensitivity and specificity describe performance conditional on the actual class. Precision and accuracy depend on the mix of events and non-events, as well as on the threshold.

A useful way to see the prevalence effect is to project a model's sensitivity and specificity to a population of size \(N\) with event prevalence \(\pi\). Assuming those conditional rates remain the same in the population, the expected counts are:

These are projections, not guarantees: they require the sensitivity and specificity to transport to the new population at the chosen threshold. In particular, a change in population or data collection can alter model performance, so the assumption should not be made silently.

Error Costs Can Change the Preferred Model

Prevalence is only one part of the decision. If a false negative is much more consequential than a false positive, a rule with higher sensitivity may be preferred, even if it creates more false alarms. If the operational burden of a false positive is large, the reverse may hold. Define \(c_{FP}\) as the cost assigned to a false positive and \(c_{FN}\) as the cost assigned to a false negative. For a validation or projected population, the total error cost is

This calculation assumes that the specified costs are appropriate and that TP and TN have no additional cost in the comparison. If true-positive or true-negative outcomes also carry different benefits or expenses, those should be included in the decision objective. Threshold selection should follow the intended objective, rather than relying on a conventional cutoff without justification.

Practical Comparison Sequence and Common Traps

• Trap: calling AUC accuracy. AUC summarizes pairwise ranking across thresholds; accuracy is a classification proportion at a particular threshold.
• Trap: overlooking prevalence. Even if sensitivity and specificity stay fixed, false-positive and false-negative counts, and therefore precision and total cost, can change with prevalence.
• Trap: comparing costs without their units or assumptions. State the costs, population, prevalence, and whether the rates are treated as stable before recommending a rule.
• Trap: treating a better AUC as an automatic decision winner. AUC does not encode the costs of errors, review capacity, or the chosen threshold. Those are separate parts of the decision.