Two Different Questions About a Model
A model that assigns each case a score can be used in two ways. A reviewer may use the score to rank cases, placing higher-scored cases earlier for attention. Alternatively, a decision rule may compare each score with a cutoff and assign a class label, such as “refer” or “do not refer.” These uses are related, but they answer different questions.
Ranking asks whether cases with the outcome of interest tend to appear above cases without it. Classification asks which cases are labelled positive at a selected cutoff, and therefore which errors result from that particular rule. A model can order cases well but produce unsuitable labels at a chosen cutoff. Conversely, two models can produce exactly the same labels at a cutoff while ordering the cases within each label group quite differently.
A score might be a predicted probability, but it need not be. For ranking, only the relative order of scores matters. For threshold classification, the cutoff and the score scale matter: a value of 0.5 has no special meaning unless the model’s score definition and the decision policy make it meaningful. The earlier material on classification thresholds treats a cutoff as a decision policy; here the key point is that the cutoff converts a score into a label, while the score itself can retain information about ordering.
Ranking Does Not Determine the Labels
Suppose a hypothetical screening team receives model scores for six applications. A score of at least 0.50 triggers a manual review. The observed outcomes are included below only to make it possible to compare the ranking with the labels.
Worked Example: Same Order, Different Cutoff Labels
The six applications have observed outcomes and Model A scores shown below. Model B reports \(0.4\) times each Model A score. For both models, the team initially applies the cutoff \(c=0.50\). Compare the rankings and the resulting classifications. Then identify the cutoff for Model B that reproduces Model A’s labels.
| Case | Observed outcome \(Y\) | Model A score | Model B score |
|---|---|---|---|
| A | 1 | 0.82 | 0.328 |
| B | 0 | 0.70 | 0.280 |
| C | 1 | 0.61 | 0.244 |
| D | 0 | 0.45 | 0.180 |
| E | 1 | 0.30 | 0.120 |
| F | 0 | 0.12 | 0.048 |
Step 1: Compare the rankings. Model A’s scores in descending order are \(0.82, 0.70, 0.61, 0.45, 0.30, 0.12\), corresponding to A, B, C, D, E, F. Model B’s scores are \(0.328, 0.280, 0.244, 0.180, 0.120, 0.048\), in that same case order. Thus, the models give exactly the same ranking.
Step 2: Apply \(c=0.50\) to Model A. Cases A, B, and C meet or exceed 0.50, so they are labelled positive. A and C have \(Y=1\), while B has \(Y=0\). Of the remaining cases D, E, and F, E has \(Y=1\) and D and F have \(Y=0\). The counts are therefore two true positives, one false positive, one false negative, and two true negatives.
Step 3: Apply \(c=0.50\) to Model B. Its largest score is 0.328, which is below 0.50. All six cases are labelled negative. There are no true positives or false positives, three false negatives (A, C, E), and three true negatives (B, D, F).
Step 4: Find a corresponding cutoff. Since Model B’s scores are 0.4 times Model A’s scores, use \(0.4 \times 0.50=0.20\). At \(c=0.20\), Model B labels A, B, and C positive, just as Model A does at 0.50.
Check. The rankings agree, but the labels at a fixed numeric cutoff do not. The cutoff must be interpreted on the model’s score scale. A common mistake is to conclude that identical rankings guarantee identical labels; this example shows that they do not. The calculation also checks directly that a corresponding cutoff, 0.20, restores the original labels.
This distinction matters whenever a score is rescaled, recalculated, or supplied by a different model. The order may remain unchanged even though the numeric values move substantially. If the task is to order cases, the unchanged ranking may be what matters. If the task is to trigger an action at a specified cutoff, the threshold rule must be assessed on the score scale actually being used.
Same Labels, Different Rankings
The reverse situation is also possible. At a chosen cutoff, two models may place exactly the same cases in the positive class and the same cases in the negative class, yet rank cases within those groups differently. The labels alone hide those differences.
Worked Example: Identical Labels, Different Intervention Order
A hypothetical insurer uses a score to order five accounts for a risk-management call. A score of at least 0.50 creates a “call” label. The observed outcome \(Y=1\) means that a specified costly event occurred during the follow-up period. Compare two models’ labels and rankings.
| Account | Observed outcome \(Y\) | Model R score | Model S score |
|---|---|---|---|
| P | 1 | 0.80 | 0.95 |
| Q | 1 | 0.60 | 0.85 |
| R | 0 | 0.95 | 0.55 |
| S | 0 | 0.40 | 0.40 |
| T | 0 | 0.20 | 0.20 |
Step 1: Apply the cutoff. Under Model R, P, Q, and R meet or exceed 0.50; S and T do not. Under Model S, P, Q, and R also meet or exceed 0.50; S and T do not. Both models therefore produce two true positives (P and Q), one false positive (R), no false negatives, and two true negatives (S and T).
Step 2: Compare score order. Model R orders the accounts R, P, Q, S, T. Model S orders them P, Q, R, S, T. In Model R, the highest-scored account is a negative case; in Model S, both observed positive cases appear above the negative case R.
Step 3: Interpret the difference. If the call team uses the scores to determine the order in which to contact accounts, the models imply different sequences even though their cutoff labels match. For example, among the first two accounts in the ranking, Model R places one observed positive and one observed negative, whereas Model S places two observed positives. The shared labels do not show this ordering distinction.
Check. Both models have three positive labels and two negative labels, with identical outcomes among those groups. Their rankings nevertheless differ. The common mistake is to treat identical labels as proof that two models prioritize cases identically; comparing the scores directly corrects that mistake.
The example does not establish that Model S is universally preferable. It shows only that the answer depends on the intended use. If the action is triggered by a cutoff, examine the labels and their consequences. If the action is to sequence cases by score, examine the order. A full assessment also needs the operational success criterion established for the problem.
Why a Monotone Score Transformation Preserves Order
A strictly increasing transformation changes the score scale without reversing the order. This gives a precise reason that ranking and cutoff decisions behave differently under rescaling.
Justification. By the definition of a strictly increasing function, whenever \(s_i>s_j\), it follows that \(g(s_i)>g(s_j)\). Applying the same property in the reverse comparison shows that a strictly increasing transformation cannot reverse the order. Also, because \(g\) is increasing, \(s_i \geq c\) implies \(g(s_i) \geq g(c)\), and \(s_i<c\) implies \(g(s_i)<g(c)\). Hence the corresponding cutoff produces the same labels. Using the old numeric cutoff rather than \(g(c)\) need not do so.
For example, multiplication by 0.4 is strictly increasing: \(s_i>s_j\) implies \(0.4s_i>0.4s_j\). But it moves a cutoff of 0.50 to 0.20. Keeping the cutoff at 0.50 after the transformation changes the decision rule. A decreasing transformation, such as multiplying by \(-1\), reverses the order instead, so it does not preserve the ranking in the same direction.
Cutoff Boundaries and Tied Scores
A threshold rule also requires a convention for equality. In this tutorial, the positive class is assigned when \(s_i \geq c\), so a score exactly equal to the cutoff is positive. A different stated convention could be used, but it must be applied consistently. Tied scores have equal position in the score ordering: the score alone does not identify one tied case as higher risk than another.
Worked Example: A Tie at the Screening Cutoff
A hypothetical review unit screens six records. Its score is used only as an ordering score, and the stated rule is positive if \(s_i \geq c\). The observed outcomes are shown so that the consequences of moving the cutoff can be counted.
| Record | Observed outcome \(Y\) | Score |
|---|---|---|
| A | 1 | 0.82 |
| B | 0 | 0.50 |
| C | 1 | 0.50 |
| D | 0 | 0.42 |
| E | 1 | 0.31 |
| F | 0 | 0.10 |
Step 1: Use \(c=0.50\). A, B, and C are positive because each score is at least 0.50. Of these, A and C have outcome 1 and B has outcome 0. D, E, and F are negative; E has outcome 1 and D and F have outcome 0. The counts are two true positives, one false positive, one false negative, and two true negatives.
Step 2: Raise the cutoff to \(c=0.51\). Only A has a score of at least 0.51. A is a true positive. B and C now join D, E, and F in the negative class. Among those five negative labels, C and E have outcome 1, while B, D, and F have outcome 0. The counts are one true positive, zero false positives, two false negatives, and three true negatives.
Step 3: Separate what changed from what did not. The score order has not changed; it is still A above the tied pair B and C, followed by D, E, and F. The labels changed because the cutoff moved from 0.50 to 0.51. At 0.50, the tie is included in the positive class; at 0.51, both tied records are excluded.
Check. The two tied cases cannot be separated using these scores alone. A common mistake is to rank B above C because one is negative and the other positive in the observed data. The model score does not support that distinction: both values are exactly 0.50. The correct calculation applies the same equality rule to both.
Using the Distinction in Model Assessment
A useful first step is to name the task before interpreting a model output. If the task is to assign labels for an action, specify the positive class, the cutoff, and the equality convention. Then evaluate the resulting classifications against observed outcomes in context. If the task is to prioritize cases, examine whether the score order places the cases of interest earlier. A classification summary at one cutoff cannot, by itself, describe every feature of that ordering.
This distinction is aligned with the SOA Exam PA (Predictive Analytics) syllabus. It is especially important in screening and intervention settings, where the same score may support an ordered queue, a cutoff-based referral rule, or both. The model’s ranking and the operational rule should not be conflated: changing the cutoff changes labels without changing scores, while a suitable strictly increasing rescaling can preserve the order but require a correspondingly rescaled cutoff.
- For ranking: state what higher scores mean and compare the order of cases.
- For classification: state the cutoff and exactly how equality is handled.
- For comparisons: check whether models have the same labels, the same order, both, or neither.
- For interpretation: do not treat a score’s numeric value as a decision without specifying the rule that uses it.
Check Your Understanding
For each problem, distinguish the ordering from the cutoff-based labels. Treat a score equal to the cutoff as positive.
- A triage model scores four files A, B, C, and D at 0.91, 0.62, 0.40, and 0.18. A revised model reports \(0.5\) times each score. State the revised ranking and the cutoff that reproduces the original labels from a cutoff of 0.60. (Answer: A, B, C, D; cutoff 0.30.)
- A review team uses cutoff 0.70. Scores and outcomes are: case P, score 0.80, \(Y=1\); Q, 0.70, \(Y=0\); R, 0.69, \(Y=1\); S, 0.20, \(Y=0\). Give the four classification counts: true positives, false positives, false negatives, and true negatives. (Answer: 1, 1, 1, 1.)
- Two models give the same positive labels at cutoff 0.50: cases U and V. Their score orders are Model A: W, U, V, X; Model B: U, V, W, X. What has been established about their labels, and what has not been established about their rankings? (Answer: The cutoff labels match; the rankings do not.)
- A screening score gives cases J, K, L, M scores of 0.74, 0.55, 0.55, and 0.30. Compare the positive cases at cutoffs 0.55 and 0.56, and state what the score can say about K versus L. (Answer: At 0.55, J, K, and L are positive; at 0.56, only J is positive; K and L are tied.)
- A model’s scores are transformed by \(g(s)=s+0.12\). The original cutoff is 0.68. State the transformed cutoff that preserves the original labels, and explain whether the ranking changes. (Answer: 0.80; the ranking is unchanged.)