From a Score to a Decision
A classification model can produce a score for each case, such as a predicted probability of a claim needing investigation. A score alone does not say what action to take. A classification threshold supplies a decision rule: cases on one side of the cutoff receive the positive label, and cases on the other side receive the negative label. In an insurance setting, for example, the positive label might mean “send for review.” The labels and consequences depend on the problem being studied.
The threshold is part of the decision policy, not an intrinsic property of the model. A model can be left unchanged while its threshold changes. That can alter both the number of cases assigned to the positive class and the mix of correct and incorrect decisions. This distinction is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
A tie convention matters. With the rule \(s \ge t\), a score equal to the cutoff is classified positive. A different convention could classify equality as negative, but it must be stated and applied consistently. If the scores are probabilities, they lie between 0 and 1. A score can also be a model output on some other scale; in that case, the cutoff still defines a rule, but the score should not be described as a probability unless that interpretation is justified.
What Changing the Cutoff Does
Suppose the scores are held fixed and the cutoff is lowered. Every case already above the old cutoff remains above the new one, and some additional cases may now be classified positive. Thus, the positive-class workload cannot decrease when the cutoff is lowered. The added cases may include actual positives and actual negatives: some false negatives can become true positives, while some true negatives can become false positives. Raising the cutoff reverses this movement. These are directional statements; the exact number and type of changed cases depend on their observed outcomes.
For a fixed set of cases, a convenient way to compare cutoffs is to make a separate confusion table at each cutoff. Check that the four cells sum to the total number of cases. Also check that actual positive and actual negative totals are unchanged across thresholds: changing a decision rule does not change what actually happened.
Worked Example: Comparing Two Cutoffs on the Same Cases
A hypothetical insurer has 12 completed cases. The score is a predicted probability of a positive outcome, and the observed outcome is recorded as 1 for positive and 0 for negative. Compare cutoffs of 0.50 and 0.30, using the rule score \( \ge t \) means positive.
| Case | Score | Observed outcome |
|---|---|---|
| A | 0.92 | 1 |
| B | 0.81 | 0 |
| C | 0.74 | 1 |
| D | 0.63 | 0 |
| E | 0.58 | 1 |
| F | 0.51 | 0 |
| G | 0.47 | 1 |
| H | 0.39 | 0 |
| I | 0.32 | 0 |
| J | 0.28 | 1 |
| K | 0.18 | 0 |
| L | 0.07 | 0 |
Method at \(t=0.50\). Cases A through F have scores at least 0.50, so there are 6 positive predictions. Among them, A, C, and E have observed outcome 1, giving 3 true positives; B, D, and F have outcome 0, giving 3 false positives. Of the six cases below 0.50, G and J are actual positives, so there are 2 false negatives. H, I, K, and L are actual negatives, so there are 4 true negatives. The check is \(3+3+2+4=12\).
Method at \(t=0.30\). Cases A through I have scores at least 0.30, so there are 9 positive predictions. A, C, E, and G are true positives; B, D, F, H, and I are false positives. J is the one false negative, and K and L are the two true negatives. The check is \(4+5+1+2=12\).
Lowering the cutoff from 0.50 to 0.30 adds 3 positive predictions. In these data, that change turns G from a false negative into a true positive, while H and I become false positives rather than true negatives. There are 5 actual positives and 7 actual negatives at either cutoff, which also checks the outcome totals. The common mistake is to say that lowering a cutoff only “catches more positives”; it also sends some additional negative cases to the positive class.
Counting Workload and Errors from Groups
When a data set is summarized by score bands, a threshold can still be evaluated if the bands line up with that threshold. Use the observed positive and negative counts in each band; do not assume that every case in a band has the same outcome. If a cutoff splits a band but individual scores and outcomes are unavailable, the exact workload and confusion counts at that cutoff cannot be recovered from the band totals alone.
Worked Example: A Review-Capacity Constraint
A hypothetical review team has historical results for 200 cases. The table gives the number of actual positives and negatives in three score bands. The team can review at most 60 cases. Compare a cutoff of 0.70 with a cutoff of 0.40, using the stated band boundaries.
| Score band | Actual positives | Actual negatives | Total cases |
|---|---|---|---|
| Score \( \ge 0.70 \) | 18 | 12 | 30 |
| 0.40 \( \le \) score \( < 0.70 \) | 20 | 30 | 50 |
| Score \( < 0.40 \) | 12 | 108 | 120 |
At \(t=0.70\). Only the first band is classified positive. Therefore, the workload is \(18+12=30\) reviews: 18 true positives and 12 false positives. The remaining bands contain \(20+12=32\) actual positives, so there are 32 false negatives. They contain \(30+108=138\) actual negatives, so there are 138 true negatives. The total is \(18+12+32+138=200\), and 30 reviews are within capacity.
At \(t=0.40\). The first two bands are classified positive. The workload is \(30+50=80\), comprising \(18+20=38\) true positives and \(12+30=42\) false positives. The lowest band gives 12 false negatives and 108 true negatives. The total is \(38+42+12+108=200\). This cutoff exceeds capacity by \(80-60=20\) reviews.
The lower cutoff finds 20 more actual positives in this set, but requires 50 additional reviews, including 30 additional actual negatives. The threshold at 0.70 fits the stated capacity; that fact alone does not establish that it is the best policy. The common mistake is to check only how many additional positives are captured and overlook the number of extra cases the operation must handle.
When a Probability Cutoff Follows from Costs
A particular probability cutoff can be derived by comparing the expected costs of the two possible decisions for one case. The derivation requires scores that can be interpreted as calibrated probabilities: for a case with predicted probability \(p\), the expected proportion of positive outcomes among cases with that probability should be \(p\). It also requires the specified costs to represent the relevant consequences of a false positive and a false negative. If those conditions do not hold, the cost-based probability rule below is not justified, although an operational cutoff can still be chosen and evaluated empirically.
Assume the cost of a false positive is \(C_{\mathrm{FP}}>0\), the cost of a false negative is \(C_{\mathrm{FN}}>0\), and correct classifications have zero cost for this comparison. If the case is classified positive, a cost is incurred only when the outcome is negative. If it is classified negative, a cost is incurred only when the outcome is positive. Therefore, the two expected costs are:
Choose the positive decision when its expected cost is no greater than the negative decision's expected cost. Solving that inequality gives the cutoff:
At equality, the two decisions have the same expected cost. Under the convention \(p \ge t\), equality is assigned positive; choosing either decision at the tie gives the same expected cost. This derivation also explains why a cutoff of 0.50 is not automatically appropriate: it follows from this particular cost comparison only when the false-positive and false-negative costs are equal.
Worked Example: Deriving a Cutoff from Case Costs
A hypothetical insurer uses a calibrated probability of a specified claim outcome to decide whether to send a case for an additional check. For this simplified decision, a false positive costs $20 and a false negative costs $80; correct decisions have zero cost. Find the cost-based cutoff and decide what to do for cases with probabilities 0.18 and 0.25.
Method. Substitute the two costs into the derived cutoff:
For \(p=0.18\), the expected cost of a positive decision is \((1-0.18)\times \$20=0.82\times \$20=\$16.40\). The expected cost of a negative decision is \(0.18\times \$80=\$14.40\). Choose negative because \(\$14.40<\$16.40\).
For \(p=0.25\), the positive-decision cost is \((1-0.25)\times \$20=0.75\times \$20=\$15.00\). The negative-decision cost is \(0.25\times \$80=\$20.00\). Choose positive because \(\$15.00<\$20.00\). At \(p=0.20\), both costs equal \(0.80\times \$20=\$16.00\), confirming the tie.
The checks agree with the threshold: 0.18 is below 0.20 and receives the negative label; 0.25 is above 0.20 and receives the positive label. A common mistake is to put the false-negative cost in the numerator. That would give \(80/100=0.80\), reversing the decision logic: the more costly it is to miss a positive case, the lower—not higher—the cutoff should be.
Choosing and Reporting a Threshold
A threshold should be reported as part of the decision policy, alongside the score definition, the tie convention, and the action triggered by a positive classification. Where relevant, state the resulting workload and the observed error counts at that cutoff. A cutoff selected for an operational reason, such as fitting a review process, is not automatically the same as one derived from a cost comparison. The cost-based derivation depends on calibrated probabilities and the stated cost assumptions; a capacity-based choice depends on the number of cases falling above the cutoff.
Threshold comparisons should use the same cases and outcomes. Otherwise, apparent changes in errors or workload may reflect a change in the data rather than a change in the policy. When reviewing a candidate rule, verify its boundary carefully: with \(s \ge t\), cases exactly equal to \(t\) count as positive. If only grouped score summaries are available, do not report exact counts for a cutoff that splits a group unless more detailed data support the calculation.
Check Your Understanding
For each problem, use the rule score \( \ge t \) means positive. Treat stated probabilities as calibrated where costs are supplied.
- Among 80 cases, 25 have scores at least 0.60, and 10 of those 25 are actual positives. There are 16 actual positives in all. At \(t=0.60\), find the four confusion counts. (Answer: TP = 10, FP = 15, FN = 6, TN = 49.)
- A set of 200 cases has 40 actual positives. At \(t=0.80\), 40 cases are positive-predicted, including 12 actual positives. At \(t=0.50\), 100 cases are positive-predicted, including 30 actual positives. Give the workload and all four confusion counts at each cutoff. (Answer: At 0.80, workload = 40, TP = 12, FP = 28, FN = 28, TN = 132; at 0.50, workload = 100, TP = 30, FP = 70, FN = 10, TN = 90.)
- A case has probability \(p=0.20\). The false-positive cost is $15 and the false-negative cost is $45, with correct decisions assigned zero cost. Find the cost-based cutoff and the preferred decision for this case. (Answer: \(t=15/(15+45)=0.25\); choose negative because 0.20 is below 0.25.)
- In a group of 500 cases, 70 have scores at least 0.65, 110 have scores from 0.40 up to but not including 0.65, and 320 have scores below 0.40. Compare review workload at \(t=0.65\) and \(t=0.40\). (Answer: 70 reviews at 0.65 and 180 at 0.40; lowering the cutoff adds 110 reviews.)
- A calibrated-probability policy has false-positive cost $30 and false-negative cost $70. Find the cutoff and state the decision at a probability exactly equal to that cutoff under the stated tie convention. (Answer: \(t=30/(30+70)=0.30\); classify positive at \(p=0.30\), with equal expected costs of $21 for either decision.)