Introduction
A classification model assigns cases to categories, such as whether a claim needs investigation or whether a policyholder is likely to lapse. Choosing a success criterion is not simply a matter of selecting a familiar metric. The criterion should reflect which errors matter, how common the positive outcome is, what action follows a prediction, and what the organization can operationally handle.
A model that identifies many positive cases may also generate too many false alarms for a small review team. A model with high overall accuracy may miss most positive cases when those cases are rare. For a classification case aligned with the SOA Exam PA (Predictive Analytics) syllabus, explain the decision context first, then justify a primary criterion and any necessary guardrails.
Match the Criterion to the Decision
A classification score and a classification decision are related but not identical. A score can rank cases from higher to lower risk. To make a positive/negative decision, the organization must also select a threshold or a fixed review capacity. Different thresholds generally produce different numbers of TPs and FPs. Consequently, a model's success criterion should reflect how its scores will actually be used.
These measures answer different questions. Sensitivity focuses on missed positives; precision focuses on the reliability of positive alerts; specificity focuses on correctly clearing negatives; and accuracy summarizes all classifications. None is universally best. For example, if a positive result triggers costly manual investigation, precision may matter because false alerts consume review effort. If a missed positive is especially harmful, sensitivity may be more important.
Prevalence is the proportion of cases that are actually positive. It affects the interpretation of classification results, particularly precision. Even if sensitivity and specificity remain unchanged, a lower prevalence generally means a smaller fraction of positive predictions are true positives. Accuracy can also be misleading when one class is much more common than the other: predicting the common class for every case may yield high accuracy while providing no useful detection of the rare class.
Worked Example: Weighing Missed Events Against False Alerts
A hypothetical insurer uses a classifier to flag claims that may require urgent review. On a test set of 1,000 claims, 50 are actually urgent. At the selected threshold, the model identifies 35 urgent claims, misses 15 urgent claims, flags 90 non-urgent claims, and correctly leaves 860 non-urgent claims unflagged. Suppose, for this decision analysis only, a missed urgent claim has a cost of $800 and a false alert costs $40. Calculate key measures and compare the model's error cost with a policy that flags no claims.
Method. First calculate the classification measures from the stated counts. Then apply the stated costs to FN and FP. The cost calculation assumes the two error costs are additive and that each error has the same cost within its category.
The total number of actual positives is \(35+15=50\), and the total number of actual negatives is \(90+860=950\). Sensitivity is \(35/50=0.7000\). Specificity is \(860/950=0.9053\). Precision is \(35/(35+90)=35/125=0.2800\). Accuracy is \((35+860)/1000=0.8950\).
The model's false-negative cost is \(15\times \$800=\$12{,}000\). Its false-positive cost is \(90\times \$40=\$3{,}600\). Total error cost is therefore \(\$12{,}000+\$3{,}600=\$15{,}600\).
If no claims are flagged, all 50 urgent claims are missed and there are no false alerts. That policy's cost is \(50\times \$800=\$40{,}000\). Under the stated costs, the model reduces error cost by \(\$40{,}000-\$15{,}600=\$24{,}400\). The relatively low precision of 0.2800 is still operationally relevant: only 28% of the flagged claims are urgent.
Check. The counts sum to \(35+15+90+860=1{,}000\), and the model's error cost includes exactly the 15 false negatives and 90 false positives. The common mistake is to call \(35/50\) precision; it is sensitivity. Precision uses all positive predictions in its denominator, so it is \(35/(35+90)\).
Prevalence, Review Capacity, and Ranking
When a team can review only a fixed number of cases, success may depend on which cases appear at the top of a ranked list rather than on performance at one fixed threshold. In that setting, evaluate the number or proportion of actual positives captured within the available review capacity. Precision among reviewed cases describes the yield; sensitivity describes what fraction of all actual positives the review list captures. These criteria are connected but not interchangeable.
A ranking measure, such as area under the receiver operating characteristic curve (ROC AUC), can summarize how well a score separates positives from negatives across possible thresholds. It does not by itself specify how many cases to review, the consequences of errors, or the value of the resulting actions. A strong ranking score is not a complete operational success criterion.
Worked Example: Comparing Models Under a Review Limit
A hypothetical claims team has capacity to review exactly 40 of 1,000 claims each week. There are 50 urgent claims in the evaluation set. Model A's 40 highest-ranked claims include 18 urgent claims; Model B's 40 highest-ranked claims include 15. The team wants to maximize urgent claims found without exceeding its capacity. Compare the models using yield and capture rate.
Method. Because capacity is fixed at 40, compare both the fraction of reviewed cases that are urgent and the fraction of all urgent cases found. The positive prevalence is \(50/1000=0.0500\), or 5%.
For Model A, 18 of the 40 reviewed cases are urgent, so its precision within the review list is \(18/40=0.4500\), or 45%. Its sensitivity is \(18/50=0.3600\), or 36%. The remaining \(40-18=22\) reviewed cases are non-urgent.
For Model B, precision within the review list is \(15/40=0.3750\), or 37.5%. Its sensitivity is \(15/50=0.3000\), or 30%. The remaining \(40-15=25\) reviewed cases are non-urgent.
Both models respect the 40-case capacity, but Model A identifies three more urgent claims in the same number of reviews. It has higher yield and captures a larger share of all urgent cases, so it is preferable for the stated objective. The decision would need reconsideration if the models differed in implementation cost or if the relative consequences of missing particular urgent cases mattered.
Check. Each model's review list contains 40 cases, and each capture rate uses the 50 actual urgent claims as its denominator. A common mistake is to compare only the number of cases reviewed; that number is fixed by the constraint and says nothing about which model uses the capacity more effectively.
From Error Costs to a Decision Threshold
If a model produces calibrated probabilities, a decision threshold can be derived from the costs of the two possible errors. Calibration means that among cases assigned probability \(p\), the positive outcome occurs at a rate close to \(p\). The following calculation assumes two possible actions, correct classifications have zero cost, a false positive costs \(C_{FP}>0\), and a false negative costs \(C_{FN}>0\). It also assumes the probabilities are appropriate for the population where the decision will be made.
Predict positive when the first expected cost is no greater than the second. Rearranging gives the cost-based threshold:
This derivation compares the two actions for one case: predicting positive incurs a false-positive cost only when the case is actually negative, while predicting negative incurs a false-negative cost only when the case is actually positive. The resulting threshold is not automatically the best operational rule. Capacity limits, intervention effects, costs not captured in the simple model, and poorly calibrated probabilities may require a different policy. If the score is only a ranking and is not a calibrated probability, this probability-threshold formula cannot be applied directly.
Worked Example: Combining Costs with a Capacity Limit
A hypothetical insurer considers a model that assigns calibrated probabilities of a future service interruption. A false alert costs $25 in unnecessary outreach; a missed interruption costs $225. In a 2,000-case evaluation set, 100 cases experience an interruption. At three candidate thresholds, the resulting counts are:
| Threshold | TP | FN | FP | TN |
|---|---|---|---|---|
| 0.50 | 30 | 70 | 20 | 1,880 |
| 0.30 | 60 | 40 | 100 | 1,800 |
| 0.10 | 70 | 30 | 180 | 1,720 |
The team can contact no more than 180 cases. Identify the cost-based threshold without the capacity limit, then select the lowest-cost listed option that meets capacity.
Method. The cost-based probability threshold is \(25/(25+225)=25/250=0.10\). Next calculate total error costs and the number of positive predictions, \(TP+FP\), for each listed threshold.
At threshold 0.50, error cost is \(70\times \$225+20\times \$25=\$15{,}750+\$500=\$16{,}250\). The number contacted is \(30+20=50\).
At threshold 0.30, error cost is \(40\times \$225+100\times \$25=\$9{,}000+\$2{,}500=\$11{,}500\). The number contacted is \(60+100=160\).
At threshold 0.10, error cost is \(30\times \$225+180\times \$25=\$6{,}750+\$4{,}500=\$11{,}250\). The number contacted is \(70+180=250\), which exceeds the limit. Thus, among the listed options that satisfy capacity, threshold 0.30 has the lower error cost and is selected.
Check. The three rows each total 2,000 cases and have 100 actual positives. The unconstrained cost calculation favors 0.10, but operational capacity rules it out. The common mistake is to apply the cost-based threshold mechanically and ignore the stated contact limit.
Build a Justified Set of Success Criteria
A useful evaluation plan distinguishes the primary objective from supporting checks. The primary criterion should directly represent the decision goal; guardrails ensure the selected model remains acceptable in practice. A report should name the positive class, evaluation population, decision rule, and any capacity or cost assumptions so that the reported metric has an unambiguous interpretation.
State what counts as positive, what the prediction triggers, and whether the model is used to rank cases or make an immediate classification.
Describe the relative consequences of false negatives and false positives, and state how common the positive outcome is in the relevant population.
Use a measure tied to the objective: for example, sensitivity when missed positives dominate, precision or yield under review capacity, or expected cost when error costs can be specified credibly.
Check capacity, acceptable false-alert burden, and relevant model properties such as calibration when probability-based decisions are intended.
Use the same population, positive-class definition, threshold or capacity rule, and cost assumptions for each candidate model.
A metric should not be selected solely because it is common or easy to calculate. For instance, accuracy can be included as context but may not be the primary criterion for a rare outcome. AUC can inform ranking quality but does not determine an operating threshold. Sensitivity and precision can each be useful, but reporting only one can conceal an important trade-off. State what the criterion rewards and what it does not capture.
Check Your Understanding
For each original scenario, identify or calculate a criterion that fits the stated decision.
- A hypothetical monitoring team evaluates 600 accounts. Of 30 accounts that later default, it flags 21. It also flags 57 of the 570 accounts that do not default. Calculate sensitivity and precision. (Answer: Sensitivity \(=21/30=0.7000\); precision \(=21/(21+57)=0.2692\).)
- A review unit can examine 25 applications. Among 500 applications, 40 are genuinely high risk. Model C places 16 high-risk applications in its top 25; Model D places 14 there. Which model has higher precision in the review list, and by how many percentage points? (Answer: Model C; \(16/25-14/25=0.0800\), or 8 percentage points.)
- A hypothetical intervention has a false-positive cost of $18 and a false-negative cost of $162. Assuming calibrated probabilities, what probability threshold follows from the two-cost rule? (Answer: \(18/(18+162)=0.1000\), or 10%.)
- A classifier is evaluated on 1,200 cases: 48 positives and 1,152 negatives. It identifies 36 positives and incorrectly flags 96 negatives. Calculate accuracy and explain why it may not be sufficient alone. (Answer: \( (36+1{,}056)/1{,}200=0.9100\), or 91%; accuracy alone does not show that sensitivity is \(36/48=0.7500\) or that the positive class is rare.)
- A hypothetical call center has capacity for 120 alerts. Threshold A produces 80 alerts at an error cost of $14,000; threshold B produces 135 alerts at an error cost of $10,500; threshold C produces 110 alerts at an error cost of $12,000. Which listed threshold minimizes cost while respecting capacity? (Answer: Threshold C; B exceeds the 120-alert limit, and C costs less than eligible threshold A.)