From Validation Results to an Operational Recommendation
A classification model does not make a business decision by itself. It assigns a score or category; an organization then decides what action to take. For a hypothetical insurer, a claim flagged as high risk might receive specialist review. The useful model is therefore not necessarily the one with the highest accuracy or the strongest average metric. The recommendation must connect validation evidence to the intended action, the costs of errors, the available capacity, and the populations affected.
The discussion below treats a positive case as a claim that meets a defined severe-claim outcome, and a positive prediction as a claim flagged for review. All examples use hypothetical validation data. Comparisons are meaningful only when the models are evaluated on the same relevant population, outcome definition, and validation period. Where those conditions are not met, apparent differences may reflect incomparable evidence rather than model performance.
Sensitivity describes how many actual positive cases the classifier catches. Specificity describes how many actual negative cases it correctly leaves unflagged. Precision answers a different operational question: among the flagged cases, what proportion are actually positive? Accuracy is the overall proportion classified correctly, but it can give a misleading impression when positives are uncommon.
Worked Example: Comparing Two Models Under a Review-Capacity Limit
A hypothetical insurer validates two claim-severity classifiers on the same 1,000 claims. There are 100 severe claims and 900 non-severe claims. At the currently proposed cutoffs, Model A has 72 TP, 28 FN, 180 FP, and 720 TN. Model B has 60 TP, 40 FN, 90 FP, and 810 TN. The review team can process at most 200 flagged claims per 1,000. For an initial cost comparison, suppose each missed severe claim has an incremental cost of $500 and each false referral has an incremental cost of $20. Identify the model that can be used at its current cutoff and compare the stated costs.
Method. First calculate the classification metrics and number of referrals. Then compare the stated error costs, while treating the review limit as a hard operational constraint.
For Model A, the number flagged is \(72+180=252\), which exceeds the capacity of 200. Its sensitivity is \(72/(72+28)=0.7200\), specificity is \(720/(720+180)=0.8000\), precision is \(72/252=0.2857\), and accuracy is \((72+720)/1000=0.7920\).
For Model B, the number flagged is \(60+90=150\), which is within capacity. Its sensitivity is \(60/(60+40)=0.6000\), specificity is \(810/(810+90)=0.9000\), precision is \(60/150=0.4000\), and accuracy is \((60+810)/1000=0.8700\).
At these cutoffs, Model A's stated error cost is \(28(500)+180(20)=14{,}000+3{,}600=\$17{,}600\). Model B's is \(40(500)+90(20)=20{,}000+1{,}800=\$21{,}800\). Thus Model A has the lower error cost under the assumptions, but it cannot be deployed at its current cutoff if the capacity limit is strict. Model B is feasible at its current cutoff and has a lower cost than flagging no claims, which would leave 100 severe claims unflagged at a cost of \(100(500)=\$50{,}000\).
Recommendation. If a decision must be made immediately using the current cutoffs, Model B is the feasible choice. This is not evidence that Model B is universally better: Model A's lower calculated error cost makes it worth evaluating at a less permissive cutoff that fits capacity. The common mistake is to choose Model A solely because its cost is lower, without checking whether the associated 252 referrals can actually be handled.
Prevalence Changes the Meaning of a Flag
The prevalence of the positive outcome is the fraction of cases in the evaluated population that are actually positive. Sensitivity and specificity are conditional on actual outcome status, whereas precision is conditional on receiving a positive prediction. As a result, the same sensitivity and specificity can produce different precision values in populations with different prevalence.
Let \(\pi=P(\text{positive})\), \(Se=P(\text{flagged}\mid\text{positive})\), and \(FPR=P(\text{flagged}\mid\text{negative})=1-\text{specificity}\). The joint probability of an actual positive that is flagged is \(Se\pi\). The probability of any flagged case is the sum of flagged actual positives and flagged actual negatives. Therefore, when the denominator is nonzero:
This is a direct application of conditional probability: the numerator is the probability of being both positive and flagged, and the denominator is the probability of being flagged. It explains why a model that performs similarly on two populations can still produce a smaller proportion of true positives in the population where severe claims are rarer.
Choosing a Threshold for Costs and Capacity
A threshold converts a model score into an action. Lowering a threshold generally flags more cases: this can catch more positives, but it can also increase unnecessary referrals. Raising it generally reduces referrals, but may miss more positives. The direction is typical for ordered risk scores; the actual validation counts should be checked rather than inferred from the threshold alone.
Worked Example: Selecting a Feasible Cutoff by Empirical Cost
A hypothetical insurer evaluates three cutoffs for a score on the same 1,000 validation claims. The positive outcome is severe claim. At cutoff 0.10, the counts are TP = 80, FN = 20, FP = 250, TN = 650. At cutoff 0.20, they are TP = 65, FN = 35, FP = 110, TN = 790. At cutoff 0.30, they are TP = 50, FN = 50, FP = 55, TN = 845. The team can review at most 180 claims. Suppose the incremental cost is $20 per false referral and $300 per missed severe claim. Find the lowest-cost cutoff that meets capacity.
Method. For each cutoff, count referrals as \(TP+FP\), check capacity, and calculate the empirical error cost as \(20(FP)+300(FN)\). These supplied costs represent the decision-maker's assumed incremental consequences; they are not universal values for insurance claims.
At 0.10, referrals are \(80+250=330\), above capacity. The cost is \(20(250)+300(20)=5{,}000+6{,}000=\$11{,}000\).
At 0.20, referrals are \(65+110=175\), within capacity. The cost is \(20(110)+300(35)=2{,}200+10{,}500=\$12{,}700\).
At 0.30, referrals are \(50+55=105\), within capacity. The cost is \(20(55)+300(50)=1{,}100+15{,}000=\$16{,}100\).
Among the feasible cutoffs, 0.20 has the lower empirical cost, so it is the recommendation under the stated costs and capacity. The cutoff at 0.10 has the lowest cost overall but is infeasible. The common mistake is to report that 0.10 minimizes cost and stop there; operational capacity is a constraint on the decision, not a secondary metric to mention after making it.
The empirical comparison is specific to the validation sample and cost assumptions. If costs are uncertain, it is useful to show how the recommendation changes under plausible alternatives. If a review costs a fixed amount for every referral rather than only for false referrals, that cost must be applied to all flagged cases, including true positives. The cost definition should match the actual decision being modeled.
Subgroup Performance and a Conditional Recommendation
An overall confusion matrix can conceal differences among groups. A subgroup assessment should state the group sizes and positive counts, then show which rate differs. A rate based on few positive cases can change substantially when only one case changes classification. Such a result should be reported clearly, but not treated as conclusive evidence about a stable underlying difference.
Worked Example: Applying a Subgroup Constraint to the Model Choice
Return to the validation results for Models A and B. The 1,000 claims include Group G, with 200 claims: 20 severe and 180 non-severe. The other group has 800 claims: 80 severe and 720 non-severe. For Model A, Group G has TP = 15, FN = 5, FP = 45, TN = 135; the other group has TP = 57, FN = 23, FP = 135, TN = 585. For Model B, Group G has TP = 16, FN = 4, FP = 18, TN = 162; the other group has TP = 44, FN = 36, FP = 72, TN = 648. Suppose the hypothetical operating requirement is a true-positive-rate gap no greater than 10 percentage points between these groups, as well as no more than 200 total referrals at the cutoff. Assess both models against both requirements.
Method. Calculate the true-positive rate (TPR), \(TP/(TP+FN)\), separately for each group. Calculate the gap as the absolute difference in those rates. Then check total referrals and compare each result with the specified constraints.
For Model A, Group G's TPR is \(15/(15+5)=0.7500\), and the other group's TPR is \(57/(57+23)=57/80=0.7125\). The gap is \(0.7500-0.7125=0.0375\), or 3.75 percentage points, within the 10-point requirement. Its total referrals are \(72+180=252\), above capacity.
For Model B, Group G's TPR is \(16/20=0.8000\), and the other group's TPR is \(44/80=0.5500\). The gap is \(0.8000-0.5500=0.2500\), or 25 percentage points, above the stated limit. Its total referrals are \(60+90=150\), within capacity.
Neither model meets both requirements at its current cutoff. Model A meets the stated TPR-gap requirement but exceeds capacity; Model B meets capacity but exceeds the TPR-gap requirement. The appropriate recommendation is not to declare one compliant based on its strongest metric. Instead, report that neither current configuration satisfies the full decision rule, and evaluate revised cutoffs or other model configurations before deployment.
There are only 20 severe claims in Group G. One additional Group G severe claim classified correctly would change its observed TPR by \(1/20=0.05\), or 5 percentage points. That sensitivity to a single case is a reason to qualify the apparent subgroup difference and seek more evidence where possible. It does not justify omitting the result: the observed gap is relevant to the stated operating requirement and should be disclosed.
The common mistake is to say that Model B has no subgroup concern because its false-positive rates are equal: both are \(18/180=72/720=0.10\). The requirement in this example is specifically about the true-positive-rate gap. Different error rates answer different questions, so the metric used in the decision rule must be calculated and reported directly.
Putting the Evidence into a Decision-Focused Report
A useful recommendation states the action, the evidence supporting it, the conditions that limit it, and what should happen next. Avoid turning a validation metric into a universal claim about model quality. A concise report can distinguish between an immediate operational choice and a longer-term model-development decision.
State what a positive prediction triggers, what counts as a positive outcome, and the relevant review capacity or operating constraints.
Give the validation population, prevalence, confusion-matrix counts, and the metrics relevant to the action. Do not compare scores obtained on materially different samples as though they were directly comparable.
Show the threshold-specific costs and constraints, including subgroup requirements if they are part of the stated operating decision.
Identify uncertainty from small subgroups, possible prevalence differences, assumed costs, or capacity. Specify what further validation or monitoring is needed before relying on the recommendation.
For example, a decision-focused conclusion could say: “At the current cutoffs, Model B is the only model within the stated review capacity, but it does not meet the hypothetical subgroup TPR-gap requirement. We do not recommend deploying either current configuration under both constraints. Evaluate revised cutoffs and validate them on a sufficiently informative sample before making a deployment decision.” This is more useful than a bare ranking because it makes the action and unresolved condition explicit.
The methods in this case workshop are aligned with the SOA Exam PA (Predictive Analytics) syllabus. The central discipline is to make every recommendation conditional on the outcome definition, validation evidence, threshold, operating constraints, and stated priorities.
Check Your Understanding
For each hypothetical problem, calculate the requested quantities and use them to reach the stated decision.
- A validation population has prevalence 5%, sensitivity 0.80, and specificity 0.90. Calculate the implied precision using the prevalence formula. (Answer: \(0.2963\))
- On 1,000 claims, a classifier has TP = 60, FN = 40, FP = 120, and TN = 780. Calculate sensitivity, precision, and accuracy. (Answer: sensitivity \(=0.6000\), precision \(=0.3333\), accuracy \(=0.8400\))
- Two cutoffs have these results: Cutoff A has TP = 70, FN = 30, FP = 140, TN = 760; Cutoff B has TP = 62, FN = 38, FP = 80, TN = 820. A team can review at most 160 cases. If an FN costs $200 and an FP costs $25, which cutoff is feasible and what is its cost? (Answer: B is feasible; its cost is \(38(200)+80(25)=\$9{,}600\). A is infeasible because it flags 210 cases.)
- A group has 30 actual positive cases, of which 21 are flagged; a comparison group has 70 actual positive cases, of which 56 are flagged. Calculate both TPRs and their absolute gap. (Answer: \(0.7000\) and \(0.8000\); gap \(=0.1000\), or 10 percentage points.)
- A classifier has sensitivity 0.75 and false-positive rate 0.10. Calculate its precision at prevalence 4% and at prevalence 12%, then explain the operational implication. (Answer: at 4%, \(0.2381\); at 12%, \(0.5056\). Precision is substantially higher at the higher prevalence, so validation precision may not transfer unchanged to a population with different prevalence.)