Communicating Trade-Offs and Residual Risks

Introduction

A model can help prioritize attention, estimate outcomes, or direct an intervention. It cannot make every prediction correct, ensure that an intervention works, or prevent disruptions in the process that uses its output. Decision-makers therefore need more than a headline performance measure: they need to understand what each proposed action costs, what it may improve, and which risks remain.

Communicating trade-offs means comparing relevant alternatives on consistent terms. Communicating residual risks means describing the errors, losses, and operational problems that can still occur after an action is taken. The aim is not to make uncertainty disappear from the report, but to show how it affects the decision.

Definition: A trade-off is a difference between alternatives in their expected benefits, costs, or risks. A residual risk is a risk that remains under a proposed action, including risks arising from prediction errors, imperfect interventions, or operational disruption. A model-supported action changes the distribution of possible outcomes; it does not guarantee a particular outcome.

Compare Decisions, Not Just Model Scores

A predictive metric describes a particular aspect of model performance. A decision comparison asks what happens when a model is used to choose an action. Those questions are related, but they are not interchangeable. For example, two thresholds may have different numbers of false positives and false negatives. Which threshold is preferable depends partly on what those errors cost and what an action can accomplish.

A useful comparison states the alternatives, the relevant costs and benefits, and the assumptions connecting predictions to actions. It should also keep unlike quantities distinct. A count of flagged cases is not a dollar saving, and a reduction in predicted risk is not automatically a reduction in realized losses. When a benefit depends on an intervention, describe the assumed intervention effect separately from the model’s predictive performance.

The following accounting approach is useful when costs can reasonably be expressed in a common unit. It is not a claim that every consequence can be reduced to money. Service quality, fairness, legal obligations, and customer impact may need to be reported alongside a financial comparison.

Decision comparison: For each alternative, identify the outcomes that matter, assign the stated or estimated consequence to each outcome, and total the consequences on a consistent basis. Label assumed effects and uncertain estimates clearly. Report both the resulting comparison and important risks that are not captured in the total.

Worked Example: Choosing a Review Threshold When Errors Have Different Costs

A hypothetical insurer evaluates two thresholds for sending claims to a specialist review. In a validation group of 1,000 claims, 80 claims have the adverse condition of interest. At threshold A, 60 of those claims are flagged and 80 claims without the condition are also flagged. At threshold B, 72 claims with the condition and 120 claims without it are flagged. Assume each flagged claim costs $25 to review, and each claim with the condition that is not flagged creates an expected $600 cost. For this calculation only, assume review prevents that $600 cost whenever a claim with the condition is flagged. Compare the total costs in this group.

Method. Count all flagged claims to calculate review costs, then count missed adverse cases to calculate the assumed residual cost.

At threshold A, flagged claims total $60+80=140$. Review cost is $140\times \$25=\$3{,}500$. The number missed is $80-60=20$, so the assumed residual cost is $20\times \$600=\$12{,}000$. Total cost is $\$3{,}500+\$12{,}000=\$15{,}500$.

At threshold B, flagged claims total $72+120=192$. Review cost is $192\times \$25=\$4{,}800$. The number missed is $80-72=8$, so the assumed residual cost is $8\times \$600=\$4{,}800$. Total cost is $\$4{,}800+\$4{,}800=\$9{,}600$.

Under these assumptions, threshold B costs $\$15{,}500-\$9{,}600=\$5{,}900$ less in the validation group. It requires 52 more reviews, costing $52\times \$25=\$1{,}300$, while reducing missed cases by 12, corresponding to $12\times \$600=\$7{,}200$ in assumed avoided cost. The net difference is $\$7{,}200-\$1{,}300=\$5{,}900$. This calculation avoids treating every classification error as equally costly.

The result is conditional on the stated assumptions, especially that review prevents the full cost for every flagged claim with the condition. Actual review may not prevent every loss, and the validation-group totals do not by themselves guarantee the same result in future claims.

Why Cost-Weighted Comparisons Work

A cost total is obtained by splitting the cases into outcome groups and adding the cost in each group. In the threshold example, the groups are flagged claims, which incur review cost, and missed adverse cases, which incur the assumed residual cost. The general calculation follows directly from that partition.

Suppose the review cost per flagged claim is $C_R$, the cost per missed adverse case is $C_M$, and review prevents the adverse cost for flagged cases with the condition. If $TP$, $FP$, and $FN$ denote true positives, false positives, and false negatives, respectively, then total cost is:

\text{Total cost}=C_R(TP+FP)+C_M FN.

There are $TP+FP$ flagged claims, each with review cost $C_R$, so their combined review cost is $C_R(TP+FP)$. There are $FN$ missed cases, each with assumed cost $C_M$, so their combined residual cost is $C_M FN$. Adding the costs of these two groups gives the expression. It does not apply unchanged if review only partly prevents losses, if review has other consequences, or if costs differ materially across cases; those effects must be included in the comparison.

This simple derivation also explains why an overall error rate may not identify the best decision. It counts errors without accounting for their consequences. A report can show both the predictive results and the cost comparison, while clearly stating which costs were assumed rather than directly observed.

Account for Intervention Benefits and Costs

A proposed action may reduce some expected loss while adding implementation or operating costs. Compare the alternatives using the same time period, population, and cost basis. If an intervention effect is an assumption, label it as one; do not present it as something established by the model’s predictions alone.

Worked Example: Comparing Two Outreach Programs

A hypothetical insurer has 1,000 accounts with total expected annual claims of $1,200,000 before an outreach program. Program A contacts 200 accounts whose baseline expected claims total $300,000. For planning, assume outreach reduces those claims by 25% and costs $20,000. Program B contacts 400 accounts whose baseline expected claims total $480,000. Assume its outreach reduces those claims by 20% and costs $44,000. Compare each program with no outreach, using expected claims plus program cost.

Method. For each program, calculate the assumed claims reduction, subtract it from baseline expected claims, and add program cost.

With no outreach, expected annual cost is $\$1{,}200{,}000$.

For Program A, the assumed claims reduction is $0.25\times \$300{,}000=\$75{,}000$. Remaining expected claims are $\$1{,}200{,}000-\$75{,}000=\$1{,}125{,}000$. Including the program cost gives $\$1{,}125{,}000+\$20{,}000=\$1{,}145{,}000$. Relative to no outreach, the assumed net saving is $\$1{,}200{,}000-\$1{,}145{,}000=\$55{,}000$.

For Program B, the assumed claims reduction is $0.20\times \$480{,}000=\$96{,}000$. Remaining expected claims are $\$1{,}200{,}000-\$96{,}000=\$1{,}104{,}000$. Including program cost gives $\$1{,}104{,}000+\$44{,}000=\$1{,}148{,}000$. Relative to no outreach, the assumed net saving is $\$1{,}200{,}000-\$1{,}148{,}000=\$52{,}000$.

Program A has the lower total expected cost under these point assumptions: $\$1{,}148{,}000-\$1{,}145{,}000=\$3{,}000$. Moving from A to B increases the assumed claims reduction by $\$96{,}000-\$75{,}000=\$21{,}000$, but increases program cost by $\$44{,}000-\$20{,}000=\$24{,}000$. The extra cost exceeds the extra reduction by $\$3{,}000$. This avoids the mistake of choosing the program with the largest gross reduction without accounting for its cost.

Both options leave substantial expected claims: $1,125,000 under A and $1,104,000 under B. The calculated reductions are planning assumptions, not guaranteed outcomes. Decision-makers would also need to consider whether the outreach can be delivered as planned and whether other consequences matter.

Include Operational Risks in the Comparison

Residual risk is not limited to prediction errors. A model may be unavailable, its inputs may fail to arrive on time, or staff may not be able to complete the recommended action. A backup process can reduce the consequences of some disruptions, but it may add cost and will not necessarily remove the risk. A decision comparison should identify the important operational events and avoid counting the same loss twice.

Expected cost is one way to compare uncertain financial outcomes when their probabilities and consequences can be stated. For a scenario with probability $p$ and cost $L$ if it occurs, expected cost is $pL$. This is a probability-weighted average, not a prediction that the exact cost will occur. It can also obscure a low-probability, high-severity event, so report that event separately when its severity matters to the decision.

Worked Example: Testing a Model Against a Manual Fallback

A hypothetical insurer compares a model-supported process with a manual process. All amounts below are in thousands of dollars per year. For the model-supported process, expected residual cost from claim errors is $68, and controls cost $18. An outage has an annual probability of 0.04 and, if it occurs, adds $250 in cost. For the manual process, expected claim-error cost is $91, disruption has probability 0.01 and severity $100, and staffing costs $30. Assume these cost components do not overlap and that the stated probabilities and costs are appropriate for this comparison.

Method. Add fixed expected costs and probability-weighted disruption costs for each process.

For the model-supported process, expected outage cost is $0.04\times \$250=\$10$ thousand. Total expected annual cost is $\$68+\$18+\$10=\$96$ thousand.

For the manual process, expected disruption cost is $0.01\times \$100=\$1$ thousand. Total expected annual cost is $\$91+\$30+\$1=\$122$ thousand. On these assumptions, the model-supported process has expected cost lower by $\$122-\$96=\$26$ thousand per year.

The comparison also shows how much outage probability would make the model-supported option tie the manual option. Let $p$ be the model-supported process’s annual outage probability. Its expected cost, in thousands, is $68+18+250p=86+250p$. Set this equal to the manual option’s 122:

86+250p=122 \qquad\Longrightarrow\qquad 250p=36 \qquad\Longrightarrow\qquad p=0.144.

Thus, under the stated cost assumptions, the alternatives tie at an outage probability of 0.144, or 14.4%. Below that value the model-supported option has lower expected cost; above it the manual process has lower expected cost. The calculation is not a guarantee that an outage will or will not occur. A report should preserve the outage probability and severity as visible residual risks rather than presenting only the $96,000 expected total.

Common Traps When Reporting Trade-Offs

Choosing by error rate alone. The smaller error rate need not correspond to lower decision cost when false positives and false negatives have different consequences.
Reporting gross benefit as net benefit. Subtract the costs of the proposed action before describing the financial comparison.
Treating an assumed effect as a model result. A predicted risk and an intervention’s ability to change that risk are separate inputs to a decision.
Calling expected cost a guaranteed cost. An expected value summarizes uncertain outcomes; individual realized costs may be higher or lower.
Hiding residual risk in an aggregate total. State material errors or operational scenarios separately when the total alone would conceal their possible severity.

A common wrong calculation is to compare only classification error rates and recommend threshold A in the first example: A has 100 errors out of 1,000, or 10.0%, while B has 128 errors, or 12.8%. That calculation correctly compares error counts, but it does not answer which threshold has lower assumed cost. The cost calculation gives $15,500 for A and $9,600 for B, because the missed adverse cases have a larger stated cost than the additional reviews. The correction is to use the decision-relevant consequences, while still reporting the error rates as predictive information.

Key takeaway: A sound trade-off report separates predictive performance, action costs, assumed intervention benefits, and residual operational risks. Compare alternatives on consistent terms, state the assumptions behind the totals, and make clear that choosing an action manages uncertainty rather than eliminating it.

Check Your Understanding

For each problem, compare the stated alternatives using the given assumptions. Treat amounts as hypothetical and do not infer benefits beyond those specified.

A review rule flags 100 cases at $30 each and leaves 12 adverse cases unflagged, each with an assumed cost of $500. A second rule flags 140 cases and leaves 6 adverse cases unflagged. What is the second rule’s review cost if its total cost is $6,000? What cost per missed case is implied by those figures? (Answer: Review cost is $4,200; the implied cost per missed case is $300.)
A program has baseline expected cost of $800,000. Option A is assumed to reduce it by 8% and costs $25,000. Option B is assumed to reduce it by 10% and costs $45,000. Which has lower total expected cost, and by how much? (Answer: A totals $761,000; B totals $765,000; A is lower by $4,000.)
A model process has fixed annual expected costs of $54,000 and an outage severity of $200,000. A manual alternative costs $82,000 annually. What outage probability makes their expected costs equal? (Answer: 0.14, or 14%.)
Two thresholds are assessed on 500 cases. Threshold X has 30 false positives and 10 false negatives; threshold Y has 50 false positives and 4 false negatives. Reviews cost $20 each and missed cases cost $400 each. Which has lower total cost under the stated assumptions? (Answer: X costs $20(30)+400(10)=\$4{,}600$; Y costs $20(50)+400(4)=\$2{,}600$, so Y is lower by $2,000.)
A control is expected to prevent $36,000 in annual losses and costs $28,000. A separate disruption has probability 0.05 and severity $120,000. Assuming the disruption cost is additional and is not reduced by the control, what is the net expected benefit after including disruption cost? (Answer: Expected disruption cost is $6,000; net expected benefit is $\$36{,}000-\$28{,}000-\$6{,}000=\$2{,}000$.)

What You'll Learn