Test Sensitivity to Claim-Cost Decisions

Why Claim-Cost Assumptions Matter

A claim-review model does not make a decision in isolation. Someone chooses a score threshold, and the resulting false positives and false negatives have consequences. A false positive may use review time on a claim that does not meet the review target; a false negative may leave a claim that does meet the target unreviewed. If the relative costs of those outcomes are uncertain, a threshold or model that looks best under one set of assumptions may not remain best under another.

This tutorial treats the target as a binary condition: whether a claim should be reviewed under a specified operational definition. A model supplies either an estimated probability of that condition or a score used to rank claims. The calculations below focus on the cost assigned to decisions, not on proving that a claim is fraudulent or deciding what action is legally appropriate. All claim scenarios and values are hypothetical.

Definition: A false positive (FP) occurs when a claim is flagged for review but does not meet the defined review target. A false negative (FN) occurs when a claim meets the target but is not flagged. Let $C_{FP}$ and $C_{FN}$ denote the incremental costs of these outcomes, respectively. For the basic calculations, correct decisions have zero incremental cost and each claim receives one decision.

These are simplifying assumptions, not universal facts. A correct flag may still consume review resources, a missed claim may have several types of downstream loss, and costs may vary by claim. If those effects matter, they should be represented in the decision costs rather than silently ignored. The purpose of sensitivity analysis is to make consequential assumptions visible and assess whether conclusions depend on them.

From Costs to a Probability Threshold

Suppose the model provides a calibrated probability $p$ that a claim meets the review target. Consider the two available actions: flag the claim or do not flag it. Under the stated cost assumptions, flagging has expected cost $C_{FP}(1-p)$, because a false positive occurs when the target is absent. Not flagging has expected cost $C_{FN}p$, because a false negative occurs when the target is present.

Choose the action with the smaller expected cost. Flag when $C_{FP}(1-p) < C_{FN}p$. Rearranging gives the cost-based threshold.

\begin{aligned} C_{FP}(1-p) &< C_{FN}p\\ C_{FP} &< p(C_{FP}+C_{FN})\\ p &> \frac{C_{FP}}{C_{FP}+C_{FN}}. \end{aligned}

Result: If $C_{FP}>0$, $C_{FN}>0$, and the model probability $p$ is calibrated for the target in the population where it will be used, the cost-minimizing rule is to flag when $p > C_{FP}/(C_{FP}+C_{FN})$. At equality, the two actions have equal expected cost. This rule does not apply as written if the probability is not calibrated, the costs omit material consequences, or the decision has additional constraints such as a fixed review capacity.

The threshold is a probability threshold under the model and assumptions, not a universal score cutoff. When false negatives cost much more than false positives, the threshold falls. When false positives are relatively costly, the threshold rises. In practice, estimated probabilities may be imperfect, costs may be ranges rather than known amounts, and review capacity may constrain the number of flags. The formula is therefore a principled benchmark, not a substitute for checking performance on relevant validation data.

Worked Example: Choosing for One Claim Under Two Cost Assumptions

A hypothetical insurer considers flagging claims for an additional review. For this example, assume the model probability is calibrated and the only incremental costs are $C_{FP}$ and $C_{FN}$. A claim has estimated probability $p=0.24$. First use $C_{FP}=\$30$ and $C_{FN}=\$90$; then consider a plausible alternative $C_{FP}=\$30$ and $C_{FN}=\$150$. Find the cost-based threshold and decision in each case.

Method. Calculate the threshold $C_{FP}/(C_{FP}+C_{FN})$, then compare it with $0.24$. As a check, calculate the expected cost of both actions.

First assumption set. The threshold is $\$30/(\$30+\$90)=30/120=0.25$. Since $0.24<0.25$, do not flag. The expected cost of flagging is $\$30(1-0.24)=\$30(0.76)=\$22.80$. The expected cost of not flagging is $\$90(0.24)=\$21.60$. Not flagging is cheaper by $\$22.80-\$21.60=\$1.20$, which checks the threshold decision.

Alternative assumption set. The threshold is $\$30/(\$30+\$150)=30/180=0.1667$, rounded to four decimal places. Since $0.24>0.1667$, flag. The expected cost of flagging remains $\$22.80$. The expected cost of not flagging is now $\$150(0.24)=\$36.00$. Flagging is cheaper by $\$36.00-\$22.80=\$13.20$.

The decision changes because the assumed cost of a missed target increased. The common mistake is to use the same threshold in both scenarios merely because the model probability did not change. Here the probability is fixed; the cost ratio, and therefore the preferred decision, changed.

Recalculate Costs on Validation Data

A probability-based rule describes an individual decision under stated assumptions. A portfolio analysis usually also needs an empirical check: apply each candidate operating threshold to the same validation records, count the resulting outcomes, and calculate their total cost. For a validation set, the simple cost total is $C_{FP}\times FP+C_{FN}\times FN$. If each claim has the same cost assumptions, dividing by the number of claims gives average cost per claim. This comparison is only informative if the validation cases represent the intended use and the thresholds were not selected by repeatedly optimizing on that same evaluation set.

Worked Example: Rechecking Two Thresholds as Costs Change

On a hypothetical validation set of 1,000 claims, 100 meet a defined review target and 900 do not. At threshold $0.15$, the model produces 80 true positives, 20 false negatives, 180 false positives, and 720 true negatives. At threshold $0.45$, it produces 60 true positives, 40 false negatives, 60 false positives, and 840 true negatives. Compare the two thresholds under two cost assumptions: first $C_{FP}=\$25$, $C_{FN}=\$100$; then $C_{FP}=\$25$, $C_{FN}=\$200$.

Check the counts. At $0.15$, $80+20=100$ target claims and $180+720=900$ non-target claims. At $0.45$, $60+40=100$ and $60+840=900$. Both thresholds are being assessed on the same 1,000 claims.

First assumption set. At threshold $0.15$, total cost is $180(\$25)+20(\$100)=\$4,500+\$2,000=\$6,500$. Average cost is $\$6,500/1,000=\$6.50$ per claim. At threshold $0.45$, total cost is $60(\$25)+40(\$100)=\$1,500+\$4,000=\$5,500$, or $\$5.50$ per claim. Under these assumptions, $0.45$ has lower validation cost by $\$1,000$.

Alternative assumption set. At threshold $0.15$, cost is $180(\$25)+20(\$200)=\$4,500+\$4,000=\$8,500$, or $\$8.50$ per claim. At threshold $0.45$, cost is $60(\$25)+40(\$200)=\$1,500+\$8,000=\$9,500$, or $\$9.50$ per claim. Now threshold $0.15$ has lower validation cost by $\$1,000$.

The lower threshold catches more target claims, but also creates more false positives. Raising the assumed false-negative cost reverses the preferred threshold. The mistake to avoid is comparing only the number of true positives or only the number of false positives: both outcome types contribute to the stated objective.

Testing Whether a Model Preference Is Robust

The same calculation can compare candidate models at specified operating points. It is essential to evaluate both models on the same validation population and to state which thresholds generated their counts. A model with more true positives may also create many more false positives; whether that trade-off is desirable depends on the cost ratio. Counts at particular operating points do not, by themselves, show that one model is generally better at every possible threshold.

Worked Example: Finding the Cost Ratio That Changes the Preferred Model

Two hypothetical claim models are evaluated on the same validation set. At the chosen operating point, Model A has $TP=70$, $FN=30$, and $FP=90$. Model B has $TP=80$, $FN=20$, and $FP=180$. Correct decisions have zero incremental cost. Compare the models first when $C_{FP}=\$20$, $C_{FN}=\$100$, and then when $C_{FP}=\$20$, $C_{FN}=\$200$. Find the break-even ratio of false-negative to false-positive cost.

First assumption set. Model A costs $90(\$20)+30(\$100)=\$1,800+\$3,000=\$4,800$. Model B costs $180(\$20)+20(\$100)=\$3,600+\$2,000=\$5,600$. Model A is cheaper by $\$800$.

Alternative assumption set. Model A costs $90(\$20)+30(\$200)=\$1,800+\$6,000=\$7,800$. Model B costs $180(\$20)+20(\$200)=\$3,600+\$4,000=\$7,600$. Model B is cheaper by $\$200$.

Break-even calculation. Let the costs be $C_{FP}$ and $C_{FN}$, without fixing their dollar values. Model A costs $90C_{FP}+30C_{FN}$, and Model B costs $180C_{FP}+20C_{FN}$. They tie when $90C_{FP}+30C_{FN}=180C_{FP}+20C_{FN}$. Subtracting the left-hand terms from the right-hand terms gives $10C_{FN}=90C_{FP}$, so $C_{FN}/C_{FP}=9$. For ratios below 9, Model A costs less; for ratios above 9, Model B costs less. At a ratio of 9, the costs tie.

The arithmetic checks against both scenarios: $100/20=5$, below 9, so A is preferred; $200/20=10$, above 9, so B is preferred. The conclusion is conditional on these operating points and validation counts. It does not establish that either model is preferable at every threshold or in a different population.

How to Make a Useful Sensitivity Check

A sensitivity analysis is most useful when it tests a defensible range of assumptions rather than selecting whichever costs produce a desired result. State what each cost represents, its unit, and the population or time period to which it applies. For example, a review cost might be measured in dollars per unnecessary investigation, while a missed-target cost might represent an estimated average consequence per missed claim. Do not add costs measured over incompatible units or time horizons.

Define the decision and target.
Specify what a flag triggers and what event makes a claim a positive case. A change in target definition changes the meaning of FP and FN.

State incremental costs.
Identify which consequences are included in $C_{FP}$ and $C_{FN}$, and which are omitted. Use consistent units and avoid counting the same consequence twice.

Recompute decisions or validation costs.
For calibrated probabilities, apply the cost-based threshold. For observed validation outcomes, calculate $C_{FP}\times FP+C_{FN}\times FN$ for each stated operating point.

Vary plausible assumptions.
Repeat the calculation over a documented range or compare break-even cost ratios. Report whether the preferred threshold or model changes, not just the result under one convenient scenario.

Qualify the conclusion.
Explain that estimated costs, validation uncertainty, probability calibration, and changes in the scored population can affect the result. A sensitivity check exposes dependence on assumptions; it does not eliminate uncertainty.

A key distinction is that the cost formula can be calculated from validation counts without calibrated probabilities, while the individual probability-threshold derivation depends on probabilities that have the stated interpretation. A raw score may rank claims but cannot automatically be treated as $p$. If the score is not a calibrated probability, use observed outcomes to assess specified thresholds or establish an appropriate probability mapping before applying the probability rule.

Common Traps and Interpretation

Reversing the cost ratio. The probability threshold is $C_{FP}/(C_{FP}+C_{FN})$, not $C_{FN}/(C_{FP}+C_{FN})$. A high false-negative cost lowers the threshold.
Optimizing a threshold and evaluating it on the same data. That can make the reported cost look too favorable. Choose or tune the threshold using training or validation data, then report a separate evaluation where feasible.
Ignoring prevalence and representativeness. Validation counts reflect the cases in the validation set. If the future mix of cases differs, expected counts and costs can differ too.
Treating assumed costs as measured truth. A break-even ratio can be more informative than one chosen dollar estimate. If plausible costs lie on both sides of the break-even value, the model preference is assumption-sensitive.
Forgetting operational constraints. A cost-minimizing threshold may exceed available review capacity. A constrained decision must account for that capacity rather than claiming the unconstrained threshold is directly implementable.

Key takeaway: A preferred claim-review threshold or model is conditional on the definition of the target, the consequences included in the costs, and the validation population. Recalculate under plausible false-positive and false-negative costs, and report any point at which the preference changes. This makes the decision transparent without presenting an assumed cost scenario as universally correct.

Check Your Understanding

For each problem, identify the relevant assumption and check whether the cost comparison supports the stated decision.

A calibrated claim probability is $0.18$. Assume $C_{FP}=\$30$ and $C_{FN}=\$120$, with correct decisions having zero incremental cost. Find the threshold and compare the expected costs of flagging and not flagging. (Answer: Threshold $=30/150=0.2000$; flag cost $=\$24.60$, do-not-flag cost $=\$21.60$, so do not flag.)
On a validation set, a threshold produces $TP=28$, $FN=12$, $FP=32$, and $TN=128$. If $C_{FP}=\$15$ and $C_{FN}=\$50$, find total and average cost. (Answer: Total $=32(\$15)+12(\$50)=\$1,080$; $200$ claims, so average $=\$5.40$ per claim.)
On the same validation set, Model A has $TP=54$, $FN=6$, $FP=40$, and Model B has $TP=57$, $FN=3$, $FP=70$. If $C_{FP}=\$10$ and $C_{FN}=\$40$, which has lower cost and by how much? (Answer: A costs $\$640$; B costs $\$820$; A is lower by $\$180$.)
Model A has $FP=90,FN=30$, and Model B has $FP=180,FN=20$ at their respective operating points. Derive the break-even value of $C_{FN}/C_{FP}$ and state which model is cheaper above that value. (Answer: Break-even ratio $=9$; Model B is cheaper when $C_{FN}/C_{FP}>9$.)
For one claim, assume a calibrated probability of $0.20$, $C_{FP}=\$40$, and $C_{FN}=\$160$. Compare the two actions, including the equality case. (Answer: Threshold $=40/200=0.2000$; both actions cost $\$32.00$ in expected cost, so the decision is tied under these assumptions.)

What You'll Learn