Set Classification Thresholds For Unequal Error Costs

Why a Probability Needs a Decision Rule

A classifier can estimate how likely an event is without deciding what action to take. For example, a hypothetical insurer might estimate the probability that a submitted device claim is fraudulent. The probability alone does not determine whether to refer the claim for review: a false referral and a missed fraudulent claim can have very different consequences. A classification threshold converts the estimated probability into a decision, and the appropriate threshold depends on the costs of the possible errors.

This tutorial considers binary outcomes, with $Y=1$ denoting the event of interest and $Y=0$ its absence. It is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. The key task is not automatically to use a threshold of 0.5, but to choose a threshold that reflects the decision costs and then assess its consequences.

Definition: Let $p=P(Y=1\mid X)$ be the probability of the positive outcome given the available predictors $X$. A threshold $t$ classifies the case as positive when $p\ge t$, and negative when $p<t$. A false positive occurs when the decision is positive but $Y=0$; a false negative occurs when the decision is negative but $Y=1$. Sensitivity is the proportion of actual positives classified positive. Specificity is the proportion of actual negatives classified negative.

Deriving the Cost-Minimizing Threshold

Suppose a false positive has cost $C_{\mathrm{FP}}>0$, a false negative has cost $C_{\mathrm{FN}}>0$, and correct classifications have zero cost. Assume these are the relevant incremental costs per case, and that $p$ is the case’s probability of $Y=1$. The conditional expected cost of deciding positive is the probability of a false positive multiplied by its cost. The conditional expected cost of deciding negative is the probability of a false negative multiplied by its cost.

L(\text{positive}\mid X)=C_{\mathrm{FP}}(1-p), \qquad L(\text{negative}\mid X)=C_{\mathrm{FN}}p.

Choose positive when the first quantity is no greater than the second. Solving the inequality gives the threshold. This short derivation also shows why the false-negative cost belongs in the numerator of the cost of a negative decision, not in the threshold numerator.

\begin{aligned} C_{\mathrm{FP}}(1-p)&\le C_{\mathrm{FN}}p\\ C_{\mathrm{FP}}&\le (C_{\mathrm{FP}}+C_{\mathrm{FN}})p\\ p&\ge \frac{C_{\mathrm{FP}}}{C_{\mathrm{FP}}+C_{\mathrm{FN}}}. \end{aligned}

Result: Under the stated two-cost setup, a cost-minimizing threshold is $t^*=C_{\mathrm{FP}}/(C_{\mathrm{FP}}+C_{\mathrm{FN}})$. Classify as positive when $p\ge t^*$. If the two actions have additional costs, or correct classifications have different costs, include those costs in the conditional expected costs before comparing the actions; this threshold formula may then change.

When the false-negative cost is much larger than the false-positive cost, $t^*$ is low: it takes less evidence to choose positive. If the false-positive cost is larger, $t^*$ is high. At exact equality, either action has the same conditional expected cost under this model. A threshold of 0.5 is optimal only when the two error costs are equal.

Worked Example: Choosing a Threshold From Error Costs

In a hypothetical screening process, referring a non-fraudulent claim for investigation costs $1,000, while failing to refer a fraudulent claim has an estimated cost of $3,000. Treat correct decisions as having zero incremental cost. A claim receives an estimated fraud probability of 0.32. Should it be referred under the cost-minimizing rule?

Method. Set $C_{\mathrm{FP}}=\$1{,}000$ and $C_{\mathrm{FN}}=\$3{,}000$. The threshold is $t^*=1{,}000/(1{,}000+3{,}000)=0.25$. Since $0.32\ge0.25$, refer the claim.

Check by comparing the conditional costs. Referring has expected cost $(1-0.32)\times \$1{,}000=0.68\times \$1{,}000=\$680$. Not referring has expected cost $0.32\times \$3{,}000=\$960$. The referral has the lower expected cost, agreeing with the threshold rule. The common mistake this avoids is using 0.5 merely because there are two classes.

Evaluating a Threshold With Outcomes

The threshold rule uses probabilities to make decisions for individual cases. To assess a threshold on cases whose outcomes are known, count the resulting errors. A true positive (TP) is an actual positive classified positive; a false negative (FN) is an actual positive classified negative. A true negative (TN) is an actual negative classified negative; a false positive (FP) is an actual negative classified positive.

\text{Sensitivity}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}, \qquad \text{Specificity}=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}.

Sensitivity is also called the true-positive rate; $1-\text{specificity}$ is the false-positive rate. For a validation set with constant costs per error, total observed loss is $\mathrm{FP}\,C_{\mathrm{FP}}+\mathrm{FN}\,C_{\mathrm{FN}}$. Dividing by the number of cases gives average observed loss per case. This is a realized validation-set measure; it is not the same as a model-based expected cost calculated from probabilities.

Worked Example: Comparing Two Thresholds on a Validation Set

A hypothetical validation set has ten claims with the following estimated fraud probabilities and known outcomes. Here $Y=1$ means fraudulent. A false referral costs $600, and a missed fraudulent claim costs $2,400. Compare thresholds 0.20 and 0.50.

The data are: A $(0.80,1)$, B $(0.65,0)$, C $(0.42,1)$, D $(0.35,0)$, E $(0.25,0)$, F $(0.18,1)$, G $(0.12,0)$, H $(0.08,1)$, I $(0.55,1)$, and J $(0.30,0)$, where each pair gives (probability, outcome). There are five actual positives (A, C, F, H, I) and five actual negatives (B, D, E, G, J).

At threshold 0.20. The predicted positives are A, B, C, D, E, I, and J. Among these, A, C, and I are TP, and B, D, E, and J are FP. The unselected cases F and H are FN, and G is TN. Thus TP = 3, FP = 4, FN = 2, and TN = 1. Sensitivity is $3/5=0.6000$, and specificity is $1/5=0.2000$. Total loss is $4(\$600)+2(\$2{,}400)=\$2{,}400+\$4{,}800=\$7{,}200$, or $\$720$ per case.

At threshold 0.50. The predicted positives are A, B, and I. A and I are TP, and B is FP. C, F, and H are FN; D, E, G, and J are TN. Therefore TP = 2, FP = 1, FN = 3, and TN = 4. Sensitivity is $2/5=0.4000$, and specificity is $4/5=0.8000$. Total loss is $1(\$600)+3(\$2{,}400)=\$600+\$7{,}200=\$7{,}800$, or $\$780$ per case.

The 0.20 threshold has lower observed loss in this validation set, at the cost of lower specificity. The cost-based threshold from the supplied costs is $600/(600+2{,}400)=0.20$, consistent with the comparison. This example avoids treating higher specificity as automatically better: the costs and the missed-positive count matter too.

Expected Loss Across a Population

Sensitivity and specificity describe error rates but do not, by themselves, determine the dollar cost. The prevalence of the positive outcome and the costs of the two error types are also needed. Let $\pi=P(Y=1)$, let $\mathrm{FNR}=1-\text{sensitivity}$, and let $\mathrm{FPR}=1-\text{specificity}$. Under fixed error costs, expected loss per case in the target population is

E[L]=C_{\mathrm{FP}}(1-\pi)\mathrm{FPR} +C_{\mathrm{FN}}\pi\mathrm{FNR}.

The first term is the probability of an actual negative multiplied by its false-positive rate and the false-positive cost. The second is the probability of an actual positive multiplied by its false-negative rate and cost. This population expression helps explain why a threshold comparison can change across settings: the class prevalence, the error costs, or the conditional performance may differ.

Worked Example: Choosing Between a Cost Rule and the 0.50 Default

A hypothetical review team has false-positive cost $1,000 and false-negative cost $3,000. For six independent cases, assume the supplied probabilities are calibrated for the population to which the decisions will be applied: 0.05, 0.18, 0.25, 0.32, 0.70, and 0.90. Correct decisions have zero incremental cost. Compare the cost-minimizing decisions with using 0.50 for every case.

Cost-based decisions. The threshold is $1{,}000/(1{,}000+3{,}000)=0.25$. At $p=0.05$, the costs of positive and negative decisions are $0.95(\$1{,}000)=\$950$ and $0.05(\$3{,}000)=\$150$; decide negative. At $p=0.18$, the costs are $\$820$ and $\$540$; decide negative. At $p=0.25$, both costs are $\$750$; either decision is optimal. Use negative at the tie for this calculation. At $p=0.32$, the costs are $\$680$ and $\$960$; decide positive. At $p=0.70$, they are $\$300$ and $\$2{,}100$; decide positive. At $p=0.90$, they are $\$100$ and $\$2{,}700$; decide positive.

The total expected cost for these six cases under the cost rule is $\$150+\$540+\$750+\$680+\$300+\$100=\$2{,}520$, or $\$420$ per case.

Using 0.50 instead. The cases with probabilities 0.70 and 0.90 are classified positive, with respective expected costs of $300 and $100. The other four are classified negative, with costs $150, $540, $750, and $0.32(\$3{,}000)=\$960$. Total expected cost is $\$300+\$100+\$150+\$540+\$750+\$960=\$2{,}800$, or about $466.67 per case. The cost rule reduces expected cost by $\$2{,}800-\$2{,}520=\$280$ across the six decisions. The common mistake is assuming 0.50 is a neutral cutoff; it disregards the stated three-to-one cost difference.

Practical Checks and Common Traps

Threshold selection is a decision step, not a way to improve the underlying probability estimates. If probabilities are poorly calibrated, the theoretical cutoff may not deliver the predicted costs. Compare candidate thresholds on validation data that represent the intended population, and calculate both the error counts and the cost under the stated cost assumptions. Do not use test outcomes repeatedly to choose a threshold and then report performance on that same test set as an unbiased final assessment.

As a threshold decreases, the set of cases classified positive can only stay the same or grow. On a fixed data set, this means TP and FP counts cannot decrease, while FN and TN counts cannot increase. Sensitivity therefore cannot decrease, and specificity cannot increase, as the threshold is lowered. These directions are useful checks, but they do not establish which threshold has the lowest loss.

Reversing the cost ratio. The threshold numerator is $C_{\mathrm{FP}}$, because the positive action incurs cost when the outcome is negative. Check it by comparing the two conditional costs.
Confusing cost with rate. Sensitivity and specificity are proportions. To obtain expected loss, combine the error rates with prevalence and the corresponding costs.
Calling the score a decision. A probability of 0.30 is an input to a decision rule; it does not imply positive classification unless the chosen threshold is at most 0.30.
Ignoring the target population. A threshold comparison on one validation set may not represent a population with different prevalence, calibration, or error costs.

Key takeaway: With calibrated probabilities, constant false-positive and false-negative costs, and zero incremental cost for correct decisions, compare $C_{\mathrm{FP}}(1-p)$ with $C_{\mathrm{FN}}p$. For a threshold under this setup, use $C_{\mathrm{FP}}/(C_{\mathrm{FP}}+C_{\mathrm{FN}})$, then evaluate threshold choices using the relevant error costs rather than accuracy alone.

Check Your Understanding

For each problem, use the stated costs and decision setup. Answers are included for checking.

A hypothetical alert has false-positive cost $400 and false-negative cost $1,600. Find the cost-minimizing threshold and the decision at probability 0.19. (Answer: threshold 0.20; classify negative.)
A validation set contains 40 actual positives and 60 actual negatives. At a selected threshold, TP = 30 and FP = 12. Find sensitivity and specificity. (Answer: sensitivity $30/40=0.7500$; specificity $(60-12)/60=0.8000$.)
False-positive cost is $500 and false-negative cost is $2,000. A case has probability 0.30. Calculate both conditional decision costs and choose the lower-cost decision. (Answer: positive cost $350; negative cost $600; classify positive.)
In a validation set, TP = 24, FN = 6, FP = 10, and TN = 60. False-positive cost is $300 and false-negative cost is $900. Calculate sensitivity, specificity, total observed loss, and average loss per case. (Answer: sensitivity 0.8000; specificity $60/70=0.8571$; total loss $10(\$300)+6(\$900)=\$8{,}400$; average loss $\$84$ across 100 cases.)
A population has prevalence $\pi=0.10$, sensitivity 0.80, and specificity 0.90. False-positive cost is $200 and false-negative cost is $1,000. Calculate expected loss per case. (Answer: $\mathrm{FPR}=0.10$, $\mathrm{FNR}=0.20$; expected loss $200(0.90)(0.10)+1{,}000(0.10)(0.20)=\$38$.)

What You'll Learn