A Threshold Decision Case With Unequal Error Costs

From Fitted Probabilities to Decisions

A logistic model produces fitted probabilities, but an action often requires a binary decision: flag a claim for review or do not flag it; offer an intervention or do not offer it. A probability threshold converts each fitted probability into a classification. The threshold should reflect the consequences of the two possible errors, not simply be assumed to be one-half.

This tutorial considers a binary outcome $Y$, where $Y=1$ denotes the event of interest and $Y=0$ denotes its absence. For case $i$, let $\hat p_i$ be the logistic model's fitted probability that $Y_i=1$. A prediction is positive when the case is classified as $Y=1$. The decision framework is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Definition: A false positive occurs when the classifier predicts $Y=1$ but the observed outcome is $Y=0$. A false negative occurs when it predicts $Y=0$ but $Y=1$ is observed. Let $C_{\mathrm{FP}}>0$ be the cost of a false positive and $C_{\mathrm{FN}}>0$ the cost of a false negative. Correct classifications are assumed to have zero error cost unless the problem specifies otherwise.

The costs are conditional on the actual outcome and the decision. For example, in a hypothetical claim-screening process, a false positive might incur the expense of an unnecessary investigation, while a false negative might mean that a claim needing review is missed. The costs and scenario in such an example are assumptions for analysis, not claims about any actual insurer.

Deriving the Cost-Based Threshold

Suppose a case has fitted probability $p$ of $Y=1$. If we predict positive, the error occurs when $Y=0$, with probability $1-p$. The expected error cost of that decision is therefore $C_{\mathrm{FP}}(1-p)$. If we predict negative, the error occurs when $Y=1$, with probability $p$, so its expected error cost is $C_{\mathrm{FN}}p$.

\begin{aligned} \text{Expected cost of predicting positive} &= C_{\mathrm{FP}}(1-p),\\ \text{Expected cost of predicting negative} &= C_{\mathrm{FN}}p. \end{aligned}

Choose the decision with the smaller expected cost. Predict positive when $C_{\mathrm{FP}}(1-p)\leq C_{\mathrm{FN}}p$. Because both costs are positive, rearranging gives $C_{\mathrm{FP}}\leq (C_{\mathrm{FP}}+C_{\mathrm{FN}})p$, and hence the cost-based threshold is:

t^*=\frac{C_{\mathrm{FP}}}{C_{\mathrm{FP}}+C_{\mathrm{FN}}}.

Decision rule: When the only costs are $C_{\mathrm{FP}}$ and $C_{\mathrm{FN}}$, predict $Y=1$ if $\hat p_i\geq t^*$, and predict $Y=0$ otherwise. At equality, the two decisions have equal expected cost; this tutorial assigns ties to the positive class. A different tie convention does not change the minimum expected cost at that exact probability.

This rule assumes that the fitted probabilities represent the relevant cases, that the two error costs apply to each case as stated, and that there is no capacity constraint or additional cost for a correct decision. If a problem instead supplies an operational threshold, use that threshold to classify; it need not equal $t^*$. A threshold selected to meet a review budget, for example, may not minimize the two-cost expected loss.

Worked Example: Deriving a Threshold and Classifying Claims

A hypothetical review model assigns each claim a fitted probability of needing additional investigation. An unnecessary investigation costs $40, while failing to investigate a claim that needs review costs $160. The fitted probabilities and observed outcomes for ten claims are listed below. Find the cost-based threshold, form the confusion matrix, and calculate the realized error cost.

Claim	Fitted probability	Observed outcome
A	0.08	0
B	0.18	1
C	0.22	0
D	0.35	1
E	0.55	0
F	0.72	1
G	0.12	1
H	0.28	0
I	0.04	0
J	0.90	1

Step 1: Find the threshold. Here $C_{\mathrm{FP}}=40$ and $C_{\mathrm{FN}}=160$. Thus $t^*=40/(40+160)=40/200=0.20$. Predict positive at probabilities of 0.20 or greater.

Step 2: Classify each claim. Claims C, D, E, F, H, and J have fitted probabilities at least 0.20 and are predicted positive. Claims A, B, G, and I are predicted negative. In particular, B and G are false negatives because their observed outcomes are 1; C, E, and H are false positives because their observed outcomes are 0.

Step 3: Complete the confusion matrix. Among the six positive predictions, D, F, and J have observed outcome 1, giving 3 true positives; C, E, and H give 3 false positives. Among the four negative predictions, A and I give 2 true negatives; B and G give 2 false negatives.

	Predicted positive	Predicted negative	Total
Observed positive	3 (true positive)	2 (false negative)	5
Observed negative	3 (false positive)	2 (true negative)	5
Total	6	4	10

Step 4: Calculate realized cost. There are 3 false positives and 2 false negatives, so the total error cost is $3(40)+2(160)=120+320=\$440$. This uses the observed confusion matrix, not the expected cost calculation used to derive the threshold.

Check. The matrix totals $3+2+3+2=10$, matching the ten claims. The realized cost is in dollars and includes only the two specified kinds of error. The method avoids treating every fitted probability below 0.50 as negative when unequal error costs imply a different threshold.

Evaluating a Selected Threshold

A confusion matrix compares predicted classes with observed outcomes. Its row totals describe the actual outcome counts, and its column totals describe the predicted counts. Use the same threshold for every case unless the decision rule explicitly varies by case.

The count of true positives is the number of cases with $Y=1$ predicted positive; true negatives have $Y=0$ and are predicted negative. The off-diagonal cells are the errors. Once these counts are known, the realized cost is $C_{\mathrm{FP}}$ times the false-positive count plus $C_{\mathrm{FN}}$ times the false-negative count.

\text{Realized error cost} = C_{\mathrm{FP}}(\text{number of false positives}) + C_{\mathrm{FN}}(\text{number of false negatives}).

A cost-based threshold minimizes expected cost case by case under the stated assumptions. It does not guarantee the lowest realized cost in every finite sample. The observed outcomes may happen to make another threshold look less costly on a particular set of cases. That sample-specific result does not invalidate the expected-cost derivation; it is one reason to distinguish a decision rule based on probabilities from a retrospective comparison using realized outcomes.

Worked Example: Comparing Thresholds on the Same Cases

A hypothetical alert system has $C_{\mathrm{FP}}=\$25$ and $C_{\mathrm{FN}}=\$100$. Ten cases have the following fitted probabilities and observed outcomes. Compare the cost-based threshold with a higher operational threshold of 0.50 by calculating each confusion matrix and realized cost.

Case	Fitted probability	Observed outcome
1	0.10	0
2	0.18	1
3	0.24	0
4	0.32	1
5	0.41	0
6	0.48	1
7	0.63	0
8	0.76	1
9	0.88	0
10	0.92	1

Step 1: Derive the cost-based threshold. The threshold is $25/(25+100)=25/125=0.20$. The specified operational threshold, 0.50, is higher than this cost-based threshold.

Step 2: Classify at 0.20. Cases 3 through 10 are predicted positive; cases 1 and 2 are predicted negative. The positive outcomes are cases 2, 4, 6, 8, and 10. Thus cases 4, 6, 8, and 10 are true positives, while case 2 is a false negative. Of the predicted positives, cases 3, 5, 7, and 9 are false positives. Case 1 is the only true negative. The counts are TP = 4, FN = 1, FP = 4, and TN = 1.

Step 3: Calculate the realized cost at 0.20. The cost is $4(25)+1(100)=100+100=\$200$.

Step 4: Classify at 0.50. Cases 7 through 10 are predicted positive. Cases 8 and 10 are true positives; cases 7 and 9 are false positives. Cases 2, 4, and 6 are false negatives, and cases 1, 3, and 5 are true negatives. The counts are TP = 2, FN = 3, FP = 2, and TN = 3.

Step 5: Calculate the realized cost at 0.50. The cost is $2(25)+3(100)=50+300=\$350$. On these cases, the threshold 0.20 has lower realized cost by $\$350-\$200=\$150$.

Check. Both matrices total ten cases. The higher threshold predicts fewer positives, reducing false positives from 4 to 2 but increasing false negatives from 1 to 3. Because a false negative costs four times as much as a false positive here, this trade-off raises realized cost. The method avoids comparing thresholds by the number of errors alone; their costs are unequal.

Common Mistake: Reversing the Cost Ratio

The numerator of the threshold is the false-positive cost. A common incorrect calculation puts the false-negative cost in the numerator. For the first example, that error would produce $160/(40+160)=0.80$, leading to a threshold of 0.80. This reverses the intended effect: when false negatives are more costly, the threshold should be lower, not higher, so that more cases are classified positive.

The direction can be checked directly from the expected costs. If $C_{\mathrm{FN}}$ increases while $C_{\mathrm{FP}}$ stays fixed, predicting negative becomes more expensive at any given $p>0$. Therefore the classifier should be more willing to predict positive, which means a lower threshold. The formula $C_{\mathrm{FP}}/(C_{\mathrm{FP}}+C_{\mathrm{FN}})$ has exactly that behavior.

Another frequent error is to call the number of predicted positives the number of true positives. A positive prediction is correct only when the observed outcome is also 1. Always compare each predicted class against the observed outcome before filling the matrix or calculating cost.

Worked Example: Checking the Error Trade-Off and Matrix Rates

A hypothetical triage model has a false-positive cost of $90 and a false-negative cost of $210. Use the cost-based threshold to classify ten cases, then calculate the realized cost, sensitivity, specificity, and positive predictive value.

Case	Fitted probability	Observed outcome
A	0.05	0
B	0.12	1
C	0.21	0
D	0.29	1
E	0.31	1
F	0.44	0
G	0.58	1
H	0.66	0
I	0.81	1
J	0.93	0

Step 1: Compute the threshold. $t^*=90/(90+210)=90/300=0.30$. Predict positive for fitted probabilities at least 0.30. The threshold is below 0.50 because a false negative is more costly than a false positive.

Step 2: Classify the cases. E, F, G, H, I, and J are predicted positive. A, B, C, and D are predicted negative. Among positive predictions, E, G, and I have observed outcome 1, so TP = 3; F, H, and J have observed outcome 0, so FP = 3. Among negative predictions, A and C are true negatives, while B and D are false negatives. Thus TN = 2 and FN = 2.

Step 3: Compute the realized cost. The cost is $3(90)+2(210)=270+420=\$690$. The matrix contains five observed positive cases and five observed negative cases, as it should: TP + FN = 3 + 2 = 5 and FP + TN = 3 + 2 = 5.

Step 4: Calculate the requested rates. Sensitivity is the fraction of observed positives classified positive: $3/(3+2)=3/5=0.60$. Specificity is the fraction of observed negatives classified negative: $2/(2+3)=2/5=0.40$. Positive predictive value is the fraction of positive predictions that are correct: $3/(3+3)=3/6=0.50$.

Check. Each rate uses a different denominator: observed positives for sensitivity, observed negatives for specificity, and predicted positives for positive predictive value. This avoids the common mistake of using the total sample size as the denominator for all three.

Practical Interpretation

A lower threshold generally increases the number of positive predictions. This can catch more actual positive cases, reducing false negatives, but it can also classify more actual negative cases as positive, increasing false positives. A higher threshold tends to make the opposite trade-off. The confusion matrix makes that trade-off visible, while the cost calculation weights the two errors according to their consequences.

When interpreting a result, state all three pieces: the threshold and its basis, the confusion-matrix counts, and the cost calculation. For instance, a statement that a model “correctly classified 80%” does not show how many costly false negatives occurred. Accuracy alone can hide an important imbalance between error types, particularly when the outcomes or error costs are unequal.

Use the cost ratio to derive the threshold only when the problem asks for the unconstrained minimum-expected-cost rule under the stated assumptions.
Use a supplied selected threshold as given when the problem specifies an operational or externally chosen rule.
Apply the threshold to fitted probabilities, then compare predictions with observed outcomes to fill the matrix.
Calculate realized cost from false-positive and false-negative counts; do not substitute expected costs for those observed counts.

Key takeaway: With only two error costs and no additional constraints, predict positive when $\hat p_i\geq C_{\mathrm{FP}}/(C_{\mathrm{FP}}+C_{\mathrm{FN}})$. Then evaluate the classifier using the observed confusion matrix and the cost attached to each type of error.

Check Your Understanding

For each problem, identify the relevant decision rule before calculating. Answers are included for checking.

A screening rule has $C_{\mathrm{FP}}=\$30$ and $C_{\mathrm{FN}}=\$120$. Find the cost-based threshold and classify a case with fitted probability 0.19 and a case with fitted probability 0.20, using the tie convention in this tutorial. (Answer: threshold $0.20$; 0.19 is predicted negative and 0.20 is predicted positive.)
A model has the following fitted probabilities and outcomes: (0.10, 0), (0.20, 1), (0.25, 0), (0.31, 1), (0.45, 0), (0.60, 1), (0.72, 0), (0.90, 1), where each pair is (probability, observed outcome). At threshold 0.25, find TP, FP, FN, and TN. (Answer: TP = 3, FP = 3, FN = 1, TN = 1.)
A confusion matrix has TP = 18, FP = 6, FN = 4, and TN = 72. If $C_{\mathrm{FP}}=\$50$ and $C_{\mathrm{FN}}=\$200$, calculate the cost-based threshold, realized error cost, sensitivity, and positive predictive value. (Answer: threshold $0.20$; realized cost $\$1{,}100$; sensitivity $18/22=0.8182$; positive predictive value $18/24=0.7500$.)
A decision has $C_{\mathrm{FP}}=\$80$ and $C_{\mathrm{FN}}=\$320$. Find the unconstrained cost-based threshold. Would a threshold of 0.40 be the cost-minimizing threshold under the two-cost assumptions? (Answer: threshold $0.20$; no, 0.40 is higher than the cost-minimizing threshold.)
At a selected threshold of 0.30, six cases have (fitted probability, outcome) pairs (0.10, 0), (0.20, 1), (0.30, 0), (0.40, 1), (0.70, 0), and (0.80, 1). If $C_{\mathrm{FP}}=\$40$ and $C_{\mathrm{FN}}=\$160$, find the confusion-matrix counts and realized error cost. (Answer: TP = 2, FP = 2, FN = 1, TN = 1; realized error cost $2(40)+1(160)=\$240$.)

What You'll Learn