Asymmetric Consequences of Prediction Errors

Why the Two Kinds of Error May Not Be Equally Serious

A prediction system can be wrong in two ways when the outcome is binary. It can flag a case even though the event does not occur, or it can fail to flag a case when the event does occur. Those errors need not have equal consequences. A hypothetical fraud team might spend time investigating an innocent transaction after a false alarm, while a missed fraud event could create a much larger loss. In a hypothetical retention program, contacting a customer who would have stayed anyway may waste a modest amount, while failing to contact a customer who then leaves may have a greater financial consequence.

The predicted probability alone does not determine which cases should be flagged. A decision also depends on what the two types of error cost, expressed in comparable units. This topic is aligned with the SOA Exam PA (Predictive Analytics) syllabus: the aim is to reason clearly about consequences when using predictions, not to treat one error type as automatically more important in every setting.

Definition: Let $E$ denote the event of interest, and let $D=1$ mean that a case is flagged for an action such as review or outreach. A false positive occurs when $D=1$ but $E$ does not occur. A false negative occurs when $D=0$ but $E$ does occur. Write $C_{\mathrm{FP}}$ for the cost assigned to a false positive and $C_{\mathrm{FN}}$ for the cost assigned to a false negative. In the basic cost comparison here, a correct decision has cost zero.

These costs should represent the consequences relevant to the decision, rather than a generic measure of model performance. For instance, a false-positive cost might include the staff time and inconvenience of a review. A false-negative cost might represent an expected financial consequence of not detecting an event. The costs need to be stated on a common scale, such as dollars per case. If relevant costs cannot reasonably be summarized in that way, a single numerical threshold may not represent the full decision.

A probability score and a decision are different things. A score $p=P(E\mid\text{available information})$ describes the estimated chance of the event for a case. A rule such as “flag if $p$ exceeds a specified threshold” converts that estimate into a decision. Changing the relative costs can change the appropriate threshold even when the probability estimates remain unchanged.

Deriving a Cost-Based Decision Threshold

Suppose the probability of the event for one case is $p$, the only two decisions are to flag or not to flag, and the two error costs are fixed. Assume the probability is appropriate for the case and the costs are measured on the same scale. If the case is flagged, it incurs a false-positive cost only when the event does not occur. If it is not flagged, it incurs a false-negative cost only when the event does occur.

\begin{aligned} \text{Expected cost of flagging} &= (1-p)C_{\mathrm{FP}},\\ \text{Expected cost of not flagging} &= pC_{\mathrm{FN}}. \end{aligned}

The first expression follows because the probability of no event is $1-p$; in that outcome, flagging is a false positive. The second follows because the probability of the event is $p$; in that outcome, not flagging is a false negative. To choose the lower expected cost, flag when

(1-p)C_{\mathrm{FP}} < pC_{\mathrm{FN}}.

Rearrange the inequality by adding $pC_{\mathrm{FP}}$ to both sides and then dividing by the positive sum of the costs:

C_{\mathrm{FP}} < p(C_{\mathrm{FP}}+C_{\mathrm{FN}}) \quad\Longleftrightarrow\quad p > \frac{C_{\mathrm{FP}}}{C_{\mathrm{FP}}+C_{\mathrm{FN}}}.

Thus, when both costs are positive, the threshold is $C_{\mathrm{FP}}/(C_{\mathrm{FP}}+C_{\mathrm{FN}})$. At equality, either decision has the same expected cost under this simplified setup. This derivation explains why a larger false-negative cost lowers the threshold: the decision-maker is willing to flag cases with smaller event probabilities to avoid more costly missed events.

Key result: Under the stated two-decision, two-error-cost assumptions, flag a case when $p$ exceeds $C_{\mathrm{FP}}/(C_{\mathrm{FP}}+C_{\mathrm{FN}})$. This is a cost-based rule, not a universal threshold. It does not by itself account for a binding review capacity, varying case-specific costs, or other decision constraints.

Worked Decisions With Different Error Costs

Worked Example: Prioritizing Hypothetical Fraud Reviews

A hypothetical insurer is considering whether to send transactions for manual fraud review. For this simplified comparison, a false positive costs $48 in review time and customer disruption; a false negative costs $252 in expected loss. A risk estimate is available for two transactions: one has estimated fraud probability $p=0.10$, and the other has $p=0.25$. Which should be flagged under the cost-based rule?

Method. Compare the expected cost of flagging with the expected cost of not flagging for each transaction. The threshold is

\frac{C_{\mathrm{FP}}}{C_{\mathrm{FP}}+C_{\mathrm{FN}}} = \frac{48}{48+252} = \frac{48}{300} = 0.16.

For the first transaction, $p=0.10$, which is below $0.16$. The expected cost of flagging is $(1-0.10)(48)=0.90(48)=\$43.20$. The expected cost of not flagging is $0.10(252)=\$25.20$. Not flagging has the lower expected cost.

For the second transaction, $p=0.25$, which is above $0.16$. The expected cost of flagging is $(1-0.25)(48)=0.75(48)=\$36.00$. The expected cost of not flagging is $0.25(252)=\$63.00$. Flagging has the lower expected cost.

Check. The false-negative cost is more than five times the false-positive cost, so a threshold below $0.5$ is sensible here; the calculated threshold is $0.16$. The common mistake is to flag only when the event is more likely than not. That rule would use $0.5$, ignoring the substantial difference between the error costs.

Worked Example: Comparing Customer-Retention Outreach Groups

A hypothetical insurer is considering outreach to customers who may not renew. In this example, a false-positive outreach costs $15 per customer, while a missed lapse has an assigned cost of $85. A group of 30 customers has estimated lapse probability $0.30$ per customer, and a group of 70 has estimated lapse probability $0.08$ per customer. Treat the probabilities and costs as constant within each group. Which group or groups should receive outreach under the basic rule, and what is the total expected cost?

Method. First calculate the threshold:

\frac{15}{15+85}=\frac{15}{100}=0.15.

For each customer in the group with probability $0.30$, flagging has expected cost $(1-0.30)(15)=0.70(15)=\$10.50$. Not flagging has expected cost $0.30(85)=\$25.50$. The group should receive outreach. For 30 customers, the group’s expected cost under outreach is $30(10.50)=\$315$.

For each customer in the group with probability $0.08$, flagging has expected cost $(1-0.08)(15)=0.92(15)=\$13.80$. Not flagging has expected cost $0.08(85)=\$6.80$. The group should not receive outreach. For 70 customers, its expected cost without outreach is $70(6.80)=\$476$.

The total expected cost of these decisions is $\$315+\$476=\$791$. As a check, outreach to both groups would cost $30(10.50)+70(13.80)=315+966=\$1{,}281$. Outreach to neither group would cost $30(25.50)+70(6.80)=765+476=\$1{,}241$. Both alternatives have greater expected cost than the group-specific decisions.

Common mistake avoided. The group of 70 has more customers, but group size alone does not make it the better outreach target. The relevant comparison is the expected cost per customer for each decision, followed by aggregation across the group.

Worked Example: Why the Same Probability Can Lead to a Different Decision

A hypothetical review unit has 100 cases, each with an estimated event probability of $0.20$. First, suppose the false-positive cost is $25 and the false-negative cost is $75. Then consider a revised assessment in which the false-positive cost is $60 and the false-negative cost remains $75. Compare the decision and total expected cost in each setting, assuming the cases are treated identically.

First cost assessment. The threshold is $25/(25+75)=25/100=0.25$. Since $0.20<0.25$, do not flag. Per case, expected cost is $0.20(75)=\$15$, so the expected cost across 100 cases is $100(15)=\$1{,}500$.

Revised cost assessment. The threshold is $60/(60+75)=60/135=0.4444$ to four decimal places. The event probability remains $0.20$, still below the threshold, so the cases remain unflagged. Per case, expected cost is $0.20(75)=\$15$, and the total expected cost remains $\$1{,}500$.

Check. In both assessments, not flagging costs $15 per case, while flagging costs $(1-0.20)(25)=\$20$ per case under the first assessment and $(1-0.20)(60)=\$48$ under the revised one. The unflagged decision is cheaper in each case. The calculation also illustrates that changing a cost does not guarantee a different decision; the probability must cross the resulting threshold.

Interpreting the Result Carefully

A cost-based threshold is useful only when its ingredients fit the decision being made. Before applying it, specify the event, the meaning of flagging, the time period to which the probability refers, and the consequences assigned to each type of error. A cost stated per investigation cannot be compared directly with a missed-event cost stated per year or per portfolio unless both are converted to a consistent basis.

The threshold also relies on the probability being suitable for the population and cases where it will be used. If a stated probability does not represent the event frequency for comparable cases, comparing expected costs using that probability can lead to a poor decision. The formula itself does not correct a probability estimate or determine whether the assigned costs are appropriate; those are inputs that need justification.

Costs may also be uncertain or differ across cases. A false alarm involving a small transaction may be less costly than one involving a complex investigation, and the expected consequence of a missed event may vary with the amount at risk. If costs vary materially, using one threshold for all cases may conceal relevant differences. A decision-maker should state whether costs are averages, case-specific estimates, or simplified assumptions.

Finally, this basic threshold treats the two correct-decision outcomes as having zero cost and uses only the two error costs. It is not appropriate without modification if correct decisions have different nonzero consequences, there are more than two possible actions, or additional operational constraints determine which cases can be handled. A derived threshold is a transparent starting point under its assumptions, not a substitute for describing the actual decision context.

Common Traps

Assuming equal costs. A threshold of $0.5$ follows from this rule only when the two error costs are equal. It is not a default that applies regardless of context.
Confusing probability with action. A probability such as $0.20$ describes estimated risk; it does not, by itself, prescribe a decision. The costs and decision rule matter.
Comparing unlike units. Costs must use a common scale and refer to a consistent unit, such as dollars per case over the relevant outcome period.
Calling every flagged case a true event. A flag is a decision based on a score; it does not establish that the event occurred.
Reporting expected costs as certain outcomes. An expected cost summarizes probability-weighted possibilities. It is not a claim that a fractional event or exact dollar amount will occur.

Key takeaway: When false positives and false negatives have different consequences, compare their expected costs rather than applying an automatic probability cutoff. State the event, decision, and costs; derive the threshold from those costs; and verify that the assumptions match the situation.

Check Your Understanding

For each question, use the basic two-error-cost setup unless the problem states otherwise. Treat all costs as dollars per case.

A review decision has false-positive cost $18 and false-negative cost $162. Find the cost-based threshold and decide whether to flag a case with event probability $0.09$. Compare both expected costs. (Answer: Threshold $0.10$; do not flag, since flagging costs $16.38 and not flagging costs $14.58.)
A case has event probability $0.12$, false-positive cost $30, and false-negative cost $170. Which decision has lower expected cost, and by how much? (Answer: Do not flag; costs are $20.40 for not flagging and $26.40 for flagging, a difference of $6.00.)
For 50 identical cases, the event probability is $0.20$, the false-positive cost is $40, and the false-negative cost is $160. Find the threshold and the total expected cost if all cases are flagged or all are not flagged. (Answer: Threshold $0.20$; the decisions tie, with total expected cost $1,600 under either choice.)
A hypothetical review team has 40 cases with event probability $0.30$ and 60 cases with probability $0.10$. The false-positive cost is $20 and the false-negative cost is $80. Choose a decision for each group and calculate total expected cost. (Answer: Threshold $0.20$; flag the first group and do not flag the second; total expected cost is $560 + $480 = $1,040.)
There are 100 identical cases, each with event probability $0.20$. Compare decisions and total expected costs under (a) false-positive cost $50 and false-negative cost $150, and (b) false-positive cost $20 and false-negative cost $150. (Answer: In (a), threshold $0.25$, so do not flag; total expected cost is $3,000. In (b), threshold $20/170=0.1176$ to four decimal places, so flag; total expected cost is $1,600.)

What You'll Learn