Asymmetric Error Costs in Classification

Why Unequal Error Costs Change Success

A classification model can make two kinds of error: it can flag a case that is actually negative, or fail to flag a case that is actually positive. These errors do not necessarily have the same consequences. In a hypothetical insurance review process, an unnecessary referral may require staff time, while a missed high-risk claim may lead to a much larger avoidable loss. A success criterion based only on counting errors would treat those outcomes as equivalent.

The appropriate comparison therefore depends on the action connected to each prediction and the consequences of being wrong. A model with more errors can be preferable if its errors are concentrated in the less costly category. Conversely, even a small number of expensive errors can outweigh many inexpensive ones. This tutorial develops a direct way to make that trade-off explicit, aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: A false positive is a negative case predicted to be positive. A false negative is a positive case predicted to be negative. Let $c_{\mathrm{FP}}$ be the incremental cost of one false positive and $c_{\mathrm{FN}}$ the incremental cost of one false negative. A cost-based success criterion compares the costs attributable to these errors, rather than assigning the same penalty to every misclassification.

Costs should be stated in a common unit, such as dollars per case or a clearly defined scale of operational impact. They should represent the consequences relevant to the decision, not simply whichever quantities are easiest to measure. For example, if a false positive causes both a review expense and a customer-service expense, both may belong in $c_{\mathrm{FP}}$, provided they are incremental consequences of that error and are not counted elsewhere.

For a transparent comparison, initially assume that each case is classified once, false-positive and false-negative costs are additive, and correct classifications have zero incremental cost. Also assume the costs are applicable to the cases being evaluated. These assumptions may be relaxed in a broader business analysis, but they must be made explicit before using a cost calculation.

Calculating Error-Related Cost

Suppose a model is evaluated on $n$ cases. Let $FP$ denote its number of false positives and $FN$ its number of false negatives. Under the assumptions above, its total error-related cost is the sum of the costs of the two error types:

C = c_{\mathrm{FP}}FP + c_{\mathrm{FN}}FN.

If every case is assigned one of the two actual classes and one of the two predicted classes, the error cost per evaluated case is $C/n$. When models are evaluated on the same number of comparable cases, their total costs can be compared directly. If sample sizes differ, comparing totals alone is generally not meaningful; use a common exposure, such as cost per case, or project each model to the same operational volume.

Justification: For case $i$, define $I_{\mathrm{FP},i}=1$ if it is a false positive and $0$ otherwise, and define $I_{\mathrm{FN},i}$ similarly. Its error cost is $c_{\mathrm{FP}}I_{\mathrm{FP},i}+c_{\mathrm{FN}}I_{\mathrm{FN},i}$. Summing over cases gives $C=c_{\mathrm{FP}}\sum_i I_{\mathrm{FP},i}+c_{\mathrm{FN}}\sum_i I_{\mathrm{FN},i}$. Since the two sums are $FP$ and $FN$, the stated formula follows. Taking expectations gives the corresponding expected cost; this uses linearity of expectation and does not require errors on different cases to be independent.

The formula is simple, but its interpretation depends on the cost assumptions. If the costs are uncertain, if a false positive can have a different consequence for different groups, or if actions have important costs even when predictions are correct, one pair of fixed error costs may not fully describe success. Such limitations should be acknowledged rather than concealed inside a single score.

Worked Comparisons

Worked Example: More Errors but Lower Total Cost

A hypothetical insurer evaluates two claim-triage models on the same 1,000 claims. A false positive generates an incremental review cost of $15. A false negative has an estimated incremental cost of $120. Model A makes 20 false positives and 4 false negatives. Model B makes 8 false positives and 8 false negatives. Which model has lower error-related cost, and what is the cost per evaluated claim?

Method. Multiply each error count by its corresponding cost, add the two components, and then divide by 1,000 for the per-claim cost.

For Model A, false-positive cost is $20\times \$15=\$300$. False-negative cost is $4\times \$120=\$480$. Therefore,

C_A=\$300+\$480=\$780,\qquad \frac{C_A}{1{,}000}=\$0.78.

For Model B, false-positive cost is $8\times \$15=\$120$. False-negative cost is $8\times \$120=\$960$. Therefore,

C_B=\$120+\$960=\$1{,}080,\qquad \frac{C_B}{1{,}000}=\$1.08.

Model B makes 16 errors, compared with Model A's 24, but its total error-related cost is $300 higher. Under the stated costs and assumptions, Model A is preferred. Common mistake avoided: choosing the model with fewer total errors without considering which type of error it makes.

Worked Example: Finding a Break-Even Error Cost

Two hypothetical review models are assessed on the same group of cases. A false positive costs $10. Model M produces 22 false positives and 4 false negatives; Model N produces 6 false positives and 8 false negatives. The cost per false negative, $c_{\mathrm{FN}}$, is not yet agreed. Find the break-even value and determine which model has lower cost below and above it.

Method. Write both total costs as functions of the unknown false-negative cost, set them equal, and then check the direction of preference on either side.

Model M's false-positive cost is $22\times \$10=\$220$, so $C_M=\$220+4c_{\mathrm{FN}}$. Model N's false-positive cost is $6\times \$10=\$60$, so $C_N=\$60+8c_{\mathrm{FN}}$. At break-even,

\$220+4c_{\mathrm{FN}}=\$60+8c_{\mathrm{FN}}, \qquad \$160=4c_{\mathrm{FN}}, \qquad c_{\mathrm{FN}}=\$40.

Check below the break-even value at $c_{\mathrm{FN}}=\$25$: $C_M=\$220+4(\$25)=\$320$, while $C_N=\$60+8(\$25)=\$260$. Thus N is cheaper below $40. Check above it at $c_{\mathrm{FN}}=\$60$: $C_M=\$220+\$240=\$460$, while $C_N=\$60+\$480=\$540$. Thus M is cheaper above $40. At $40, both cost $\$380$.

The break-even cost is $40 per false negative; N is preferred below $40, M above $40, and they tie at $40. Common mistake avoided: reversing the preference by looking only at which model has fewer false positives. The direction must be checked by substituting a value on each side of the break-even point.

Worked Example: Combining Several Consequences Into an Error Cost

A hypothetical claims team compares two models on 500 cases. Each false positive requires an estimated $24 of staff review and creates an additional $16 of customer-handling expense. Each false negative has an estimated incremental consequence of $180. Model R has 7 false positives and 3 false negatives; Model S has 3 false positives and 5 false negatives. Compare their costs per case.

Method. First combine the stated false-positive consequences, then calculate each model's total and per-case error cost using the same assumptions.

The false-positive cost is $\$24+\$16=\$40$ per error. For Model R, false-positive cost is $7\times \$40=\$280$, and false-negative cost is $3\times \$180=\$540$. Hence $C_R=\$280+\$540=\$820$, giving a per-case cost of $\$820/500=\$1.64$.

For Model S, false-positive cost is $3\times \$40=\$120$, and false-negative cost is $5\times \$180=\$900$. Hence $C_S=\$120+\$900=\$1{,}020$, giving a per-case cost of $\$1{,}020/500=\$2.04$. Model R has 10 total errors and Model S has 8, yet R's total cost is $200 lower, or $0.40 lower per case. Under the stated assumptions, R has lower error-related cost.

Check. The difference is $\$1{,}020-\$820=\$200$, and $\$200/500=\$0.40$ per case, consistent with the difference between the two per-case costs. Common mistake avoided: omitting a relevant consequence from the false-positive cost or adding it twice.

Defining Success Before Comparing Models

A cost-based criterion is useful only when the costs correspond to the decision being made. The earlier material on linking predictions to actions and outcomes emphasizes that a prediction has value through what follows from it. A false positive that triggers no action may have little consequence; a false positive that blocks a customer or consumes scarce expert review capacity may have a material one. Likewise, a false negative is costly only to the extent that the missed case leads to an adverse outcome that could have been prevented or reduced.

Before using unequal costs to define success, specify:

The error definitions: identify the positive class and what counts as a false positive or false negative.
The consequence being valued: state whether each cost is direct expense, expected loss, staff effort converted to cost, or another defined quantity.
The comparison unit: use the same cases, exposure, or operational volume for the models being compared.
The scope of the criterion: clarify whether costs cover only classification errors or also implementation and other operational expenses.
The source and uncertainty of the costs: explain whether they are measured, estimated, or based on a policy judgment.

A model comparison is conditional on these choices. If decision-makers disagree about the cost of a missed positive, the break-even calculation can show where their preferences change, as in the second example. It does not establish which cost estimate is correct. That estimate needs a defensible basis and, when uncertain, should be tested over a plausible range.

Common Mistakes and Checks

Giving both error types equal weight by default. Equal weight is an assumption, not a neutral fact. Use it only when the consequences support it.
Comparing error counts as though they were costs. A count of one error in each category does not imply equal impact.
Reversing a break-even conclusion. After solving for equality, substitute one value below and one above the break-even point to verify the direction.
Mixing units. Do not add dollars per case to total dollars, or compare a total on one cohort with a per-case amount on another.
Presenting uncertain costs as exact facts. Estimated consequences should be labeled as estimates, with their limitations made clear.
Ignoring consequences outside the chosen definition. A lower error-related cost does not automatically imply lower total operating cost if other costs differ between models.

Key takeaway: Unequal error costs mean classification success should reflect the consequences of false positives and false negatives, not merely the number of errors. State the costs and comparison unit, calculate each model consistently, and check break-even preferences on both sides of any solved value.

Check Your Understanding

For each problem, use only the stated costs and assumptions. Treat costs as additive and correct classifications as having zero incremental cost.

A hypothetical screening tool is evaluated on 300 cases. It makes 6 false positives and 2 false negatives. Each false positive costs $15 and each false negative costs $100. Find its total error-related cost and cost per case. (Answer: $290 total; approximately $0.9667 per case.)
On the same validation cases, Model U has 12 false positives and 3 false negatives, while Model V has 5 false positives and 5 false negatives. A false positive costs $20 and a false negative costs $70. Compare total costs. (Answer: Both cost $450; they tie under the stated criterion.)
Model R has 14 false positives and 2 false negatives; Model S has 5 false positives and 5 false negatives. A false positive costs $10, and the false-negative cost is $c_{\mathrm{FN}}$. Find the break-even value and state the preference on either side. (Answer: Break-even at $30; S is cheaper below $30, R is cheaper above $30, and they tie at $30.)
Two models are evaluated on the same 800 cases. Model X has 9 false positives and 7 false negatives; Model Y has 4 false positives and 9 false negatives. Costs are $25 per false positive and $150 per false negative. Which model is cheaper, and by how much per case? (Answer: X costs $1,275 and Y costs $1,450; X is cheaper by $175 total, or $0.21875 per case.)
Model A has 30 false positives and 2 false negatives; Model B has 10 false positives and 6 false negatives. Each false positive costs $8, and each false negative costs $c_{\mathrm{FN}}$. Find the break-even cost and the lower-cost model when $c_{\mathrm{FN}}=\$25$. (Answer: Break-even at $40; at $25, A costs $290 and B costs $230, so B is cheaper.)

What You'll Learn