Expected Error Cost From a Confusion Matrix

Why Count Errors by Their Consequences?

A classifier can make two different kinds of mistakes: it can flag a case that is actually negative, or fail to flag a case that is actually positive. In many practical settings, these errors do not have the same consequences. For example, in a hypothetical screening program, a false alarm may require a follow-up review, while a missed positive case may lead to a much larger loss. Counting errors alone does not capture that difference.

A confusion matrix supplies the numbers of each kind of outcome at a specified classification threshold. If the costs of false positives and false negatives are also stated, those counts can be converted into a total decision loss and an average loss per evaluated case. This cost-based comparison is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: A false positive (FP) is a case predicted positive when its actual class is negative. A false negative (FN) is a case predicted negative when its actual class is positive. Let $C_{\mathrm{FP}}$ be the cost of one false positive and $C_{\mathrm{FN}}$ the cost of one false negative. Under the assumption that correct classifications have zero cost and each error has a fixed, additive cost, the total error cost is $FP\,C_{\mathrm{FP}}+FN\,C_{\mathrm{FN}}$. The average error cost per evaluated case is the total error cost divided by the number of evaluated cases.

Read the Confusion Matrix Before Calculating

Use the actual class to identify which cases are false positives and false negatives. The following layout makes the distinction explicit.

	Predicted positive	Predicted negative
Actually positive	True positive (TP)	False negative (FN)
Actually negative	False positive (FP)	True negative (TN)

The total number of evaluated cases is $N=TP+FN+FP+TN$. Only FP and FN contribute to the error-cost formula under the stated assumptions. True positives and true negatives contribute zero direct classification cost. This does not mean correct classifications can never have operational costs; it means those costs are outside this particular calculation unless the problem specifies them.

The total error cost answers, “What is the combined cost for this evaluated group?” The average error cost per case answers, “How much cost is associated with one evaluated case on average?” The denominator for the average is all evaluated cases, not just the cases misclassified.

Formula and Justification

Suppose each false positive incurs the same cost $C_{\mathrm{FP}}$, and each false negative incurs the same cost $C_{\mathrm{FN}}$. Let $I_j^{\mathrm{FP}}$ equal 1 if case $j$ is a false positive and 0 otherwise; define $I_j^{\mathrm{FN}}$ in the same way. The cost assigned to case $j$ is $C_{\mathrm{FP}}I_j^{\mathrm{FP}}+C_{\mathrm{FN}}I_j^{\mathrm{FN}}$. Summing over cases gives the total.

L=\sum_{j=1}^{N}\left(C_{\mathrm{FP}}I_j^{\mathrm{FP}}+C_{\mathrm{FN}}I_j^{\mathrm{FN}}\right) =FP\,C_{\mathrm{FP}}+FN\,C_{\mathrm{FN}}

The equality follows because exactly $FP$ of the false-positive indicators equal 1, exactly $FN$ of the false-negative indicators equal 1, and all other indicators equal 0. Dividing the total by $N$ gives the average cost per evaluated case:

\bar{L}=\frac{L}{N} =\frac{FP\,C_{\mathrm{FP}}+FN\,C_{\mathrm{FN}}} {TP+FN+FP+TN}

When the confusion matrix describes a sample used to estimate performance in a larger population, $L/N$ is an empirical estimate of average cost per case. Interpreting it as a future expected cost depends on the evaluation cases being relevant to the population in which the classifier will be used, and on the error costs being appropriate for that use. A validation-set cost is not automatically a reliable forecast if the population mix or operating conditions change.

Important distinction: Total cost depends on both the per-case error pattern and the number of cases. Average cost per case adjusts for evaluation-set size. For a future group of $M$ similar cases, $M\bar{L}$ is a projected total only if the validation estimate is applicable to that group.

Worked Cost Comparisons

Worked Example: Comparing Two Classifiers on the Same Cases

A hypothetical equipment inspection program evaluates 500 components. A missed defective component costs $180, and an unnecessary inspection of a sound component costs $25. Classifier A has $TP=32$, $FN=8$, $FP=18$, and $TN=442$. Classifier B has $TP=35$, $FN=5$, $FP=30$, and $TN=430$. Calculate the total and average error cost for each classifier, then compare them.

Method. Multiply each error count by its stated cost, add the two cost components, and divide by all 500 evaluated components. The common mistake to avoid is selecting a classifier by looking only at its number of errors.

For Classifier A, the false-positive cost is $18\times\$25=\$450$. The false-negative cost is $8\times\$180=\$1{,}440$. Therefore,

L_A=\$450+\$1{,}440=\$1{,}890, \qquad \bar{L}_A=\frac{\$1{,}890}{500}=\$3.78

For Classifier B, the false-positive cost is $30\times\$25=\$750$, and the false-negative cost is $5\times\$180=\$900$. Therefore,

L_B=\$750+\$900=\$1{,}650, \qquad \bar{L}_B=\frac{\$1{,}650}{500}=\$3.30

Classifier B has more total errors: $30+5=35$, compared with $18+8=26$ for A. Nevertheless, its total cost is lower by $\$1{,}890-\$1{,}650=\$240$, because it makes fewer of the much more costly false-negative errors. On these cases and under these costs, B has the lower decision loss. The arithmetic checks: each matrix sums to 500, and each average multiplied by 500 returns its total.

Worked Example: Comparing Different Evaluation-Set Sizes

Two hypothetical regional teams evaluate the same type of claim-screening classifier, but their validation sets have different sizes. A false positive costs $8 and a false negative costs $70. Team A reports $TP=72$, $FN=48$, $FP=90$, $TN=990$. Team B reports $TP=54$, $FN=36$, $FP=60$, $TN=750$. Calculate each total cost and average cost per case. Which team’s evaluated classifier has the lower observed cost per case?

Method. Calculate each matrix total, then calculate both total and per-case costs. The common mistake is comparing the totals without accounting for the different numbers of cases.

Team A evaluated $72+48+90+990=1{,}200$ cases. Its false-positive cost is $90\times\$8=\$720$, and its false-negative cost is $48\times\$70=\$3{,}360$. Thus,

L_A=\$720+\$3{,}360=\$4{,}080, \qquad \bar{L}_A=\frac{\$4{,}080}{1{,}200}=\$3.40

Team B evaluated $54+36+60+750=900$ cases. Its false-positive cost is $60\times\$8=\$480$, and its false-negative cost is $36\times\$70=\$2{,}520$. Thus,

L_B=\$480+\$2{,}520=\$3{,}000, \qquad \bar{L}_B=\frac{\$3{,}000}{900} =\$3.\overline{3}

Team B has the lower observed average cost per case: approximately $3.33 compared with $3.40. Team B also has the lower total cost, but that total comparison alone is less informative because Team B evaluated 300 fewer cases. The per-case comparison is the relevant one for comparing these observed rates of cost. Rounded to the nearest cent, Team B’s average is $3.33; the displayed repeating decimal shows the unrounded value used in the comparison.

Worked Example: Applying Unequal Costs Carefully

A hypothetical review unit evaluates 240 applications. A false positive leads to an avoidable review costing $45; a false negative leads to an estimated loss of $260. Classifier A produces 12 false positives and 3 false negatives. Classifier B produces 20 false positives and 2 false negatives. Calculate each classifier’s total and average error cost, and identify the lower-cost result.

Method. Apply the cost attached to each error type separately, then divide by the same 240 cases. The common mistake to avoid is treating one false positive as equivalent to one false negative.

For A, false positives cost $12\times\$45=\$540$, and false negatives cost $3\times\$260=\$780$. Hence $L_A=\$540+\$780=\$1{,}320$, and $\bar{L}_A=\$1{,}320/240=\$5.50$ per case.

For B, false positives cost $20\times\$45=\$900$, and false negatives cost $2\times\$260=\$520$. Hence $L_B=\$900+\$520=\$1{,}420$, and $\bar{L}_B=\$1{,}420/240=\$5.9166\ldots$ per case, or $5.92 to the nearest cent.

Classifier A has fewer total errors, $12+3=15$, than B’s $20+2=22$, and it also has lower total cost by $\$100$. The cost result is not established just by comparing error counts: A’s three false negatives alone cost $780. Checking the calculation, both averages use 240—not the number of errors—as their denominator.

Common Mistakes and Practical Checks

The cost calculation is simple, but a small interpretation error can reverse a comparison. Before reporting a result, check the matrix orientation, the costs, and the denominator.

Reversing FP and FN. Locate each count using both its actual class and its predicted class. Do not infer the error type from a row or column label without checking how the matrix is arranged.
Using the same cost for both errors without support. Apply the cost stated for each error type. If the problem gives equal costs, use them; otherwise, do not assume equality.
Dividing by the number of errors. That produces average cost per error, not average cost per evaluated case. The latter uses $N=TP+FN+FP+TN$.
Comparing totals across different portfolio sizes. A larger evaluation set can have a larger total cost even if its per-case performance is better. Report the total for budget impact and the average for a normalized comparison.
Calling a validation result a guaranteed forecast. A projected future total uses the validation average only when the future cases and error costs are sufficiently comparable.

Common Mistake: Averaging Over Errors Instead of Cases

Suppose a matrix has 10 false positives and 5 false negatives, with costs $20 and $100 respectively, among 500 evaluated cases. The total is $10\times\$20+5\times\$100=\$200+\$500=\$700$. A mistaken calculation divides by the 15 errors and reports $\$700/15=\$46.67$ as the average cost per case. That number is average cost per error, not per evaluated case. The correct per-case cost is $\$700/500=\$1.40$. Use the denominator that matches the quantity requested.

Key takeaway: For fixed additive error costs, calculate total loss as $FP\,C_{\mathrm{FP}}+FN\,C_{\mathrm{FN}}$. Divide by all evaluated cases to obtain average cost per case. Compare totals when the question concerns the full group’s cost; compare averages when normalizing across groups of different sizes.

Check Your Understanding

For each problem, identify the relevant error counts, apply the stated costs, and use the requested denominator.

A hypothetical maintenance model is assessed on 400 machines. It produces 16 false positives and 7 false negatives. A false positive costs $30 and a false negative costs $140. What are the total and average error costs per machine? (Answer: Total cost $1,460; average cost $3.65 per machine.)
Two classifiers are evaluated on the same 300 cases. False positives cost $10 and false negatives cost $90. Classifier A has $FP=24$, $FN=4$; Classifier B has $FP=10$, $FN=6$. Calculate both total costs and state which is lower. (Answer: A costs $600; B costs $640; A is lower by $40.)
A team evaluates a classifier on 800 cases and reports $TP=120$, $FN=40$, $FP=80$, and $TN=560$. A false positive costs $15 and a false negative costs $75. Calculate total cost and average cost per case. (Answer: Total cost $4,200; average cost $5.25 per case.)
Group X has 30 false positives and 10 false negatives among 1,000 cases. Group Y has 18 false positives and 8 false negatives among 600 cases. Costs are $12 per false positive and $60 per false negative. Which group has the lower average cost per case? (Answer: X costs $960 total, or $0.96 per case; Y costs $696 total, or $1.16 per case; X is lower per case.)
On the same 500 evaluated cases, Model A has 15 false positives and 9 false negatives, while Model B has 25 false positives and 5 false negatives. False positives cost $20 and false negatives cost $160. Calculate each total cost and determine which model has lower cost. (Answer: A costs $1,740; B costs $1,300; B is lower by $440.)

What You'll Learn