Choosing Criteria When Error Costs Are Uncertain

Why Uncertain Error Costs Matter

A model comparison can appear straightforward when the cost of each kind of error is known. In practice, a decision-maker may know that a false negative is more consequential than a false positive without being able to assign either one a precise dollar value. A single assumed cost can then make a model seem preferable for reasons that are not supported by the available information.

The appropriate response is not to ignore costs or to choose whichever metric is easiest to calculate. Instead, state plausible cost ranges, compare candidate models across those ranges, and identify the assumptions under which the preferred model changes. This makes the evaluation more defensible: it shows what is supported by the data and what depends on uncertain judgments.

Definition: An error-cost range is a stated interval of plausible costs for a particular type of prediction error. A decision is robust over the stated ranges if the same candidate has lower cost for every combination of costs in those ranges. A decision is sensitive if the preferred candidate changes for at least two plausible combinations.

The ranges need a reasoned interpretation. They might reflect uncertainty in the financial impact of an error, differences in how a decision is implemented, or uncertainty about how a stakeholder values outcomes. They are not probability distributions unless probabilities over cost values have been specified. In particular, the midpoint of a range is not automatically the most likely cost.

For classification, let $C_{FP}$ be the cost of one false positive and $C_{FN}$ the cost of one false negative. For a fixed evaluation sample, let $FP_j$ and $FN_j$ be the corresponding error counts for candidate $j$. Its total error cost is

L_j = FP_j C_{FP} + FN_j C_{FN}.

This is an evaluation of errors under the specified costs, not a complete measure of every benefit or operational consequence. If costs differ by case, the costs should be applied at the case level rather than treating all errors of a type as identical.

Compare Candidates Across Cost Ranges

A useful first step is to compare two candidates algebraically. The difference in their costs is a function of the uncertain costs. Setting that difference equal to zero gives a break-even condition: the cost combinations at which the candidates tie. On either side of that condition, one candidate has lower cost.

The justification follows directly from adding the cost of each error. If candidate A has $FP_A$ false positives and $FN_A$ false negatives, its total cost is $FP_A C_{FP}+FN_A C_{FN}$. Subtracting candidate B's corresponding total gives

L_A-L_B = (FP_A-FP_B)C_{FP} + (FN_A-FN_B)C_{FN}.

If this difference is negative, A has lower cost; if positive, B has lower cost. Setting it to zero identifies the break-even combinations. The result does not require assigning probabilities to the costs: it is a comparison conditional on each possible cost combination.

Worked Example: A Classifier with Uncertain Error Costs

A hypothetical screening program evaluates two classifiers on the same 1,000 cases. Classifier A produces 40 false positives and 20 false negatives; classifier B produces 70 false positives and 8 false negatives. Based on the decision context, the cost of a false positive is considered to be between $20 and $50, and the cost of a false negative between $80 and $200. Determine whether one classifier is preferred throughout these ranges.

Method. Write each classifier's total cost using the same error-cost definitions, then compare their difference. This avoids choosing a model from sensitivity or specificity alone when the relative consequences of the errors are uncertain.

For A, $L_A=40C_{FP}+20C_{FN}$. For B, $L_B=70C_{FP}+8C_{FN}$. Subtract:

L_A-L_B=-30C_{FP}+12C_{FN}.

A is preferred when this difference is less than zero: $-30C_{FP}+12C_{FN}<0$, or $C_{FN}/C_{FP}<30/12=2.5$. B is preferred when the ratio is greater than 2.5; they tie when it equals 2.5.

At the lower-cost combination, $C_{FP}=\$20$ and $C_{FN}=\$80$, the ratio is $80/20=4$, so B is preferred. The costs are $L_A=40(20)+20(80)=800+1{,}600=\$2{,}400$ and $L_B=70(20)+8(80)=1{,}400+640=\$2{,}040$.

At $C_{FP}=\$50$ and $C_{FN}=\$80$, the ratio is $80/50=1.6$, so A is preferred: $L_A=40(50)+20(80)=2{,}000+1{,}600=\$3{,}600$, while $L_B=70(50)+8(80)=3{,}500+640=\$4{,}140$. The preferred classifier changes within the plausible ranges, so neither is robustly preferred. The common mistake this comparison avoids is treating one selected cost pair as if it settled the decision for all plausible costs.

Checking the break-even ratio against the ranges is often more informative than testing just one pair of costs. Here $C_{FN}/C_{FP}$ can range from $80/50=1.6$ to $200/20=10$, and this interval crosses 2.5. Therefore, there are plausible cost combinations on both sides of the break-even condition. This establishes sensitivity without assuming a probability distribution for the costs.

Uncertainty in Numeric-Error Costs

For numeric predictions, the same principle applies when stakeholders place different values on different error sizes. Suppose a hypothetical evaluation explicitly assigns a cost of $a$ dollars per unit of absolute error and $b$ dollars per squared unit of error. For a model with errors $e_1,\ldots,e_n$, define the illustrative total cost as

L = a\sum_{i=1}^{n}|e_i| + b\sum_{i=1}^{n}e_i^2.

This is a specified cost function, not a universal formula for prediction loss. The coefficients have different units: $a$ is dollars per unit of error, while $b$ is dollars per squared unit of error. They must be chosen or bounded in a way that gives the two terms a meaningful interpretation. The formula is not appropriate merely because both error summaries are available; it applies only when this combined cost function represents the decision being evaluated.

Worked Example: Choosing Between Numeric Predictions Across Cost Ranges

A hypothetical forecasting team compares two models on three cases. Model A has errors $0, 0, 5$; Model B has errors $2, 2, 2$. The decision-maker uses the stated cost function $L=a\sum |e_i|+b\sum e_i^2$, with $a$ plausibly between $8 and $12 per unit and $b$ between $0.40 and $1.00 per squared unit. Determine whether the preference is robust.

Method. Calculate both error summaries for each model, derive the cost difference, and examine its sign across the coefficient ranges. This avoids assuming that a smaller absolute-error total must decide a comparison when squared error also has an explicit cost.

For A, the absolute-error total is $|0|+|0|+|5|=5$, and the squared-error total is $0^2+0^2+5^2=25$. For B, the totals are $|2|+|2|+|2|=6$ and $2^2+2^2+2^2=12$. Thus, $L_A=5a+25b$, $L_B=6a+12b$, and

$$ L_A-L_B=-a+13b. $$

A is preferred if $-a+13b<0$, equivalently $b/a<1/13$. At $a=\$12$ and $b=\$0.40$, $L_A=5(12)+25(0.40)=60+10=\$70$, while $L_B=6(12)+12(0.40)=72+4.80=\$76.80$; A is preferred.

At $a=\$8$ and $b=\$1.00$, $L_A=5(8)+25(1)=40+25=\$65$, while $L_B=6(8)+12(1)=48+12=\$60$; B is preferred. The break-even ratio $1/13$, approximately 0.0769, lies within the plausible ratio range: its minimum is $0.40/12$, approximately 0.0333, and its maximum is $1/8=0.125$. The preference is therefore sensitive. The common mistake avoided here is to treat the terms as interchangeable without defining their costs and units.

Use Ranges to Identify Robust Choices

A candidate can be robustly preferred even when the exact costs are unknown. For a total cost that is linear in nonnegative error costs, its minimum and maximum over a rectangular range of costs occur at range endpoints. This follows because increasing a cost cannot decrease the total when its associated error count is nonnegative. Evaluating relevant endpoints can therefore reveal whether one candidate remains cheaper throughout the specified ranges.

The next example compares three hypothetical classifiers. It also illustrates why an arithmetic check matters: each error count must be multiplied by its corresponding cost, and the minimum and maximum calculations must use the appropriate endpoints.

Worked Example: Testing Three Classifiers Across Cost Bounds

A hypothetical claims review evaluates three classifiers on the same cases. A has 20 false positives and 20 false negatives; B has 45 false positives and 10 false negatives; C has 10 false positives and 35 false negatives. A false positive is assigned a cost between $30 and $50, and a false negative between $60 and $90. Find each candidate's minimum and maximum total error cost and assess whether any candidate is robustly preferred.

Method. Since each error count is nonnegative, the minimum for each classifier uses both minimum costs and its maximum uses both maximum costs. Then compare cost intervals and, where they overlap, use the pairwise cost difference to check whether the preference can change.

For A, the minimum is $20(30)+20(60)=600+1{,}200=\$1{,}800$, and the maximum is $20(50)+20(90)=1{,}000+1{,}800=\$2{,}800$.

For B, the minimum is $45(30)+10(60)=1{,}350+600=\$1{,}950$, and the maximum is $45(50)+10(90)=2{,}250+900=\$3{,}150$.

For C, the minimum is $10(30)+35(60)=300+2{,}100=\$2{,}400$, and the maximum is $10(50)+35(90)=500+3{,}150=\$3{,}650$. In particular, the minimum is $2,400, not $3,000.

These intervals overlap, so the bounds alone do not establish one winner. Compare A and B directly: $L_B-L_A=25C_{FP}-10C_{FN}$. At $C_{FP}=\$30$, $C_{FN}=\$90$, this is $750-900=-\$150$, so B costs $150 less than A. At $C_{FP}=\$50$, $C_{FN}=\$60$, it is $1{,}250-600=\$650$, so A costs $650 less than B. The preferred choice changes, so neither A nor B is robustly preferred over the full ranges.

C is more expensive than A throughout these ranges. In fact, $L_C-L_A=-10C_{FP}+15C_{FN}$. Its smallest value over the bounds is $-10(50)+15(60)=-500+900=\$400$, which is positive. Thus C costs at least $400 more than A for every allowed cost combination. A dominates C on this specified cost comparison, but A is not robustly best overall because B can be cheaper. The common mistake avoided is to infer a robust winner just because one candidate has the lowest minimum cost.

How to Present a Cost-Sensitive Conclusion

A careful conclusion distinguishes findings that hold across the ranges from findings that depend on uncertain assumptions. It should state the evaluated error types, the range used for each cost, the comparison criterion, and whether the preferred candidate changes. If the choice is sensitive, give the break-even condition or an example of cost combinations on either side. Do not describe a scenario as robust unless the comparison supports that claim over the full stated ranges.

Ranges also have limits. A range that is too narrow can conceal important uncertainty, while a range with no defensible basis can create the appearance of rigor without adding useful evidence. If the ranges are correlated—for example, if a high false-positive cost is plausible only when the false-negative cost is low—then not every endpoint combination may be plausible. The sensitivity analysis should use the combinations that the decision context actually permits, rather than silently assuming independence.

Finally, a cost comparison does not by itself show that costs have been measured accurately, that the evaluation cases represent future use, or that all relevant consequences have been included. It answers a more limited question: given the specified errors and cost assumptions, which candidate has lower calculated cost? That distinction is essential when communicating a result to decision-makers.

Key takeaway: When error costs are uncertain, use plausible ranges and identify the conditions under which candidate rankings change. Report a robust preference only when it holds throughout the relevant ranges; otherwise, make the sensitivity explicit.

Check Your Understanding

For each question, compare candidates using the stated costs or ranges. Answers are provided so you can check the calculation.

Two classifiers have these errors: A has 12 false positives and 8 false negatives; B has 20 false positives and 3 false negatives. If a false positive costs $25 and a false negative costs $70, which classifier has lower total cost? (Answer: B, with $710 versus A's $860.)
Classifier A has 18 false positives and 6 false negatives; classifier B has 10 false positives and 10 false negatives. At what value of $C_{FN}/C_{FP}$ do they tie, and does a plausible ratio range from 1.5 to 3 imply a robust preference? (Answer: They tie at 2; no, the preference changes across the range.)
Two numeric models have absolute-error and squared-error totals of 7 and 25 for A, and 8 and 18 for B. With cost $L=a\sum|e_i|+b\sum e_i^2$, $a=\$10$ and $b=\$0.50$, which has lower cost? (Answer: A, with $82.50 versus B's $89.00.)
False-positive cost ranges from $20 to $40 and false-negative cost from $50 to $80. Candidate A has 15 false positives and 10 false negatives; B has 5 false positives and 20 false negatives. Is A robustly preferred on total error cost? (Answer: Yes; $L_A-L_B=10C_{FP}-10C_{FN}$, whose maximum is $10(40)-10(50)=-\$100$, so A is always cheaper.)

What You'll Learn