When a Good Metric Stops Answering the Right Question
A model can keep producing predictions of similar quality while the decision around those predictions becomes less successful. For example, a false alarm may become more expensive after an investigation process changes, or a missed event may matter less after the business adopts another safeguard. In either case, a metric calculated the same way as before may no longer represent the consequences that now matter.
The key distinction is between predictive performance and decision value. Predictive performance describes how predictions compare with observed outcomes. Decision value describes whether acting on those predictions improves the outcome of interest, given current costs, available capacity, and priorities. A performance measure can remain useful for monitoring predictions while being inadequate for choosing an operational policy. These distinctions are aligned with the SOA Exam PA (Predictive Analytics) syllabus.
From Prediction Errors to Decision Costs
For a binary outcome, call an event positive and its absence negative. A positive prediction when the event does not occur is a false positive (FP); a negative prediction when the event does occur is a false negative (FN). A true positive (TP) and a true negative (TN) are correct classifications. Accuracy is the fraction of all cases classified correctly:
Accuracy treats each error as one error. It does not say whether one error type causes larger costs, whether the cost per error has changed, or whether an available action can prevent or reduce a loss. If the business objective is to minimize the costs of false positives and false negatives, define \(C_{\mathrm{FP}}\) and \(C_{\mathrm{FN}}\) as the costs of those errors. For a set of cases with observed outcomes, total empirical error cost is:
This calculation assumes that the two stated costs are applicable to every false positive and false negative in the evaluation set, that outcomes are known, and that other costs are either negligible or deliberately excluded. If costs vary by case, an average cost applied to every case may be misleading; case-specific costs should be used when available. The calculation also describes observed costs on the evaluation data, not a guarantee of future costs.
Worked Example: Accuracy and Error Costs Choose Different Models
A hypothetical claims operation evaluates two flagging models on the same 200 claims. The observed results are:
| Model | TP | FN | FP | TN |
|---|---|---|---|---|
| A | 16 | 4 | 24 | 156 |
| B | 14 | 6 | 10 | 170 |
The current estimated cost is $50 per false positive and $500 per false negative. Compare accuracy and total error cost, then determine which model has lower cost.
Method. Calculate each model's accuracy from its correct classifications, then calculate its weighted error cost using the current cost assumptions.
For Model A, the total is \(16+4+24+156=200\), and the number correct is \(16+156=172\). Thus its accuracy is \(172/200=0.86\), or 86%. Its total error cost is \(24($50)+4($500)=$1,200+$2,000=$3,200\).
For Model B, the total is \(14+6+10+170=200\), and the number correct is \(14+170=184\). Thus its accuracy is \(184/200=0.92\), or 92%. Its total error cost is \(10($50)+6($500)=$500+$3,000=$3,500\).
Model B has higher accuracy, but Model A has lower error cost by \($3,500-$3,200=$300\). Under the stated cost objective, Model A is preferred. As a check, Model A has more errors overall (28 versus 16), but most of the additional errors are false positives, which are relatively inexpensive under these assumptions.
Common mistake avoided. Selecting Model B solely because its accuracy is higher ignores the fact that a false negative costs ten times as much as a false positive in this scenario.
Why Changing Costs Can Change the Best Choice
The previous comparison depends on its cost assumptions. If a new process makes false-positive investigations more expensive, the preferred model can change even though neither model's predictions have changed. Recalculate decision costs whenever the relevant business consequences change; do not assume that a historical ranking still applies.
Worked Example: Rechecking a Model Choice After a Cost Change
Use the same hypothetical model results as above. A change in the review process increases the false-positive cost from $50 to $200 per case; the false-negative cost remains $500. Which model now has lower total error cost?
Method. Hold the observed error counts fixed and apply the revised cost assumptions to each model.
For Model A, the revised total is \(24($200)+4($500)=$4,800+$2,000=$6,800\).
For Model B, the revised total is \(10($200)+6($500)=$2,000+$3,000=$5,000\).
Model B now has lower cost by \($6,800-$5,000=$1,800\). Its higher accuracy did not change; what changed was the decision objective represented by the cost calculation. The arithmetic also checks against the direction of the change: Model A has 14 more false positives than Model B, so raising the false-positive cost disadvantages Model A.
Common mistake avoided. Reusing Model A's earlier ranking because it was preferred under the old cost schedule would carry forward a decision that no longer reflects the stated business conditions.
A Cost-Based Decision Threshold
A threshold turns a predicted probability into a binary action. Its appropriate value depends on what the action costs and what errors it creates. The following result applies to a simple setting: each case has a known predicted probability \(p\) of a positive event; the decision is either to classify the case as positive or negative; correct classifications have zero cost relative to this comparison; and the only error costs are fixed \(C_{\mathrm{FP}}>0\) and \(C_{\mathrm{FN}}>0\).
To justify the threshold, compare the expected costs of the two classifications. If the classification is positive, it is wrong only when the event does not occur, with probability \(1-p\), so its expected cost is \(C_{\mathrm{FP}}(1-p)\). If the classification is negative, it is wrong only when the event occurs, with probability \(p\), so its expected cost is \(C_{\mathrm{FN}}p\). Choose positive when its expected cost is smaller:
This threshold is not universal. It does not directly apply when correct decisions have important costs or benefits, costs vary by case, the action has an implementation cost, or the business must choose among more than two actions. Those situations require comparing the expected consequences of the available actions, not inserting a familiar threshold without checking its assumptions.
Worked Example: A Cost Change Moves the Decision Threshold
A hypothetical screening team assigns a positive classification when it expects to take an action that prevents a costly event. Under the simple two-error-cost model, the estimated cost of a false positive is $240 and the cost of a false negative is $960. A case has a predicted event probability of \(p=0.35\). Determine the threshold and classification. Then repeat if the false-positive cost rises to $600 while the false-negative cost stays at $960.
Method. Derive the threshold from the two expected error costs, then compare \(p\) with it. As a check, calculate both expected costs for the case.
Initially, the threshold is \(240/(240+960)=240/1,200=0.20\). Since \(0.35>0.20\), classify the case as positive. The expected cost of positive classification is \((1-0.35)($240)=0.65($240)=$156\). The expected cost of negative classification is \(0.35($960)=$336\). Positive classification is lower by $180, confirming the threshold decision.
After the change, the threshold is \(600/(600+960)=600/1,560=0.3846\) to four decimal places. Since \(0.35<0.3846\), classify the case as negative. The positive-classification expected cost is \(0.65($600)=$390\), while the negative-classification expected cost remains \(0.35($960)=$336\). Negative classification is lower by $54, which checks the revised choice.
Common mistake avoided. Using \(C_{\mathrm{FN}}/(C_{\mathrm{FP}}+C_{\mathrm{FN}})\) as the positive threshold reverses the cost relationship. The derivation shows that the false-positive cost belongs in the numerator because it is the cost incurred by choosing positive when the event is absent.
Capacity Constraints and Operational Priorities
A classification metric may also be a poor guide when the operation cannot act on every case. Suppose a team can investigate only a fixed number of cases in a period. The relevant question is no longer just which cases exceed a common classification threshold. It is which available actions produce the greatest expected benefit within the capacity limit.
For a simple prioritization calculation, suppose reviewing case \(j\) costs \(K_j\), and a successful review prevents a loss of \(L_j\) if the event occurs. Let \(p_j\) be the predicted event probability and \(q_j\) the probability that review successfully prevents that loss, conditional on the event. Under these stated assumptions, the expected net benefit of review is:
The formula follows because the loss is prevented only if the event occurs (probability \(p_j\)) and the review prevents it (conditional probability \(q_j\)); the expected prevented loss is therefore \(p_jq_jL_j\). Subtract the review cost paid for the case. If capacity permits only a specified number of reviews and cases are otherwise independent for this calculation, prioritize the permitted cases with the greatest positive expected net benefits. If reviews interact, or capacity and costs have more complicated constraints, this simple ranking may not be sufficient.
Worked Example: Ranking Actions Under a Review Limit
A hypothetical risk team can review two cases this week. Its estimated probability of an event, event loss, review effectiveness, and review cost are:
| Case | Event probability | Loss if event occurs | Review effectiveness | Review cost |
|---|---|---|---|---|
| A | 0.50 | $400 | 0.80 | $40 |
| B | 0.30 | $2,000 | 0.80 | $100 |
| C | 0.25 | $1,500 | 0.80 | $50 |
| D | 0.20 | $500 | 0.80 | $20 |
Assume review effectiveness is the probability that review prevents the loss if the event occurs, and assume the listed review cost is incurred for every reviewed case. Choose two cases to review using expected net benefit.
Method. Calculate \(p_jq_jL_j-K_j\) for each case and rank the results. A higher event probability alone is not the decision rule because severity and review cost also differ.
For A, expected prevented loss is \(0.50(0.80)($400)=$160\), so net benefit is \($160-$40=$120\).
For B, expected prevented loss is \(0.30(0.80)($2,000)=$480\), so net benefit is \($480-$100=$380\).
For C, expected prevented loss is \(0.25(0.80)($1,500)=$300\), so net benefit is \($300-$50=$250\).
For D, expected prevented loss is \(0.20(0.80)($500)=$80\), so net benefit is \($80-$20=$60\).
The ranking is B ($380), C ($250), A ($120), and D ($60). Review B and C, the top two positive net benefits. The result checks against the capacity limit: exactly two cases are selected. Reviewing A and B merely because they have the two highest event probabilities would overlook the larger expected value of reviewing C.
Common mistake avoided. Prioritizing by probability alone ignores both the financial severity of the event and the cost of the intervention. This calculation is only as useful as its assumptions: if the estimated probabilities, losses, or review effectiveness do not reflect current operations, the ranking can also become stale.
Maintaining a Useful Metric
A metric should be interpreted in the context of the decision it is meant to support. When the objective changes, revisit the metric and its assumptions rather than assuming that a familiar measure remains an adequate summary. In practice, check whether:
- The costs or consequences of false positives and false negatives have changed.
- The available action, its effectiveness, or its operating cost has changed.
- The number of cases the operation can handle has changed.
- The organization now gives greater weight to a different outcome or constraint.
- The assumptions used to calculate costs still describe the current population and process.
A metric can still be informative without being the final decision rule. For example, accuracy can describe the fraction of correct classifications, while cost calculations address the consequences assigned to different errors. Clear reporting states which question each measure answers and which assumptions connect it to operational success. Where costs are uncertain, report how the decision changes under plausible alternatives instead of presenting one cost-based ranking as certain.
Check Your Understanding
For each problem, identify the relevant objective before calculating. Assume all stated costs and probabilities apply to the cases described.
- Two models are assessed on 300 cases. Model R has TP = 24, FN = 6, FP = 36, TN = 234. Model S has TP = 21, FN = 9, FP = 15, TN = 255. If a false positive costs $40 and a false negative costs $200, calculate both total error costs and identify the lower-cost model. (Answer: R costs $2,640; S costs $2,400; Model S has the lower cost by $240.)
- Under the two-error-cost threshold assumptions, a false positive costs $180 and a false negative costs $720. A case has predicted event probability 0.25. Find the threshold and choose a classification; verify by comparing the two expected costs. (Answer: Threshold 0.20; classify positive; expected costs are $135 for positive and $180 for negative.)
- A team can review two cases. Case A has event probability 0.40, loss $500, review effectiveness 0.50, and review cost $40. Case B has probability 0.25, loss $1,600, effectiveness 0.75, and cost $80. Case C has probability 0.30, loss $900, effectiveness 0.80, and cost $50. Calculate each expected net benefit and choose two cases. (Answer: A $60, B $220, C $166; review B and C.)
- Model A has 15 false positives and 5 false negatives; Model B has 5 false positives and 10 false negatives. The false-negative cost is fixed at $300, while the false-positive cost is \(c\) dollars. Find the cost \(c\) at which the models tie, then identify the lower-cost model when \(c=$250\). (Answer: They tie at \(c=$150\); at \(c=$250\), Model B costs $4,250 and is lower than Model A at $5,250.)