Why Metric-Selection Traps Matter
A model can appear successful under a reported score and still fail the task it was built to support. This often happens not because the score was calculated incorrectly, but because the score answers the wrong question. A high accuracy score may conceal missed rare events; a probability model may be evaluated at a cutoff different from the one used in practice; or two models may be declared comparable using scores that represent different objectives.
The remedy is to connect the evaluation criterion to the intended use of the prediction. State what outcome is being predicted, what action follows from a prediction, which errors matter, and how performance will be summarized. This is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Trap 1: Optimizing Accuracy for a Rare Outcome
For a binary outcome, accuracy is the proportion of evaluated cases classified correctly. It is easy to calculate and interpret, but it gives equal weight to each case. When one class is much more common than the other, a model that always predicts the common class can achieve high accuracy while detecting none of the rare class.
Let \(p\) be the proportion of actual positive cases, sensitivity the proportion of actual positives predicted positive, and specificity the proportion of actual negatives predicted negative. On the same evaluation population, accuracy is the proportion of positives correctly classified plus the proportion of negatives correctly classified. Therefore,
This identity shows why accuracy can be dominated by the more prevalent class. If every case is predicted negative, sensitivity is zero and specificity is one, so accuracy is \(1-p\). When \(p\) is small, that value can be close to one.
Worked Example: High Accuracy With No Rare-Event Detection
A hypothetical screening model is evaluated on 1,000 records. There are 20 actual positive cases and 980 actual negative cases. Model A predicts every record negative. Model B identifies 12 of the positive cases, misses 8, and incorrectly labels 38 negative cases as positive. Compare their accuracy and sensitivity. The evaluation set is the full 1,000 records; no population weighting is applied.
Method. Calculate correct classifications divided by 1,000 for accuracy, and correctly identified positives divided by 20 for sensitivity. This directly checks whether accuracy alone reflects the rare-event objective.
For Model A, all 980 actual negatives are correct and all 20 positives are missed:
For Model B, 12 positives are correctly identified. Of the 980 negatives, 38 are incorrectly labeled positive, leaving \(980-38=942\) correctly labeled negative:
Model A has higher accuracy, but it detects none of the positive cases. Model B has lower accuracy and detects 60% of them. Whether Model B is preferable depends on the purpose and costs of the errors; the calculations alone do not establish that. Common mistake: selecting Model A solely because \(0.980>0.954\). The correction is to report accuracy alongside a measure that represents positive-case detection when detecting positives is part of the goal.
The formula above follows by splitting the evaluation records into actual positives and actual negatives. A fraction \(p\) of the records are positive, and a fraction equal to sensitivity of those are correct; their contribution to overall accuracy is \(p(\text{Sensitivity})\). The remaining fraction \(1-p\) are negative, and a fraction equal to specificity of those are correct; their contribution is \((1-p)(\text{Specificity})\). Adding these disjoint contributions gives the stated accuracy. The identity assumes that the measures refer to the same binary outcome, evaluation population, and classification rule.
Trap 2: Ignoring the Decision Cutoff
Many classification models produce a score or estimated probability rather than a class label. To make a positive/negative decision, an operating cutoff is applied: cases at or above the cutoff are labeled positive, and cases below it are labeled negative. Changing the cutoff changes the resulting classifications, so it can change accuracy, sensitivity, specificity, and other classification measures even when the underlying scores do not change.
Consequently, a metric calculated at an arbitrary cutoff may not describe the decisions the organization will actually make. The cutoff should reflect the planned decision rule, such as a documented review policy. A score for ranking cases and a measure of decisions made at a particular cutoff answer different questions.
Worked Example: Reporting Performance at the Intended Cutoff
A hypothetical claims triage team evaluates predicted probabilities for 10 claims. Their outcomes and scores are: positive at 0.90, negative at 0.80, positive at 0.70, negative at 0.60, positive at 0.45, negative at 0.35, positive at 0.25, negative at 0.20, positive at 0.10, and negative at 0.05. The operational rule is to flag a claim as positive when its score is at least 0.25. Compare the confusion counts and accuracy at cutoffs 0.50 and 0.25.
Method. At each cutoff, classify scores at or above it as positive, then count true positives, false positives, false negatives, and true negatives. The correction to the common cutoff trap is to report the result at the stated operational cutoff, not only at a familiar default.
At cutoff 0.50, the scores 0.90, 0.80, 0.70, and 0.60 are classified positive. Among these, two are actual positives and two are actual negatives. The five actual positives include three missed cases, and the remaining three actual negatives are correctly classified. Thus \(TP=2\), \(FP=2\), \(FN=3\), and \(TN=3\), giving:
At cutoff 0.25, the scores 0.90, 0.80, 0.70, 0.60, 0.45, 0.35, and 0.25 are classified positive. Four are actual positives and three are actual negatives. One positive is missed, and two negatives are correctly classified. Thus \(TP=4\), \(FP=3\), \(FN=1\), and \(TN=2\), giving:
At the operational cutoff, sensitivity is 0.800 and accuracy is 0.600. The cutoff of 0.50 gives different results and does not describe the stated operating rule. Common mistake: calculating at 0.50 by habit and presenting that result as the team’s expected classification performance. The correction is to state the cutoff with the metric and use the cutoff that corresponds to the decision.
Trap 3: Comparing Unlike Objectives
A comparison is meaningful only when the candidate models are evaluated against a common criterion on comparable evaluation data. Different metrics can reward different prediction behavior. For numeric predictions, mean absolute error (MAE) averages absolute errors, while root mean squared error (RMSE) squares errors before averaging and then takes a square root. Both are expressed in the units of the outcome, but RMSE gives greater influence to large errors.
A lower value is preferred for both MAE and RMSE when smaller prediction errors are the stated objective. That shared direction does not make their numerical values interchangeable. In particular, comparing one model’s MAE directly with another model’s RMSE does not establish which model performs better.
Worked Example: A Large Error Changes the Ranking
A hypothetical insurer compares two models on the same five evaluation cases. For each case, the observed claim amount is $10. Model A predicts $10, $10, $10, $10, and $14; Model B predicts $11 for all five cases. Compare both models using MAE and RMSE. Treat the five cases as equally weighted, and report dollar results to the nearest cent.
Method. First calculate the five absolute errors for each model. Then calculate their mean for MAE and the square root of their mean squared value for RMSE. This avoids the trap of treating the two criteria as if they were the same objective.
Model A’s absolute errors are \(0,0,0,0,4\) dollars. Therefore:
Model B’s absolute errors are \(1,1,1,1,1\) dollars. Therefore:
Model A has the lower MAE, while Model B has the lower RMSE. The arithmetic checks: Model A’s large error raises its squared-error score substantially, even though its average absolute error is smaller. If the stated objective is minimizing MAE, Model A is preferred; if it is minimizing RMSE, Model B is preferred. Common mistake: concluding that Model A is better because \(\$0.80\) is less than Model B’s RMSE of \(\$1.00\). That compares unlike criteria. Compare both models using the same prespecified measure.
Trap 4: Treating One Score as Universally Best
No single score is best for every task. A criterion reflects what errors matter, what actions follow predictions, and which population is being evaluated. Accuracy may be appropriate when correct and incorrect classifications have similar importance and class balance is not a concern. It may be inadequate when missed positives are especially consequential. Likewise, an error measure that penalizes large numeric errors heavily is appropriate only when that emphasis matches the use of the prediction.
A decision-based evaluation can make the consequences explicit. Suppose false negatives have loss 8 units and false positives have loss 1 unit, while correct decisions have loss 0. These losses are hypothetical and must be justified for the application; they are not universal values. For a set of decisions, total loss is the sum of the loss attached to each error type. Accuracy and this loss measure can rank models differently.
Worked Example: Accuracy and Decision Loss Disagree
A hypothetical review operation evaluates two policies on the same 100 records. Policy A has 8 true positives, 10 false positives, 2 false negatives, and 80 true negatives. Policy B has 5 true positives, 3 false positives, 5 false negatives, and 87 true negatives. The stated hypothetical loss is 8 units per false negative and 1 unit per false positive; correct classifications have zero loss. Calculate accuracy and average loss per record for each policy.
Method. Calculate accuracy from true positives and true negatives, and calculate decision loss from the specified costs. This exposes the common mistake of assuming that the model with higher accuracy must also be preferable for the decision.
For Policy A, the counts sum to \(8+10+2+80=100\). Its accuracy is \((8+80)/100=0.880\). Its total loss is \(2(8)+10(1)=16+10=26\), so average loss is \(26/100=0.26\) units per record.
For Policy B, the counts sum to \(5+3+5+87=100\). Its accuracy is \((5+87)/100=0.920\). Its total loss is \(5(8)+3(1)=40+3=43\), so average loss is \(43/100=0.43\) units per record.
Accuracy favors Policy B, but the stated decision loss favors Policy A. Under these assumed costs, Policy A has the lower average loss. The result would need reconsideration if the costs or decision requirements changed. Common mistake: calling Policy B better based only on its 0.920 accuracy. The correction is to use the criterion tied to the stated consequence of errors, while reporting other relevant measures as context.
A Practical Check Before Reporting a Metric
Before accepting a reported score, trace it back to the decision it is meant to support. A compact audit helps catch errors without turning metric selection into a search for a universally superior measure.
Confirm which outcome or quantity is predicted and which records the score summarizes. A score has limited meaning without this context.
For classification, state the cutoff used to turn scores into decisions. Do not assume that a conventional cutoff matches the actual operating policy.
Check whether the measure reflects rare outcomes, large errors, unequal consequences, or another stated requirement. Avoid relying on a convenient score that does not address the goal.
Use the same evaluation data, metric definitions, and classification cutoff where applicable. Explain the trade-off if different valid criteria favor different models.
A defensible evaluation plan does not need to report every possible metric. It does need to state the main criterion and explain why it fits the task. Additional measures can reveal trade-offs or weaknesses, but they should not be presented as interchangeable evidence of the same objective. These checks help prevent metric-selection errors before a model comparison is communicated.
Check Your Understanding
For each problem, use the stated evaluation data and assumptions. Answers are included so you can check your calculations.
- A binary classifier is evaluated on a population with 3% positives. Its sensitivity is 0.700 and its specificity is 0.900. Using the accuracy identity, calculate accuracy. What does the result say about interpreting accuracy alone? (Answer: Accuracy is \(0.03(0.700)+0.97(0.900)=0.894\), or 0.8940; accuracy alone does not state sensitivity, which is 0.700 here.)
- A screening set has 200 records, including 20 positives and 180 negatives. A rule predicts every record negative. Calculate accuracy and sensitivity, and identify the metric that exposes its failure to find positive cases. (Answer: Accuracy \(=180/200=0.900\); sensitivity \(=0/20=0\); sensitivity exposes the missed positives.)
- On 100 records, a classifier has \(TP=18\), \(FP=12\), \(FN=6\), and \(TN=64\). Calculate accuracy, sensitivity, and precision, where precision is the proportion of predicted positives that are actual positives. (Answer: Accuracy \(=(18+64)/100=0.820\); sensitivity \(=18/(18+6)=0.750\); precision \(=18/(18+12)=0.600\).)
- For three numeric predictions, a model’s errors are \(0,2,2\). Calculate MAE and RMSE, rounding to three decimal places. (Answer: MAE \(=(0+2+2)/3=1.333\); RMSE \(=\sqrt{(0^2+2^2+2^2)/3}=\sqrt{8/3}\approx1.633\), in the units of the predicted quantity.)
- Two policies are evaluated on 100 cases. Policy X has 6 true positives, 5 false positives, 4 false negatives, and 85 true negatives. Policy Y has 9 true positives, 20 false positives, 1 false negative, and 70 true negatives. If a false negative costs 10 units, a false positive costs 1 unit, and correct decisions cost zero, which policy has higher accuracy, and which has lower average loss? (Answer: X accuracy \(=(6+85)/100=0.910\), loss \(=(4(10)+5)/100=0.450\) units per record. Y accuracy \(=(9+70)/100=0.790\), loss \(=(1(10)+20)/100=0.300\) units per record. X has higher accuracy; Y has lower average loss under the stated costs.)