Diagnosing Metric and Threshold Conflicts in a Classification Case

Why Classification Results Can Conflict

Two classifiers can give different answers to “which model is better” without either calculation being wrong. One may rank cases more effectively across possible cutoffs, while another may make fewer mistakes at a particular cutoff. If false positives and false negatives have different consequences, the model with fewer total errors may still have the greater decision cost.

The diagnosis therefore needs to keep three questions separate: how well does each model rank cases, what errors does it make at the stated threshold, and what are those errors worth in the decision context? The examples below use hypothetical validation data. They assume both models are evaluated on the same cases, with the same definition of the positive outcome.

Definition: For a binary outcome, a true positive (TP) is a positive case classified as positive; a false negative (FN) is a positive case classified as negative; a false positive (FP) is a negative case classified as positive; and a true negative (TN) is a negative case classified as negative. A threshold turns a model’s score or predicted probability into a class: classify as positive when the score is at least the threshold.

The confusion matrix records results at one chosen threshold. Common measures derived from it include accuracy, the proportion classified correctly; sensitivity (also called recall), the proportion of actual positives found; specificity, the proportion of actual negatives correctly classified; and precision, the proportion of predicted positives that are actual positives.

\text{Accuracy}=\frac{TP+TN}{TP+FN+FP+TN},\quad \text{Sensitivity}=\frac{TP}{TP+FN},\quad \text{Specificity}=\frac{TN}{TN+FP},\quad \text{Precision}=\frac{TP}{TP+FP}.

AUC summarizes ranking across thresholds rather than performance at one selected threshold. Its empirical interpretation is the fraction of positive-negative case pairs for which the positive case receives the higher score, counting a tied pair as one-half. A higher AUC indicates better pairwise ranking on the evaluated cases; it does not by itself establish lower error cost at the operating threshold.

Definition: If a false positive has cost $C_{FP}$ and a false negative has cost $C_{FN}$, the total cost on a validation set is $FP\,C_{FP}+FN\,C_{FN}$. Cost per case is this total divided by the number of cases. This comparison assumes the stated costs apply to the evaluated cases and that other consequences are either negligible or included in those costs.

Worked Examples: Diagnose the Conflict

Worked Example: Higher AUC but Higher Cost

A hypothetical screening program validates two classifiers on 1,000 cases, of which 100 have the positive outcome. At threshold 0.50, Model A has $TP=70$, $FN=30$, $FP=90$, and $TN=810$. Model B has $TP=60$, $FN=40$, $FP=40$, and $TN=860$. Their validation AUCs are 0.82 and 0.77, respectively. A false positive costs $30 and a false negative costs $120. Compare their ranking, threshold measures, and cost.

Method. First calculate threshold measures from each confusion matrix, then apply the stated error costs. This avoids treating AUC as if it were a cost calculation.

For Model A, accuracy is $(70+810)/1{,}000=0.8800$, sensitivity is $70/100=0.7000$, specificity is $810/900=0.9000$, and precision is $70/(70+90)=70/160=0.4375$.

For Model B, accuracy is $(60+860)/1{,}000=0.9200$, sensitivity is $60/100=0.6000$, specificity is $860/900=0.9556$, and precision is $60/(60+40)=0.6000$.

The cost for Model A is $90($30)+30($120)=$2{,}700+$3{,}600=$6{,}300$, or $6.30 per case. The cost for Model B is $40($30)+40($120)=$1{,}200+$4{,}800=$6{,}000$, or $6.00 per case.

Model A has the higher AUC, and it finds a greater share of positives at this threshold. Model B nevertheless has higher accuracy, precision, and specificity, and its total stated error cost is $300 lower. Common mistake avoided: choosing Model A solely because its AUC is higher. That choice ignores the threshold-specific errors and the unequal costs in this case. The result supports Model B at the stated threshold and costs; it does not prove that Model B has better ranking or will be preferable under different costs.

Worked Example: A Threshold Change Reverses the Cost Comparison

Two hypothetical classifiers are assessed on the same 200 cases, including 40 positive cases. Model C has AUC 0.79 and Model D has AUC 0.83. Their confusion-matrix counts are shown below at two thresholds. A false positive costs $20 and a false negative costs $100. Compare costs at each threshold.

Model and threshold	TP	FN	FP	TN
C, 0.20	32	8	24	136
D, 0.20	30	10	10	150
C, 0.50	25	15	8	152
D, 0.50	24	16	4	156

At threshold 0.20, Model C’s cost is $24($20)+8($100)=$480+$800=$1{,}280$. Model D’s cost is $10($20)+10($100)=$200+$1{,}000=$1{,}200$. D has $80 lower cost at this threshold.

At threshold 0.50, Model C’s cost is $8($20)+15($100)=$160+$1{,}500=$1{,}660$. Model D’s cost is $4($20)+16($100)=$80+$1{,}600=$1{,}680$. C has $20 lower cost at this threshold.

Thus, D’s higher AUC does not make it the lower-cost choice at both thresholds. The threshold comparison itself reverses: D is less costly at 0.20, while C is less costly at 0.50. Raising a threshold can reduce false positives while increasing false negatives, as these counts show. Common mistake avoided: reporting a threshold-free AUC ranking as though it settles the operating-threshold decision. The two thresholds give different costs, and neither comparison establishes the cost at an unreported threshold.

Worked Example: Compare Weighted Cost With Raw Error Count

A hypothetical claims triage team evaluates two models on 500 cases, including 50 cases that genuinely require urgent review. At the chosen threshold, Model E has $TP=40$, $FN=10$, $FP=30$, $TN=420$; Model F has $TP=35$, $FN=15$, $FP=10$, $TN=440$. Their AUCs are 0.86 for E and 0.90 for F. A false positive costs $15 in unnecessary review, while a false negative costs $90 in delayed attention. Compare accuracy, cost, and savings against classifying every case as negative.

Model E has $40+30=70$ predicted positives and $10+420=430$ predicted negatives. Its accuracy is $(40+420)/500=0.9200$, sensitivity is $40/50=0.8000$, and precision is $40/70=0.5714$. Its cost is $30($15)+10($90)=$450+$900=$1{,}350$.

Model F has $35+10=45$ predicted positives and $15+440=455$ predicted negatives. Its accuracy is $(35+440)/500=0.9500$, sensitivity is $35/50=0.7000$, and precision is $35/45=0.7778$. Its cost is $10($15)+15($90)=$150+$1{,}350=$1{,}500$.

Classifying every case as negative would produce 50 false negatives and no false positives, for a cost of $50($90)=$4{,}500$. Relative to that baseline, Model E reduces cost by $$4{,}500-$1{,}350=$3{,}150$, while Model F reduces it by $$4{,}500-$1{,}500=$3{,}000$. F has higher AUC and accuracy, but E has lower cost under the specified threshold and costs. Common mistake avoided: selecting F because it has fewer total errors (25 rather than 40). The errors are not equally costly: E’s additional false positives are more than offset by its five fewer false negatives.

Deriving a Cost-Based Threshold

A threshold based on costs can be derived when the model outputs calibrated probabilities. Calibration means that among cases assigned probability $p$, approximately a proportion $p$ have the positive outcome. For one case, let $p$ be its predicted probability of a positive outcome. Predicting positive incurs expected false-positive cost $(1-p)C_{FP}$; predicting negative incurs expected false-negative cost $pC_{FN}$. Choose positive when its expected cost is lower.

(1-p)C_{FP}<pC_{FN} \quad\Longleftrightarrow\quad C_{FP}<p(C_{FP}+C_{FN}) \quad\Longleftrightarrow\quad p>\frac{C_{FP}}{C_{FP}+C_{FN}}.

At equality, the two actions have the same expected cost. For the costs in the first example, the threshold is $30/(30+120)=0.20$. This is a theoretical decision rule under the stated assumptions, not a guarantee that a model’s observed validation cost will be lowest at exactly 0.20: finite samples create variation, and the model’s probabilities and operational constraints matter.

The rule does not apply unchanged if probabilities are not calibrated, if costs vary by case, if there is a binding capacity limit, or if taking action has additional costs or benefits not included in $C_{FP}$ and $C_{FN}$. In those situations, the decision rule must reflect those features. Nor should a numeric score from an uncalibrated model automatically be treated as a probability.

Confirm comparability.
Check that the models use the same validation cases, positive-outcome definition, and evaluation period. AUC or confusion-matrix differences can be misleading if the evaluated populations differ.

Separate ranking from threshold performance.
Read AUC as a ranking summary. Read each confusion matrix as a result at its stated threshold; do not infer one from the other.

Apply the decision costs.
Calculate $FP\,C_{FP}+FN\,C_{FN}$ for each model-threshold pair. If relevant, compare the result with simple baseline decisions such as predicting every case negative.

State the scope of the conclusion.
Name the threshold and costs that support the comparison. A different threshold, cost structure, or population may change the preferred operating point.

Common Traps in a Metric and Threshold Diagnosis

Using accuracy as decision value. Accuracy counts errors equally. It does not account for unequal consequences of false positives and false negatives.
Comparing AUC with a single-threshold measure as if they were interchangeable. AUC evaluates ranking across thresholds; sensitivity, specificity, precision, and cost depend on a particular threshold.
Ignoring the positive-class prevalence. Precision and accuracy depend on the mix of positive and negative cases. If validation prevalence differs from the population where decisions will be made, these measures and the resulting workload may not transfer directly.
Choosing a threshold from costs without checking assumptions. The cost-based probability cutoff presumes calibrated probabilities and the specified cost structure. A cutoff applied to an uncalibrated score has no such probability interpretation.
Reporting only a winner. A useful diagnosis states which metric favors which model, the threshold used for confusion-matrix comparisons, and how the stated costs affect the result.

The practical conclusion is conditional rather than universal: identify whether the decision is about ranking, performance at an operating threshold, or cost under specified consequences. Then compare like with like, show the calculation, and state what would need to change for the conclusion to change. This diagnostic approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

For each original scenario, distinguish the relevant metric from the decision calculation. Probabilities are reported to four decimals where needed.

A hypothetical model is evaluated on 300 cases, of which 30 are positive. At its threshold, $TP=24$, $FN=6$, $FP=27$, and $TN=243$. A false positive costs $10 and a false negative costs $70. Calculate the total cost and the cost-based probability threshold, assuming calibrated probabilities and no other decision costs. (Answer: Total cost = $27($10)+6($70)=$690$; threshold $=10/(10+70)=0.1250$.)
On 400 cases with 80 positives, Model G has $TP=60$, $FN=20$, $FP=48$, and $TN=272$. Calculate its sensitivity, specificity, precision, and accuracy. (Answer: Sensitivity $=60/80=0.7500$; specificity $=272/320=0.8500$; precision $=60/108=0.5556$; accuracy $=332/400=0.8300$.)
A calibrated model assigns one case a positive probability of 0.30. A false-positive cost is $18 and a false-negative cost is $42. Compare the expected costs of predicting positive and negative for this case. (Answer: Positive $=(1-0.30)($18)=$12.60$; negative $=0.30($42)=$12.60$; the actions are tied, and the threshold is $18/(18+42)=0.3000$.)
Two models have AUCs 0.88 and 0.84. At a stated threshold, the first has $TP=45$, $FN=15$, $FP=30$, $TN=210$; the second has $TP=40$, $FN=20$, $FP=10$, $TN=230$. If false positives cost $12 and false negatives cost $50, which has lower total cost at this threshold? (Answer: First $=30($12)+15($50)=$1{,}110$; second $=10($12)+20($50)=$1{,}120$; the first is lower by $10 at this threshold despite the second having fewer false positives.)
A hypothetical validation set has 25 positive and 175 negative cases. A model produces $TP=20$, $FN=5$, $FP=35$, $TN=140$. If false positives cost $8 and false negatives cost $64, calculate the model’s cost and compare it with predicting every case negative. (Answer: Model cost $=35($8)+5($64)=$600$; all-negative cost $=25($64)=$1{,}600$; the model reduces cost by $1,000 on this validation set.)

What You'll Learn