Resolving Conflicting Classification Metrics

Why Classification Metrics Can Disagree

Two classifiers can be evaluated on the same cases and still appear to have different strengths. One may identify more actual positives, while another makes fewer false alarms or labels more cases correctly overall. These are not necessarily contradictory results: each metric emphasizes a different aspect of performance. A sound comparison begins by identifying the decision the model will support and the cost of each kind of error.

For this tutorial, assume a binary outcome with one explicitly designated positive class, such as a hypothetical claim requiring additional review. A classifier assigns each case a predicted label at a specified threshold. Evaluate competing classifiers on the same labeled data, with the same positive class and a clearly stated threshold. Changing any of these conditions can change the comparison.

Definition: For a binary classification evaluation, true positive (TP) is an actual positive classified as positive; false negative (FN) is an actual positive classified as negative; false positive (FP) is an actual negative classified as positive; and true negative (TN) is an actual negative classified as negative. The total number of evaluated cases is $N=TP+FN+FP+TN$.

These four counts form the confusion matrix. They are the foundation for the common classification metrics. The following definitions assume that the relevant denominator is nonzero. If a classifier predicts no positives, for example, precision is undefined rather than zero: there are no predicted-positive cases over which to calculate it.

\begin{aligned} \text{Accuracy} &= \frac{TP+TN}{N}, & \text{Sensitivity} &= \frac{TP}{TP+FN},\\ \text{Specificity} &= \frac{TN}{TN+FP}, & \text{Precision} &= \frac{TP}{TP+FP}. \end{aligned}

Accuracy is the proportion of all cases classified correctly. Sensitivity, also called the true positive rate, is the proportion of actual positives detected. Specificity, also called the true negative rate, is the proportion of actual negatives correctly rejected. Precision, also called positive predictive value, is the proportion of predicted positives that are actual positives. Sensitivity and precision are not interchangeable: sensitivity conditions on the actual positive group, while precision conditions on the predicted positive group.

Prevalence Changes the Meaning of Accuracy and Precision

The prevalence in an evaluation set is the proportion of its cases that are actually positive. If $\pi=(TP+FN)/N$, then accuracy can be written as a weighted average of sensitivity and specificity:

\text{Accuracy} =\pi\,\text{Sensitivity}+(1-\pi)\,\text{Specificity}.

To justify this identity, split the correctly classified cases into the actual-positive and actual-negative groups. The probability of a correct classification is the probability of being positive and correctly classified, plus the probability of being negative and correctly classified. Those terms are $\pi\,\text{Sensitivity}$ and $(1-\pi)\,\text{Specificity}$, respectively. Their sum is accuracy. Thus, accuracy gives more weight to the performance on whichever class is more common.

Precision is also affected by prevalence. For a population with prevalence $\pi$, sensitivity $s$, and specificity $c$, the proportion of all cases that are true positives is $\pi s$; the proportion that are false positives is $(1-\pi)(1-c)$. Therefore, when the denominator is positive,

\text{Precision} =\frac{\pi s}{\pi s+(1-\pi)(1-c)}.

Even if sensitivity and specificity stay fixed, precision can fall when the positive class becomes rarer. A classifier may therefore have the same ability to distinguish the classes but generate a smaller share of correct flags in a low-prevalence setting. When comparing models, check whether the evaluation prevalence resembles the intended use population.

Worked Example: Accuracy Versus Detection of a Rare Event

A hypothetical review team evaluates two classifiers on the same 200 records. There are 40 actual positives and 160 actual negatives. Classifier A has $TP=32$, $FN=8$, $FP=40$, and $TN=120$. Classifier B has $TP=28$, $FN=12$, $FP=16$, and $TN=144$. Compare their accuracy, sensitivity, specificity, and precision. Name the method and identify the mistake it avoids.

Method. Calculate each metric directly from its defining confusion-matrix denominator. The actual-positive total is $40$, the actual-negative total is $160$, and the total is $200$.

For classifier A:

Accuracy is $(32+120)/200=152/200=0.7600$.
Sensitivity is $32/(32+8)=32/40=0.8000$.
Specificity is $120/(120+40)=120/160=0.7500$.
Precision is $32/(32+40)=32/72=0.4444$, to four decimal places.

For classifier B:

Accuracy is $(28+144)/200=172/200=0.8600$.
Sensitivity is $28/(28+12)=28/40=0.7000$.
Specificity is $144/(144+16)=144/160=0.9000$.
Precision is $28/(28+16)=28/44=0.6364$, to four decimal places.

Classifier B has higher accuracy, specificity, and precision. Classifier A has higher sensitivity, so it detects a larger share of actual positives. The calculations are consistent with the accuracy identity: prevalence is $40/200=0.20$. For A, $0.20(0.80)+0.80(0.75)=0.16+0.60=0.76$. For B, $0.20(0.70)+0.80(0.90)=0.14+0.72=0.86$.

Check. Each set of four counts sums to 200, and each calculated accuracy matches the weighted sensitivity-specificity identity. The common mistake avoided is announcing one classifier as simply “better” from accuracy alone: that would hide A’s advantage in detecting actual positives.

What AUC Adds—and What It Does Not

Accuracy, sensitivity, specificity, and precision describe classifications made at a particular threshold. If a model produces scores, the threshold converts those scores into predicted labels. AUC, the area under the receiver operating characteristic curve, instead summarizes how well the scores rank positive cases above negative cases across thresholds. AUC does not depend on choosing one particular classification threshold.

Definition: For a set of labeled cases with scores, AUC is the proportion of positive-negative case pairs in which the positive case has the higher score, counting a tied pair as one-half. Equivalently, it is the area under the ROC curve, which plots sensitivity against the false positive rate $1-\text{specificity}$ as the threshold varies.

This definition assumes there is at least one actual positive and at least one actual negative, so that positive-negative pairs exist. AUC measures ranking, not the accuracy of the labels produced at a particular threshold, nor whether scores are calibrated probabilities. A higher AUC can be useful evidence when ranking cases is the goal, but it does not by itself establish that a classifier meets an operational requirement for false alarms or missed positives.

Worked Example: AUC and Threshold Metrics Tell Different Stories

On a hypothetical six-case evaluation set, three cases are positive and three are negative. Classifier C gives positive-case scores $0.49, 0.48, 0.47$, and negative-case scores $0.46, 0.45, 0.44$. Classifier D gives positive-case scores $0.90, 0.80, 0.20$, and negative-case scores $0.85, 0.40, 0.10$. At a threshold of $0.50$, compare the confusion-matrix metrics and AUC. Scores at or above the threshold are classified as positive.

Method. First count threshold-based classifications. Then calculate AUC by comparing every positive score with every negative score. There are $3\times3=9$ pairs for each classifier.

For C, all three positive scores exceed all three negative scores. It wins all $9$ pairs, so its AUC is $9/9=1.0000$. At threshold $0.50$, none of its scores reach the threshold. Thus $TP=0$, $FN=3$, $FP=0$, and $TN=3$. Accuracy is $3/6=0.5000$, sensitivity is $0/3=0.0000$, and specificity is $3/3=1.0000$. Precision is undefined because $TP+FP=0$.

For D, the positive score $0.90$ exceeds all three negative scores, giving $3$ wins. The positive score $0.80$ exceeds $0.40$ and $0.10$, giving $2$ wins. The positive score $0.20$ exceeds only $0.10$, giving $1$ win. Therefore, AUC is $(3+2+1)/9=6/9=0.6667$, rounded to four decimal places.

At threshold $0.50$, D classifies $0.90$ and $0.80$ as positive, and $0.85$ as positive. Therefore $TP=2$, $FN=1$, $FP=1$, and $TN=2$. Accuracy is $(2+2)/6=0.6667$; sensitivity is $2/3=0.6667$; specificity is $2/3=0.6667$; and precision is $2/(2+1)=0.6667$, each rounded to four decimal places.

Check. Both matrices sum to six, and the AUC pair counts use all nine possible positive-negative pairs. C ranks every positive above every negative, but at the stated threshold it predicts no positives. The example shows why AUC and threshold-based metrics can rank classifiers differently; it does not make either metric an automatic choice for every decision.

Match the Metric to the Decision

The preferred evidence depends on what happens after a prediction. If missing a positive case is especially harmful, sensitivity may be central. If every flagged case requires an expensive investigation, precision and the number of false positives matter. If the principal concern is correctly rejecting negatives, specificity is relevant. Accuracy is informative when overall correct classification is the objective and the class mix makes that summary meaningful. AUC is relevant when ranking positives above negatives across possible thresholds is important.

Metrics alone do not encode the relative costs of errors. If costs are supplied, compare the consequences using the same evaluation counts and the stated costs. For example, a decision rule that assigns cost $C_{FN}$ to each false negative and $C_{FP}$ to each false positive has total error cost

C_{FN}\,FN+C_{FP}\,FP.

This calculation is appropriate only when those costs apply to the cases being evaluated and the comparison uses a relevant population. It does not account for other costs unless they are specified. In practice, a defensible comparison should state the positive class, evaluation population, threshold for threshold-based metrics, and the decision the metric is intended to inform.

Worked Example: Choosing Evidence When False Negatives Cost More

A hypothetical organization evaluates two classifiers on 200 cases, including 40 actual positives and 160 actual negatives. Model P has $TP=34$, $FN=6$, $FP=32$, and $TN=128$. Model Q has $TP=30$, $FN=10$, $FP=12$, and $TN=148$. Each false negative has an assigned cost of $\$500$, and each false positive has an assigned cost of $\$25$. Compare accuracy and total specified error cost. Which model has the lower cost under these assumptions?

Method. Calculate accuracy from correct classifications, then multiply each error count by its stated cost and add. This compares the models against the supplied decision criterion rather than selecting a metric without reference to consequences.

For P, accuracy is $(34+128)/200=162/200=0.8100$. Its specified error cost is $6(\$500)+32(\$25)=\$3{,}000+\$800=\$3{,}800$.

For Q, accuracy is $(30+148)/200=178/200=0.8900$. Its specified error cost is $10(\$500)+12(\$25)=\$5{,}000+\$300=\$5{,}300$.

Model Q has higher accuracy, but Model P has the lower specified error cost by $\$5{,}300-\$3{,}800=\$1{,}500$. Under the stated costs and evaluation population, P is preferred on this cost criterion.

Check. The matrices each sum to 200, and both cost calculations include false negatives and false positives exactly once. The common mistake avoided is treating higher accuracy as equivalent to lower decision cost.

Common Metric Traps

Using the wrong denominator. Sensitivity divides by actual positives, $TP+FN$; precision divides by predicted positives, $TP+FP$. Swapping these answers a different question.
Ignoring prevalence. A majority-class classifier can have high accuracy while detecting no positives. Report the class mix and examine class-specific metrics.
Treating undefined precision as zero. If no cases are predicted positive, the precision denominator is zero. State that precision is undefined; the classifier’s zero sensitivity and the absence of positive predictions are still informative.
Comparing AUC with a threshold metric as if they were the same quantity. AUC summarizes score ranking across thresholds; accuracy, sensitivity, specificity, and precision refer to classifications at the stated threshold.
Changing the evaluation basis. A comparison is difficult to interpret if candidates use different cases, different definitions of the positive class, or different thresholds without saying so. In particular, precision can change when prevalence changes.

Key takeaway: No single classification metric captures every form of performance. Calculate the metric from its correct denominator, check prevalence and the evaluation conditions, and use the result that addresses the stated decision. AUC provides ranking evidence; threshold-based metrics describe the consequences of a particular classification rule.

Check Your Understanding

For each problem, use the positive class and evaluation conditions stated. Report probabilities to four decimal places where needed.

A classifier is evaluated on 100 records. Its counts are $TP=18$, $FN=2$, $FP=12$, and $TN=68$. Find accuracy, sensitivity, specificity, and precision. (Answer: accuracy $0.8600$, sensitivity $0.9000$, specificity $0.8500$, precision $0.6000$.)
In a hypothetical set of 500 cases, 15 are positive. A classifier predicts every case negative. Find its accuracy, sensitivity, specificity, and precision. (Answer: accuracy $485/500=0.9700$, sensitivity $0.0000$, specificity $1.0000$, and precision undefined because it predicts no positives.)
A review model identifies 24 true positives and 36 false positives. What is its precision? Explain what the denominator represents. (Answer: $24/(24+36)=0.4000$; the denominator is the 60 cases predicted positive.)
Two positive cases have scores $0.80$ and $0.40$, and two negative cases have scores $0.70$ and $0.30$. Calculate AUC using positive-negative pairs. (Answer: $3/4=0.7500$; the higher positive score wins three of the four pairs.)
Two hypothetical models are evaluated on the same set. Model A has $TP=30$, $FN=10$, $FP=20$, and $TN=140$; Model B has $TP=25$, $FN=15$, $FP=8$, and $TN=152$. If each false negative costs $\$100$ and each false positive costs $\$20$, calculate each model’s specified error cost and select the lower-cost model. (Answer: A costs $10(\$100)+20(\$20)=\$1{,}400$; B costs $15(\$100)+8(\$20)=\$1{,}660$; Model A has the lower specified cost by $\$260$.)

What You'll Learn