Common Errors in Classification Metric Calculations

Why Classification Calculations Go Wrong

Classification metrics are often simple ratios, but a correct calculation depends on knowing exactly what each cell represents and which group forms the denominator. A matrix with the axes reversed can produce plausible-looking but incorrect answers. Precision and sensitivity are especially easy to confuse because both use true positives in the numerator, but they divide by different totals.

The calculations below use a consistent convention: rows are the actual class, and columns are the predicted class. Always check the convention stated in a problem before reading its matrix. “Positive” means the event class of interest, such as a hypothetical claim requiring review; it does not necessarily mean a favorable outcome.

Definition: A true positive (TP) is an actual positive classified as positive; a false negative (FN) is an actual positive classified as negative; a false positive (FP) is an actual negative classified as positive; and a true negative (TN) is an actual negative classified as negative. With actual classes in rows and predicted classes in columns, the matrix is:

\begin{array}{c|cc} & \text{Predicted positive} & \text{Predicted negative} \\ \hline \text{Actual positive} & TP & FN \\ \text{Actual negative} & FP & TN \end{array}

The row totals describe actual classes: \(TP+FN\) actual positives and \(FP+TN\) actual negatives. The column totals describe predictions: \(TP+FP\) predicted positives and \(FN+TN\) predicted negatives. These totals provide a quick check before calculating any metric.

Match Each Metric to Its Denominator

The standard metrics below are defined for a specified positive class and a fixed classification threshold. A ratio is undefined if its denominator is zero; it should not silently be reported as zero.

Definition: Sensitivity (also called recall or true positive rate) is the proportion of actual positives classified as positive. Specificity (true negative rate) is the proportion of actual negatives classified as negative. Precision (positive predictive value) is the proportion of predicted positives that are actual positives. Accuracy is the proportion of all cases classified correctly.

\text{Sensitivity}=\frac{TP}{TP+FN}, \qquad \text{Specificity}=\frac{TN}{TN+FP}

\text{Precision}=\frac{TP}{TP+FP}, \qquad \text{Accuracy}=\frac{TP+TN}{TP+FN+FP+TN}

A useful verbal check is to ask what cases the metric begins with. Sensitivity starts with actual positives; specificity starts with actual negatives; precision starts with predicted positives; and accuracy starts with all cases. If the denominator does not match that group, the calculation is answering a different question.

Worked Example: Find the Denominator Before Calculating

A hypothetical review classifier produces the following results on 100 cases. Calculate sensitivity, specificity, precision, accuracy, and F1. State the method and identify the denominator trap.

\begin{array}{c|cc|c} & \text{Predicted positive} & \text{Predicted negative} & \text{Actual total}\\ \hline \text{Actual positive} & 12 & 18 & 30\\ \text{Actual negative} & 8 & 62 & 70\\ \hline \text{Predicted total} & 20 & 80 & 100 \end{array}

Method. Read the row and column totals first. There are \(30\) actual positives, \(70\) actual negatives, \(20\) predicted positives, and \(100\) total cases. Here \(TP=12\), \(FN=18\), \(FP=8\), and \(TN=62\).

Sensitivity is \(TP/(TP+FN)=12/(12+18)=12/30=0.4000\). Specificity is \(TN/(TN+FP)=62/(62+8)=62/70=0.8857\). Precision is \(TP/(TP+FP)=12/(12+8)=12/20=0.6000\).

Accuracy is \((TP+TN)/100=(12+62)/100=74/100=0.7400\). The F1 score, the harmonic mean of precision and sensitivity when both are defined, can be calculated directly from the matrix:

F_1=\frac{2TP}{2TP+FP+FN} =\frac{24}{24+8+18} =\frac{24}{50} =0.4800.

Check. Sensitivity \(0.4000\) means 12 of 30 actual positives were found; precision \(0.6000\) means 12 of 20 predicted positives were correct. The common mistake is to use \(TP+FN=30\) for precision. That gives \(12/30=0.4000\), which is sensitivity, not precision. The metrics are different because their denominators answer different questions.

Prevalence Changes Precision

Prevalence is the proportion of cases that are actually positive. Sensitivity and specificity condition on actual class, so if those rates remain fixed, they do not change merely because prevalence changes. Precision, however, conditions on a positive prediction. Its value depends on how many actual positives and negatives enter the population.

Derivation: Let \(\pi=P(\text{actual positive})\), let sensitivity be \(Se\), and let specificity be \(Sp\). In a population of size \(N\), the expected number of true positives is \(N\pi Se\). The expected number of false positives is \(N(1-\pi)(1-Sp)\), because the false-positive rate is \(1-Sp\). Divide true positives by all predicted positives to obtain precision.

\text{Precision} =\frac{N\pi Se}{N\pi Se+N(1-\pi)(1-Sp)} =\frac{\pi Se}{\pi Se+(1-\pi)(1-Sp)}.

The common factor \(N\) cancels, but prevalence does not. Consequently, a classifier can have unchanged sensitivity and specificity and still have substantially different precision in two populations. This is why a precision value from one population should not be transferred uncritically to another with a different event rate.

Worked Example: Same Rates, Different Prevalence

A hypothetical screening rule has sensitivity \(0.8000\) and specificity \(0.9000\). Compare its confusion matrices and precision in two populations of 10,000 cases: one with prevalence \(0.0200\), and one with prevalence \(0.2000\). The method is to apply the fixed rates to the actual class totals; the trap is assuming precision is fixed because the classifier’s rates are fixed.

Population A. With prevalence \(0.0200\), there are \(10{,}000(0.0200)=200\) actual positives and \(10{,}000-200=9{,}800\) actual negatives. Sensitivity \(0.8000\) gives \(TP=200(0.8000)=160\) and \(FN=200-160=40\). Specificity \(0.9000\) gives \(TN=9{,}800(0.9000)=8{,}820\) and \(FP=9{,}800-8{,}820=980\). Therefore precision is \(160/(160+980)=160/1{,}140=0.1404\), to four decimal places.

Population B. With prevalence \(0.2000\), there are \(10{,}000(0.2000)=2{,}000\) actual positives and \(8{,}000\) actual negatives. Then \(TP=2{,}000(0.8000)=1{,}600\), \(FN=400\), \(TN=8{,}000(0.9000)=7{,}200\), and \(FP=800\). Precision is \(1{,}600/(1{,}600+800)=1{,}600/2{,}400=0.6667\), to four decimal places.

Check. Both populations have sensitivity \(0.8000\) and specificity \(0.9000\), but their precision values are \(0.1404\) and \(0.6667\). In each matrix, the row totals equal 10,000 and the true positives plus false negatives equal the actual-positive total.

Threshold Conventions and Tied Scores

A score-based classifier must state how scores are converted into labels. A common convention predicts positive when the score is greater than or equal to a threshold, \(s\geq t\); another uses \(s>t\). The two rules agree for scores not equal to \(t\), but they can classify every case tied at \(t\) differently. When a problem specifies a convention, use it consistently.

Worked Example: The Equality Case Changes the Matrix

A hypothetical model assigns scores to four actual positives and three actual negatives. The positive-case scores are \(0.8, 0.6, 0.5, 0.2\); the negative-case scores are \(0.7, 0.5, 0.1\). Calculate the confusion-matrix metrics at threshold \(t=0.5\), first using \(s\geq t\), then using \(s>t\). The method is to classify each score before counting cells; the trap is forgetting that two cases are tied at the cutoff.

Using \(s\geq0.5\). Among the four actual positives, scores \(0.8,0.6,0.5\) are predicted positive, so \(TP=3\) and \(FN=1\). Among the three actual negatives, scores \(0.7,0.5\) are predicted positive, so \(FP=2\) and \(TN=1\). Thus sensitivity is \(3/4=0.7500\), specificity is \(1/3=0.3333\), and precision is \(3/(3+2)=0.6000\).

Using \(s>0.5\). The tied scores of \(0.5\) are now classified negative. The positive scores \(0.8,0.6\) give \(TP=2\), while the remaining two positives give \(FN=2\). Only the negative score \(0.7\) is predicted positive, so \(FP=1\) and \(TN=2\). Sensitivity is \(2/4=0.5000\), specificity is \(2/3=0.6667\), and precision is \(2/(2+1)=0.6667\).

Check. Both matrices contain four actual positives and three actual negatives. The change in the metrics is not an arithmetic inconsistency: it follows from the different equality rule applied to the two observations tied at \(0.5\).

AUC Is About Ranking, Not One Threshold

The area under the receiver operating characteristic curve (AUC) summarizes how well a score ranks positive cases above negative cases across thresholds. It is not the accuracy at a selected threshold, nor does it specify sensitivity, specificity, or precision at that threshold.

Definition: For a randomly selected actual positive and a randomly selected actual negative, AUC is the probability that the positive receives a higher score than the negative, with a tie counted as one-half of a correct ranking. This interpretation assumes scores are oriented so that larger scores indicate stronger evidence for the positive class.

The tie rule follows by assigning half credit to a tied positive-negative pair. If there are \(n_+\) positives and \(n_-\) negatives, there are \(n_+n_-\) such pairs. The AUC estimate is the total pairwise credit divided by \(n_+n_-\): a win contributes \(1\), a loss contributes \(0\), and a tie contributes \(0.5\). This gives a ranking summary without choosing one operating threshold.

Worked Example: Calculate AUC and Count Ties Correctly

A hypothetical ranking model gives four positive cases scores \(0.9,0.7,0.4,0.4\) and three negative cases scores \(0.8,0.4,0.2\). Calculate AUC by comparing every positive-negative pair. The method is pairwise ranking credit; the trap is either omitting tied pairs or counting each tie as a full win.

There are \(4(3)=12\) positive-negative pairs. The positive score \(0.9\) beats all three negative scores, contributing \(3\). The positive score \(0.7\) beats \(0.4\) and \(0.2\), but not \(0.8\), contributing \(2\). Each positive score \(0.4\) ties the negative score \(0.4\) and beats \(0.2\), contributing \(0.5+1=1.5\). The two such positive cases contribute \(2(1.5)=3\).

\text{AUC}=\frac{3+2+3}{12}=\frac{8}{12}=0.6667

Check. The pairwise credits sum to \(8\) out of \(12\), giving AUC \(0.6667\) to four decimal places. AUC describes ranking across the score ordering; it does not say that a particular threshold has accuracy \(0.6667\).

A Practical Error-Checking Routine

A few deliberate checks prevent many classification-metric mistakes. Use them especially when a problem gives a matrix without clearly naming its orientation, reports percentages rather than counts, or asks for a metric after a threshold change.

Write down the axis convention.
Identify whether rows or columns are actual classes. Translate the four cells into TP, FN, FP, and TN before using a formula.

Check the totals.
Confirm that TP+FN is the actual-positive count, FP+TN is the actual-negative count, and the grand total agrees with the data.

Say the denominator aloud.
For sensitivity use actual positives; for specificity use actual negatives; for precision use predicted positives; for accuracy use all cases.

Check the condition that generated the labels.
Record whether equality at the threshold is positive or negative, and apply the rule to every tied score.

Match the interpretation to the metric.
Keep prevalence in view when interpreting precision, and describe AUC as ranking performance rather than threshold-specific classification performance.

These checks are useful because errors can produce plausible values. For example, swapping sensitivity and precision may still give a number between zero and one; checking the denominator’s meaning is more reliable than checking whether the answer “looks reasonable.” These distinctions are aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

For each problem, identify the relevant denominator or comparison rule before calculating.

A classifier has \(TP=24\), \(FN=16\), \(FP=10\), and \(TN=50\). Calculate sensitivity, precision, and accuracy to four decimal places. (Answer: sensitivity \(=0.6000\), precision \(=0.7059\), accuracy \(=0.7400\).)
A screening rule has sensitivity \(0.9000\), specificity \(0.8000\), and prevalence \(0.1000\). Use the prevalence formula to calculate precision to four decimal places. (Answer: \(0.3333\).)
There are three positive scores \(0.9,0.6,0.5\) and two negative scores \(0.6,0.2\). At threshold \(0.6\), use the rule \(s\geq0.6\) to find TP, FN, FP, and TN. (Answer: \(TP=2,\ FN=1,\ FP=1,\ TN=1\).)
Four positive scores are \(0.9,0.8,0.7,0.6\), and one negative score is \(0.6\). Calculate AUC using half credit for a tie. (Answer: \((1+1+1+0.5)/4=0.8750\).)
A model has AUC \(0.82\). Does this mean its accuracy at a particular threshold is \(0.82\)? State what the AUC does mean. (Answer: No. It means a randomly selected positive-negative pair is ranked correctly with probability \(0.82\), counting ties as half.)

What You'll Learn