Why a High Accuracy Score May Not Mean Success
Accuracy is easy to calculate and easy to explain: it is the fraction of predictions that are correct. But that fraction gives every observation the same weight. If positive outcomes are rare, a classifier can be correct on nearly everyone by predicting the common, or majority, class every time. Its accuracy may look impressive even though it never identifies a positive case.
This matters when the positive class is the one an organization needs to find: for example, a hypothetical insurer’s claims requiring an additional review, or customers likely to cancel a service. The relevant question is not simply “How many predictions were correct?” It is “Does the measure reflect the action or outcome the model is meant to support?” Accuracy answers the first question, not necessarily the second. This distinction is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
The prevalence, written \(\pi\), is the proportion of cases that truly belong to the positive class. Sensitivity is the proportion of positive cases correctly predicted positive; specificity is the proportion of negative cases correctly predicted negative. These quantities help show how prevalence affects accuracy. They do not, by themselves, say whether the model is useful for a particular decision.
The Majority-Class Baseline
A simple baseline predicts the majority class for every case. If the positive class has prevalence \(\pi\) and \(\pi<0.5\), then the negative class is the majority. Predicting negative for everyone gives accuracy \(1-\pi\): all negative cases are correct, and all positive cases are missed. More generally, the accuracy of the best constant-class prediction is \(\max(\pi,1-\pi)\), assuming there are no ties that change which class is the majority.
This baseline is useful because it puts a seemingly large accuracy figure in context. If positive cases make up only 2% of a population, a classifier that predicts negative for everyone already has 98% accuracy. A more complex model should not be judged successful merely because its accuracy is high; compare it with this basic reference and ask what the model achieves on the cases that matter.
The identity follows by splitting all cases into the two actual classes. A proportion \(\pi\) are positive; among them, the correct fraction is sensitivity. A proportion \(1-\pi\) are negative; among them, the correct fraction is specificity. The two disjoint groups together make up the full population, so adding their contributions gives the overall correct fraction. In particular, the all-negative baseline has sensitivity \(0\), specificity \(1\), and therefore accuracy \(1-\pi\).
Worked Example: A Strong-Looking All-Negative Baseline
A hypothetical review team evaluates a classifier on 1,000 claims. Twenty claims actually require additional review, and 980 do not. The classifier predicts that no claim requires review. Calculate its accuracy and explain what the result does and does not establish.
Method. Count correct predictions using the confusion-matrix definition. The observed positive class has 20 cases, so prevalence is \(20/1000=0.0200\).
All 20 positive cases are predicted negative: \(\mathrm{FN}=20\), \(\mathrm{TP}=0\). All 980 negative cases are predicted negative: \(\mathrm{TN}=980\), \(\mathrm{FP}=0\). The total is \(20+0+980+0=1{,}000\).
The result checks against the baseline formula: \(1-\pi=1-0.0200=0.9800\). The classifier is correct for 980 cases, but it identifies none of the 20 claims needing review. Thus, 98.00% accuracy alone does not establish that it serves the review objective.
Common mistake avoided: treating “98% correct” as evidence that the classifier detects the target cases well. It does not: its sensitivity is \(0/20=0.0000\). The high score is explained by the large number of negative cases.
What Accuracy Gives Equal Weight To
In the accuracy calculation, each case contributes either one correct prediction or one incorrect prediction. The measure does not distinguish a false negative from a false positive, and it does not express the consequences of either error. Nor does it identify which class is being predicted correctly. Two models can therefore have different patterns of errors and different practical value even when their accuracy figures are close.
This does not make accuracy mathematically incorrect. It makes accuracy a limited summary. If the goal is simply to maximize the number of correctly classified cases and the consequences of errors are comparable, accuracy may be appropriate. If the goal is to identify rare positive cases, or if missing a positive case is more consequential than reviewing a negative one, accuracy alone does not represent that goal. The operational objective and the consequences of errors must guide interpretation.
The comparison should be explicit. Report the accuracy, identify the majority-class baseline, and state how many positive cases were found or missed. When error consequences are known, calculate an objective that reflects them as well. That calculation can lead to a different model choice than accuracy does.
Worked Example: Lower Accuracy, More Positive Cases Found
A hypothetical screening team compares two classifiers on 1,000 applicants. There are 20 positive cases and 980 negative cases. Classifier A predicts every case negative. Classifier B identifies 12 positive cases, misses 8, and incorrectly flags 40 negative cases. Compare their accuracies and state the trade-off.
Method. Complete the counts for each classifier and apply the same accuracy formula. For A, \(\mathrm{TN}=980\), \(\mathrm{FN}=20\), \(\mathrm{TP}=0\), and \(\mathrm{FP}=0\), giving accuracy \(980/1{,}000=0.9800\), or 98.00%.
For B, the 20 positive cases split into 12 identified and 8 missed, so \(\mathrm{TP}=12\) and \(\mathrm{FN}=8\). The 980 negative cases split into 40 incorrectly flagged and \(980-40=940\) correctly not flagged, so \(\mathrm{FP}=40\) and \(\mathrm{TN}=940\). Check the total: \(12+8+40+940=1{,}000\).
Classifier B’s accuracy is 2.80 percentage points lower than A’s: \(98.00\%-95.20\%=2.80\%\). However, B identifies 12 positive cases, while A identifies none. Whether this is a worthwhile trade-off cannot be decided from accuracy alone; it depends on the purpose of screening and the consequences of the additional flags and missed cases.
Check. B’s errors are \(8+40=48\), so it has \(1{,}000-48=952\) correct predictions, matching its numerator. Its sensitivity is \(12/20=0.6000\). A’s sensitivity is \(0/20=0.0000\).
Common mistake avoided: selecting A automatically because its accuracy is higher. That decision treats the accuracy ranking as if it answered the screening objective, even though A finds no positive cases.
Prevalence Changes the Meaning of the Same Model
Accuracy depends not only on how a classifier performs within each actual class, but also on the mix of classes in the evaluated population. This follows directly from the weighted formula: sensitivity is weighted by prevalence, and specificity is weighted by one minus prevalence. When positives are rare, specificity can dominate the overall accuracy because most cases are negative.
Consequently, accuracy values from populations with different prevalences are not automatically comparable as summaries of the same class-specific performance. A change in accuracy may reflect a change in class mix, a change in classifier performance, or both. State the evaluation population’s prevalence when interpreting the score.
Worked Example: The Same Classifier at Two Prevalences
A hypothetical classifier has sensitivity \(0.7000\) and specificity \(0.9500\). Calculate its expected accuracy in two populations: one with 2% positive cases and another with 20% positive cases. Compare each result with the all-negative baseline for that population.
Method. Use the weighted accuracy identity. At 2% prevalence, the positive contribution is \(0.0200\times0.7000=0.0140\). The negative contribution is \(0.9800\times0.9500=0.9310\). Therefore, accuracy is \(0.0140+0.9310=0.9450\), or 94.50%. The all-negative baseline is \(1-0.0200=0.9800\), or 98.00%.
At 20% prevalence, the positive contribution is \(0.2000\times0.7000=0.1400\). The negative contribution is \(0.8000\times0.9500=0.7600\). Therefore, accuracy is \(0.1400+0.7600=0.9000\), or 90.00%. The all-negative baseline is \(1-0.2000=0.8000\), or 80.00%.
The classifier’s sensitivity and specificity are unchanged, yet its accuracy differs because the prevalence differs. At 2% prevalence its accuracy is below the all-negative baseline, while at 20% it is above that baseline. Neither comparison, by itself, decides whether using the classifier is worthwhile; each only describes overall correct classification relative to the specified reference.
Check. Both computed accuracies lie between 0 and 1. At the lower prevalence, the high specificity applies to 98% of cases and contributes \(0.9310\) to accuracy, explaining why the overall score remains high despite sensitivity of only \(0.7000\).
Common mistake avoided: assuming that a model with fixed sensitivity and specificity must have fixed accuracy. Accuracy is prevalence-weighted, so the population mix matters.
When Error Consequences Matter
If the decision objective assigns different consequences to different errors, accuracy is especially incomplete. A false negative might mean that a case needing attention is not reviewed; a false positive might mean spending time on an unnecessary review. The relative consequences depend on the hypothetical or real application. Accuracy counts both errors as one incorrect case, regardless of those consequences.
A cost calculation can make the distinction concrete. Suppose the stated objective is to minimize error cost, with a specified cost for each false negative and false positive and no additional cost for correct classifications. Then total error cost is the number of false negatives multiplied by their cost, plus the number of false positives multiplied by their cost. This objective should not be inferred from accuracy: it must be supplied by the decision context.
Worked Example: Accuracy Ranking Versus Error Cost
A hypothetical audit team evaluates two classifiers on 2,000 accounts, of which 30 are positive. It assigns a cost of $500 to each false negative and $20 to each false positive; correct classifications have zero error cost. Model A predicts every account negative. Model B has 24 true positives, 6 false negatives, 100 false positives, and 1,870 true negatives. Compare accuracy and total error cost.
Method. First calculate the confusion-matrix totals and accuracy. For A, \(\mathrm{TN}=1{,}970\), \(\mathrm{FN}=30\), \(\mathrm{TP}=0\), and \(\mathrm{FP}=0\); the counts sum to 2,000. Its accuracy is \(1{,}970/2{,}000=0.9850\), or 98.50%.
For B, the counts sum to \(24+6+100+1{,}870=2{,}000\). Its correct predictions are \(24+1{,}870=1{,}894\), so its accuracy is \(1{,}894/2{,}000=0.9470\), or 94.70%.
Now calculate the stated error cost. Model A has 30 false negatives and no false positives, so its cost is \(30\times\$500+0\times\$20=\$15{,}000\). Model B’s cost is \(6\times\$500+100\times\$20=\$3{,}000+\$2{,}000=\$5{,}000\).
Under accuracy, A ranks higher by \(98.50\%-94.70\%=3.80\) percentage points. Under the specified error-cost objective, B has $10,000 lower cost. Thus the two criteria rank the models differently. The cost calculation does not prove that B is preferable under every operational objective; it shows that the stated costs lead to a different comparison than accuracy does.
Common mistake avoided: treating the number of errors as their total consequence. A has 30 errors and B has 106, but the specified cost of B’s errors is lower because its error types and their assigned costs differ.
A Practical Interpretation Checklist
When you encounter a high accuracy score on imbalanced data, interpret it in a fixed order:
Record the number and proportion of positive and negative cases in the evaluation data.
If positives are the minority, the all-negative baseline is \(1-\pi\). Compare the reported accuracy with it.
Count positive cases found and missed, as well as negative cases incorrectly flagged. An overall fraction does not reveal that pattern.
Ask what action the predictions support and whether errors have different consequences. Use an objective that represents those consequences when they are specified.
The central lesson is not that accuracy should never be used. It is that accuracy measures the fraction correct, not the importance of being correct on a particular class or the value of the actions that follow. A strong interpretation names both the score and its limitation. For example: “Accuracy is 98%, compared with a 98% all-negative baseline; the classifier identifies no positive cases.” That statement is more informative than the accuracy figure alone.
Check Your Understanding
For each problem, calculate the requested quantity and interpret it in context.
- A hypothetical equipment-monitoring data set contains 500 inspections, including 15 that identify a fault. A classifier predicts no faults for any inspection. What is its accuracy, and how many faults does it identify? (Answer: Accuracy \(=485/500=0.9700\), or 97.00%; it identifies 0 of 15 faults.)
- In a population with 4% positive cases, a classifier has sensitivity \(0.7500\) and specificity \(0.9000\). Calculate its accuracy and the all-negative baseline accuracy. (Answer: Classifier accuracy \(=(0.0400)(0.7500)+(0.9600)(0.9000)=0.8940\), or 89.40%; baseline \(=0.9600\), or 96.00%.)
- A data set has 60 positive and 940 negative cases. A classifier predicts 42 positives correctly, misses 18 positives, incorrectly predicts 47 negatives as positive, and correctly predicts 893 negatives. Calculate its accuracy and compare it with the all-negative baseline. (Answer: Accuracy \(=(42+893)/1{,}000=0.9350\), or 93.50%; the all-negative baseline is \(940/1{,}000=0.9400\), or 94.00%.)
- Two classifiers are evaluated on 1,000 cases with 25 positives. Model A has 20 true positives and 900 true negatives. Model B has 15 true positives and 940 true negatives. For each model, calculate accuracy and the number of positive cases identified. Which model has higher accuracy, and which identifies more positives? (Answer: A has \((20+900)/1{,}000=0.9200\), or 92.00%, and identifies 20 positives; B has \((15+940)/1{,}000=0.9550\), or 95.50%, and identifies 15 positives. B has higher accuracy; A identifies more positives.)
- A hypothetical review objective assigns a cost of $300 to each false negative and $25 to each false positive. Model C has 5 false negatives and 40 false positives; Model D has 12 false negatives and 10 false positives. Calculate each model’s error cost. Does the lower cost necessarily mean the same model has higher accuracy? (Answer: C costs \(5(\$300)+40(\$25)=\$2{,}500\); D costs \(12(\$300)+10(\$25)=\$3{,}850\). C has lower stated error cost. This information alone does not establish which model has higher accuracy; the numbers of true positives and true negatives, or the class totals together with the error counts, are also needed.)