Why Accuracy Needs a Reference Point
Accuracy is an intuitive classification metric: it is the fraction of cases for which a model's predicted class matches the observed class. When the two classes occur at similar rates, this fraction can provide a useful first summary. Under substantial class imbalance, however, a classifier can achieve high accuracy largely by predicting the common class. That score alone may say little about whether the classifier identifies the rare outcome that matters to the decision.
A useful first comparison is the majority-class baseline: the accuracy obtained by always predicting the more common class. If a model does not exceed this simple reference, its overall proportion correct has not improved on a constant prediction rule. If it does exceed the reference, that still does not prove that the rare class is detected well; the confusion-matrix counts must be examined in context.
The class designated positive is the outcome of interest, such as a fraudulent claim or a future readmission. The negative class is its complement. Those labels must be stated before interpreting the counts; changing which outcome is called positive changes the class-specific interpretation, even though the total number classified correctly remains the same.
Constructing the Majority-Class Baseline
Suppose a held-out evaluation set contains \(n_+\) positive cases and \(n_-\) negative cases, with \(N=n_++n_-\). A constant rule that predicts the more numerous class is correct on every case in that class and wrong on every case in the other class. Its accuracy is therefore the larger class count divided by the total:
For a fair model comparison, the model and reference must be evaluated on the same cases. There is a practical distinction about how the constant rule is chosen. As a descriptive reference, one can calculate what the majority-class rule would achieve given the evaluation-set class counts. For a deployable benchmark, choose the constant predicted class using training data, then apply that fixed rule to the held-out cases. Do not inspect the held-out outcomes to choose a rule and then present it as if it had been fixed in advance.
When the training and evaluation sets have the same majority class, both approaches use that class. If the majority differs between them, the training-defined rule's test accuracy is the number of test cases in the class it predicts, divided by the test-set total. The simple descriptive maximum on the test set is then not the performance of that training-defined rule.
Why Imbalance Can Dominate Accuracy
Accuracy combines the results for the two classes, but it does not give them equal weight. Let the positive-class prevalence in an evaluation set be \(n_+/N\). Sensitivity is the fraction of actual positive cases predicted positive, \(TP/n_+\), and specificity is the fraction of actual negative cases predicted negative, \(TN/n_-\). These definitions apply when both class counts are nonzero. Substituting \(TP=n_+(TP/n_+)\) and \(TN=n_-(TN/n_-)\) into the accuracy formula gives:
This identity explains the main limitation. If positive cases are rare, the negative-class term receives most of the weight. A classifier can therefore have a high accuracy even when its sensitivity is low. If a class count is zero, its corresponding class-specific rate is undefined; the count-based accuracy formula remains valid, but this weighted expression should be interpreted only for classes that are present.
Worked Comparisons
Worked Example: A Screening Model Below the Majority Baseline
A hypothetical screening program evaluates 1,000 clients for a rare condition. There are 20 positive and 980 negative cases. The training data's majority class is negative, so a constant benchmark predicts negative for everyone. On the held-out set, a model produces \(TP=8\), \(FN=12\), \(TN=970\), and \(FP=10\). Calculate both accuracies and explain the comparison.
Method. First verify the confusion-matrix total: \(8+12+970+10=1{,}000\). The positive total is \(TP+FN=8+12=20\), and the negative total is \(TN+FP=970+10=980\), agreeing with the stated class counts.
The model correctly classifies \(TP+TN=8+970=978\) cases. Thus its accuracy is \(978/1{,}000=0.978\), or 97.8%. It makes \(FP+FN=10+12=22\) errors, giving an error rate of \(22/1{,}000=0.022\), or 2.2%.
The negative-predicting baseline is correct on all 980 negative cases and incorrect on the 20 positive cases. Its accuracy is \(980/1{,}000=0.980\), or 98.0%. The model's accuracy is 0.2 percentage points lower than the baseline.
The comparison does not mean the model found nothing useful: it correctly identifies 8 of the 20 positive cases. But its overall accuracy is below the constant rule's accuracy. The common mistake is to call 97.8% strong performance without noticing that the baseline already achieves 98.0%. This example avoids that mistake by computing both values on the same cases and then inspecting the counts.
Worked Example: A High Score With No Positive Detections
A hypothetical claims team evaluates a model on 2,000 claims. The held-out set has 40 claims with a specified costly event and 1,960 without it. The model predicts negative for every claim. Find its accuracy, the majority-class baseline accuracy, and the number of positive cases it detects.
Method. Since every prediction is negative, all 1,960 actual negatives are true negatives, and all 40 actual positives are false negatives. Therefore, \(TN=1{,}960\), \(FN=40\), \(TP=0\), and \(FP=0\). The counts sum to \(1{,}960+40=2{,}000\).
The model's number correct is \(TP+TN=0+1{,}960=1{,}960\). Its accuracy is \(1{,}960/2{,}000=0.980\), or 98.0%. Its error rate is \(40/2{,}000=0.020\), or 2.0%.
The majority class is negative, so the majority-class baseline also predicts negative for every claim. Its accuracy is \(1{,}960/2{,}000=98.0\%\). The model detects \(TP=0\) of the 40 positive cases; the high score reflects the large number of negatives, not detection of the costly event.
A common mistake is to interpret 98.0% accuracy as evidence that most relevant events are found. The counts show the opposite: every positive event is missed. This example avoids that error by translating the constant rule into its confusion-matrix counts before interpreting the percentage.
Worked Example: Comparing Two Models With the Same Prevalence
A hypothetical monitoring system is tested on 5,000 accounts, of which 100 have the rare outcome and 4,900 do not. Both models are evaluated on these same accounts. Model A has \(TP=40\), \(FN=60\), \(TN=4{,}800\), \(FP=100\). Model B has \(TP=5\), \(FN=95\), \(TN=4{,}895\), \(FP=5\). Calculate each accuracy and the majority-class baseline, then state what the accuracy ranking does and does not show.
Method. Check Model A's total: \(40+60+4{,}800+100=5{,}000\). Its correct predictions are \(TP+TN=40+4{,}800=4{,}840\), so its accuracy is \(4{,}840/5{,}000=0.968\), or 96.8%.
Check Model B's total: \(5+95+4{,}895+5=5{,}000\). Its correct predictions are \(5+4{,}895=4{,}900\), so its accuracy is \(4{,}900/5{,}000=0.980\), or 98.0%.
The majority class is negative, with 4,900 cases. Thus the baseline accuracy is \(4{,}900/5{,}000=98.0\%\). Model B matches the baseline, while Model A is 1.2 percentage points below it. Model B has the higher accuracy, but it identifies only 5 of the 100 positive cases, compared with 40 for Model A.
The trap is to conclude from the accuracy ranking alone that Model B is better for every purpose. The calculation supports only a comparison of overall proportions correct on these cases; it does not establish which model is preferable for a decision where identifying positive cases is important. This example avoids that overclaim by reporting the positive counts alongside accuracy.
Common Calculation and Interpretation Traps
The denominator in accuracy is the number of all evaluated cases, not the number of predicted positives, the number of actual positives, or the number of negatives alone. In the screening example, the incorrect calculation \(978/(TP+TN+FP)=978/988\), approximately 99.0%, omits the 12 false negatives from the denominator. The correct calculation is \(978/(8+970+10+12)=978/1{,}000=97.8\%\).
Other checks help prevent misleading comparisons:
- Use the same evaluation set. Accuracy values based on different case mixes may differ because their class proportions differ, even if class-specific behavior is similar.
- Report the baseline rule. State which class it predicts and how that class was selected. For a test-set evaluation of a learned constant rule, select the class using training data.
- Do not equate high accuracy with rare-event detection. A large negative-class count can dominate the total, as the all-negative example demonstrates.
- Keep counts with percentages. The confusion matrix makes visible how many positive cases were correctly identified or missed.
Accuracy and its majority-class comparison are useful screening summaries, not a complete decision analysis. Their value is in showing whether the model improves on a simple constant prediction in overall correctness. The operational importance of missed positives and false alarms depends on the application; the percentage alone does not encode those consequences.
Check Your Understanding
For each problem, calculate accuracy and compare it with the stated majority-class reference. Interpret the result in terms of the rare outcome.
- A hypothetical equipment alert set has 30 actual failures and 570 non-failures. A model has \(TP=18\), \(FN=12\), \(TN=540\), and \(FP=30\). What is its accuracy, and what is the accuracy of a rule that always predicts non-failure? (Answer: Model accuracy \(558/600=93.0\%\); baseline accuracy \(570/600=95.0\%\).)
- A hypothetical benefit review set has 25 positive cases and 975 negative cases. A classifier predicts negative for every case. Find its accuracy and the number of positive cases it detects. (Answer: Accuracy \(975/1{,}000=97.5\%\); it detects 0 of 25 positive cases.)
- A hypothetical inspection set contains 80 positive and 920 negative units. A model has \(TP=60\), \(FN=20\), \(TN=870\), and \(FP=50\). Calculate its accuracy and the majority-class baseline accuracy. (Answer: Model accuracy \(930/1{,}000=93.0\%\); baseline accuracy \(920/1{,}000=92.0\%\).)
- On 2,500 hypothetical accounts, there are 100 positive and 2,400 negative outcomes. Model A has \(TP=70\), \(FN=30\), \(TN=2{,}280\), \(FP=120\). Model B has \(TP=20\), \(FN=80\), \(TN=2{,}390\), \(FP=10\). Calculate both accuracies and the majority-class baseline, then identify which model has the higher accuracy. (Answer: Model A \(2{,}350/2{,}500=94.0\%\); Model B \(2{,}410/2{,}500=96.4\%\); baseline \(2{,}400/2{,}500=96.0\%\); Model B has higher accuracy.)
- A held-out set has 40 positive and 160 negative cases. A model has \(TP=24\), \(FN=16\), \(TN=136\), and \(FP=24\). A candidate reports accuracy as \((TP+TN)/(TP+TN+FP)\). Identify the denominator error and give the correct model and baseline accuracies. (Answer: The denominator omits \(FN\); correct model accuracy is \(160/200=80.0\%\), and the negative-majority baseline is \(160/200=80.0\%\).)