Why Class Imbalance Changes the Comparison
A binary outcome is imbalanced when one class is much more common than the other. In a hypothetical insurance screening task, for example, most policies might have no large loss while only a small fraction have one. A classifier can then appear accurate simply by predicting the common outcome for everyone. That may be a poor result if identifying the uncommon outcome matters.
A classification tree assigns a class using rules that partition predictor space into regions. A \(k\)-nearest neighbors classifier assigns a class using the labels of nearby observations. Their methods differ, but a fair comparison requires the same evaluation observations and the same definition of a positive prediction. The earlier material on classification thresholds explains how a threshold determines a predicted class; here, the focus is on comparing the resulting classifications when the positive class is uncommon.
The confusion matrix records actual classes in rows and predicted classes in columns. Its row totals are the numbers of actual positive and actual negative cases. These totals—not the model's predicted class counts—are the denominators for sensitivity and specificity.
| Predicted positive | Predicted negative | |
|---|---|---|
| Actual positive | TP | FN |
| Actual negative | FP | TN |
Balanced Accuracy and Its Rationale
Balanced accuracy gives the two classes equal weight, regardless of their frequencies in the evaluation data. It is the arithmetic mean of sensitivity and specificity:
The equal weighting is the defining feature. Ordinary accuracy weights each correct classification according to how often its actual class occurs. Balanced accuracy instead summarizes how often the model is correct within each class, then averages those two rates. This makes it useful for diagnosing a model that performs well on the majority class but poorly on the minority class.
The relationship to class-specific error rates follows directly from their definitions. Since sensitivity is one minus the false-negative rate, and specificity is one minus the false-positive rate, substituting into the balanced-accuracy formula gives:
Thus, balanced accuracy is one minus the average of the two class-specific error rates. This identity explains why a large error rate on the minority class cannot be hidden merely because that class is rare. It does not, by itself, determine which errors matter more in a particular decision; that depends on the application.
Worked Example: Comparing a Tree and Neighbors on One Test Set
A hypothetical insurer evaluates a tree and a \(k\)-nearest neighbors classifier on the same 500 policies. An unusually large claim is the positive outcome. There are 25 positive and 475 negative outcomes. The tree has TP \(=15\), FN \(=10\), TN \(=425\), and FP \(=50\). The neighbor classifier has TP \(=20\), FN \(=5\), TN \(=380\), and FP \(=95\). Compare accuracy and balanced accuracy. Use exact rates in intermediate calculations and round final rates to four decimal places.
Check the totals. The actual positive total is \(15+10=25\) for the tree and \(20+5=25\) for the neighbor classifier. The actual negative total is \(425+50=475\) for the tree and \(380+95=475\) for the neighbor classifier. Each model is evaluated on all 500 policies.
Tree accuracy. The tree classifies \(15+425=440\) policies correctly, so its accuracy is \(440/500=0.88\).
Tree class-specific rates. Sensitivity is \(15/25=0.6000\). Specificity is \(425/475=17/19=0.8947368\ldots\). Using those exact fractions,
The tree's balanced accuracy is \(0.7474\) to four decimals.
Neighbor-classifier accuracy. It classifies \(20+380=400\) policies correctly, so its accuracy is \(400/500=0.80\).
Neighbor-classifier class-specific rates. Sensitivity is \(20/25=0.80\), and specificity is \(380/475=0.80\). Its balanced accuracy is \((0.80+0.80)/2=0.8000\).
The tree has the higher overall accuracy, \(0.88\) versus \(0.80\), while the neighbor classifier has the higher balanced accuracy, \(0.8000\) versus \(0.7474\). The tree misses \(10\) of the \(25\) positive cases, whereas the neighbor classifier misses \(5\). The neighbor classifier also has more false positives, \(95\) versus \(50\). Method and mistake avoided: compare both class-specific rates on the same evaluation set, rather than selecting the tree from accuracy alone. Do not round specificity before using it to calculate balanced accuracy.
Why the Majority-Class Rule Can Mislead
A majority-class rule always predicts the more common class. When the positive class is rare, this rule can achieve high accuracy while having zero sensitivity. It is an important reference point: a candidate model should be assessed against the problem's objectives, not assumed useful just because its accuracy is high.
Worked Example: High Accuracy With No Positive Detection
A hypothetical monitoring program evaluates 1,000 accounts, of which 40 have a specified adverse event and 960 do not. A rule predicts “no event” for every account. A tree instead produces TP \(=24\), FN \(=16\), TN \(=864\), and FP \(=96\). Calculate accuracy and balanced accuracy for both rules, and explain what the comparison shows.
All-negative rule. It predicts all 1,000 accounts as negative. Thus TP \(=0\), FN \(=40\), TN \(=960\), and FP \(=0\). Its accuracy is \((0+960)/1{,}000=0.9600\). Its sensitivity is \(0/40=0\), and its specificity is \(960/960=1\). Therefore its balanced accuracy is \((0+1)/2=0.5000\).
Tree. Its actual positive count is \(24+16=40\), and its actual negative count is \(864+96=960\). Its accuracy is \((24+864)/1{,}000=888/1{,}000=0.8880\). Sensitivity is \(24/40=0.6000\), and specificity is \(864/960=0.9000\). Its balanced accuracy is \((0.6000+0.9000)/2=0.7500\).
The all-negative rule has greater accuracy, \(0.9600\), because it correctly labels all 960 common negative cases. Yet it identifies none of the 40 positive cases. The tree has lower accuracy but detects 24 positive cases and has balanced accuracy \(0.7500\). Method and mistake avoided: calculate sensitivity and specificity as well as accuracy. The high accuracy of the all-negative rule is not evidence that it can screen for the event.
When the Metrics Disagree
A metric summarizes performance; it does not make the decision objective disappear. A tree and a neighbor classifier can have the same balanced accuracy while making different kinds of errors. In that situation, compare their sensitivity and specificity directly. If false negatives and false positives have different consequences, the preferred model can depend on those consequences, even when one model has higher accuracy.
Worked Example: Equal Balanced Accuracy, Different Error Profiles
A hypothetical review team tests two classifiers on 200 records: 20 are positive and 180 are negative. The tree yields TP \(=14\), FN \(=6\), TN \(=162\), and FP \(=18\). The neighbor classifier yields TP \(=16\), FN \(=4\), TN \(=144\), and FP \(=36\). Compare the models, rounding final rates to four decimals.
Tree. It classifies \(14+162=176\) records correctly, so accuracy is \(176/200=0.8800\). Sensitivity is \(14/20=0.7000\). Specificity is \(162/180=0.9000\). Balanced accuracy is \((0.7000+0.9000)/2=0.8000\).
Neighbor classifier. It classifies \(16+144=160\) records correctly, so accuracy is \(160/200=0.8000\). Sensitivity is \(16/20=0.8000\). Specificity is \(144/180=0.8000\). Balanced accuracy is \((0.8000+0.8000)/2=0.8000\).
The balanced accuracies are equal, but the error profiles differ: the neighbor classifier misses \(4\) positives rather than \(6\), while the tree produces \(18\) false positives rather than \(36\). The tree has higher ordinary accuracy because it correctly classifies more negative records. Method and mistake avoided: report the component rates along with balanced accuracy. Declaring a universal winner from the tied summary metric would discard information that may matter to the decision.
Common Exam Traps
- Using the predicted-positive count as the sensitivity denominator. Sensitivity is \(TP/(TP+FN)\), using all actual positives. The predicted-positive total is \(TP+FP\), which is a different quantity.
- Relying on accuracy under imbalance. A high accuracy can result from correctly classifying the majority class while missing most or all minority cases.
- Rounding a component rate too early. In the first worked example, the tree's exact specificity is \(425/475=17/19\), not the rounded decimal \(0.8947\). Its balanced accuracy is calculated from exact rates as \((15/25+425/475)/2=71/95=0.7473684\ldots\), or \(0.7474\) to four decimals. Averaging a rounded specificity can produce a slightly different intermediate result.
- Comparing models on different evaluation cases. Differences in class mix or case difficulty can confound the comparison. Use the same evaluation set and the same positive-class definition when comparing classifiers.
- Treating balanced accuracy as a decision rule. It gives the two classes equal weight; it does not encode a specific financial or operational cost for each kind of error.
Check Your Understanding
For each problem, identify the positive class and use the stated evaluation counts. Round final rates to four decimals where needed.
- A hypothetical detector is tested on 250 cases, including 20 positives and 230 negatives. It has TP \(=12\), FN \(=8\), TN \(=207\), and FP \(=23\). Calculate accuracy, sensitivity, specificity, and balanced accuracy. (Answer: accuracy \(=0.8760\), sensitivity \(=0.6000\), specificity \(=0.9000\), balanced accuracy \(=0.7500\).)
- Among 500 hypothetical records, 30 are positive and 470 are negative. A rule predicts every record negative. Calculate its accuracy and balanced accuracy, and state its sensitivity. (Answer: accuracy \(=0.9400\), balanced accuracy \(=0.5000\), sensitivity \(=0.0000\).)
- On the same 400 hypothetical cases, there are 40 positives and 360 negatives. A tree has TP \(=28\), FN \(=12\), TN \(=324\), FP \(=36\); a neighbor classifier has TP \(=32\), FN \(=8\), TN \(=288\), FP \(=72\). Calculate each model's accuracy and balanced accuracy. (Answer: tree accuracy \(=0.8800\) and balanced accuracy \(=0.8000\); neighbor accuracy \(=0.8000\) and balanced accuracy \(=0.8000\).)
- A team evaluates two models on 200 cases, with 16 positives and 184 negatives. The tree has TP \(=12\), FN \(=4\), TN \(=166\), FP \(=18\). The neighbor classifier has TP \(=14\), FN \(=2\), TN \(=147\), FP \(=37\). Calculate both accuracies and balanced accuracies. Which model has higher accuracy, and which has higher balanced accuracy? (Answer: tree accuracy \(=0.8900\), balanced accuracy \(=19/23=0.8261\); neighbor accuracy \(=0.8050\), balanced accuracy \(=77/92=0.8370\); the tree has higher accuracy and the neighbor classifier has higher balanced accuracy.)
- A model has sensitivity \(=0.70\) and specificity \(=0.90\) on an evaluation set containing both actual classes. Calculate its false-negative rate, false-positive rate, and balanced accuracy. Explain why the positive-class prevalence is not needed for this calculation. (Answer: false-negative rate \(=0.3000\), false-positive rate \(=0.1000\), balanced accuracy \(=0.8000\); balanced accuracy averages the two class-specific rates, so it does not require their class counts once those rates are known.)