Evaluating Classifiers Under Imbalance

Why Class Imbalance Changes the Evaluation

In multi-omics prediction, a clinically important outcome may be uncommon in the population where a classifier will be used. A test set can therefore contain many more negative than positive cases. Under this imbalance, a classifier that mostly predicts the common class can achieve high accuracy while missing many positive cases. Conversely, a model with useful sensitivity and specificity may have low precision when the positive outcome is rare: many flagged cases can be false positives simply because there are so few true cases to find.

The distinction to keep in view is between class-conditional performance and prevalence-sensitive performance. Sensitivity and specificity describe prediction behavior within the true positive and true negative groups. Precision, negative predictive value, and accuracy combine that behavior with the proportion of each group in the evaluated population. The prevalence used for interpretation must therefore be stated, not left implicit.

Confusion Counts and Metric Definitions

Fix a binary outcome, a prediction threshold, and a held-out set. Let \(Y=1\) denote the positive outcome and let a positive prediction mean that the classifier's score meets or exceeds the prespecified threshold. The four confusion-matrix counts are:

True positives (TP): actual positives classified as positive.
False negatives (FN): actual positives classified as negative.
False positives (FP): actual negatives classified as positive.
True negatives (TN): actual negatives classified as negative.

The total number of held-out observations is \(n=TP+FN+FP+TN\). The observed positive fraction is \((TP+FN)/n\); it estimates prevalence in the population represented by that test set, but it need not equal the prevalence in a target clinical population.

Definition: For a specified outcome, threshold, and evaluation population, sensitivity is \(TP/(TP+FN)\), the fraction of actual positives classified positive. Specificity is \(TN/(TN+FP)\), the fraction of actual negatives classified negative. Precision, also called positive predictive value (PPV), is \(TP/(TP+FP)\), the fraction of positive predictions that are truly positive. Negative predictive value (NPV) is \(TN/(TN+FN)\). Accuracy is \((TP+TN)/n\). Balanced accuracy is \((\text{sensitivity}+\text{specificity})/2\). The F1 score is \(2TP/(2TP+FP+FN)\), equivalently the harmonic mean of precision and sensitivity when both are defined.

These ratios require the relevant denominators to be nonzero. For example, if there are no actual positives, sensitivity cannot be estimated from that test set; if the classifier makes no positive predictions, precision is undefined. Such cases should be reported as undefined rather than silently assigned a value.

Sensitivity and specificity are conditional on the true class. Precision and NPV are conditional on the prediction. This change in conditioning is central: the latter pair depends on how common the true classes are in the population being considered.

Why Precision Depends on Prevalence

Let \(\pi=P(Y=1)\) be the positive-outcome prevalence in the population of interest. Suppose the classifier's sensitivity is \(Se=P(\widehat{Y}=1\mid Y=1)\) and its specificity is \(Sp=P(\widehat{Y}=0\mid Y=0)\) at the fixed threshold. Then the probability of a true positive prediction is \(Se\pi\), while the probability of a false positive prediction is \((1-Sp)(1-\pi)\). Dividing the true-positive probability by the probability of any positive prediction gives:

PPV(\pi)=P(Y=1\mid \widehat{Y}=1) =\frac{Se\pi}{Se\pi+(1-Sp)(1-\pi)}.

The same reasoning yields the negative predictive value:

NPV(\pi)=\frac{Sp(1-\pi)}{(1-Se)\pi+Sp(1-\pi)}.

These are applications of the definition of conditional probability. In the PPV expression, the numerator is the fraction of the population that is both positive and correctly flagged. The denominator adds that fraction to the fraction of the population that is negative but incorrectly flagged. If prevalence falls while sensitivity and specificity stay fixed, false positives can make up a larger share of all positive predictions.

Key result: If sensitivity and specificity are stable across populations, PPV and NPV can be recalculated for a new prevalence using the formulas above. This is not a guarantee that sensitivity and specificity really are stable: differences in disease stage, case mix, measurement, or clinical workflow can change class-conditional performance.

Worked Examples

Worked Example: Reading a Held-Out Confusion Matrix

Suppose a hypothetical classifier is evaluated once on 1,000 held-out patients. The outcome is present in 50 patients. The classifier correctly flags 40 of them and incorrectly flags 90 of the 950 patients without the outcome. Thus \(TP=40\), \(FN=10\), \(FP=90\), and \(TN=860\); the counts sum to 1,000.

Sensitivity is \(40/(40+10)=40/50=0.800\), or 80.0%. Specificity is \(860/(860+90)=860/950=0.9053\), or 90.5%. Precision is \(40/(40+90)=40/130=0.3077\), or 30.8%. NPV is \(860/(860+10)=860/870=0.9885\), or 98.8%.

Accuracy is \((40+860)/1{,}000=0.900\), or 90.0%. Balanced accuracy is \((0.800+0.9053)/2=0.8526\), or 85.3%. The F1 score is \(2(40)/(2(40)+90+10)=80/180=0.4444\), or 44.4%.

The test-set prevalence is \(50/1{,}000=5.00\%\). In this hypothetical held-out population, a positive prediction corresponds to an outcome in about 30.8% of flagged patients, despite 80.0% sensitivity. The high NPV also does not establish that the classifier is suitable for ruling out the outcome: the sensitivity shows that it missed 10 of the 50 positive cases.

Worked Example: Recalculating Precision at a Target Prevalence

Suppose a hypothetical classifier has held-out sensitivity \(Se=0.800\) and specificity \(Sp=0.9053\), as in the preceding calculation. The target clinical population has a positive prevalence of \(\pi=0.010\), or 1.00%. If class-conditional performance transports to that population, its expected precision is:

PPV(0.010) =\frac{0.800(0.010)} {0.800(0.010)+(1-0.9053)(1-0.010)} =\frac{0.00800}{0.00800+0.09375} \approx 0.0786.

The estimated PPV is about 7.86%, not the 30.8% observed in the 5%-prevalence test set. To see the scale in a hypothetical target population of 10,000 people, 1% prevalence means 100 actual positives and 9,900 actual negatives. At the stated sensitivity, about 80 positives would be flagged and 20 missed. At the stated specificity, the false-positive rate is \(1-0.9053=0.0947\), so about \(9{,}900(0.0947)=937.5\) negatives would be flagged falsely. The expected precision is therefore \(80/(80+937.5)=0.0786\). These are expected counts, so they need not be integers.

This calculation is conditional on the transport assumption: the target population has the specified prevalence, and the classifier's sensitivity and specificity remain the same there. If the target population differs in ways that change those rates, using prevalence alone cannot correct the evaluation.

Worked Example: High Accuracy From Predicting the Majority Class

In a hypothetical held-out set of 10,000 patients, 100 have the positive outcome and 9,900 do not. Consider a classifier that predicts negative for everyone. Its counts are \(TP=0\), \(FN=100\), \(FP=0\), and \(TN=9{,}900\).

Accuracy is \((0+9{,}900)/10{,}000=0.990\), or 99.0%. Specificity is \(9{,}900/9{,}900=1.00\), or 100%. But sensitivity is \(0/100=0\%\), and balanced accuracy is \((0+1.00)/2=0.500\), or 50.0%. Precision is undefined because the classifier made no positive predictions; it is not 0% or 100%.

The 99.0% accuracy is exactly the fraction of patients in the majority class. It does not show useful detection of the outcome: the classifier misses every positive case. This example is why accuracy should be read alongside class-specific rates and the intended use of the predictions.

What the Metrics Can and Cannot Compare

Sensitivity and specificity are often useful for comparing threshold-specific behavior across populations because their definitions condition on the true class. But they are not immune to population differences. For example, if a target population contains milder cases than the test set, sensitivity may change; if its negative cases include conditions that resemble the outcome, specificity may change. A prevalence adjustment to PPV or NPV is defensible only when the class-conditional rates used in the calculation are relevant to the target.

Accuracy weights sensitivity by prevalence and specificity by the prevalence of negatives:

\text{Accuracy}(\pi)=Se\pi+Sp(1-\pi).

Thus accuracy changes with prevalence even if both class-conditional rates remain fixed. Balanced accuracy averages those two rates with equal weight, so it does not use the population class proportions directly. It can make a majority-class-only classifier's weakness visible, but it does not express the clinical consequences of false positives and false negatives.

F1 combines sensitivity and precision, and therefore depends on prevalence through precision. It excludes true negatives from its formula. It may be useful for some ranking or operational goals, but it is not a universal replacement for accuracy or balanced accuracy. Precision-recall summaries also depend on prevalence; a precision-recall curve from one population should not be treated as if its precision values automatically apply to a population with a different positive fraction. ROC summaries use sensitivity and false-positive rate across thresholds, but neither a ROC summary nor one threshold-specific pair of rates supplies the target population's PPV by itself.

Evaluation Practice and Failure Modes

Before interpreting a confusion matrix, verify that the outcome definition, positive class, prediction threshold, and held-out evaluation set are clear. The threshold should be chosen without using the held-out outcomes; otherwise the reported performance may be optimistic. The held-out subjects should also be independent at the unit relevant to the prediction task, and preprocessing or feature selection must not use held-out outcome information. These conditions make the estimates an honest evaluation of the specified pipeline, but do not make a nonrepresentative test set representative of a target population.

Report the confusion counts alongside the rates. Counts reveal how many positive cases support the sensitivity estimate and how many negative cases support specificity. State the test-set prevalence and, if the clinical question concerns a different population, state that target prevalence and the assumption behind any recalculation. In a multi-omics setting, where a cohort can be selected or enriched for cases, the test-set class fraction may differ substantially from the fraction encountered in routine use.

Accuracy treated as evidence of case detection: a high value may be mostly correct classification of the majority class. Check sensitivity and the class counts.
Precision transported without adjustment: a test-set PPV is tied to its prevalence. Recalculate it for an explicit target prevalence, assuming relevant sensitivity and specificity transfer.
Prevalence adjustment treated as a cure for population shift: changing \(\pi\) in the formula does not repair altered sensitivity or specificity. Investigate whether target case mix and measurement conditions are comparable.
Undefined rates filled in by convention: if a denominator is zero, report the metric as undefined and give the underlying counts.
Thresholds changed after inspecting the held-out labels: the resulting confusion matrix no longer gives an unbiased evaluation of a prespecified classifier. Threshold choice and held-out assessment must be kept separate.

Common mistake: “The model has 99% accuracy, so it detects the rare outcome well.” This conclusion is not justified: a classifier that predicts the majority class for everyone can attain 99% accuracy when prevalence is 1%, while having zero sensitivity. Report class-specific rates, predictive values for the relevant prevalence, and the confusion counts.

Check Your Understanding

Answer each question, then compare with the solutions below. Assume rates are evaluated at a fixed threshold unless stated otherwise.

A held-out set has \(TP=24\), \(FN=6\), \(FP=30\), and \(TN=540\). Calculate sensitivity, specificity, precision, and test-set prevalence.
A classifier has sensitivity 0.90 and specificity 0.95. Calculate its PPV at prevalence 0.20 and at prevalence 0.01, assuming those class-conditional rates apply in both populations.
Why can a classifier have high accuracy but low balanced accuracy when the positive outcome is rare?
If no actual positive cases occur in a held-out set, which metric cannot be estimated from its confusion matrix, and why?

Solutions. (1) Sensitivity is \(24/(24+6)=0.800\). Specificity is \(540/(540+30)=0.9474\). Precision is \(24/(24+30)=0.4444\). Prevalence is \((24+6)/(24+6+30+540)=30/600=0.0500\), or 5.00%.

(2) At prevalence 0.20, \(PPV=0.90(0.20)/[0.90(0.20)+0.05(0.80)]=0.18/0.22=0.8182\). At prevalence 0.01, \(PPV=0.90(0.01)/[0.90(0.01)+0.05(0.99)]=0.009/0.0585=0.1538\). The lower PPV at lower prevalence follows from the larger share of false positives among positive predictions.

(3) Accuracy weights correct positives and correct negatives by their population frequencies. When negatives dominate, predicting most cases negative can yield high accuracy. Balanced accuracy gives sensitivity and specificity equal weight, so poor sensitivity lowers it even when specificity is high.

(4) Sensitivity cannot be estimated because its denominator, \(TP+FN\), is zero. The set contains no observed positive cases with which to measure the fraction detected.