Prevalence Shifts and Misleading Accuracy

Why Accuracy Can Mislead

A classifier can appear highly accurate simply because it usually predicts the more common class. This is especially important when the outcome of interest is rare. A hypothetical screening model, for example, might correctly classify many people who do not have a condition while missing a large share of those who do. Its overall accuracy can still look impressive because the non-condition group dominates the population.

To interpret performance, keep the positive class explicit and examine the class prevalence alongside the confusion matrix. Accuracy is one useful summary, but it does not say which class was classified correctly. Sensitivity, specificity, and precision help answer that question from different perspectives.

Definition: Let the positive class be the outcome of interest in a population of size \(N\). Its prevalence is \(\pi=P(Y=1)\), where \(Y=1\) denotes a positive outcome. Sensitivity is the proportion of actual positives classified as positive, and specificity is the proportion of actual negatives classified as negative. Accuracy is the proportion of all cases classified correctly. Precision (also called positive predictive value) is the proportion of positive predictions that are actual positives.

Write sensitivity as \(Se\) and specificity as \(Sp\). The confusion-matrix counts implied by a population with prevalence \(\pi\) are:

\begin{aligned} \text{Actual positives} &= N\pi, & TP &= N\pi Se, & FN &= N\pi(1-Se),\\ \text{Actual negatives} &= N(1-\pi), & TN &= N(1-\pi)Sp, & FP &= N(1-\pi)(1-Sp). \end{aligned}

Here \(TP\), \(FN\), \(TN\), and \(FP\) denote true positives, false negatives, true negatives, and false positives. If the rates do not imply whole-number counts, these are expected counts, not a literal partition of a finite data set. In an observed sample, the counts are integers and the corresponding rates may differ slightly from the stated or estimated rates.

How Prevalence Enters the Measures

Accuracy combines correct classifications from both classes. Precision focuses only on predictions labelled positive. Their population formulas are:

\begin{aligned} \text{Accuracy} &= \pi Se + (1-\pi)Sp,\\ \text{Precision} &= \frac{\pi Se}{\pi Se+(1-\pi)(1-Sp)}. \end{aligned}

The precision formula requires a nonzero probability of a positive prediction. If the model never predicts positive, precision is undefined: there are no positive predictions over which to calculate a proportion. Prevalence also affects the negative predictive value, the proportion of negative predictions that are actual negatives, but the central issue here is the interpretation of positive detection.

Why the formulas hold: Among all cases, the fraction that are both actual positives and predicted positive is \(\pi Se\). The fraction that are actual negatives and predicted negative is \((1-\pi)Sp\). Adding these disjoint groups gives the accuracy formula. Positive predictions consist of true positives, with fraction \(\pi Se\), and false positives, with fraction \((1-\pi)(1-Sp)\). Dividing the true-positive fraction by the total positive-prediction fraction gives precision. Thus, changing \(\pi\) changes these population measures even when \(Se\) and \(Sp\) are held fixed.

This result explains a seeming paradox: a classifier's accuracy can rise as prevalence falls even though its sensitivity and specificity have not changed. When positives become rarer, correct negative classifications make up more of the population. Precision can fall sharply at the same time because the supply of actual positives shrinks while false positives may remain numerous.

Worked Example: Strong Accuracy but Modest Precision

A hypothetical screening program evaluates 8,000 people. The outcome prevalence is 0.5%, and a classifier has sensitivity 75% and specificity 99%. Find the expected confusion counts, accuracy, and precision. The method is to calculate class sizes first, then apply the conditional rates within each class.

1. Find the number in each actual class. The positive group has \(8{,}000(0.005)=40\) people. The negative group has \(8{,}000(0.995)=7{,}960\) people.

2. Calculate the expected confusion counts. True positives are \(40(0.75)=30\), and false negatives are \(40(0.25)=10\). False positives are \(7{,}960(1-0.99)=7{,}960(0.01)=79.6\). True negatives are \(7{,}960(0.99)=7{,}880.4\). The fractional negative-class counts are expected values.

3. Calculate accuracy. Correct classifications total \(30+7{,}880.4=7{,}910.4\), so accuracy is \(7{,}910.4/8{,}000=0.9888\), or 98.88%.

4. Calculate precision. There are \(30+79.6=109.6\) expected positive predictions. Precision is \(30/109.6=0.2737226\ldots\), or 27.37% to two decimal places.

The model is highly accurate overall, but fewer than three in ten of its positive predictions are true positives under these assumptions. The common mistake is to treat 99% specificity as meaning that 99% of positive predictions are correct. Specificity is conditional on actually being negative; precision is conditional on being predicted positive. The expected counts check: \(30+10=40\), and \(79.6+7{,}880.4=7{,}960\).

Prevalence Shifts Change the Interpretation

Suppose sensitivity and specificity are the same in two populations, but prevalence differs. Their accuracy and precision need not match. This can happen when a validation sample has a different class mix from the population where predictions will be used. A case-control sample, for example, may deliberately include more positive cases than occur in the target population. Its raw accuracy and precision describe that sample's prevalence, not automatically the target population's.

The formulas above can be used to calculate the measures at a specified target prevalence, provided the sensitivity and specificity are appropriate for that target setting. Holding those rates constant is an assumption for the calculation, not a guarantee: differences in case mix, measurement, or operating conditions can also change sensitivity and specificity.

Worked Example: Same Conditional Rates, Different Prevalence

A hypothetical classifier has sensitivity 90% and specificity 96%. Compare its accuracy and precision in two populations, one with 8% prevalence and one with 1% prevalence. Use a population proportion calculation; the rates are assumed to remain the same in both populations.

At 8% prevalence: The true-positive fraction is \(0.08(0.90)=0.072\), and the false-negative fraction is \(0.08(0.10)=0.008\). The false-positive fraction is \(0.92(0.04)=0.0368\), and the true-negative fraction is \(0.92(0.96)=0.8832\). Accuracy is \(0.072+0.8832=0.9552\), or 95.52%. Precision is \(0.072/(0.072+0.0368)=0.072/0.1088=0.6617647\ldots\), or 66.18% to two decimal places.

At 1% prevalence: The true-positive fraction is \(0.01(0.90)=0.009\), and the false-negative fraction is \(0.01(0.10)=0.001\). The false-positive fraction is \(0.99(0.04)=0.0396\), and the true-negative fraction is \(0.99(0.96)=0.9504\). Accuracy is \(0.009+0.9504=0.9594\), or 95.94%. Precision is \(0.009/(0.009+0.0396)=0.009/0.0486=0.185185\ldots\), or 18.52% to two decimal places.

Accuracy is slightly higher in the lower-prevalence population because the large negative class is classified correctly at a high rate. Precision is much lower because there are fewer actual positives to contribute true positives, while false positives remain possible among the much larger negative class. Equal sensitivity and specificity therefore do not imply equal precision.

Use More Than Overall Accuracy

For a rare positive outcome, report the class-specific quantities needed to understand performance. Sensitivity describes how many actual positives are detected; specificity describes how many actual negatives are correctly rejected; precision describes how often a positive prediction is right at the stated prevalence. The confusion matrix makes the underlying error counts visible. If a single summary is required, balanced accuracy is sometimes useful:

\text{Balanced accuracy}=\frac{Se+Sp}{2}.

Balanced accuracy gives equal weight to sensitivity and specificity, rather than weighting each class according to how common it is. It can reveal weak positive-class detection hidden by ordinary accuracy. It is not a replacement for all other measures: it does not report the fraction of positive predictions that are correct, and it does not encode the relative consequences of false positives and false negatives. The appropriate set of measures depends on the decision being supported.

Worked Example: A Rare-Class Baseline

A hypothetical monitoring system classifies 5,000 events, of which 2% are positive. Compare a rule that predicts every event negative with a second classifier having sensitivity 80% and specificity 90%. Calculate accuracy and balanced accuracy for both, and precision for the second classifier.

Rule that predicts every event negative: There are \(5{,}000(0.02)=100\) actual positives and \(4{,}900\) actual negatives. It has \(TP=0\), \(FN=100\), \(FP=0\), and \(TN=4{,}900\). Accuracy is \(4{,}900/5{,}000=98\%\). Sensitivity is \(0/100=0\%\), specificity is \(4{,}900/4{,}900=100\%\), and balanced accuracy is \((0+1)/2=50\%\). Precision is undefined because there are no positive predictions.

Classifier with sensitivity 80% and specificity 90%: It has \(TP=100(0.80)=80\) and \(FN=100(0.20)=20\). Among the 4,900 actual negatives, \(FP=4{,}900(0.10)=490\) and \(TN=4{,}900(0.90)=4{,}410\). Accuracy is \((80+4{,}410)/5{,}000=4{,}490/5{,}000=89.8\%\). Balanced accuracy is \((0.80+0.90)/2=85\%\). Precision is \(80/(80+490)=80/570=0.1403509\ldots\), or 14.04% to two decimal places.

The always-negative rule has higher accuracy but detects no positives. The second classifier detects 80 of the 100 actual positives, but many of its positive predictions are false positives. Neither accuracy nor balanced accuracy alone answers every operational question. The counts check: the second classifier has \(80+20=100\) actual positives and \(490+4{,}410=4{,}900\) actual negatives.

Reporting Performance Under a Prevalence Shift

When presenting accuracy, state the population or sample prevalence to which it applies. When positive outcomes are rare, pair overall accuracy with at least sensitivity and specificity, and include precision when the reliability of positive predictions matters. A confusion matrix or clearly labelled expected counts can help readers see how many errors each measure summarizes.

If a validation set has prevalence \(\pi_{\text{val}}\) but the target population has prevalence \(\pi_{\text{target}}\), do not simply carry the validation accuracy or precision over unchanged. If sensitivity and specificity are valid for the target setting, recompute the target-population summaries using \(\pi_{\text{target}}\). If those conditional rates may also change, the available evidence does not justify a prevalence-only adjustment; report that uncertainty rather than implying the recalculated metrics are established performance.

A frequent exam trap is to compare two accuracy values without checking their class prevalences. A second is to call a high accuracy proof of strong detection of the rare class. In both cases, return to the class-specific counts or rates, identify the prevalence, and state precisely which conditional question each measure answers. This interpretation is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

For each problem, identify the role of prevalence and distinguish overall accuracy from positive-class detection.

A hypothetical classifier is used on 10,000 cases with prevalence 2%, sensitivity 80%, and specificity 95%. Find the expected confusion counts, accuracy, and precision. (Answer: \(TP=160\), \(FN=40\), \(FP=490\), \(TN=9{,}310\); accuracy \(=94.70\%\); precision \(=24.62\%\).)
A hypothetical screening model has prevalence 0.2%, sensitivity 70%, and specificity 99.5% in 50,000 cases. Find the expected counts and accuracy and precision, rounding rates to two decimal places. (Answer: \(TP=70\), \(FN=30\), \(FP=249.5\), \(TN=49{,}650.5\); accuracy \(=99.44\%\); precision \(=21.91\%\).)
A rule predicts every case negative in a population with prevalence 3%. Find its accuracy, sensitivity, specificity, balanced accuracy, and precision. (Answer: accuracy \(=97\%\); sensitivity \(=0\%\); specificity \(=100\%\); balanced accuracy \(=50\%\); precision is undefined because there are no positive predictions.)
A classifier has sensitivity 85% and specificity 97%. Calculate accuracy and precision when prevalence is 10%, then when it is 2%. (Answer: At 10%, accuracy \(=95.80\%\) and precision \(=75.89\%\); at 2%, accuracy \(=96.76\%\) and precision \(=36.64\%\).)
A validation sample has prevalence 2%, sensitivity 80%, and specificity 90%. The target population has prevalence 5%. Assuming sensitivity and specificity transfer unchanged, calculate target accuracy and precision. Also state the validation-sample accuracy. (Answer: target accuracy \(=89.50\%\), target precision \(=29.63\%\), and validation-sample accuracy \(=89.80\%\).)

What You'll Learn