Prevalence Effects on Classification Measures

Why Prevalence Changes the Meaning of a Classification Metric

A classification model may be used to flag a rare event, such as a hypothetical claim requiring additional review. Even if the model’s ability to detect events and avoid false alarms stays the same, the fraction of cases that actually contain the event can change. That change affects how much a positive flag should be trusted and can also change overall accuracy.

The key is to distinguish measures calculated within the event and non-event groups from measures that combine those groups. Sensitivity and specificity are conditional on the actual outcome. Precision and accuracy depend on how many cases belong to each outcome group, so they can move when prevalence changes.

Definition: Let \(Y=1\) denote that the event occurs and \(Y=0\) that it does not. Let \(\hat{Y}=1\) denote a positive model classification and \(\hat{Y}=0\) a negative classification at a specified threshold. The prevalence is \(\pi=P(Y=1)\). Sensitivity is \(P(\hat{Y}=1\mid Y=1)\), and specificity is \(P(\hat{Y}=0\mid Y=0)\). Precision, also called positive predictive value, is \(P(Y=1\mid\hat{Y}=1)\). Negative predictive value is \(P(Y=0\mid\hat{Y}=0)\). Accuracy is \(P(\hat{Y}=Y)\).

The same quantities can be described with a confusion matrix. A true positive (TP) is an event case classified positive; a false negative (FN) is an event case classified negative. A true negative (TN) is a non-event case classified negative; a false positive (FP) is a non-event case classified positive. Thus precision is TP divided by all positive classifications, while accuracy is the fraction of all classifications that are correct.

\text{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}},\qquad \text{NPV}=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FN}},\qquad \text{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{N}.

These formulas require the relevant denominator to be nonzero. For example, precision is undefined if the model makes no positive classifications. In practice, report that condition rather than treating the undefined value as zero.

How Prevalence Enters the Formulas

Write sensitivity as \(Se\) and specificity as \(Sp\). The false-positive rate is \(1-Sp\), because among cases with \(Y=0\), every case is either classified negative or classified positive. For a population of size \(N\), the expected proportions in the confusion matrix are determined by prevalence and the two conditional rates.

\begin{aligned} \frac{\mathrm{TP}}{N}&=\pi Se, & \frac{\mathrm{FN}}{N}&=\pi(1-Se),\\ \frac{\mathrm{TN}}{N}&=(1-\pi)Sp, & \frac{\mathrm{FP}}{N}&=(1-\pi)(1-Sp). \end{aligned}

For example, the positive group consists of true positives from the event population and false positives from the non-event population. Its size as a proportion of all cases is \(\pi Se+(1-\pi)(1-Sp)\). Dividing the true-positive proportion by that total gives precision. This is Bayes’ rule expressed using the classification rates.

\text{Precision} =\frac{\pi Se}{\pi Se+(1-\pi)(1-Sp)}.

The same count argument gives negative predictive value and accuracy:

\text{NPV} =\frac{(1-\pi)Sp}{(1-\pi)Sp+\pi(1-Se)},\qquad \text{Accuracy}=\pi Se+(1-\pi)Sp.

These equations assume the prevalence, sensitivity, and specificity refer to the same population, outcome definition, classification threshold, and observation period. They also show a useful distinction: if \(Se\) and \(Sp\) truly remain fixed, a prevalence change alone does not alter either conditional rate, but it does alter precision, NPV, and accuracy.

Key result: At fixed sensitivity and specificity, increasing prevalence increases precision and decreases NPV. Accuracy changes at a rate determined by the difference between sensitivity and specificity: as prevalence rises, accuracy rises if \(Se>Sp\), falls if \(Se<Sp\), and stays constant if \(Se=Sp\).

Worked Examples

Worked Example: A Rare Event Becomes More Common

A hypothetical review model flags claims for additional investigation. At one threshold, its sensitivity is \(0.80\) and its specificity is \(0.95\). Compare two hypothetical populations of 10,000 claims: one has event prevalence \(2\%\), and the other has prevalence \(8\%\). Assume the same sensitivity and specificity apply in both populations. Calculate precision and accuracy in each.

Method. Convert prevalence into event and non-event counts, then apply sensitivity to event cases and specificity to non-event cases. This avoids treating precision as though it were sensitivity.

Population 1: prevalence \(2\%\). There are \(10{,}000(0.02)=200\) event claims and \(10{,}000-200=9{,}800\) non-event claims. The true positives are \(200(0.80)=160\), so false negatives are \(200-160=40\). The true negatives are \(9{,}800(0.95)=9{,}310\), so false positives are \(9{,}800-9{,}310=490\).

The model makes \(160+490=650\) positive classifications. Precision is \(160/650=0.2462\), or \(24.62\%\) to two decimal places. Accuracy is \((160+9{,}310)/10{,}000=9{,}470/10{,}000=0.9470\), or \(94.70\%\).

Population 2: prevalence \(8\%\). There are \(10{,}000(0.08)=800\) event claims and \(9{,}200\) non-event claims. The true positives are \(800(0.80)=640\), and the false negatives are \(800-640=160\). The true negatives are \(9{,}200(0.95)=8{,}740\), and the false positives are \(9{,}200-8{,}740=460\).

There are \(640+460=1{,}100\) positive classifications, so precision is \(640/1{,}100=0.5818\), or \(58.18\%\) to two decimal places. Accuracy is \((640+8{,}740)/10{,}000=9{,}380/10{,}000=0.9380\), or \(93.80\%\).

Check. In both populations, sensitivity is \(0.80\) and specificity is \(0.95\). Precision rises because a positive flag is more likely to come from an event case when events are more common. Accuracy falls because the model is more accurate on non-event cases than on event cases, and the event group has become a larger share of the population. A common mistake is to infer that unchanged sensitivity means unchanged precision; the calculations show why that is false.

Worked Example: Precision, NPV, and Accuracy at Two Prevalence Levels

A hypothetical screening rule has sensitivity \(0.90\) and specificity \(0.80\). Consider two separate populations of 5,000 cases, with event prevalence \(10\%\) and \(30\%\), respectively. For each population, calculate precision, NPV, and accuracy. Assume the conditional rates remain fixed.

At \(10\%\) prevalence: There are \(5{,}000(0.10)=500\) event cases and \(4{,}500\) non-event cases. The true positives are \(500(0.90)=450\), and the false negatives are \(500-450=50\). The true negatives are \(4{,}500(0.80)=3{,}600\), and the false positives are \(4{,}500-3{,}600=900\).

Precision is \(450/(450+900)=450/1{,}350=0.3333\), or \(33.33\%\) to two decimal places. NPV is \(3{,}600/(3{,}600+50)=3{,}600/3{,}650=0.9863\), or \(98.63\%\). Accuracy is \((450+3{,}600)/5{,}000=4{,}050/5{,}000=0.8100\), or \(81.00\%\).

At \(30\%\) prevalence: There are \(5{,}000(0.30)=1{,}500\) event cases and \(3{,}500\) non-event cases. The true positives are \(1{,}500(0.90)=1{,}350\), and the false negatives are \(1{,}500-1{,}350=150\). The true negatives are \(3{,}500(0.80)=2{,}800\), and the false positives are \(3{,}500-2{,}800=700\).

Precision is \(1{,}350/(1{,}350+700)=1{,}350/2{,}050=0.6585\), or \(65.85\%\). NPV is \(2{,}800/(2{,}800+150)=2{,}800/2{,}950=0.9492\), or \(94.92\%\). Accuracy is \((1{,}350+2{,}800)/5{,}000=4{,}150/5{,}000=0.8300\), or \(83.00\%\). Percentages are rounded to two decimal places.

Check. Precision rises and NPV falls as prevalence increases. Accuracy rises here because sensitivity (\(0.90\)) exceeds specificity (\(0.80\)). The common mistake is to say that accuracy must fall whenever prevalence rises. The accuracy formula shows that its direction depends on which conditional rate is larger.

Comparing Periods With Different Case Mix

A raw accuracy comparison can be misleading when periods have different event prevalence. Consider two periods with different prevalence and different conditional classification rates. A useful diagnostic is to calculate each period’s accuracy at a common reference prevalence. This holds the event/non-event mix constant for the calculation, so the comparison focuses on the periods’ sensitivity and specificity rather than their differing prevalence.

This standardization is a comparison tool, not a claim that either period actually had the reference prevalence. It requires a defensible reference prevalence and conditional rates estimated on comparable outcomes and thresholds. It does not by itself eliminate differences in case mix within the event or non-event groups.

Worked Example: Standardizing Accuracy to a Common Prevalence

Two hypothetical periods each contain 10,000 cases. In Period A, prevalence is \(10\%\), sensitivity is \(0.99\), and specificity is \(0.80\). In Period B, prevalence is \(60\%\), sensitivity is \(0.80\), and specificity is \(0.99\). Compare their observed accuracies, then calculate each period’s accuracy standardized to a common prevalence of \(60\%\).

Observed Period A accuracy: The count-based calculation has 1,000 event cases and 9,000 non-event cases. Its true positives are \(1{,}000(0.99)=990\); its true negatives are \(9{,}000(0.80)=7{,}200\). Therefore, accuracy is \((990+7{,}200)/10{,}000=8{,}190/10{,}000=0.8190\), or \(81.90\%\).

Observed Period B accuracy: There are 6,000 event cases and 4,000 non-event cases. The true positives are \(6{,}000(0.80)=4{,}800\); the true negatives are \(4{,}000(0.99)=3{,}960\). Accuracy is \((4{,}800+3{,}960)/10{,}000=8{,}760/10{,}000=0.8760\), or \(87.60\%\). On raw accuracy alone, Period B appears better.

Standardize both to \(60\%\) prevalence: Use the accuracy formula, \(\pi Se+(1-\pi)Sp\). For Period A, the standardized accuracy is \(0.60(0.99)+0.40(0.80)=0.594+0.320=0.9140\), or \(91.40\%\). For Period B, it is \(0.60(0.80)+0.40(0.99)=0.480+0.396=0.8760\), or \(87.60\%\).

Check. At the common prevalence, Period A has the higher standardized accuracy. The reversal occurs because Period A’s high sensitivity is more valuable when event cases make up \(60\%\) of the reference population. This does not prove Period A is preferable for every operational purpose: the relative consequences of false positives and false negatives also matter. The example isolates prevalence standardization, rather than deciding which error is more costly.

Assumptions and Common Traps

The calculations above illustrate a prevalence effect under a controlled comparison: sensitivity and specificity are treated as fixed while prevalence changes. In real monitoring, that condition may not hold. A new period can have different types of event cases or non-event cases, different measurement or labeling processes, or a changed classification threshold. Any of these can alter sensitivity or specificity as well as prevalence. Therefore, a shift in precision is consistent with a prevalence change, but precision alone cannot establish that prevalence was the only cause.

When comparing periods, state the outcome definition, threshold, period, and denominator for each measure. Check whether the event rate changed, and report the conditional rates alongside prevalence-based measures where the data support them. If a period’s labels are incomplete or immature, its observed prevalence and performance measures may not yet be comparable.

A frequent calculation error is to multiply sensitivity by the number of all cases rather than by the number of event cases. For example, with 10,000 cases, \(2\%\) prevalence, and \(80\%\) sensitivity, the correct true-positive count is \(10{,}000(0.02)(0.80)=160\), not \(10{,}000(0.80)=8{,}000\). Sensitivity is conditional on \(Y=1\); it is not the fraction of the entire population flagged. Likewise, specificity applies to the non-event count, not to all cases.

The topic is aligned with the SOA Exam PA (Predictive Analytics) syllabus. The practical lesson is to interpret metrics in context: a percentage without its underlying event rate, outcome definition, and classification threshold can support an incomplete or misleading comparison.

State the classification setup.
Identify the positive outcome, the positive prediction, and the threshold used.

Establish the case mix.
Calculate prevalence as event cases divided by all cases for the population and period being assessed.

Separate conditional and prevalence-based measures.
Interpret sensitivity and specificity within outcome groups; interpret precision, NPV, and accuracy in light of prevalence.

Qualify comparisons.
If prevalence differs, show that difference or use a clearly stated common reference prevalence, while checking whether the conditional rates are comparable.

Check Your Understanding

For each question, use the stated outcome definition and threshold. Round probabilities to four decimal places where needed.

A hypothetical detector has prevalence \(4\%\), sensitivity \(0.75\), and specificity \(0.90\). Calculate precision and accuracy. (Answer: Precision \(=0.2381\); accuracy \(=0.8940\).)
A hypothetical population has 2,000 cases, prevalence \(3\%\), sensitivity \(0.80\), and specificity \(0.95\). Find TP, FN, TN, FP, precision, and accuracy. (Answer: TP \(=48\), FN \(=12\), TN \(=1{,}843\), FP \(=97\); precision \(=0.3310\); accuracy \(=0.9455\).)
A classifier has sensitivity \(0.90\), specificity \(0.90\), and precision \(0.50\). Under the fixed-rate formula, what prevalence is implied? (Answer: \(\pi=0.1000\), or \(10\%\).)
Sensitivity is \(0.70\) and specificity is \(0.95\). Calculate accuracy at prevalence \(2\%\) and \(10\%\), assuming the rates stay fixed. Explain the direction of change. (Answer: Accuracy falls from \(0.9450\) to \(0.9250\), a decrease of \(0.0200\), because sensitivity is below specificity.)
At a common reference prevalence of \(40\%\), Period A has sensitivity \(0.85\) and specificity \(0.90\); Period B has sensitivity \(0.90\) and specificity \(0.80\). Calculate standardized accuracy for each and identify the higher value. (Answer: Period A \(=0.40(0.85)+0.60(0.90)=0.8800\); Period B \(=0.40(0.90)+0.60(0.80)=0.8400\); Period A is higher.)

What You'll Learn