Prevalence Shift and Classification Performance

Why Prevalence Changes the Meaning of a Classification Result

A classifier can have the same ability to distinguish positive from negative cases in two populations and still produce very different predictive values. The reason is that the proportion of positive cases—the prevalence—changes how many true and false results arise in the population. This distinction matters whenever a model is moved from a development sample to a population where the event is more or less common.

For example, a hypothetical screening rule may identify most people who have a condition and correctly classify most people who do not. If the condition is rare, however, even a modest false-positive rate applied to the much larger negative group can produce many false positives. Sensitivity and specificity describe performance conditional on the actual class; predictive values describe the class composition conditional on the model's prediction.

Definition: Let \(Y=1\) denote the positive class and \(Y=0\) the negative class. Prevalence is \(\pi=P(Y=1)\). For a fixed classification threshold, sensitivity is \(Se=P(\widehat{Y}=1\mid Y=1)\), and specificity is \(Sp=P(\widehat{Y}=0\mid Y=0)\). Positive predictive value is \(PPV=P(Y=1\mid\widehat{Y}=1)\), and negative predictive value is \(NPV=P(Y=0\mid\widehat{Y}=0)\). A pure prevalence shift changes \(\pi\) while leaving the class-conditional distributions of the model score unchanged.

Under a pure prevalence shift, sensitivity and specificity at a fixed score threshold stay the same: they are conditional on the actual class. Predictive values and overall accuracy generally change because the relative sizes of the positive and negative groups change. If the class-conditional score distributions change as well—for example, because the new population contains different kinds of positive cases—then sensitivity and specificity need not remain stable.

Recalculating Performance at a New Prevalence

For a population of size \(N\), the expected numbers of positive and negative cases are \(N\pi\) and \(N(1-\pi)\). If sensitivity and specificity are known at a fixed threshold, the expected confusion-matrix counts follow directly. The formulas below describe expected counts; they need not be integers when used with a population size that gives fractional expectations.

\begin{aligned} TP &= N\pi Se, & FN &= N\pi(1-Se),\\ TN &= N(1-\pi)Sp, & FP &= N(1-\pi)(1-Sp). \end{aligned}

Dividing the relevant expected counts gives the predictive values and accuracy at the target prevalence:

\begin{aligned} PPV &= \frac{\pi Se}{\pi Se+(1-\pi)(1-Sp)},\\ NPV &= \frac{(1-\pi)Sp}{(1-\pi)Sp+\pi(1-Se)},\\ \text{Accuracy} &= \pi Se+(1-\pi)Sp. \end{aligned}

The first expression shows why a low prevalence can reduce \(PPV\): the numerator, true positives, shrinks with \(\pi\), while false positives arise from the negative group, whose size is proportional to \(1-\pi\). The accuracy formula is a prevalence-weighted average of sensitivity and specificity, not their unweighted average except in special cases.

Important condition: These calculations transfer sensitivity and specificity to a target population only when the class-conditional performance at the chosen threshold is applicable there. A change in prevalence alone does not guarantee that condition. If the makeup of either class changes, recalculate or validate the conditional performance rather than assuming it is constant.

Worked Example: Recomputing Predictive Values After a Screening Population Changes

A hypothetical screening rule has sensitivity \(Se=0.90\) and specificity \(Sp=0.95\) at its chosen threshold. Compare its expected performance in two populations of 10,000 people: one with prevalence \(0.20\), and one with prevalence \(0.02\). Assume the class-conditional performance is unchanged.

Method. For each population, calculate expected class counts, apply sensitivity and specificity to obtain the confusion matrix, and then divide the appropriate counts to find the requested measures. This avoids treating predictive values as fixed properties of the classifier.

Population with prevalence \(0.20\). There are \(10{,}000(0.20)=2{,}000\) positive cases and \(10{,}000(0.80)=8{,}000\) negative cases. Thus

TP=2{,}000(0.90)=1{,}800,\quad FN=2{,}000(0.10)=200,

and

TN=8{,}000(0.95)=7{,}600,\quad FP=8{,}000(0.05)=400.

The total predicted positive count is \(1{,}800+400=2{,}200\), and the total predicted negative count is \(7{,}600+200=7{,}800\). Therefore

PPV=\frac{1{,}800}{2{,}200}=0.8182,\quad NPV=\frac{7{,}600}{7{,}800}=0.9744,

and accuracy is \((1{,}800+7{,}600)/10{,}000=0.9400\).

Population with prevalence \(0.02\). There are \(10{,}000(0.02)=200\) positive cases and \(9{,}800\) negative cases. The expected counts are

TP=200(0.90)=180,\quad FN=200(0.10)=20,\quad TN=9{,}800(0.95)=9{,}310,\quad FP=9{,}800(0.05)=490.

There are \(180+490=670\) predicted positives and \(9{,}310+20=9{,}330\) predicted negatives. Hence

PPV=\frac{180}{670}=0.2687,\quad NPV=\frac{9{,}310}{9{,}330}=0.9979,\quad \text{Accuracy}=\frac{180+9{,}310}{10{,}000}=0.9490.

Sensitivity remains \(0.9000\) and specificity remains \(0.9500\) by the stated assumption. Yet \(PPV\) falls from \(0.8182\) to \(0.2687\), while \(NPV\) rises from \(0.9744\) to \(0.9979\). Accuracy rises slightly because the target population is mostly negative, and specificity is higher than sensitivity. Common mistake avoided: carrying the source population's predictive values into the new population instead of recomputing them from the target prevalence.

Why the Predictive Values Change

The change can be justified directly from conditional probability. A positive prediction can arise either from a true positive, with probability \(\pi Se\), or from a false positive, with probability \((1-\pi)(1-Sp)\). These outcomes are mutually exclusive and together make up all positive predictions. Dividing the true-positive probability by the total probability of a positive prediction gives \(PPV\).

\begin{aligned} P(Y=1\mid\widehat{Y}=1) &=\frac{P(Y=1,\widehat{Y}=1)}{P(\widehat{Y}=1)}\\ &=\frac{P(Y=1)P(\widehat{Y}=1\mid Y=1)} {P(Y=1)P(\widehat{Y}=1\mid Y=1)+P(Y=0)P(\widehat{Y}=1\mid Y=0)}\\ &=\frac{\pi Se}{\pi Se+(1-\pi)(1-Sp)}. \end{aligned}

The same reasoning gives \(NPV\): a negative prediction is either a true negative, with probability \((1-\pi)Sp\), or a false negative, with probability \(\pi(1-Se)\). The numerator for \(NPV\) is the true-negative probability; its denominator is the probability of any negative prediction.

This derivation also makes an important point about prevalence: sensitivity and specificity alone do not determine predictive values. The prevalence is also required. Conversely, a predictive value observed in one population should not be treated as a transferable measure when the target prevalence differs.

Adjusting a Calibrated Probability for a New Prevalence

A model may output a probability calibrated in a source population. Under a pure prevalence shift, that probability can be adjusted if the class-conditional score distributions stay the same and the source probability is correctly calibrated. The adjustment is most simply expressed in odds. It is not a direct multiplication of the probability by a prevalence ratio.

\frac{p_t}{1-p_t} = \frac{p_s}{1-p_s} \frac{\pi_t/(1-\pi_t)}{\pi_s/(1-\pi_s)}

Here \(p_s\) is the source calibrated probability for a particular score, \(p_t\) is its target-population probability, and \(\pi_s,\pi_t\) are the source and target prevalences. This adjustment applies only under the stated pure-shift and calibration assumptions. It does not repair poor calibration or a change in the relationship between scores and outcomes within either class.

Worked Example: Updating a Score Probability Under a Rare-Event Shift

A hypothetical model is calibrated in a source population with prevalence \(\pi_s=0.10\). For one case, it reports source probability \(p_s=0.40\). The model is to be used in a target population with prevalence \(\pi_t=0.02\). Assume a pure prevalence shift. A decision is positive when the target probability is at least \(0.15\). Should this case receive a positive decision?

Method. Convert the source probability to odds, multiply by the ratio of target prior odds to source prior odds, and convert the resulting odds back to a probability.

The source posterior odds are

\frac{p_s}{1-p_s}=\frac{0.40}{0.60}=\frac{2}{3}.

The source prevalence odds are \(0.10/0.90=1/9\), and the target prevalence odds are \(0.02/0.98=1/49\). The prior-odds adjustment factor is therefore

\frac{1/49}{1/9}=\frac{9}{49}.

The target posterior odds are

\frac{2}{3}\frac{9}{49}=\frac{6}{49}.

Odds \(6/49\) correspond to probability \(6/(6+49)=6/55=0.1091\), to four decimal places. Since \(0.1091<0.15\), the case does not receive a positive decision under the target-population cutoff. Common mistake avoided: multiplying \(0.40\) directly by \(0.02/0.10\) gives \(0.08\), but that is not the correct probability adjustment. The prior-prevalence correction acts on odds, not directly on probability.

Prevalence Can Change Which Threshold Is Preferred

A fixed threshold's sensitivity and specificity may remain stable under pure prevalence shift, but the consequences of its errors change. If false negatives and false positives have different costs, the preferred operating point can change as prevalence changes. This is a population-level decision: compare the expected costs at the target prevalence, rather than assuming the threshold preferred in the source population remains best.

For false-negative cost \(C_{FN}\) and false-positive cost \(C_{FP}\), the expected error cost per case is

C(\pi)=C_{FN}\pi(1-Se)+C_{FP}(1-\pi)(1-Sp).

The first term is the prevalence-weighted false-negative probability multiplied by its cost; the second is the prevalence-weighted false-positive probability multiplied by its cost. This comparison assumes the costs and the relevant class-conditional operating characteristics are applicable to the target population.

Worked Example: Comparing Two Thresholds at a Lower Prevalence

A hypothetical service has two available thresholds. Threshold A has sensitivity \(0.95\) and specificity \(0.90\). Threshold B has sensitivity \(0.80\) and specificity \(0.99\). A false negative costs 10 cost units and a false positive costs 1 cost unit. Compare the expected error costs per 10,000 cases at prevalence \(0.10\) and prevalence \(0.01\). Assume each threshold's sensitivity and specificity remain stable.

Method. For each threshold and prevalence, calculate false-negative and false-positive counts, multiply by their respective costs, and add. Using counts makes the prevalence effect explicit.

At prevalence \(0.10\): There are 1,000 positive and 9,000 negative cases. For A, \(FN=1{,}000(0.05)=50\) and \(FP=9{,}000(0.10)=900\). Its cost is \(50(10)+900(1)=1{,}400\) cost units. For B, \(FN=1{,}000(0.20)=200\) and \(FP=9{,}000(0.01)=90\). Its cost is \(200(10)+90(1)=2{,}090\) cost units. Threshold A has the lower cost.

At prevalence \(0.01\): There are 100 positive and 9,900 negative cases. For A, \(FN=100(0.05)=5\) and \(FP=9{,}900(0.10)=990\). Its cost is \(5(10)+990(1)=1{,}040\) cost units. For B, \(FN=100(0.20)=20\) and \(FP=9{,}900(0.01)=99\). Its cost is \(20(10)+99(1)=299\) cost units. Threshold B now has the lower cost. Check: at each prevalence, the positive and negative case counts sum to 10,000. The preferred threshold changes because the lower prevalence greatly reduces the number of false negatives while increasing the relative influence of false positives.

Interpretation and Common Traps

Prevalence shift is a reason to revisit model interpretation and decisions, not a reason to assume every performance measure has changed. Under a pure shift, sensitivity and specificity at a fixed threshold remain stable, while \(PPV\), \(NPV\), and accuracy can change. If the model score distributions conditional on class remain unchanged, the ROC operating points also remain the same; nevertheless, a different prevalence can alter the usefulness of an operating point for a particular decision.

Several practical checks help prevent incorrect conclusions:

Separate conditional rates from population rates. Sensitivity and specificity condition on the actual outcome; predictive values condition on the prediction. They answer different questions.
Use target prevalence for target-population predictive values. A predictive value from a case-control or otherwise altered-prevalence sample is not automatically the target-population predictive value.
Do not equate stable discrimination with stable decisions. Even if the score's class-conditional behavior is stable, an altered prevalence can change predictive values, error counts, expected cost, and the appropriate probability cutoff.
Check whether the shift is truly only prevalence. If positive or negative cases differ in their score distributions between populations, the transferred sensitivity and specificity may be wrong.

For exam-style calculations, write down the target prevalence, sensitivity, specificity, and any error costs before substituting values. Then identify whether the question asks for a conditional measure, such as sensitivity, or a population-dependent measure, such as \(PPV\), accuracy, or expected cost. This is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

For each problem, assume any stated sensitivity and specificity apply to the target population. Give probabilities to four decimal places where relevant.

A hypothetical classifier has \(Se=0.80\) and \(Sp=0.90\). In 20,000 cases with prevalence \(0.05\), calculate \(TP\), \(FN\), \(FP\), \(TN\), \(PPV\), and accuracy. (Answer: \(TP=800,\ FN=200,\ FP=1{,}900,\ TN=17{,}100;\ PPV=0.2963;\) accuracy \(=0.8950\).)
A target population has prevalence \(0.40\), sensitivity \(0.75\), and specificity \(0.80\). Calculate accuracy. (Answer: \(0.40(0.75)+0.60(0.80)=0.7800\).)
A source model is calibrated at prevalence \(0.20\) and assigns one case probability \(p_s=0.60\). Under a pure prevalence shift to \(0.05\), calculate the adjusted probability. (Answer: source odds \(=1.5\), prevalence-odds factor \(=(0.05/0.95)/(0.20/0.80)=4/19\), target odds \(=6/19\), and target probability \(=6/25=0.2400\).)
At prevalence \(0.02\), compare two thresholds on 10,000 cases. A has \(Se=0.90, Sp=0.95\); B has \(Se=0.70, Sp=0.99\). A false negative costs 8 units and a false positive costs 1 unit. Which threshold has lower expected cost, and what is that cost? (Answer: A costs \(20(8)+490=650\) units; B costs \(60(8)+98=578\) units, so B has the lower cost.)
A model is moved to a population where prevalence is lower. What must be true for its sensitivity and specificity at a fixed threshold to remain unchanged, and do \(PPV\) and \(NPV\) necessarily remain unchanged? (Answer: The class-conditional score distributions and the threshold must be unchanged; \(PPV\) and \(NPV\) generally change with prevalence.)

What You'll Learn