Why Prevalence Matters at a Fixed Threshold
A binary classifier turns a score into a decision. For example, a hypothetical insurer might use a model score to flag policies for review, or a health service might use one to identify cases for follow-up. The decision can affect people even when the score itself is informative, so a group comparison needs more than a single summary number.
One important distinction is between how often the outcome occurs and how the classifier performs among people with and without the outcome. The first is prevalence; the second is described by conditional rates such as sensitivity and false-positive rate. At a fixed threshold, different prevalence can change the share of positive decisions that are correct, even when sensitivity and false-positive rate are identical.
The positive predictive value (PPV) is \(P(Y=1\mid D=1)\): among positive decisions, the proportion that have the outcome. The negative predictive value (NPV) is \(P(Y=0\mid D=0)\): among negative decisions, the proportion that do not have the outcome. These predictive values condition on the decision, whereas sensitivity and specificity condition on the actual outcome.
How Prevalence Enters Predictive Values
Write sensitivity as \(r=P(D=1\mid Y=1)\) and false-positive rate as \(f=P(D=1\mid Y=0)\). The law of total probability gives the chance of a positive decision: a positive decision can come from someone with the outcome or someone without it. Bayes’ theorem then gives the PPV.
The numerator is the fraction of the population that has the outcome and receives a positive decision. The denominator is the fraction receiving any positive decision, including false positives. This explains why prevalence must be in the calculation: it determines how much of the population belongs to each outcome group before the classifier is applied.
Similarly, if specificity is \(s=P(D=0\mid Y=0)\), then the probability of a negative decision is \(s(1-\pi)+(1-r)\pi\). Therefore:
For fixed, nonzero \(r\) and \(f\), increasing prevalence increases PPV. To see why, write its denominator as \(f+(r-f)\pi\). Differentiating the PPV expression with respect to \(\pi\) gives:
Thus, with sensitivity and false-positive rate held fixed, higher prevalence produces higher PPV. The corresponding NPV generally falls as prevalence rises, because a negative decision is less likely to come from the larger outcome-positive share. These results do not say that the classifier’s sensitivity or false-positive rate must change with prevalence. Those are conditional rates; predictive values are not.
Worked Example: Same Error Rates, Different Prevalence
In a hypothetical review program, two groups each contain 1,000 people. At the same decision threshold, the classifier has sensitivity 0.80 and false-positive rate 0.10 in both groups. Outcome prevalence is 10% in Group A and 30% in Group B. Calculate the confusion counts, PPV, NPV, and positive-decision rate for each group. The method is to apply the conditional rates within the outcome-positive and outcome-negative counts, then use the appropriate predictive-value denominators. This avoids confusing a conditional rate with a predictive value.
Group A. There are \(1{,}000(0.10)=100\) outcome-positive people and \(1{,}000-100=900\) outcome-negative people. Sensitivity 0.80 gives \(100(0.80)=80\) true positives and \(100-80=20\) false negatives. False-positive rate 0.10 gives \(900(0.10)=90\) false positives and \(900-90=810\) true negatives.
There are \(80+90=170\) positive decisions and \(20+810=830\) negative decisions. Thus, \(\operatorname{PPV}=80/170=0.4706\) and \(\operatorname{NPV}=810/830=0.9759\). The positive-decision rate is \(170/1{,}000=0.1700\).
Group B. There are \(1{,}000(0.30)=300\) outcome-positive people and \(700\) outcome-negative people. True positives are \(300(0.80)=240\), false negatives are \(300-240=60\), false positives are \(700(0.10)=70\), and true negatives are \(700-70=630\).
There are \(240+70=310\) positive decisions and \(60+630=690\) negative decisions. Thus, \(\operatorname{PPV}=240/310=0.7742\), \(\operatorname{NPV}=630/690=0.9130\), and the positive-decision rate is \(310/1{,}000=0.3100\).
Check. In each group, true positives plus false negatives equal the outcome-positive count, and false positives plus true negatives equal the outcome-negative count. Sensitivity is \(80/100=240/300=0.80\), and false-positive rate is \(90/900=70/700=0.10\). The predictive values differ despite those matching conditional rates because prevalence differs. The displayed proportions are rounded to four decimal places.
Fixed Threshold Does Not Mean Fixed Error Rates
A threshold specifies the score value at which a decision changes; it does not guarantee the same sensitivity or false-positive rate in every group. Those rates also depend on the score distributions among outcome-positive and outcome-negative people. If those conditional distributions differ, the same threshold can capture different fractions of each outcome class.
This distinction is essential when interpreting group comparisons. Prevalence, by itself, changes PPV and NPV when sensitivity and specificity are held constant. It does not, by itself, change sensitivity or false-positive rate under that same condition. But in observed data, groups may differ in both prevalence and conditional score distributions. A fixed threshold then produces differences in predictive values and in error rates, and the calculations alone do not identify the reason for the conditional-rate differences.
Worked Example: Comparing Groups at One Threshold
A hypothetical service uses the same score threshold, \(t=0.60\), for two groups of 500 people each. In Group C, prevalence is 20%; among outcome-positive people, 80% meet the threshold, and among outcome-negative people, 10% meet it. In Group D, prevalence is 10%; the corresponding proportions are 70% and 20%. Calculate each group’s confusion counts, sensitivity, false-positive rate, and PPV. The method is to calculate each group’s outcome counts first and then apply its own conditional rates. This avoids assuming that a common threshold forces common error rates.
Group C. There are \(500(0.20)=100\) outcome-positive and \(400\) outcome-negative people. Of the 100 outcome-positive people, \(100(0.80)=80\) are true positives and 20 are false negatives. Of the 400 outcome-negative people, \(400(0.10)=40\) are false positives and 360 are true negatives. Therefore, sensitivity is \(80/100=0.80\), false-positive rate is \(40/400=0.10\), and \(\operatorname{PPV}=80/(80+40)=80/120=0.6667\).
Group D. There are \(500(0.10)=50\) outcome-positive and \(450\) outcome-negative people. True positives are \(50(0.70)=35\), false negatives are 15, false positives are \(450(0.20)=90\), and true negatives are 360. Therefore, sensitivity is \(35/50=0.70\), false-positive rate is \(90/450=0.20\), and \(\operatorname{PPV}=35/(35+90)=35/125=0.2800\).
Check. The counts in each group sum to 500, and the computed conditional rates match the stated proportions. Both groups used \(t=0.60\), but their sensitivity and false-positive rate differ. Those differences cannot be attributed to prevalence alone: the problem specifies different conditional rates at that threshold. The PPV difference reflects both prevalence and conditional-rate differences. Proportions are rounded to four decimal places.
Threshold Changes Alter the Classification Counts
For a fixed set of scores, raising the threshold can only keep the same number of positive decisions or reduce it. It may reduce false positives, but it may also reduce true positives. The resulting changes in sensitivity, false-positive rate, and predictive values depend on how many outcome-positive and outcome-negative scores lie between the old and new thresholds.
The next example holds the group’s population and scores fixed while changing the threshold. It illustrates why a performance measure should always be reported with its threshold, outcome definition, and population context.
Worked Example: Raising a Review Threshold
A hypothetical classifier is applied to 400 cases, of which 80 have the outcome and 320 do not. Among outcome-positive cases, 60 scores are at least 0.70, 12 are from 0.50 up to but not including 0.70, and 8 are below 0.50. Among outcome-negative cases, the corresponding counts are 16, 48, and 256. Compare thresholds 0.50 and 0.70. The method is to combine score bands that meet each threshold, then recalculate each measure from its defining denominator. This avoids assuming that a higher threshold improves every measure.
At \(t=0.50\). The 60 and 12 outcome-positive cases are predicted positive, so there are 72 true positives and \(80-72=8\) false negatives. The 16 and 48 outcome-negative cases are predicted positive, so there are 64 false positives and \(320-64=256\) true negatives. Sensitivity is \(72/80=0.9000\), false-positive rate is \(64/320=0.2000\), and PPV is \(72/(72+64)=72/136=0.5294\). There are 136 positive decisions, or \(136/400=0.3400\) of all cases.
At \(t=0.70\). There are 60 true positives and \(80-60=20\) false negatives. There are 16 false positives and \(320-16=304\) true negatives. Sensitivity is \(60/80=0.7500\), false-positive rate is \(16/320=0.0500\), and PPV is \(60/(60+16)=60/76=0.7895\). There are 76 positive decisions, or \(76/400=0.1900\) of all cases.
Check. At each threshold, the four counts sum to 400. Raising the threshold reduced both the number of positive decisions and the false-positive rate, but also lowered sensitivity. PPV rose for these particular score counts; it is not guaranteed to rise for every possible score distribution. Proportions are rounded to four decimal places.
Reading Group Comparisons Carefully
A fairness or performance review should state what each rate means and what it does not establish. A higher PPV in one group means that positive decisions in that group more often coincide with the defined outcome in the observed data. It does not, on its own, show that the classifier is more accurate in every sense, that the threshold is appropriate, or that the decisions cause a particular harm.
Likewise, a difference in sensitivity or false-positive rate is a difference in errors conditional on the observed outcome labels. Its interpretation depends on the population, outcome measurement, data collection, and decision context. Earlier material on measurement bias, selection bias, and mapping decisions to potential harms is relevant when deciding what the observed labels and consequences mean.
- Report the threshold and define which outcome is positive.
- Give the prevalence and, where possible, the underlying confusion counts alongside rates.
- Keep denominators explicit: sensitivity divides by outcome-positive cases; false-positive rate divides by outcome-negative cases; PPV divides by positive decisions.
- Do not infer that prevalence caused an observed difference in conditional error rates without checking the score distributions and data.
A second trap is to claim that the same threshold guarantees the same false-positive rate or sensitivity. It does not: those rates depend on where the scores fall within each outcome class. Conversely, if sensitivity and false-positive rate really are held constant, a prevalence difference alone is enough to change PPV and NPV. Keeping these statements separate is central to sound interpretation and is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each question, identify the relevant denominator before calculating. Give probabilities to four decimal places.
- A hypothetical screening group has 2,000 people, prevalence 5%, sensitivity 0.90, and specificity 0.95. Calculate the four confusion counts and PPV. (Answer: TP = 90, FN = 10, FP = 95, TN = 1,805; PPV = \(90/185=0.4865\).)
- At a fixed threshold, prevalence is 25%, sensitivity is 0.72, and false-positive rate is 0.12. Calculate PPV and the positive-decision rate. (Answer: PPV = \(0.18/(0.18+0.09)=0.6667\); positive-decision rate = \(0.27\).)
- Two groups each contain 800 people and have the same sensitivity, 0.80, and false-positive rate, 0.10. Their prevalences are 15% and 30%. Calculate the PPV for each group. (Answer: at 15%, \(TP=96\), \(FP=68\), so PPV = \(96/164=0.5854\); at 30%, \(TP=192\), \(FP=56\), so PPV = \(192/248=0.7742\).)
- In 500 cases, 50 have the outcome and 450 do not. At a lower threshold there are 40 true positives and 45 false positives. At a higher threshold there are 35 true positives and 18 false positives. Calculate sensitivity, false-positive rate, and PPV at both thresholds. (Answer: lower threshold: sensitivity \(40/50=0.8000\), false-positive rate \(45/450=0.1000\), PPV \(40/85=0.4706\); higher threshold: sensitivity \(35/50=0.7000\), false-positive rate \(18/450=0.0400\), PPV \(35/53=0.6604\).)
- A classifier has sensitivity 0.80 and false-positive rate 0.10. What prevalence gives a PPV of 0.50? (Answer: solving \(0.50=0.80\pi/[0.80\pi+0.10(1-\pi)]\) gives \(0.45\pi=0.05\), so \(\pi=1/9=0.1111\).)