Why Prevalence Changes the Meaning of Error Counts
A false positive and a false negative describe different mistakes: one labels a non-event as an event, while the other misses an event that occurred. Their practical meaning depends partly on how often the event occurs in the population being classified. If the event is common, there are many actual events on which false negatives can occur. If it is rare, even a small false-positive rate can produce a noticeable number of false alarms because it applies to many non-events.
This tutorial focuses on that connection between prevalence and classification errors. The calculations help translate rates into counts and explain what those counts mean in context. They are aligned with the SOA Exam PA (Predictive Analytics) syllabus.
The population and period must be explicit. For example, “the proportion of policies with a specified claim outcome during the next year” defines a different prevalence from the proportion with that outcome over five years. A prevalence measured in a model-development sample also need not equal the prevalence in the population where a model will be used.
Start With the Actual Outcome Groups
A confusion matrix organizes classifications by the actual outcome and the predicted class. A true positive (TP) is an actual positive classified as positive; a false negative (FN) is an actual positive classified as negative. A false positive (FP) is an actual negative classified as positive; a true negative (TN) is an actual negative classified as negative.
| Predicted positive | Predicted negative | |
|---|---|---|
| Actual positive | TP | FN |
| Actual negative | FP | TN |
For a population of \(N\) individuals, the actual positive group contains \(N\pi\) individuals and the actual negative group contains \(N(1-\pi)\). Given sensitivity \(s\) and specificity \(c\), the expected counts (or exact counts when the supplied rates and group sizes produce integers) are:
These formulas apply when the prevalence, sensitivity, and specificity refer to the same target population and outcome definition. If rates are estimated from another setting, using them for a new population is a projection that assumes those class-specific rates carry over; the resulting counts may not match what is observed. For a projected count that is not an integer, interpret it as an expected count, not a literal fraction of a person.
Worked Example: Errors for a Common Outcome
A hypothetical screening program classifies 2,000 eligible members for a common outcome during the coming year. The outcome prevalence is 30%, sensitivity is 85%, and specificity is 90%. Assume these rates apply to this population. Calculate the four confusion-matrix counts and each class-specific error rate.
Method. First divide the population into actual outcome groups, then apply sensitivity within the positive group and specificity within the negative group.
Actual positives: \(2{,}000(0.30)=600\). Sensitivity is 0.85, so \(\mathrm{TP}=600(0.85)=510\), and \(\mathrm{FN}=600(1-0.85)=600(0.15)=90\).
Actual negatives: \(2{,}000(1-0.30)=2{,}000(0.70)=1{,}400\). Specificity is 0.90, so \(\mathrm{TN}=1{,}400(0.90)=1{,}260\), and \(\mathrm{FP}=1{,}400(1-0.90)=1{,}400(0.10)=140\).
The check is \(510+90=600\) actual positives, \(1{,}260+140=1{,}400\) actual negatives, and \(510+90+1{,}260+140=2{,}000\) individuals. The false-negative rate is \(90/600=15\%\); the false-positive rate is \(140/1{,}400=10\%\). Common mistake avoided: dividing false negatives by all 2,000 members would not give the false-negative rate. Its denominator is the actual-positive group.
Why the Prevalence-Weighted Error Formula Holds
The overall fraction classified incorrectly combines errors from two different actual-outcome groups. It is not enough to know sensitivity and specificity alone to determine that fraction: the sizes of the two groups matter. Under a fixed population size \(N\), prevalence \(\pi\), sensitivity \(s\), and specificity \(c\), the total number of errors is the sum of false negatives and false positives.
The derivation follows directly from the group sizes: the false-negative rate applies to the \(\pi\) fraction who are actual positives, while the false-positive rate applies to the \((1-\pi)\) fraction who are actual negatives. Their weighted contributions add. This explains why the same two class-specific error rates can result in different total error fractions when prevalence differs.
The equation is a summary of how the two types of errors contribute to the total; it does not say that the errors have the same consequences. If a decision has different consequences for false positives and false negatives, those consequences require separate consideration, as in cost-based decision analysis.
Worked Example: A Rare Outcome and False Alarms
A hypothetical program screens 40,000 accounts for a rare adverse outcome during a defined period. Prevalence is 0.5%, sensitivity is 80%, and specificity is 99.5%. Assume these rates apply to the stated population. Find the four counts, then compare the false-positive and false-negative counts.
Method. Calculate the actual-positive and actual-negative group sizes before applying the corresponding rates.
There are \(40{,}000(0.005)=200\) actual positives. Therefore, \(\mathrm{TP}=200(0.80)=160\) and \(\mathrm{FN}=200(0.20)=40\).
There are \(40{,}000(1-0.005)=40{,}000(0.995)=39{,}800\) actual negatives. The false-positive rate is \(1-0.995=0.005\), so \(\mathrm{FP}=39{,}800(0.005)=199\). Thus, \(\mathrm{TN}=39{,}800(0.995)=39{,}601\).
The check is \(160+40=200\), \(199+39{,}601=39{,}800\), and the four counts sum to 40,000. There are 199 false positives and 40 false negatives, even though the false-positive rate of 0.5% is much smaller than the false-negative rate of 20%. The reason is that the false-positive rate applies to 39,800 actual negatives, while the false-negative rate applies to only 200 actual positives. Common mistake avoided: comparing 0.5% directly with 20% and assuming the false-positive count must be smaller ignores the different group sizes.
Same Class-Specific Rates, Different Prevalence
A useful comparison holds sensitivity and specificity fixed while changing prevalence. This isolates the effect of the sizes of the actual outcome groups. It does not guarantee that rates will remain fixed in real settings; it shows what follows if they do.
Worked Example: Comparing Two Populations
A hypothetical classification rule has sensitivity 80% and specificity 95%. It is applied to two populations of 2,000 people each. Population A has prevalence 5%, and Population B has prevalence 25%. Assume the same sensitivity and specificity hold in both populations. Calculate the error counts, error fractions, and the share of errors that are false positives.
Population A. There are \(2{,}000(0.05)=100\) actual positives and \(2{,}000(0.95)=1{,}900\) actual negatives. The positive group gives \(\mathrm{TP}=100(0.80)=80\) and \(\mathrm{FN}=100(0.20)=20\). The negative group gives \(\mathrm{TN}=1{,}900(0.95)=1{,}805\) and \(\mathrm{FP}=1{,}900(0.05)=95\). Total errors are \(20+95=115\), so the error fraction is \(115/2{,}000=0.0575\), or 5.75%. The share of errors that are false positives is \(95/115=0.8261\), or 82.61% when expressed as a percentage.
Population B. There are \(2{,}000(0.25)=500\) actual positives and \(2{,}000(0.75)=1{,}500\) actual negatives. The positive group gives \(\mathrm{TP}=500(0.80)=400\) and \(\mathrm{FN}=500(0.20)=100\). The negative group gives \(\mathrm{TN}=1{,}500(0.95)=1{,}425\) and \(\mathrm{FP}=1{,}500(0.05)=75\). Total errors are \(100+75=175\), so the error fraction is \(175/2{,}000=0.0875\), or 8.75%. The share of errors that are false positives is \(75/175=0.4286\), or 42.86%.
Each set of counts checks against its group sizes and totals 2,000. The class-specific rates are identical in both populations, but Population B has more false negatives because it has more actual positives. Population A has more false positives because it has more actual negatives. Common mistake avoided: interpreting a difference in error counts as evidence that the classification rule’s sensitivity or specificity changed, without first accounting for the changed prevalence.
Practical Interpretation
Prevalence provides the denominator context needed to interpret error counts. A count of 100 false negatives alone says little: it could represent a small fraction of a large positive group or a large fraction of a small one. Likewise, a false-positive count must be read in relation to the number of actual negatives. In a report, give the population size and period, prevalence, and the class-specific rates or their denominators alongside the counts.
Keep the following distinctions clear:
- Prevalence describes the actual-outcome mix. It is not a measure of how often the model predicts positive.
- Sensitivity and specificity use different denominators. Sensitivity is calculated among actual positives; specificity is calculated among actual negatives.
- Counts depend on both rates and group sizes. Applying the same rate to a larger group produces more expected cases of that type of error.
- Prevalence may differ between evaluation and deployment. Counts projected using a new prevalence assume the class-specific rates remain applicable there.
For an actuarial application, a hypothetical classifier might flag policies for review based on a defined future event. If that event is uncommon, the actual non-event group may be large, so even a modest false-positive rate can yield many unnecessary reviews. If the event is common, missed events may form a larger count even when the false-negative rate is unchanged. The model’s error counts describe operational volume; the consequences of each type of error must still be interpreted for the decision being made.
Check Your Understanding
For each problem, calculate using the stated population and rates. Treat given rates as applicable to that population.
- A hypothetical classifier is used on 1,200 members. Prevalence is 10%, sensitivity is 75%, and specificity is 90%. Find TP, FN, TN, and FP. (Answer: TP = 90, FN = 30, TN = 972, FP = 108.)
- A screening rule has a false-positive rate of 2%. It is applied to 5,000 individuals, of whom 4,000 are actual negatives. How many false positives are expected, and what is specificity? (Answer: 80 false positives; specificity = 98%.)
- A rare outcome has prevalence 1% among 10,000 people. Sensitivity is 90%, and specificity is 99%. Find the false-negative and false-positive counts, and state which is larger. (Answer: FN = 10, FP = 99; false positives are larger.)
- A classification rule has sensitivity 70% and specificity 90%. For a population of 1,000 with prevalence 20%, calculate the total errors and the error fraction. (Answer: FN = 60, FP = 80, total errors = 140, error fraction = 14%.)
- Two populations each contain 4,000 individuals and have the same sensitivity of 80% and specificity of 95%. Their prevalences are 2% and 10%. Find the false-positive and false-negative counts in each population. (Answer: At 2% prevalence, FP = 196 and FN = 16; at 10% prevalence, FP = 180 and FN = 80.)