Precision, Prevalence, and Follow-Up Work

Why Positive Predictions Need Context

A positive prediction is useful only in relation to what happens after it. In a hypothetical screening program, for example, each positive result may prompt a costly inspection or a specialist review. If many positive predictions are false alarms, the follow-up workload can be substantial even when the model identifies most of the cases that truly need attention.

The share of positive predictions that are correct is called precision. Its practical meaning depends strongly on prevalence: the proportion of the target population that truly has the outcome of interest. It also matters how many positive cases have actually been reviewed. A precision estimate based on a small number of reviews can be much less stable than the same estimate based on many reviews.

This distinction is aligned with the SOA Exam PA (Predictive Analytics) syllabus: a metric should be interpreted in the operational setting where predictions will be used, rather than treated as an isolated score.

Definition: For a binary outcome, a true positive is a case that is truly positive and predicted positive; a false positive is truly negative but predicted positive. Precision, also called positive predictive value, is the proportion of predicted-positive cases that are truly positive: $$ \text{Precision} = P(\text{truly positive} \mid \text{predicted positive}) = \frac{\text{true positives}}{\text{true positives}+\text{false positives}}. $$ The denominator is the number of positive predictions, not the number of truly positive cases.

From Prevalence to Expected Follow-Up Counts

Let $N$ be the number of cases to which a model is applied. Let $\pi$ be prevalence, $Se$ be sensitivity, and $Sp$ be specificity. Sensitivity is the proportion of truly positive cases predicted positive; specificity is the proportion of truly negative cases predicted negative. For calculations using these quantities, assume the outcome is binary, the rates apply to the target population at the chosen prediction threshold, and the cases are evaluated over the same period and under the same outcome definition.

If the rates are treated as known, multiplying them by the expected numbers of positive and negative cases gives expected counts. These may be fractional; they represent averages over repeated populations, not literal fractions of an individual case.

\begin{aligned} \text{Expected true positives} &= N\pi Se,\\ \text{Expected false negatives} &= N\pi(1-Se),\\ \text{Expected false positives} &= N(1-\pi)(1-Sp),\\ \text{Expected true negatives} &= N(1-\pi)Sp. \end{aligned}

Consequently, the precision implied by prevalence and the two classification rates is

\text{Precision} = \frac{\pi Se}{\pi Se+(1-\pi)(1-Sp)}.

The population size $N$ cancels from this ratio. It affects the number of cases requiring follow-up, but not the expected precision when prevalence, sensitivity, and specificity are held fixed. In practice, however, $N$ and the number of cases reviewed affect how precisely the rate can be estimated.

Why the formula holds: Among $N$ cases, $N\pi$ are truly positive on average, and a fraction $Se$ of those are predicted positive, giving $N\pi Se$ true positives. The remaining $N(1-\pi)$ cases are truly negative; a fraction $1-Sp$ of those are predicted positive, giving $N(1-\pi)(1-Sp)$ false positives. All predicted positives consist of these two groups. Dividing the true-positive count by their sum gives the precision formula.

Worked Example: Estimating Follow-Up Volume in a Low-Prevalence Screen

A hypothetical program screens 10,000 accounts for a condition that affects 2% of accounts. At the operating threshold, sensitivity is 90% and specificity is 95%. Estimate the true positives, false positives, total positive predictions, and precision. Also interpret the expected follow-up workload.

Method. Partition the population into truly positive and truly negative cases, then apply sensitivity and specificity. This avoids assuming that a high sensitivity means most positive predictions are correct.

Truly positive cases: $10{,}000(0.02)=200$. Of these, $200(0.90)=180$ are true positives, and $200(0.10)=20$ are false negatives.

Truly negative cases: $10{,}000(0.98)=9{,}800$. The false-positive rate is $1-0.95=0.05$, so $9{,}800(0.05)=490$ are false positives. The true-negative count is $9{,}800(0.95)=9{,}310$.

There are $180+490=670$ positive predictions. Therefore,

\text{Precision} = \frac{180}{670}=0.2687 = 26.87\%.

The expected follow-up workload is 670 positive cases per 10,000 screened. On average, $670/180=3.72$ positive cases must be followed up for each true positive identified. The count check is $180+20+490+9{,}310=10{,}000$, as required. Counts are exact for this constructed calculation. Precision is rounded to four decimal places as a probability and two decimal places as a percentage.

Common mistake avoided: Using $180/200=90\%$ as precision gives sensitivity, not precision. The denominator for precision is all positive predictions, including the 490 false positives, so the correct calculation is $180/(180+490)$.

Why Prevalence Changes Precision

Sensitivity and specificity describe performance conditional on the true outcome. Precision reverses the conditioning: it asks about the true outcome given a positive prediction. The mix of truly positive and truly negative cases therefore matters. When prevalence is low, even a modest false-positive rate can create many false positives because it is applied to a large group of truly negative cases.

For fixed sensitivity and specificity, the formula shows that precision increases as prevalence increases. This does not mean a model has become better at distinguishing cases. It means that a positive prediction is more likely to come from the truly positive group when that group makes up a larger share of the population.

Worked Example: Same Model, Different Prevalence

A hypothetical model has sensitivity 80% and specificity 98%. Compare its expected precision in two populations of 5,000 cases: one with 1% prevalence and one with 10% prevalence. Calculate counts in each population and state what stays the same.

Method. Calculate the expected counts separately for each prevalence. The sensitivity and specificity remain fixed, but the underlying proportions of positive and negative cases differ.

Population A, prevalence 1%: There are $5{,}000(0.01)=50$ truly positive cases, so true positives are $50(0.80)=40$, and false negatives are $50(0.20)=10$. There are $5{,}000(0.99)=4{,}950$ truly negative cases. The false-positive rate is $1-0.98=0.02$, giving $4{,}950(0.02)=99$ false positives and $4{,}950(0.98)=4{,}851$ true negatives. Positive predictions total $40+99=139$, so precision is $40/139=0.2878$, or 28.78%.

Population B, prevalence 10%: There are $5{,}000(0.10)=500$ truly positive cases, giving $500(0.80)=400$ true positives and $500(0.20)=100$ false negatives. There are $5{,}000(0.90)=4{,}500$ truly negative cases, giving $4{,}500(0.02)=90$ false positives and $4{,}500(0.98)=4{,}410$ true negatives. Positive predictions total $400+90=490$, so precision is $400/490=0.8163$, or 81.63%.

The model identifies 80% of the truly positive cases in both populations and correctly rejects 98% of the truly negative cases in both. Yet precision differs substantially: 28.78% versus 81.63%. In Population A, there are 99 false alarms for 40 true positives; in Population B, there are 90 false alarms for 400 true positives. The count checks are $40+10+99+4{,}851=5{,}000$ and $400+100+90+4{,}410=5{,}000$. Probabilities are rounded to four decimal places and percentages to two decimal places.

Common mistake avoided: Carrying a precision estimate from one population into another without checking prevalence can misstate expected follow-up value. The rates alone do not determine precision; the target population's prevalence is also needed.

How Many Reviewed Cases Support the Estimate?

In an evaluation, precision is usually estimated from cases whose outcomes have been confirmed. If $n$ positive predictions are reviewed and $x$ are confirmed positive, the observed precision is $x/n$. This is a sample proportion, so it varies from one reviewed sample to another even when the underlying process is stable.

When reviewed positive cases are independent and representative of the positive predictions to which the estimate will be applied, an approximate standard error is

SE(\widehat{\text{Precision}}) \approx \sqrt{\frac{\widehat{\text{Precision}}(1-\widehat{\text{Precision}})}{n}}, \qquad \widehat{\text{Precision}}=\frac{x}{n}.

This normal approximation is most useful when the sample contains a reasonable number of confirmed positives and negatives. It can be unreliable for very small samples or estimates near 0 or 1; an appropriate binomial interval is preferable in those situations. The approximation also does not fix selection bias: if reviewed cases are not representative of the positive predictions of interest, increasing their number alone does not guarantee a useful estimate.

Worked Example: Interpreting Precision from a Limited Review

A hypothetical team reviews 120 cases that a model flagged positive. It confirms that 36 truly meet the outcome definition. Estimate precision, the expected confirmed cases per 300 comparable positive reviews, and an approximate 95% interval for precision. Assume the reviewed cases are independent and representative.

Method. Use the observed proportion for precision. Use the standard-error approximation for a rough interval, recognizing that the result describes sampling uncertainty under the stated assumptions.

The observed precision is $36/120=0.30$, or 30%. The other $120-36=84$ reviewed cases are false positives. The expected number of confirmed cases in 300 comparable reviews is $300(0.30)=90$.

The approximate standard error is

\sqrt{\frac{0.30(1-0.30)}{120}} = \sqrt{\frac{0.21}{120}} = \sqrt{0.00175} \approx 0.0418.

The approximate 95% interval is $0.30\pm1.96(0.0418)$. The margin is approximately $0.0819$, so the interval is $[0.2181,\,0.3819]$, or about 21.81% to 38.19%. The normal approximation is reasonable here as a rough calculation because there are 36 confirmed positives and 84 confirmed negatives. The point estimate and interval are rounded to four decimal places in probability form.

A larger review may narrow sampling uncertainty. For example, if 300 of 1,000 comparable positive predictions are confirmed, the estimated precision is still 30%, but the approximate standard error is $\sqrt{0.21/1{,}000}\approx0.0145$, smaller than 0.0418. The estimate is more stable, not necessarily higher. The calculation for the original review checks: $36+84=120$.

Common mistake avoided: Treating 30% as a guaranteed rate for the next 300 cases ignores sampling variation and assumes the next cases are comparable. The estimate is useful for planning, but it is not a certainty about any particular batch.

Using Precision to Describe Follow-Up Work

Precision can be translated into an intuitive workload measure. If precision is $p>0$, then the expected number of positive predictions to review per confirmed true positive is $1/p$. This is a ratio of expected counts under a stable process, not a promise about the order in which cases will be confirmed.

For example, a precision of 25% corresponds to about four positive reviews per confirmed case. A precision of 80% corresponds to about 1.25 reviews per confirmed case. These summaries are meaningful only when the outcome definition, prediction threshold, follow-up process, and population remain relevant to the decision.

A model can have a respectable precision but still produce more positive predictions than a team can review. Conversely, a modest precision may still be useful if each confirmed case has substantial value and follow-up is inexpensive. Whether a particular level is acceptable depends on the operational success criteria and costs; precision alone does not establish that a policy is worthwhile.

Key takeaway: Precision answers, “Of the cases predicted positive, what fraction are truly positive?” Estimate it with the true-positive count divided by all positive predictions. To anticipate its value in a target population, account for prevalence as well as sensitivity and specificity. To judge confidence in an observed estimate, consider how many representative positive predictions were reviewed.

Check Your Understanding

For each hypothetical scenario, calculate the requested quantities and interpret what the counts imply for positive-case follow-up.

A program applies a model to 2,000 cases. Prevalence is 5%, sensitivity is 80%, and specificity is 90%. Find the expected true positives, false positives, total positive predictions, and precision. (Answer: 80 true positives, 190 false positives, 270 positive predictions, precision $80/270=0.2963$, or 29.63%; 3.375 positive reviews per true positive.)
A model is used on 10,000 cases with 2% prevalence, 75% sensitivity, and 96% specificity. Find the expected number of positive predictions and precision. (Answer: 150 true positives and 392 false positives, so 542 positive predictions; precision $150/542=0.2768$, or 27.68%.)
A hypothetical review finds 55 confirmed cases among 250 positive predictions. Calculate observed precision and the number of positive reviews per confirmed case. (Answer: precision $55/250=0.2200$, or 22.00%; $250/55\approx4.55$ reviews per confirmed case.)
A screen covers 10,000 cases at 3% prevalence, with sensitivity 90% and specificity 97%. Calculate true positives, false positives, and precision. (Answer: 270 true positives and $9{,}700(0.03)=291$ false positives; precision $270/561=0.4813$, or 48.13%.)
Using the same sensitivity and specificity as Question 4, prevalence rises to 6% in a population of 10,000. Recalculate the positive-prediction count and precision. Explain what changed. (Answer: 540 true positives and $9{,}400(0.03)=282$ false positives, giving 822 positive predictions and precision $540/822=0.6569$, or 65.69%. Precision rises because prevalence is higher; sensitivity and specificity are unchanged.)

What You'll Learn