What Does a Positive Prediction Mean?
A binary predictive model may be used to flag cases for review, identify applications likely to meet a criterion, or screen for a future event. In each setting, a positive prediction can lead to additional scrutiny, an intervention, or a decision. One useful question is: among the cases flagged as positive, what proportion actually have the outcome?
That proportion is the positive predictive value (PPV). Comparing PPV across groups asks whether a positive prediction has the same observed reliability for each group. This is a different question from whether groups experience the same rates of model errors across all people with and without the outcome. The distinction is central when interpreting fairness measures and is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
The denominator is the number of cases predicted positive, not the number of actual positive outcomes and not the total number of people in the group. A PPV of 0.80 means that, among cases predicted positive in that group, 80% had the positive outcome according to the observed labels. Equivalently, the false discovery proportion among predicted positives is \(1-\operatorname{PPV}_g\).
PPV is conditional on a positive prediction. It does not, by itself, describe how often the model misses positive outcomes, how often it flags people without the outcome, or how many people receive a positive prediction. Those questions require other rates and counts. In particular, PPV is not the same quantity as the probability estimate produced for an individual case.
Calculating PPV From Group-Specific Outcomes
To calculate PPV from evaluation data, first fix the outcome definition, the model decision rule, the groups, and the evaluation population. Then classify each case as a true positive, false positive, false negative, or true negative. The outcomes must be observed in a sufficiently comparable way for the calculation to be meaningful. If labels are missing or measured differently across groups, the calculated values may not reflect comparable underlying outcomes.
Worked Example: Equal PPV With Different Error Rates
A hypothetical service uses a model to flag accounts for follow-up. After the outcomes are observed, the evaluation counts are as follows. Calculate each group’s PPV and compare the result with its sensitivity and false-positive rate.
| Group | True positive | False positive | False negative | True negative |
|---|---|---|---|---|
| A | 72 | 18 | 28 | 182 |
| B | 48 | 12 | 32 | 108 |
Method. Calculate PPV using true positives divided by all positive predictions. For Group A, the number predicted positive is \(72+18=90\), so $$ \operatorname{PPV}_A=\frac{72}{72+18}=\frac{72}{90}=0.8000. $$ For Group B, the number predicted positive is \(48+12=60\), so $$ \operatorname{PPV}_B=\frac{48}{48+12}=\frac{48}{60}=0.8000. $$ Both groups have a PPV of 0.8000, so predictive parity holds for these observed counts.
To check what this does not establish, calculate sensitivity as true positives divided by all actual positives. Group A has \(72+28=100\) actual positives, giving \(72/100=0.7200\). Group B has \(48+32=80\) actual positives, giving \(48/80=0.6000\). Their sensitivities differ.
The false-positive rates are calculated among actual negatives. Group A has \(18+182=200\) actual negatives, so its false-positive rate is \(18/200=0.0900\). Group B has \(12+108=120\) actual negatives, so its false-positive rate is \(12/120=0.1000\). These rates differ as well. Thus equal PPV does not imply equalized odds, which requires equal true-positive rates and equal false-positive rates across groups. The common mistake is to treat equality of one fairness measure as equality of all group error rates. Check: both PPVs are 0.8000, while both sensitivity and false-positive rate differ.
Why PPV Depends on More Than Error Rates
PPV can also be expressed using three group-specific quantities: prevalence, sensitivity, and false-positive rate. Here, prevalence is the proportion of the group with \(Y=1\); sensitivity is the probability of a positive prediction among those with \(Y=1\); and the false-positive rate is the probability of a positive prediction among those with \(Y=0\). The identity follows from conditioning on whether the outcome occurred.
To justify the formula, the probability of both a positive outcome and a positive prediction is \(P(Y=1,\hat{Y}=1\mid G=g)=\pi_gs_g\). The probability of a positive prediction with a negative outcome is \(P(Y=0,\hat{Y}=1\mid G=g)=(1-\pi_g)f_g\). These two disjoint events make up all positive predictions. Dividing the first probability by their sum gives the conditional probability of a positive outcome among positive predictions:
This identity explains why comparing PPV is not the same as comparing sensitivity and false-positive rate. PPV depends on all three quantities, including the group’s prevalence. The formula applies to a binary outcome and a binary positive/negative decision in the specified evaluation population. It does not apply as written if the positive-decision probability is zero, because there are then no positive predictions for which to calculate PPV.
Worked Example: Equalized Odds but Unequal PPV
Two hypothetical screening groups each contain 1,000 people. In both groups, sensitivity is 0.75 and the false-positive rate is 0.10. In Group C, 20% have the outcome; in Group D, 10% have it. Calculate PPV in each group, first from counts and then check the calculation using the formula.
Group C. Of 1,000 people, \(0.20(1{,}000)=200\) have the outcome and \(1{,}000-200=800\) do not. The number of true positives is \(0.75(200)=150\), and the number of false negatives is \(200-150=50\). The number of false positives is \(0.10(800)=80\), and the number of true negatives is \(800-80=720\). There are \(150+80=230\) positive predictions, so $$ \operatorname{PPV}_C=\frac{150}{230}=0.6522 $$ to four decimal places.
Group D. Of 1,000 people, \(0.10(1{,}000)=100\) have the outcome and \(900\) do not. There are \(0.75(100)=75\) true positives and \(100-75=25\) false negatives. There are \(0.10(900)=90\) false positives and \(900-90=810\) true negatives. There are \(75+90=165\) positive predictions, so $$ \operatorname{PPV}_D=\frac{75}{165}=0.4545 $$ to four decimal places.
The formula gives the same checks. For Group C, \(0.20(0.75)/[0.20(0.75)+0.80(0.10)]=0.15/0.23=0.6522\). For Group D, \(0.10(0.75)/[0.10(0.75)+0.90(0.10)]=0.075/0.165=0.4545\). Sensitivity and false-positive rate are equal across these groups, but PPV is not. The common mistake is to conclude that equal error rates require equal PPV. They do not: the groups have different proportions with the outcome, changing the composition of their positive predictions. Check: the count-based and formula-based results agree in both groups.
Interpreting Predictive Parity Carefully
Predictive parity focuses on the reliability of positive predictions. In a setting where a positive flag leads to a costly or burdensome review, a lower PPV means that a greater share of flagged cases did not have the observed outcome. In another setting, a positive prediction may trigger access to a helpful service; there, the meaning of a false positive and the relevant fairness concern may differ. Interpretation depends on the decision and the potential harms, not only the numerical comparison.
A PPV comparison should be reported with its denominator and evaluation context. For example, “the PPV was 0.70” is incomplete without saying which group, which outcome, which decision threshold, and which evaluation data were used. A comparison should also make clear whether it reports observed estimates or claims about population values. Counts matter: the same estimated PPV can be based on very different numbers of positive predictions.
Worked Example: Comparing Groups and Pooling Counts
In a hypothetical review program, two groups each have 300 evaluated cases. For Group K, the counts are 54 true positives, 36 false positives, 46 false negatives, and 164 true negatives. For Group L, they are 63 true positives, 17 false positives, 37 false negatives, and 183 true negatives. Calculate the group PPVs, their difference in percentage points, and the overall PPV when the groups are combined.
Method. Group K has \(54+36=90\) positive predictions, giving $$ \operatorname{PPV}_K=\frac{54}{90}=0.6000. $$ Group L has \(63+17=80\) positive predictions, giving $$ \operatorname{PPV}_L=\frac{63}{80}=0.7875. $$ The difference is \(0.7875-0.6000=0.1875\), or 18.75 percentage points. Group L has the higher observed PPV.
Across both groups, there are \(54+63=117\) true positives and \(90+80=170\) positive predictions. Therefore the combined PPV is \(117/170=0.6882\) to four decimal places. The unweighted average of the two group PPVs would instead be \((0.6000+0.7875)/2=0.69375\), which rounds to 0.6938. That is not the combined PPV because the groups contribute different numbers of positive predictions. The correct pooled calculation weights each group by its positive-prediction count. Check: the combined denominator, 170, equals the total number of positive predictions across both groups.
Common Traps and Responsible Use
- Using the wrong denominator. PPV is \(\operatorname{TP}/(\operatorname{TP}+\operatorname{FP})\). Dividing by all actual positives calculates sensitivity, not PPV.
- Confusing predictive parity with equalized odds. Predictive parity compares PPV; equalized odds compares both sensitivity and false-positive rate. Neither criterion is a substitute name for the other.
- Taking a simple average when pooling. Combined PPV is total true positives divided by total positive predictions. It is generally not the unweighted average of group PPVs.
- Interpreting equal values as proof of fairness. Equal PPV addresses one conditional outcome of a positive decision. It does not establish that the decision is appropriate, that outcomes are measured without bias, or that other error rates or harms are equal.
- Ignoring the evaluation design. If outcomes are observed only for selected cases, or outcome definitions differ across groups, the comparison may describe the observed records rather than all cases subject to the decision.
The practical question is not only whether two PPVs match, but what consequence the metric is intended to monitor. Report the group-specific numerators and denominators, define the positive outcome and decision, and describe what the comparison can and cannot support. Predictive parity is one fairness diagnostic; its usefulness depends on the decision context and the quality and comparability of the evaluation data.
Check Your Understanding
For each problem, use the stated groups and outcome definition. Round probability answers to four decimal places where needed.
- A hypothetical triage model produces 42 true positives and 18 false positives for Group M. Calculate its PPV and the fraction of positive predictions that are false positives. (Answer: PPV \(=42/60=0.7000\); false discovery proportion \(=18/60=0.3000\).)
- Group A has prevalence 0.25, sensitivity 0.80, and false-positive rate 0.10. Group B has prevalence 0.10 with the same sensitivity and false-positive rate. Calculate each PPV using the prevalence formula. (Answer: Group A \(=0.20/(0.20+0.075)=0.7273\); Group B \(=0.08/(0.08+0.09)=0.4706\).)
- A group has 120 positive predictions and a PPV of 0.75. Find the number of true positives and false positives. (Answer: 90 true positives and 30 false positives.)
- Group R has 45 true positives and 15 false positives. Group S has 36 true positives and 24 false positives. Calculate both PPVs and state whether predictive parity holds for these observed counts. (Answer: Group R \(=45/60=0.7500\); Group S \(=36/60=0.6000\); predictive parity does not hold.)
- Group U has 40 true positives and 10 false positives; Group V has 30 true positives and 30 false positives. Calculate the pooled PPV and compare it with the unweighted average of the group PPVs. (Answer: pooled PPV \(=70/110=0.6364\); unweighted average \(=(0.8000+0.5000)/2=0.6500\), so they differ.)