Why Group-Specific Error Rates Matter
An overall performance measure can conceal different experiences across groups. For a model that classifies an application as requiring further review, the same overall accuracy could coexist with one group having a higher rate of eligible applications incorrectly flagged and another having a higher rate of ineligible applications incorrectly missed. A useful fairness finding therefore needs more than an overall score: it identifies the groups, the errors, and the question the selected measure is meant to answer.
This tutorial considers a binary classification decision at a specified threshold. The actual positive and actual negative refer to the observed outcome used to evaluate the prediction; the predicted positive and predicted negative are the model’s classifications. The practical meaning of “positive” depends on the application. In a hypothetical screening process, for example, positive might mean “refer for manual review,” not “confirmed misconduct.”
The denominators are essential. FPR is conditional on being actually negative; FNR is conditional on being actually positive. Neither rate is the proportion of all group members who were misclassified. That overall error proportion is \((\mathrm{FP}_g+\mathrm{FN}_g)/N_g\), where \(N_g\) is the total group size, and answers a different question.
What Different Fairness Measures Ask
There is no single error-rate comparison that answers every fairness question. The measure must be named and connected to the concern being assessed. The following definitions describe common statistical comparisons; they are not, by themselves, a complete legal, ethical, or operational judgment.
| Measure | Comparison across groups | Question it addresses |
|---|---|---|
| Equalized odds | FPRs and TPRs are equal (equivalently, FPRs and FNRs are equal). | Among actual negatives, are positive classifications equally frequent? Among actual positives, are positive classifications equally frequent? |
| Equal opportunity | TPRs are equal (equivalently, FNRs are equal). | Among actual positives, are positive classifications equally frequent? |
| Selection-rate parity | The proportion classified positive is equal. | Are members of each group selected or referred at the same rate, regardless of actual outcome? |
| Predictive parity | The positive predictive values are equal, where \( \mathrm{PPV}_g=\mathrm{TP}_g/(\mathrm{TP}_g+\mathrm{FP}_g) \). | Among those classified positive, is the proportion actually positive equal? |
Equal opportunity is narrower than equalized odds: it compares the rate of correctly identifying actual positives, but does not require equal FPRs. Selection-rate parity uses all group members as its denominator, while predictive parity conditions on being classified positive. These measures can therefore tell different stories about the same model. A finding that one measure is equal does not establish that all group experiences or consequences are equal.
Worked Example: Reporting Unequal Error Rates
Worked Example: Reporting Unequal Error Rates
A hypothetical service uses a fixed model threshold to flag applications for manual review. In each of two groups, 50 applications are later confirmed to meet the review criterion and 150 are confirmed not to meet it. For Group North, the model has 40 true positives, 10 false negatives, 30 false positives, and 120 true negatives. For Group South, it has 35 true positives, 15 false negatives, 15 false positives, and 135 true negatives. Calculate each group’s FPR and FNR, compare them, and describe the practical counts.
Method. Calculate FNR using the actual-positive count, and FPR using the actual-negative count. Do not divide either error count by the group total.
For North, there are \(40+10=50\) actual positives and \(30+120=150\) actual negatives:
For South, there are \(35+15=50\) actual positives and \(15+135=150\) actual negatives:
The FNR is 10 percentage points higher in South (30% versus 20%), while the FPR is 10 percentage points higher in North (20% versus 10%). In these hypothetical evaluation data, the model missed 10 actual positives in North and 15 in South; it falsely flagged 30 actual negatives in North and 15 in South. The counts describe observed cases, while the rates allow comparison relative to each actual-outcome group.
Because both rates differ, equalized odds is not met in this sample. Equal opportunity is also not met because the FNRs, and therefore TPRs, differ. The appropriate report is not simply that one group “does worse”: North has the higher false-flag rate among actual negatives, while South has the higher miss rate among actual positives. These are distinct practical consequences.
Common mistake avoided. Dividing North’s 30 false positives by all 200 applications gives 15%, but that is not North’s FPR. The FPR is \(30/150=20\%\), because it asks what fraction of actual negatives were falsely flagged.
Why the Denominators Follow from the Definitions
The rates are conditional proportions. For a group \(g\), let \(Y\) denote the actual binary outcome and \(\widehat{Y}\) the model classification at the stated threshold. A false positive occurs when \(Y=0\) and \(\widehat{Y}=1\); an actual negative is any case with \(Y=0\). Thus the empirical conditional probability is the number of cases satisfying both conditions divided by the number satisfying the conditioning condition:
The denominator is FP plus TN because those are exactly the actual negatives in group \(g\). Replacing it with the whole group size would instead estimate the fraction of all group members who were false positives. By the same reasoning, conditioning on actual positives gives:
This justification also explains why the rates are not interchangeable with error counts. Counts depend on how many people are in the relevant actual-outcome category; conditional rates standardize by that category’s size.
Worked Example: Rates and Counts in Groups of Different Sizes
Worked Example: Rates and Counts in Groups of Different Sizes
A hypothetical model classifies cases for a follow-up process. In Group Cedar, there are 40 actual positives and 160 actual negatives; the confusion counts are TP = 32, FN = 8, FP = 16, and TN = 144. In Group Maple, there are 100 actual positives and 900 actual negatives; the counts are TP = 70, FN = 30, FP = 180, and TN = 720. Calculate both error rates and the total number of errors in each group. Explain why the error counts alone would be a poor group comparison.
Method. First confirm the actual-outcome totals from each confusion matrix, then calculate conditional rates and total errors separately.
For Cedar, the actual-positive total is \(32+8=40\), and the actual-negative total is \(16+144=160\). Therefore:
Cedar has \(8+16=24\) total errors. For Maple, the actual-positive total is \(70+30=100\), and the actual-negative total is \(180+720=900\). Therefore:
Maple has \(30+180=210\) total errors. Maple has many more errors in absolute number, but its group contains five times as many cases as Cedar: \(1{,}000\) versus \(200\). The normalized rates also show that both Maple error rates are higher: its FNR is 10 percentage points higher and its FPR is 10 percentage points higher. Reporting both counts and rates gives context: counts describe the observed workload or number of affected cases, while rates compare error frequency within the relevant actual-outcome groups.
Equalized odds is not met in these results because neither pair of corresponding rates is equal. The report should also state that these are estimates from the evaluated cases, not guarantees about future performance. Where subgroup counts are limited, uncertainty around the rates can be substantial; apparent differences or similarities should not be presented with more precision than the evidence supports.
Common mistake avoided. Comparing 210 Maple errors with 24 Cedar errors and concluding that the model is nearly nine times worse for Maple ignores the group-size difference. The comparison of 21% overall error for Maple (\(210/1{,}000\)) with 12% for Cedar (\(24/200\)) is informative as an overall error comparison, but it does not replace the FPR and FNR comparisons, which condition on actual outcome.
Worked Example: Equal Error Rates Do Not Mean Equal Selection Rates
Worked Example: Equal Error Rates Do Not Mean Equal Selection Rates
A hypothetical model selects cases for a program. Group Harbor has 100 actual positives and 100 actual negatives, with TP = 80, FN = 20, FP = 20, and TN = 80. Group Meadow has 50 actual positives and 150 actual negatives, with TP = 40, FN = 10, FP = 30, and TN = 120. Compare equalized odds, selection rates, and PPV. State what each comparison does and does not establish.
Method. Calculate the conditional error and true-positive rates first. Then calculate selection rate as all predicted positives divided by group size, and PPV as true positives divided by predicted positives.
For Harbor:
The number selected is \(80+20=100\), so the selection rate is \(100/200=0.50\). Harbor’s PPV is \(80/(80+20)=80/100=0.80\).
For Meadow:
The number selected is \(40+30=70\), so the selection rate is \(70/200=0.35\). Meadow’s PPV is \(40/(40+30)=40/70\approx0.5714\), or about 57.14%.
The observed FPRs and TPRs match, so equalized odds is satisfied in these evaluation data. The selection rates do not match (50% versus 35%), and the PPVs do not match (80% versus about 57.14%). Thus equalized odds answers a question about error rates conditional on actual outcome; it does not imply equal selection rates or equal proportions of actual positives among selected cases. These figures do not, on their own, determine which difference is acceptable or why it arose.
Common mistake avoided. It would be incorrect to infer that equal error rates force equal selection rates. Here the groups have different proportions of actual positives, and the same conditional rates yield different numbers and proportions selected. Each measure must be reported for the question it addresses.
How to Present a Fairness Finding
A clear report makes the comparison reproducible and keeps the conclusion proportional to the evidence. State the population and evaluation period, the group definitions, the positive outcome, and the threshold used to generate classifications. Then give each group’s confusion counts or enough denominators to interpret the rates. Report rates with denominators and, where appropriate, percentage-point differences; avoid describing a difference only as a relative increase when the underlying rates are small.
Explain what “positive” means, what the model’s positive classification triggers, and which observed outcome is used as the reference.
Use FNR or TPR to assess misses among actual positives; use FPR to assess false flags among actual negatives. If the concern is equal selection or equal precision among selected cases, report the corresponding measure separately.
For example, report “15 of 50 actual positives were missed (FNR 30%),” not only “the FNR was 30%.” This makes the amount of evidence visible.
Translate errors into the hypothetical decision’s consequences, and qualify the finding by the evaluation population, data quality, and uncertainty. A measured disparity is a reason to investigate and explain, not a complete causal explanation or verdict.
These rates depend on the chosen threshold and on the observed outcome used to define actual positives and negatives. They also depend on whether the evaluation data adequately represent the population in which the model will be used. Report the threshold and evaluation context; do not imply that a rate observed in one sample is fixed for every future setting. When a subgroup has few actual positives or negatives, its corresponding FNR or FPR may be imprecise, so include an appropriate uncertainty caveat rather than treating small numerical differences as conclusive.
Check Your Understanding
For each hypothetical evaluation, identify the requested comparison and keep the conditioning denominator clear.
- A group has 24 false positives and 96 true negatives. Calculate its FPR. (Answer: \(24/(24+96)=0.2000\), or 20.00%.)
- A group has 18 false negatives and 72 true positives. Calculate its FNR and TPR. (Answer: FNR \(=18/(18+72)=0.2000\); TPR \(=72/90=0.8000\).)
- Group A has 30 actual positives, of whom 24 are classified positive. Group B has 80 actual positives, of whom 64 are classified positive. Does the observed pair meet equal opportunity? (Answer: Yes; both TPRs are \(0.8000\), equivalently both FNRs are \(0.2000\).)
- Group Pine has 40 true positives and 10 false positives. Group Birch has 18 true positives and 12 false positives. Calculate each PPV and state whether predictive parity is observed. (Answer: Pine \(=40/(40+10)=0.8000\); Birch \(=18/(18+12)=0.6000\); no, the PPVs differ.)
- Group Ridge has 45 predicted positives among 150 members; Group Lake has 54 predicted positives among 240 members. Calculate selection rates and state whether selection-rate parity is observed. (Answer: Ridge \(=45/150=0.3000\); Lake \(=54/240=0.2250\); no, the rates differ by 7.50 percentage points.)