Why Compare Errors by Group?
A model can influence consequential decisions even when it does not make the final decision itself. For example, a hypothetical insurer might use a model to flag applications for additional review. If the flag rate differs across groups, that difference alone does not show which groups are more often flagged incorrectly. To examine errors, we need to distinguish cases where the recorded outcome is positive from cases where it is negative.
Equalized odds is a group fairness criterion for a binary prediction. It asks whether a model has the same true-positive rate and the same false-positive rate across the groups being compared. These rates answer different questions: one concerns positive cases that are correctly flagged, and the other concerns negative cases that are incorrectly flagged.
The true-positive rate (TPR) is \(P(\hat{Y}=1\mid Y=1,A=a)\), the proportion of recorded positive cases predicted positive.
The false-positive rate (FPR) is \(P(\hat{Y}=1\mid Y=0,A=a)\), the proportion of recorded negative cases predicted positive.
The model satisfies equalized odds across the groups being compared if both rates are equal across those groups: \(P(\hat{Y}=1\mid Y=1,A=a)\) is the same for every group \(a\), and \(P(\hat{Y}=1\mid Y=0,A=a)\) is also the same for every group \(a\).
The outcome \(Y\) is the reference used to calculate these rates. In an audit, it might be a documented event or a recorded determination. It should not automatically be treated as an error-free measure of the underlying condition: limitations in the outcome labels can affect the interpretation of the calculated rates.
Reading the Confusion Matrix
A confusion matrix places the recorded outcome in one dimension and the model prediction in the other. “Positive” refers to the category coded as \(1\); its practical meaning depends on the application. For a referral model, for example, \(\hat{Y}=1\) might mean “refer for review,” while \(Y=1\) might mean that a specified condition was recorded.
| Recorded outcome | Predicted positive \(\hat{Y}=1\) | Predicted negative \(\hat{Y}=0\) | Recorded-outcome total |
|---|---|---|---|
| \(Y=1\) | True positive (TP) | False negative (FN) | TP + FN |
| \(Y=0\) | False positive (FP) | True negative (TN) | FP + TN |
The denominator for TPR is the number of recorded positive cases, \(TP+FN\). The denominator for FPR is the number of recorded negative cases, \(FP+TN\). Both rates are conditional on the recorded outcome, not on the model's prediction. Calculate them separately for each group using that group's counts.
Worked Example: Equal Error Rates With Different Group Counts
A hypothetical review system is evaluated for two groups. In Group A, the confusion matrix has \(TP=72\), \(FN=18\), \(FP=20\), and \(TN=80\). In Group B, it has \(TP=64\), \(FN=16\), \(FP=30\), and \(TN=120\). Calculate both rates for each group and decide whether the observed rates satisfy equalized odds.
Method. For each group, divide true positives by all recorded positive cases to get TPR, and divide false positives by all recorded negative cases to get FPR.
For Group A, the recorded positive total is \(72+18=90\), so \(\mathrm{TPR}_A=72/90=0.8000\). Its recorded negative total is \(20+80=100\), so \(\mathrm{FPR}_A=20/100=0.2000\).
For Group B, the recorded positive total is \(64+16=80\), so \(\mathrm{TPR}_B=64/80=0.8000\). Its recorded negative total is \(30+120=150\), so \(\mathrm{FPR}_B=30/150=0.2000\).
The TPR difference is \(0.8000-0.8000=0.0000\), and the FPR difference is \(0.2000-0.2000=0.0000\). Thus the observed rates are equal, so this comparison satisfies equalized odds. The conclusion is about these empirical rates; it does not establish that the underlying population rates are exactly equal.
Check. Every denominator is a recorded-outcome total: 90 and 80 for the positive cases, and 100 and 150 for the negative cases. This avoids using a prediction total as a denominator.
Why Equalized Odds Requires Two Comparisons
The definition can be understood by considering the two possible recorded outcomes separately. Among cases with \(Y=1\), a model makes a false negative when it predicts zero. Therefore, comparing TPRs compares how often each group’s recorded positive cases are correctly predicted positive. Among cases with \(Y=0\), a model makes a false positive when it predicts one. Comparing FPRs compares how often each group’s recorded negative cases are incorrectly predicted positive.
For group \(a\), there are \(TP_a+FN_a\) cases with \(Y=1\). Of these, \(TP_a\) receive a positive prediction, so the positive-prediction fraction is \(TP_a/(TP_a+FN_a)\), the TPR.
For group \(a\), there are \(FP_a+TN_a\) cases with \(Y=0\). Of these, \(FP_a\) receive a positive prediction, so the positive-prediction fraction is \(FP_a/(FP_a+TN_a)\), the FPR.
Equalized odds holds only if the conditional positive-prediction fractions match across groups in both outcome strata: the \(Y=1\) fractions match and the \(Y=0\) fractions match.
This gives a compact probability statement: equalized odds requires that the distribution of \(\hat{Y}\), conditional on \(Y\), does not vary by group. In symbols, \(\hat{Y}\) and \(A\) are conditionally independent given \(Y\), for the groups and outcome labels included in the comparison. The statement is equivalent to checking both rates because \(\hat{Y}\) is binary: when the conditional probability of \(\hat{Y}=1\) matches, the conditional probability of \(\hat{Y}=0\), its complement, matches as well.
Worked Example: Detecting a Difference in Both Error Rates
A hypothetical organization uses a model to flag applications for a specified review. Here, \(Y=1\) means the application was recorded as having a defined eligibility violation, and \(\hat{Y}=1\) means the model flags the application. Group C has \(TP=54\), \(FN=6\), \(FP=36\), and \(TN=144\). Group D has \(TP=68\), \(FN=12\), \(FP=30\), and \(TN=170\). Calculate and interpret the differences.
For Group C, the recorded positive total is \(54+6=60\), giving \(\mathrm{TPR}_C=54/60=0.9000\). The recorded negative total is \(36+144=180\), giving \(\mathrm{FPR}_C=36/180=0.2000\).
For Group D, the recorded positive total is \(68+12=80\), giving \(\mathrm{TPR}_D=68/80=0.8500\). The recorded negative total is \(30+170=200\), giving \(\mathrm{FPR}_D=30/200=0.1500\).
Taking Group C minus Group D, the TPR difference is \(0.9000-0.8500=0.0500\), or 5 percentage points. The FPR difference is \(0.2000-0.1500=0.0500\), also 5 percentage points. Both rates differ, so equalized odds is not satisfied by these observed rates.
In context, among applications recorded as having the specified violation, the model's flag rate is 5 percentage points higher for Group C. Among applications recorded as having no specified violation, its flag rate is also 5 percentage points higher for Group C. These statements name the correct outcome group for each rate.
Check. Group C’s TPR denominator is 60, not \(TP+FP=90\); its FPR denominator is 180, not the total number flagged. The rates therefore compare the same recorded-outcome strata across groups.
Interpreting Differences in a Consequential Decision
A rate difference is a description of model performance relative to the recorded outcome. In a consequential setting, interpretation should name the group, the outcome stratum, the model action, and the size of the difference. “Group F has a higher error rate” is incomplete: it does not identify which error or which cases are in the denominator.
Worked Example: Translating Rate Differences Into Context
Suppose a hypothetical application-screening model flags an application for further review when \(\hat{Y}=1\). The recorded outcome \(Y=1\) means that a specified violation was found. Group F has \(TP=64\), \(FN=16\), \(FP=30\), and \(TN=170\). Group G has \(TP=70\), \(FN=30\), \(FP=40\), and \(TN=160\). Calculate the rates and state what their differences mean.
For Group F, there are \(64+16=80\) recorded positive cases, so \(\mathrm{TPR}_F=64/80=0.8000\). There are \(30+170=200\) recorded negative cases, so \(\mathrm{FPR}_F=30/200=0.1500\).
For Group G, there are \(70+30=100\) recorded positive cases, so \(\mathrm{TPR}_G=70/100=0.7000\). There are \(40+160=200\) recorded negative cases, so \(\mathrm{FPR}_G=40/200=0.2000\).
The TPR difference, F minus G, is \(0.8000-0.7000=0.1000\), or 10 percentage points. The FPR difference is \(0.1500-0.2000=-0.0500\): Group F’s FPR is 5 percentage points lower than Group G’s. Since both rates differ, the observed rates do not satisfy equalized odds.
Correctly interpreted: among applications recorded as having the specified violation, the model's referral rate is 10 percentage points higher for Group F. Among applications recorded as having no specified violation, its referral rate is 5 percentage points lower for Group F. The second comparison concerns recorded negatives; it is not a comparison among the applications recorded as having a violation.
Check. Group G has more false positives in count (40 versus 30), but both groups have 200 recorded negatives. Its FPR is therefore higher, \(0.2000\) versus \(0.1500\), consistent with the rate difference above.
Common Mistakes and Limits
- Using the wrong denominator. TPR uses \(TP+FN\); FPR uses \(FP+TN\). Dividing false positives by all predicted positives gives a different quantity, not FPR.
- Checking only one rate. Equal TPRs do not establish equalized odds if FPRs differ. Both conditional comparisons are required.
- Reversing the outcome stratum. TPR describes cases with \(Y=1\); FPR describes cases with \(Y=0\). State the recorded outcome explicitly when translating a difference into words.
- Confusing counts with rates. Groups can have different numbers of errors because their group sizes differ, even when their rates match. Conversely, similar counts need not imply similar rates.
- Overstating what the audit establishes. Equalized observed rates do not establish that the labels are accurate, the decision process is harmless, or the model is fair in every relevant sense. The conclusion also depends on which groups, labels, threshold, and evaluation population were selected.
For a clean comparison, report the group-specific confusion-matrix counts, both rates, and clearly defined rate differences. If the evaluation uses a particular threshold, state that threshold or the rule that determines it; changing the prediction rule can change both error rates. A measured difference should prompt contextual examination of the decision and the recorded outcomes, rather than being treated as a complete account of fairness.
Check Your Understanding
For each problem, calculate the requested rates from the recorded-outcome totals. Report percentage-point differences where requested.
- A hypothetical permit-review model has Group M counts \(TP=42\), \(FN=18\), \(FP=24\), \(TN=96\), and Group N counts \(TP=63\), \(FN=27\), \(FP=15\), \(TN=85\). Calculate both rates for each group and decide whether equalized odds holds. (Answer: Group M has TPR \(42/60=0.7000\) and FPR \(24/120=0.2000\); Group N has TPR \(63/90=0.7000\) and FPR \(15/100=0.1500\). TPRs match, but FPRs differ by 0.0500, so equalized odds does not hold.)
- For a hypothetical equipment-inspection flag, Group K has \(TP=36\), \(FN=24\), \(FP=10\), \(TN=90\), and Group L has \(TP=45\), \(FN=15\), \(FP=20\), \(TN=80\). Find the TPR and FPR differences, taking K minus L. (Answer: K has TPR \(36/60=0.6000\) and FPR \(10/100=0.1000\); L has TPR \(45/60=0.7500\) and FPR \(20/100=0.2000\). The differences are \(-0.1500\) for TPR and \(-0.1000\) for FPR.)
- In a hypothetical review setting, Group R has 50 recorded positive cases, of which 40 are predicted positive, and 150 recorded negative cases, of which 30 are predicted positive. Group S has 80 recorded positive cases, of which 64 are predicted positive, and 120 recorded negative cases, of which 18 are predicted positive. Calculate both rates and determine whether the observed rates satisfy equalized odds. (Answer: R has TPR \(40/50=0.8000\) and FPR \(30/150=0.2000\); S has TPR \(64/80=0.8000\) and FPR \(18/120=0.1500\). The TPRs match, but the FPRs differ by 0.0500, so equalized odds does not hold.)
- A hypothetical eligibility model has the same TPR, \(0.7500\), for Groups U and V. Among 200 recorded negative cases in U, 36 are predicted positive. Among 160 recorded negative cases in V, 24 are predicted positive. Calculate the FPR for each group and state whether equalized odds holds. (Answer: U has FPR \(36/200=0.1800\); V has FPR \(24/160=0.1500\). Despite matching TPRs, the FPRs differ by 0.0300, so equalized odds does not hold.)
- A hypothetical screening rule has Group W counts \(TP=56\), \(FN=14\), \(FP=18\), and \(TN=102\). Group Z has 90 recorded positive cases and 100 recorded negative cases. To match W’s observed TPR and FPR, how many true positives and false positives would Z need, respectively? (Answer: W’s TPR is \(56/70=0.8000\), and its FPR is \(18/120=0.1500\). Z would need \(0.8000(90)=72\) true positives and \(0.1500(100)=15\) false positives.)