Why Compare Group Selection Rates?
A predictive model can influence who receives an offer, referral, review, or other opportunity. A useful first audit question is whether the proportion selected differs across groups. A disparate impact ratio summarizes one such difference as a relative comparison. It can flag a pattern worth investigating, but it does not explain why the rates differ or establish that a decision process is fair or unfair.
The ratio must be read with its direction in view. The analyst needs to state which group’s selection rate is in the numerator and which group is the reference. A ratio below one then means the numerator group has a lower selection rate than the reference group; a ratio above one means it has a higher rate.
In a dataset, this rate is the observed fraction selected within the group. It can also be described as the empirical conditional proportion of a positive decision given group membership. The ratio is dimensionless: it compares proportions, not counts. Therefore, the groups’ denominators matter when calculating each rate, even though the final ratio contains no units.
Calculation and Interpretation
To calculate the ratio, first calculate each group’s rate using that group’s own denominator. Then divide the numerator group’s rate by the reference group’s rate. A ratio of \(0.75\) means the numerator group’s observed selection rate is 75% of the reference rate, or 25% lower relative to that reference. It does not mean the rate is lower by 75 percentage points.
Worked Example: Comparing Two Groups in a Hypothetical Referral Screen
A hypothetical referral screen selects 72 of 120 applicants in Group Cedar and 54 of 120 applicants in Group Maple. Calculate the Maple-to-Cedar ratio and interpret it. Also compare the rates using a percentage-point difference.
Method. Calculate the selection rate within each group, then divide Maple’s rate by Cedar’s. This avoids the common mistake of comparing the raw numbers selected without accounting for group size.
Cedar’s selection rate is \(72/120=0.60\), or 60%. Maple’s selection rate is \(54/120=0.45\), or 45%.
Using Cedar as the reference, the ratio is \(0.45/0.60=0.75\). Maple’s observed selection rate is therefore 75% of Cedar’s rate, which is 25% lower relative to Cedar’s rate.
The percentage-point difference is \(0.45-0.60=-0.15\), or 15 percentage points lower for Maple. The ratio and the difference describe the same rates on different scales: the ratio is relative, while the difference is absolute.
Check. Reversing the order gives the Cedar-to-Maple ratio \(0.60/0.45=1.3333\) to four decimal places. This is the reciprocal of \(0.75\), as expected.
Why the Ratio Has This Form
The selection rate is a proportion: the count selected divided by the number eligible or considered in that group. A ratio of two such proportions compares the selection likelihoods on a relative scale. Writing \(D=1\) for a positive decision and \(D=0\) otherwise makes the relationship explicit: the empirical rate in group \(g\) estimates the conditional probability of selection given membership in \(g\).
The formula follows directly from the question being asked: “How large is group \(g\)’s selection rate relative to the reference group’s rate?” A relative comparison divides one quantity by the other. For instance, if both rates are multiplied by the same positive factor, their ratio remains unchanged; a percentage-point difference generally changes when rates change by that factor. That is why a ratio can communicate relative disparity, but cannot replace reporting the underlying rates.
Reference Group and Threshold Choices
The ratio depends on the reference group. If two group rates are 0.60 and 0.45, comparing the second with the first gives \(0.75\), while comparing the first with the second gives \(1.3333\). Neither calculation is wrong: they answer opposite directional questions. A report should name the groups and the reference explicitly rather than presenting an unlabeled ratio.
The ratio also depends on the rule that converts a model score into a decision. For a score-based screen, the threshold determines which records count as selected. Raising or lowering the threshold can change each group’s selection rate by different amounts, so the ratio can change as well. A ratio is therefore a summary of a particular population, decision definition, and threshold—not a permanent property of the model.
Worked Example: Comparing Ratios at Two Thresholds
In a hypothetical eligibility screen, Group Alder has 250 applicants and Group Birch has 200. At threshold 50, 100 Alder applicants and 60 Birch applicants are selected. At threshold 65, 60 Alder applicants and 24 Birch applicants are selected. Calculate the rates and the Alder-to-Birch ratio at both thresholds. Then calculate \(R_{50}-R_{65}\), where \(R_t\) is the Alder-to-Birch ratio at threshold \(t\).
Method. Use the same group as the numerator and the same reference group at each threshold. The ratio must be recomputed from the rates at that threshold; comparing counts or silently reversing the reference would change the quantity.
At threshold 50, Alder’s rate is \(100/250=0.4000\), and Birch’s rate is \(60/200=0.3000\). Thus \(R_{50}=0.4000/0.3000=1.3333\) to four decimal places.
At threshold 65, Alder’s rate is \(60/250=0.2400\), and Birch’s rate is \(24/200=0.1200\). Thus \(R_{65}=0.2400/0.1200=2.0000\).
The requested contrast is \(R_{50}-R_{65}=1.3333-2.0000=-0.6667\), rounded to four decimal places. The negative result means the Alder-to-Birch ratio is 0.6667 lower at threshold 50 than at threshold 65; equivalently, the ratio rises by 0.6667 when moving from 50 to 65. This is not a percentage-point change in selection rate.
For comparison, using Birch as the reference produces the reciprocal ratios: \(0.3000/0.4000=0.7500\) at threshold 50 and \(0.1200/0.2400=0.5000\) at threshold 65. Their threshold contrast in the same order is \(0.7500-0.5000=0.2500\), not \(-0.6667\), because this is the reverse ratio. The calculation checks both the stated reference and the direction of subtraction.
Reading a Ratio Without Overclaiming
A ratio can be a useful screening indicator: it compresses a comparison of observed selection rates into one number and can help identify a disparity for further review. It does not, by itself, establish the cause of a rate difference, determine whether the model is appropriate, or settle the broader fairness question. Interpretation depends on the decision’s purpose, the definition of the groups and selection, the data used, and the potential harms identified for the setting.
For example, a ratio close to one means the two observed rates are close relative to one another. It does not mean both rates are high, that everyone had comparable access to the process, or that the decisions were accurate. Conversely, a ratio far from one identifies a relative difference, but the ratio alone does not tell the analyst whether it arises from model behavior, an upstream process, or data limitations. Those possibilities require context and investigation rather than inference from the ratio alone.
Worked Example: How the Reference Changes the Summary
A hypothetical program records these selections: Group Juniper, 200 of 400; Group Larch, 120 of 300; and Group Moss, 40 of 200. Calculate all three rates, then report the ratios for Juniper and Moss using Larch as the reference. Finally, calculate the Moss-to-Juniper ratio.
Method. Calculate each rate from its own group count and denominator before forming any ratios. This avoids treating a larger group’s selected count as a higher selection rate automatically.
Juniper’s rate is \(200/400=0.50\). Larch’s rate is \(120/300=0.40\). Moss’s rate is \(40/200=0.20\).
Using Larch as the reference, Juniper’s ratio is \(0.50/0.40=1.25\), while Moss’s ratio is \(0.20/0.40=0.50\). Using Juniper as the reference instead, the Moss-to-Juniper ratio is \(0.20/0.50=0.40\).
The ratios differ because their reference rates differ: Moss’s rate is half of Larch’s and 40% of Juniper’s. All three calculations are consistent with the same underlying rates. A report that gives a ratio without naming its reference would leave its meaning ambiguous.
Common Exam Traps and Reporting Practice
- Using counts instead of rates. Compare \(S_g/N_g\), not \(S_g\) alone. Groups of different sizes cannot be compared fairly by selected counts alone.
- Leaving the reference unstated. “The ratio is 0.75” is incomplete. State which group’s rate is divided by which reference rate.
- Confusing relative and absolute differences. A ratio of 0.75 means 25% lower relative to the reference rate; it does not mean 75 percentage points lower.
- Reversing a comparison or subtraction. \(R_{g:r}=1/R_{r:g}\) when both rates are positive. When comparing thresholds, write the contrast in the requested order before calculating it.
- Treating one threshold’s result as universal. State the decision rule or threshold used. A different threshold can produce different group rates and a different ratio.
- Calling the ratio a complete fairness judgment. It is a descriptive comparison of selection rates, not a complete explanation or verdict.
Check Your Understanding
For each problem, identify the reference group and keep the direction of the comparison explicit.
- A hypothetical screen selects 48 of 80 applicants in Group Pine and 36 of 90 in Group Reed. Find the Pine-to-Reed ratio and the difference in rates in percentage points. (Answer: Rates are 0.6000 and 0.4000; ratio \(=1.5000\); Pine’s rate is 20 percentage points higher.)
- Group Slate’s selection rate is 0.36 and Group Gold’s is 0.48. Find the Slate-to-Gold ratio and interpret it. (Answer: \(0.36/0.48=0.7500\); Slate’s rate is 25% lower relative to Gold’s.)
- At threshold 40, Group Elm’s rate is 0.50 and Group Fir’s is 0.25. At threshold 55, the rates are 0.30 and 0.20. Find the Elm-to-Fir ratios and the threshold contrast \(R_{40}-R_{55}\). (Answer: \(R_{40}=2.0000\), \(R_{55}=1.5000\), and \(R_{40}-R_{55}=0.5000\).)
- Three groups have rates of 0.72, 0.54, and 0.36. Calculate the third group’s ratio using the first group as reference, then using the second group as reference. (Answer: \(0.36/0.72=0.5000\) and \(0.36/0.54=0.6667\), to four decimal places.)
- A reference group has 0 selected out of 70, while another group has 12 selected out of 60. Is the second-to-reference ratio defined? (Answer: No. The reference rate is zero, so division by the reference rate is undefined; report the rates and explain the limitation rather than giving a numeric ratio.)