Why a Neutral Predictor Can Raise a Fairness Concern
A predictive model does not need to include a sensitive attribute explicitly to produce different predictions across groups. A location code, service history, or pattern of prior transactions may be associated with a sensitive attribute, or may reflect unequal access to services and opportunities. If a model uses that information, its predictions can reproduce or amplify patterns represented in the data.
That possibility calls for investigation, not an automatic conclusion. A feature's association with a group does not, by itself, establish that the feature is inappropriate, that the model caused a disparity, or that a particular decision is unlawful. A sound report identifies the plausible pathway, shows relevant evidence, and states what remains uncertain.
What the Evidence Can and Cannot Show
Let \(G\) denote a group, \(Z\) a candidate predictor, and \(D\) a binary model decision, such as whether a case is flagged for review. Examining whether \(Z\) and \(G\) are associated can show that the predictor carries group information. Examining model outputs across groups can show how the model's decisions differ. These are descriptive findings. Neither one alone identifies why the association exists or establishes a causal effect.
The relationship between a predictor and group membership may arise through several routes. The predictor may be a close substitute for a sensitive attribute; it may reflect historical decisions; or it may measure a real difference in circumstances that itself results from unequal conditions. It can also be associated with group membership for reasons that do not create a material model effect. The analyst should therefore distinguish three questions: whether a proxy relationship is plausible, whether using the feature changes model outputs or errors, and whether that change is justified for the intended decision.
The law of total expectation provides a useful way to examine group differences in a binary decision. For a discrete \(Z\), the decision rate in group \(g\) is a weighted average of the decision rates within the levels of \(Z\). For a continuous \(Z\), the corresponding expression is an integral when the conditional density exists. More generally, the identity is a conditional expectation:
The reason is that the event \(D=1\) can be partitioned by the values of \(Z\). Within each group, its overall probability is the average of the conditional decision probability at each value of \(Z\), weighted by how common that value is in the group. The sum is the discrete form of that average; the integral is the continuous form. This identity does not assume that \(Z\) causes \(D\), or that \(Z\) is the only factor associated with the decision.
Worked Example: Separating Group Composition From Within-Level Differences
A hypothetical review model flags cases when they need additional assessment. A reviewer compares groups A and B across two levels of a location-based predictor, \(Z\). In group A, 70% of cases have \(Z=1\) and 30% have \(Z=2\); the flag rates are 10% and 50%, respectively. In group B, 20% have \(Z=1\) and 80% have \(Z=2\); the corresponding flag rates are 20% and 60%. Calculate the overall flag rate in each group, then standardize group B to group A's distribution of \(Z\). State what this calculation does and does not establish.
Method. Apply the discrete weighted-average identity. The common mistake is to compare conditional rates in just one level, or to use the wrong group's feature distribution in the overall rate.
Group A overall rate. Weight each conditional flag rate by its share of group A:
Group A's overall flag rate is 22%.
Group B overall rate. Use group B's own shares:
Group B's overall flag rate is 52%, an observed difference of \(0.52-0.22=0.30\), or 30 percentage points.
Standardized group B rate. Reweight group B's conditional rates using group A's distribution of \(Z\):
With this chosen standardization, the difference between group B's standardized rate and group A's rate is \(0.32-0.22=0.10\), or 10 percentage points. The remaining 20 percentage points of the original difference arise from replacing group B's \(Z\) distribution with group A's distribution in this calculation: \(0.52-0.32=0.20\). The components add back to the observed difference, \(0.10+0.20=0.30\).
Check and interpretation. The weighted rates use shares that sum to 1 in each group, and the decomposition reconciles to the original 30-point gap. It is a descriptive decomposition, not a causal allocation: the result depends on the chosen reference distribution, and other variables may also differ across groups. The calculation flags a possible feature-composition pathway for further review; it does not prove that \(Z\) caused the difference or that the model is discriminatory.
Reviewing a Candidate Proxy
A useful review combines information about the feature with evidence about the model's behavior. Begin with the feature's meaning and data provenance: how it was created, what it measures in practice, and whether historical decisions or access patterns shaped its values. Then assess its association with sensitive groups, using appropriate data and measures. A strong association may make proxy use plausible, but association alone is not a reasoned recommendation.
Next examine whether the feature materially affects predictions or decisions. Compare model behavior with and without the feature using an appropriate validation design, and examine relevant outcomes and errors across groups. A variable-importance result can help identify a feature for investigation, but importance is not unique attribution: correlated predictors can share or substitute for predictive information. A small change in overall performance also does not guarantee that subgroup effects are small.
The review should ask whether the feature has a defensible connection to the intended prediction, whether it is measured consistently, and whether less concerning alternatives provide comparable predictive value. It should also document uncertainty, data limitations, and the intended use of the model. Where evidence is incomplete, that limitation belongs in the recommendation. Consultation with appropriate legal, compliance, and governance specialists may be necessary; a statistical analysis alone does not determine legal permissibility.
Name the candidate feature, the sensitive attribute or structural disadvantage it may reflect, and the decision context. Identify this as a hypothesis to investigate, not a finding of causation.
Review its definition, source, timing, missingness, and any historical processes that influenced its values. A label that sounds neutral may summarize prior access or decisions.
Examine how the feature varies across groups and how predictions, decisions, and relevant errors vary with and without it. Interpret subgroup results in light of sample size and data quality.
Determine whether the feature is connected to the intended task, whether a less concerning alternative is available, and what the evidence cannot establish. Document the basis for any recommendation.
Worked Example: A Service-Distance Feature in a Missed-Appointment Model
A hypothetical clinic uses a model to estimate missed-appointment risk. One predictor is travel distance to the clinic. The model does not include a sensitive attribute, but an audit finds that 80% of group A and 20% of group B live in the longer-distance category. The model's average predicted risk is 10% for the shorter-distance category and 30% for the longer-distance category, regardless of group. Calculate the average predicted risk by group and describe the evidence needed before recommending the distance feature.
Method. Use the law of total expectation with the group-specific distribution of distance. The common mistake is to conclude that omitting a sensitive attribute prevents group-linked effects, or to interpret predicted risk as the observed missed-appointment rate.
Group A. The longer-distance share is 0.80, so the shorter-distance share is \(1-0.80=0.20\):
Group A's average predicted risk is 26%.
Group B. The longer-distance share is 0.20, so the shorter-distance share is \(1-0.20=0.80\):
Group B's average predicted risk is 14%. The difference is \(0.26-0.14=0.12\), or 12 percentage points. Under the stated model outputs, the difference follows from the groups' different distance distributions; it is not evidence that the model uses the sensitive attribute directly. It also does not show whether either group's predictions are accurate.
Review and check. The calculation uses the stated 20% and 80% shares correctly, and each weighted average lies between 10% and 30%. Before recommending the feature, the analyst should examine how distance is measured, whether it is a relevant and reliable predictor for the intended use, observed outcomes and errors by group, the effect of removing or replacing the feature on held-out data, and whether distance reflects barriers that the decision process should not reinforce. Predicted risks are not observed event rates; calibration and error evidence are needed to assess their relationship to outcomes. The calculation identifies a potential proxy pathway, not a final fairness or causality finding.
Common Exam Trap: Treating Association as a Verdict
A tempting but incorrect argument is: “The feature is strongly associated with group membership, so it is discriminatory and must be removed.” This skips the necessary questions about what the feature measures, how it affects predictions, its connection to the task, the quality of the evidence, and available alternatives. The opposite argument is also unsound: “The sensitive attribute is absent from the model, so there can be no group-related concern.” A correlated predictor can still carry group information.
A more defensible report separates findings from conclusions. For example: “Longer travel distance is more common in group A, and the model's predicted risk is higher for that category. This creates a plausible pathway for group differences in average predictions. The current analysis does not establish that distance causes the difference or whether the predictions are equally accurate. We recommend reviewing the feature's relevance and data provenance, and evaluating subgroup outcomes and a model without the feature before deciding whether its use is justified.” This wording is specific without claiming more than the evidence supports.
Worked Example: Weighing a Small Performance Gain Against a Proxy Signal
A hypothetical insurer tests a location category as a predictor of claim severity. On the same held-out evaluation sample, the model's root mean squared error is $1,230 without the feature and $1,190 with it. In the evaluation data, 28% of group A and 12% of group B receive a high-severity flag. Calculate the relative reduction in root mean squared error and the difference in flag rates. Then state a measured recommendation. Assume the group counts are 400 for group A and 600 for group B.
Method. Calculate the relative error reduction using the model without the feature as the baseline, and compare the group flag rates in percentage points. The common mistake is to present a lower overall error as sufficient evidence that the feature should be retained.
Error reduction. The absolute reduction is \(1{,}230-1{,}190=\$40\). Relative to the baseline:
The model with the feature has a 3.3% lower root mean squared error on this evaluation, rounded to one decimal place.
Flag-rate difference. The rate difference is \(28\%-12\%=16\) percentage points. For a count check, group A has \(400(0.28)=112\) high-severity flags, and group B has \(600(0.12)=72\). The different group sizes do not change the comparison of rates; the rates are 28% and 12%, respectively.
Recommendation and check. The calculations support reporting a modest overall error reduction alongside a 16-point difference in flag rates. They do not establish that the location feature caused the difference, that the difference is unjustified, or that errors are equal across groups. A measured recommendation is to defer a final endorsement until the feature's meaning and relevance, subgroup error and outcome evidence, and a comparable evaluation of alternatives have been reviewed. The stated counts reproduce the rates, and the error reduction uses the correct baseline. A lower overall error alone is not enough to resolve the proxy concern.
Reporting a Defensible Conclusion
A useful conclusion states the observed association, the model behavior examined, and the limits of the analysis separately. Avoid labels such as “fair” or “unfair” unless the report defines the criterion and supports the claim with relevant evidence. If group data are sparse or incomplete, say so; an apparent absence of a difference may reflect limited evidence rather than evidence of no difference.
The recommendation should be proportionate to what was learned. Depending on the findings, it may call for further validation, feature review, comparison with an alternative, or a decision not to recommend use until key evidence is available. The essential discipline is to avoid treating either a neutral name or a statistical association as the whole answer. Proxy analysis is a structured review of how information enters a model and what the resulting evidence can support.
Check Your Understanding
For each problem, calculate what is requested and state the appropriate limit on interpretation.
- A binary flag rate is 15% for \(Z=1\) and 45% for \(Z=2\). Group C has 60% at \(Z=1\) and 40% at \(Z=2\); group D has 30% at each level and 40% at \(Z=3\). The flag rate for \(Z=3\) is 5%. Calculate both overall rates and their difference. (Answer: Group C: \(0.60(0.15)+0.40(0.45)=0.27\), or 27%; group D: \(0.30(0.15)+0.30(0.45)+0.40(0.05)=0.20\), or 20%; C exceeds D by 7 percentage points. This descriptive difference does not establish causation.)
- For a continuous predictor \(Z\) uniformly distributed from 0 to 1 in a group, suppose \(P(D=1\mid Z=z,G=g)=0.10+0.20z\). Find the group's overall decision rate. (Answer: \(\int_0^1(0.10+0.20z)\,dz=0.10+0.10=0.20\), or 20%.)
- A model predicts risk of 8% for nearby locations and 28% for distant locations. Group E is 75% distant; group F is 25% distant. Calculate average predicted risk by group and the difference. (Answer: E: \(0.25(0.08)+0.75(0.28)=0.23\), or 23%; F: \(0.75(0.08)+0.25(0.28)=0.13\), or 13%; the difference is 10 percentage points. It is a difference in predicted risk, not proof of a difference in observed outcomes.)
- A candidate proxy is associated with group membership, but removing it changes overall validation error very little. Name two additional pieces of evidence needed before making a recommendation. (Answer: Examples include subgroup outcome and error results; the feature's provenance and practical meaning; evidence of its relevance to the intended task; or comparison with a suitable alternative. Association and overall error alone do not settle the recommendation.)
- In a hypothetical severity model, root mean squared error falls from $840 to $810 when a location feature is added. The high-severity flag rates are 22% and 14% in two groups. Calculate the relative error reduction and the flag-rate difference, then state one conclusion the figures do not justify. (Answer: \((840-810)/840=30/840\approx0.03571\), or 3.6% to one decimal place; the flag-rate difference is 8 percentage points. These figures alone do not prove that the location feature caused the difference or that the feature is justified.)