Evaluating Fairness Interventions in Context

Evaluate the Decision, Not Just the Metric

A fairness intervention can improve one group comparison while changing another error rate, the number of people referred for review, or the cost of missed cases. A decision-maker therefore needs more than a statement such as “the true positive rate gap fell.” The relevant question is whether the intervention produces an acceptable combination of outcomes for the people affected and for the process that must deliver the decision.

This is a contextual evaluation, not a search for one universally correct fairness statistic. Earlier material on group error rates, predictive parity, calibration, and threshold effects supplies useful diagnostics. Here, those diagnostics are considered alongside the purpose of the decision, the consequences of different errors, uncertainty in estimated effects, and operational feasibility. The examples use hypothetical screening and referral systems; their figures are invented for instruction.

Definition: An intervention is a specified change to a model or decision process, such as applying a different decision threshold or changing how cases are referred. To evaluate it in context, compare the intervention with a clearly stated alternative using (1) decision-relevant outcomes, (2) impacts for affected groups, (3) uncertainty in the estimates, and (4) operational consequences. The acceptable balance among these dimensions depends on the decision’s purpose and the consequences borne by people affected by it.

Start With Consequences and a Defined Comparison

A comparison is meaningful only when the alternatives and outcomes are explicit. Define the baseline policy, the proposed intervention, the population and evaluation period, and the outcome used to judge the decision. Then identify which errors matter and why. For example, a false negative might mean a person needing assistance is not referred, while a false positive might mean a person who does not need assistance receives an unnecessary review. The relative consequences should be justified by the decision context rather than chosen merely to make one policy appear better.

When a decision-maker has agreed on costs per event, a simple cost summary can make tradeoffs visible. Suppose each false negative has cost $c_{\mathrm{FN}}$, each false positive has cost $c_{\mathrm{FP}}$, and each referral has cost $c_{\mathrm{R}}$. For observed counts, the specified total cost is:

C=c_{\mathrm{FN}}N_{\mathrm{FN}}+ c_{\mathrm{FP}}N_{\mathrm{FP}}+ c_{\mathrm{R}}N_{\mathrm{R}}.

This is a decision aid, not a claim that all consequences are naturally reducible to dollars. A cost summary omits consequences that were not specified, and its conclusion depends on the values assigned to each event. It should be accompanied by group-level results and operational information.

Worked Comparisons

Worked Example: Compare Two Referral Policies by Specified Cost

A hypothetical program compares policies R and S on the same evaluation population. Policy R produces 4 false negatives, 18 false positives, and 25 referrals. Policy S produces 2 false negatives, 27 false positives, and 25 referrals. The decision team has specified costs of $400 per false negative, $40 per false positive, and $8 per referral. Which policy has the lower specified cost, and by how much?

Method. Apply the stated cost formula to each policy. This comparison assumes the three cost amounts are the agreed values for this decision and that the reported event counts use the same population and definitions under both policies.

For policy R, the false-negative cost is $4(400)=\$1{,}600$, the false-positive cost is $18(40)=\$720$, and the referral cost is $25(8)=\$200$. Therefore:

C_R=1{,}600+720+200=\$2{,}520.

For policy S, the false-negative cost is $2(400)=\$800$, the false-positive cost is $27(40)=\$1{,}080$, and the referral cost is $25(8)=\$200$. Therefore:

C_S=800+1{,}080+200=\$2{,}080.

The difference is $2{,}520-2{,}080=\$440$, so S has the lower specified cost by $440. The calculation checks because S has two fewer false negatives, saving $2(400)=\$800$, but nine more false positives, adding $9(40)=\$360$; referrals are unchanged. The net saving is $800-360=\$440$.

Common mistake avoided: Do not add or compare partial costs and then infer the total. In particular, policy S’s total is $2,080, not $2,480. Also, the result is conditional on the specified costs; it does not establish that S is fairer or preferable under every relevant outcome.

Worked Example: Assess a Threshold Change Across Groups

A hypothetical service evaluates 100 people in group A, including 40 who need assistance and 60 who do not. Its baseline decisions are 32 true positives, 8 false negatives, 12 false positives, and 48 true negatives. In group B, 20 of 100 people need assistance and 80 do not; baseline decisions are 10 true positives, 10 false negatives, 8 false positives, and 72 true negatives. A proposed threshold change for group B yields 14 true positives, 6 false negatives, 16 false positives, and 64 true negatives. Group A’s decisions do not change. The stated costs are $300 per false negative and $30 per false positive. Compare error rates, specified costs, and referral workload.

Method. For each group, calculate the true positive rate as true positives divided by people who need assistance, and the false positive rate as false positives divided by people who do not. Treat a referral as any positive decision, so referrals equal true positives plus false positives. These rates describe this evaluation population; they do not by themselves determine the appropriate policy.

For group A, the true positive rate is $32/40=0.80$, and the false positive rate is $12/60=0.20$. For group B under the baseline, the corresponding rates are $10/20=0.50$ and $8/80=0.10$. The baseline group gap in true positive rates is $0.80-0.50=0.30$; the gap in false positive rates is $0.20-0.10=0.10$.

After the intervention, group B’s true positive rate is $14/20=0.70$, and its false positive rate is $16/80=0.20$. Group A remains at 0.80 and 0.20. Thus, the true positive rate gap falls to $0.80-0.70=0.10$, and the false positive rate gap falls to $0.20-0.20=0$. But group B’s positive predictive value changes from $10/(10+8)=10/18=0.5556$ to $14/(14+16)=14/30=0.4667$, rounded to four decimals. A smaller rate gap does not mean every group-level outcome improved.

For the specified costs, group A costs $8(300)+12(30)=2{,}400+360=\$2{,}760$. Group B’s baseline cost is $10(300)+8(30)=3{,}000+240=\$3{,}240$. Total baseline cost is $2{,}760+3{,}240=\$6{,}000$. Under the intervention, group B costs $6(300)+16(30)=1{,}800+480=\$2{,}280$; total cost becomes $2{,}760+2{,}280=\$5{,}040$. The specified saving is $960.

Baseline referrals are $32+12=44$ in group A and $10+8=18$ in group B, for 62 total. After the intervention, referrals are 44 in A and $14+16=30$ in B, for 74 total: 12 additional referrals. The cost and rate calculations check against the changed counts: group B has four fewer false negatives and eight more false positives, producing a cost change of $-4(300)+8(30)=-\$960$.

Common mistake avoided: Do not report only the reduced true positive rate gap. The decision-maker must also see the changed positive predictive value, the specified costs, and the extra review workload. Whether the additional referrals are feasible and whether the error tradeoff is acceptable require contextual judgment.

Uncertainty Changes How Strongly to Conclude

Group rates are estimates when calculated from a sample. A difference between two observed rates may reflect a real policy effect, sampling variation, or both. State the denominators and consider uncertainty before describing a small numerical difference as decisive. For a rate based on $x$ successes among $n$ independent observations with a common success probability $p$, the binomial model is $X\sim\operatorname{Binomial}(n,p)$, and the estimated rate is $\hat p=x/n$. Its estimated standard error is $\sqrt{\hat p(1-\hat p)/n}$. This normal-approximation standard error is a rough summary; it can be unreliable with small samples or rates near zero or one.

Worked Example: Interpret an Uncertain Group Difference

A hypothetical evaluation uses separate, independent validation samples of people who need assistance. Under the baseline, 10 of 20 people in group B are correctly referred. Under an intervention, 14 of a different 20 people in group B are correctly referred. Estimate the change in the true positive rate and use a rough normal-approximation interval to assess its precision.

Method. Treat each sample’s correct-referral indicator as binomial, with a common probability within that sample. The samples are stated to be independent, so the estimated variance of the difference is the sum of the two estimated variances. This calculation would not be appropriate as written for paired decisions on the same people, because the paired outcomes would be dependent.

The baseline rate is $\hat p_0=10/20=0.50$; the intervention rate is $\hat p_1=14/20=0.70$. The estimated increase is $0.70-0.50=0.20$. The estimated standard error of the difference is:

\widehat{\operatorname{SE}}(\hat p_1-\hat p_0) = \sqrt{\frac{0.70(0.30)}{20}+\frac{0.50(0.50)}{20}} = \sqrt{0.0105+0.0125} = \sqrt{0.023} \approx 0.1517.

A rough 95% normal-approximation interval is the estimate plus or minus $1.96$ standard errors. Its margin is $1.96(0.1517)\approx0.2973$, giving $0.20\pm0.2973$, or approximately $[-0.0973,\,0.4973]$. The interval is wide and includes zero. The observed increase is 20 percentage points, but these small samples do not support a precise estimate of the change.

The calculation checks in scale: each estimated rate is based on only 20 relevant people, so substantial uncertainty is plausible. Because this is a small-sample Wald approximation, the interval should be treated as illustrative rather than definitive; a suitable interval method or a larger evaluation may be needed for a consequential decision.

Common mistake avoided: Do not interpret a point estimate as a certain policy effect, and do not confuse a lack of precision with proof that the policies are equivalent. The interval expresses uncertainty about the difference under the stated model; it does not settle whether the intervention’s benefits justify its costs.

A Structured Evaluation in Practice

A disciplined comparison makes assumptions and tradeoffs visible to technical and non-technical decision-makers. It also prevents an attractive aggregate result from concealing an adverse impact on a group or an infeasible implementation burden.

Specify the decision and alternatives.
Describe the baseline and intervention, who is evaluated, what a positive decision means, and the evaluation period. Keep the comparison population and outcome definitions consistent.

Choose relevant outcomes.
Report the group metrics relevant to the decision, such as true positive rates, false positive rates, or predictive values. Explain which errors or outcomes create material consequences in this setting.

Show uncertainty and subgroup effects.
Give denominators and suitable uncertainty summaries. Check whether a headline result hides a substantial impact for a smaller or intersectional group; do not treat an unstable estimate as a settled ranking.

Assess delivery and limitations.
Estimate workload, staffing, implementation demands, and any stated costs. Note what the evaluation does not measure and which assumptions drive the recommendation.

Operational consequences are part of the evaluation, not administrative details to postpone. If an intervention increases referrals, ask whether timely and consistent review is available. If it reduces one kind of error by increasing another, identify who bears each consequence. If monetary costs are used, state whose costs they represent and which harms remain outside the calculation. A policy with an attractive rate comparison may be impractical if it cannot be implemented as evaluated.

Common Traps and the Decision Summary

Treating one metric as a universal objective. A smaller difference in one group rate does not establish overall fairness or suitability. Report the other decision-relevant outcomes and explain the tradeoff.
Comparing rates without denominators. The same percentage-point difference can have very different precision in large and small groups. Include group counts and uncertainty.
Using an unexplained cost score. Costs encode judgments about consequences. State the values, their rationale, and important effects that have not been monetized.
Ignoring the operating process. More referrals may require added capacity; insufficient capacity can change the real-world consequences of the proposed policy.
Overstating what a validation comparison proves. A measured difference is tied to the evaluated population, definitions, and assumptions. It is evidence for a decision, not a guarantee of outcomes in every setting.

A sound conclusion names the intervention, summarizes its benefits and costs for affected groups, describes uncertainty, and states the operational conditions under which it could be used. If decision-makers disagree about the acceptable tradeoff, the analysis should make that disagreement explicit rather than disguising it as a technical choice of metric. This balanced approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

For each hypothetical problem, identify the relevant comparison and check the arithmetic before deciding what the results imply.

A referral rule has 3 false negatives, 10 false positives, and 20 referrals. The stated costs are $250 per false negative, $30 per false positive, and $5 per referral. Find its specified total cost. (Answer: $3(250)+10(30)+20(5)=\$1{,}150$.)
Policy A has 6 false negatives and 12 false positives; policy B has 4 false negatives and 22 false positives. Referrals are equal and have no cost in this comparison. Each false negative costs $180 and each false positive costs $20. Which policy has lower specified error cost, and by how much? (Answer: A costs $6(180)+12(20)=\$1{,}320$; B costs $4(180)+22(20)=\$1{,}160$; B is lower by $\$160$.)
In a group with 30 people who need assistance, a policy correctly refers 21. What is its true positive rate? If a proposed change correctly refers 24 of the same 30-person denominator, what is the rate change in percentage points? (Answer: $21/30=0.70$; $24/30=0.80$; the increase is 10 percentage points.)
A group has 12 true positives and 8 false positives under one policy. Under another, it has 15 true positives and 15 false positives. Calculate positive predictive value under both policies. Does the second policy’s additional true positives alone show that it is preferable? (Answer: First $12/(12+8)=0.6000$; second $15/(15+15)=0.5000$; no, the predictive value falls and other outcomes and consequences must be considered.)
A decision team estimates a 0.10 difference in group rates but reports that the estimate is highly uncertain because one group has a small denominator. What should it do before presenting the difference as evidence that one intervention is better? (Answer: Report the group denominators and an appropriate uncertainty summary, avoid treating the point estimate as conclusive, and weigh the evidence alongside decision outcomes and operational consequences.)

What You'll Learn