Why an Actuarial Fairness Audit Needs More Than One Metric
A model can perform differently across groups even when its overall accuracy looks acceptable. It can also have similar error rates while producing different probabilities of a positive outcome, or be calibrated on one population but not another. A fairness audit therefore needs to examine the data, predictions, and consequences together rather than treating one metric as a complete verdict.
This tutorial uses a hypothetical insurer considering a model to identify commercial policies for optional loss-prevention outreach. A positive decision means an offer of outreach, not a price increase, coverage restriction, or claim denial. The distinction matters: a metric gap can have different consequences depending on what a prediction causes. The case is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Define the Audit Before Comparing Groups
Let \(Y=1\) denote that a policy has the audited event within the specified follow-up period, and \(Y=0\) otherwise. Let \(\hat{Y}=1\) denote that the model’s score meets a chosen threshold and the policy is selected for outreach. In a real audit, the event definition, observation period, threshold, population, and decision process would need to be documented before the results are interpreted.
For a group \(g\), the true-positive rate is the share of actual events selected, and the false-positive rate is the share of non-events selected. The positive predictive value is the share of selected policies that actually have the event. These measures have different denominators, so they answer different questions.
The measures are descriptive estimates on an audit sample, not guarantees about future performance. Their uncertainty depends on sample size, especially when a group or an intersection of groups has few observations. Before comparing them, check that each group is measured over a comparable period and that outcome labels are sufficiently complete and consistently defined.
Worked Example: Compare Error Rates and Predictive Value
In a hypothetical held-out audit sample, each of two groups contains 1,000 policies. The following table gives observed outcomes and outreach decisions. Calculate each group’s true-positive rate, false-positive rate, and positive predictive value. State what the comparison does and does not show.
| Group | True positive | False negative | False positive | True negative |
|---|---|---|---|---|
| A | 120 | 80 | 80 | 720 |
| B | 60 | 40 | 90 | 810 |
Group A. There are \(120+80=200\) actual events. Thus \(\operatorname{TPR}_A=120/200=0.6000\). There are \(80+720=800\) non-events, so \(\operatorname{FPR}_A=80/800=0.1000\). The model selects \(120+80=200\) policies, of which 120 have the event; therefore \(\operatorname{PPV}_A=120/200=0.6000\).
Group B. There are \(60+40=100\) actual events, giving \(\operatorname{TPR}_B=60/100=0.6000\). There are \(90+810=900\) non-events, giving \(\operatorname{FPR}_B=90/900=0.1000\). The model selects \(60+90=150\) policies, of which 60 have the event, so \(\operatorname{PPV}_B=60/150=0.4000\).
At this threshold, the groups have equal observed true-positive and false-positive rates, but different positive predictive values. The comparison does not establish that the decision is fair: outreach counts, calibration, data quality, consequences, and uncertainty still matter. Method: calculate each rate using its own denominator. Common mistake avoided: dividing false positives by all policies rather than by actual non-events.
Why Equal Error Rates Can Coexist With Different Predictive Values
The previous result is not contradictory. Predictive value depends both on error rates and on how common the event is in each group. Write \(\pi_g=P(Y=1\mid G=g)\) for group \(g\)’s event prevalence. By the definition of conditional probability, the proportion of the group that is both an event and selected is \(\pi_g\operatorname{TPR}_g\). The proportion that is a non-event and selected is \((1-\pi_g)\operatorname{FPR}_g\). Selected policies consist of these two disjoint cases, so dividing the first by their sum gives the positive predictive value:
In the table, the event prevalences are \(200/1{,}000=0.20\) for A and \(100/1{,}000=0.10\) for B. Substituting the shared rates, \(0.60\) and \(0.10\), gives \(0.20(0.60)/[0.20(0.60)+0.80(0.10)]=0.12/0.20=0.60\) for A. For B, it gives \(0.10(0.60)/[0.10(0.60)+0.90(0.10)]=0.06/0.15=0.40\). The results match the counts.
This identity explains why, when prevalences differ, equal error rates do not generally imply equal predictive values. It does not say which metric should take priority. That is a decision about context, objectives, constraints, and the harms associated with errors—not an algebraic conclusion.
Worked Example: Check Representation and a Possible Proxy
A hypothetical model was developed using 8,000 policies from group A and 2,000 from group B. A later audit cohort contains 1,000 policies from each group. A location-related feature is recorded for 70% of A and 25% of B in the development data. Calculate the group shares, the representation change, and weights that would make the development group shares match the audit cohort. Then explain what these results say about the location feature.
Development representation. The development total is \(8{,}000+2{,}000=10{,}000\). Group A’s share is \(8{,}000/10{,}000=0.80\), and group B’s is \(2{,}000/10{,}000=0.20\).
Audit representation. The audit total is \(1{,}000+1{,}000=2{,}000\), so each group’s share is \(1{,}000/2{,}000=0.50\). The share of A has decreased by \(0.80-0.50=0.30\), or 30 percentage points; B’s share has increased by the same amount.
Illustrative reweighting. To give both groups equal total weight in the development data, multiply A’s records by \(0.50/0.80=0.625\) and B’s by \(0.50/0.20=2.5\). The weighted totals become \(8{,}000(0.625)=5{,}000\) and \(2{,}000(2.5)=5{,}000\). This changes group representation in the weighted calculation; it does not add new observations or ensure that outcomes, measurement quality, or within-group characteristics match the audit cohort.
Proxy risk. The location-related feature occurs in \(70\%-25\%=45\) percentage points more of A’s development records than B’s. Its prevalence ratio is \(0.70/0.25=2.8\). It is therefore associated with group membership in these data and could carry group-related information into predictions, even if a sensitive attribute is not used directly. The figures do not prove that the feature is an invalid proxy or that it causes a particular decision; that requires examining its meaning, use, and consequences. Method: compare both data composition and feature prevalence. Common mistake avoided: assuming that removing a group label removes all group-related information from the model.
Calibration, Decisions, and Practical Consequences
A probability score is calibrated within a group when, among policies in that group receiving a given predicted probability, the observed event frequency is approximately that probability. In finite data, calibration is commonly assessed in score bands: compare the average predicted probability with the observed event rate in each group and band. A single overall calibration result can hide group-level differences, while small bands can produce unstable estimates.
Calibration is not the same as having equal error rates at a threshold. A threshold converts scores into decisions; it can alter true-positive and false-positive rates without changing the score itself. When groups have different event prevalence, the derivation above shows why a common threshold with similar error rates can still produce different predictive values. Conversely, similar predictive values do not by themselves establish equal error rates or equal access to the opportunity associated with a positive decision.
The audit must also ask what errors mean in this setting. If outreach is genuinely optional and useful, a false positive may mainly consume staff time and impose an unwanted contact. A false negative may mean a policyholder misses a helpful service. If a prediction instead changes price, eligibility, or claim handling, the stakes are materially different, and the same metric pattern would warrant a different level of caution. These are hypothetical consequences; an actual organization would need evidence about costs, benefits, consent, and operational practice.
Worked Example: Translate Audit Counts Into Outreach Consequences
Suppose, solely for this hypothetical calculation, each correctly selected policy with the event yields an estimated $120 of value from useful outreach, while every outreach offer costs $20 to deliver. Use the audit counts above to calculate net value for each group and per policy. Do not treat these illustrative values as observed industry results.
Group A. There are 120 true positives, so the estimated value from useful outreach is \(120\times\$120=\$14{,}400\). There are 200 total offers, costing \(200\times\$20=\$4{,}000\). The illustrative net value is \(\$14{,}400-\$4{,}000=\$10{,}400\), or \(\$10{,}400/1{,}000=\$10.40\) per audited policy.
Group B. There are 60 true positives, so the estimated value is \(60\times\$120=\$7{,}200\). There are 150 offers, costing \(150\times\$20=\$3{,}000\). The illustrative net value is \(\$7{,}200-\$3{,}000=\$4{,}200\), or \(\$4{,}200/1{,}000=\$4.20\) per audited policy.
The calculation identifies a possible difference in aggregate value under the stated assumptions; it is not a fairness measure and does not show that one group should receive less service. It omits uncertainty, any benefits of outreach to policies without the event, burdens on recipients, and the cost of events missed by the model. Method: apply the same explicit per-offer cost and per-true-positive value to each group’s counts. Common mistake avoided: interpreting a higher aggregate value as a justification for withholding an otherwise beneficial service.
Privacy and a Context-Specific Recommendation
Group comparisons may require sensitive attributes that should not be exposed broadly or used casually in individual decisions. A responsible audit defines why each attribute is needed, limits access to authorized reviewers, separates audit data from operational access where feasible, and reports only the level of detail needed for a decision. Very small intersectional groups or rare outcomes can create disclosure risks even in summary tables; suppressing or combining a small cell may reduce that risk, but can also hide disparities. The choice and its effect on interpretation should be recorded.
Record what a positive decision triggers, who can be affected, and how false positives and false negatives matter in this particular use.
Review group representation, outcome definitions, score calibration, error rates, and sample sizes. Investigate plausible proxies rather than assuming that omission of a sensitive field resolves proxy risk.
State which use, if any, the evidence supports; restrict access and adverse uses as appropriate; and specify what remains uncertain and what must be monitored.
For the hypothetical optional outreach use, a cautious recommendation could be to permit a limited, monitored pilot only if the service is genuinely non-adverse, recipients can decline it, and the model does not determine price, eligibility, or claim outcomes. The observed equal true-positive and false-positive rates at this threshold are relevant, but the lower positive predictive value for B, the representation shift, and the possible location proxy require investigation. Review calibration by group and relevant subgroups, confirm that outcome labels and follow-up are comparable, and examine whether offers impose unequal burdens. Restrict sensitive audit data to authorized reviewers and avoid reporting identifying small cells.
This recommendation is conditional, not a declaration that the model is fair. The audit cannot establish that the event labels are unbiased, that the illustrative outreach benefits are real, or that the observed rates will persist in a new population. Sampling variability, unmeasured group differences, proxy use, and changes over time remain limitations. If outreach becomes adverse or its consequences differ from the assumed context, the decision should be reassessed before that use.
Check Your Understanding
For each problem, calculate the requested quantities before checking the answer. All situations and values are hypothetical.
- A group has 300 policies: 60 true positives, 20 false negatives, 44 false positives, and 176 true negatives. Find its true-positive rate, false-positive rate, and positive predictive value. (Answer: TPR \(=0.7500\), FPR \(=0.2000\), PPV \(=0.5769\).)
- Two groups have the same TPR of 0.70 and FPR of 0.20. Their event prevalences are 0.30 and 0.10. Use the predictive-value formula to find PPV for each group. (Answer: prevalence 0.30 gives PPV \(=0.6000\); prevalence 0.10 gives PPV \(=0.2800\).)
- A development sample is 70% group A and 30% group B; the audit population is 40% A and 60% B. Find group-specific weights that match those shares. (Answer: weight A by \(0.40/0.70\approx0.5714\); weight B by \(0.60/0.30=2\).)
- An audit table contains an intersectional cell with four people, one positive outcome, and a distinctive combination of attributes. Give one privacy-conscious reporting action and one limitation it creates. (Answer: suppress or combine the cell in public reporting while documenting the choice; suppression reduces disclosure risk but conceals the cell’s group-specific rate.)
- A hypothetical outreach program records 50 true positives and 100 total offers in a group. Assume $90 of value per true positive and $15 cost per offer. Calculate the illustrative net value and explain why it is not itself a fairness conclusion. (Answer: \(50(\$90)-100(\$15)=\$3{,}000\); it depends on assumed values and omits other consequences, so it does not alone establish whether decisions are fair.)