Why the Metric Must Match the Question
A fairness review can produce different conclusions depending on what it measures. A comparison of premium distributions asks whether groups receive similar rate assignments. A comparison of classification errors asks whether a model makes certain mistakes at similar rates. A calibration check asks whether predicted costs correspond to observed costs within groups. These are related but distinct questions; passing one check does not imply passing the others.
For insurance applications, the comparison also needs a clearly defined outcome. A rate might be the modeled expected loss cost, or it might be the full premium after expenses and other adjustments. A binary decision might mean that a policy is assigned to a particular rate tier, or that a claim is predicted to exceed a specified threshold. The analyst must define the quantity, population, time period, and group comparison before calculating a metric.
Demographic Parity: Comparing Assigned Rates
For a binary decision \(D\), demographic parity (also called statistical parity) holds when the probability of receiving the decision is the same in each group \(G=g\):
The meaning of \(D=1\) depends on the analysis. For example, it could mean assignment to a specified premium tier. The result can change if the analyst instead defines \(D=1\) as receiving a premium below a chosen threshold. For a rate with several possible values, parity can be assessed by comparing the full distribution across rate bands, rather than only comparing one threshold or the mean premium.
The calculation is a group proportion. For group \(g\), the assigned-decision rate is the number in that group receiving \(D=1\), divided by the total number in that group. The group denominator matters: comparing counts instead of proportions can create a difference simply because groups have different sizes.
Worked Example: Comparing Three Premium Tiers
Hypothetical scenario. Two groups each contain 120 policies. A proposed model assigns each policy to a $600, $900, or $1,200 annual premium tier. Group A has 30 policies at $600, 30 at $900, and 60 at $1,200. Group B has 30 at $600, 40 at $900, and 50 at $1,200. Compare the tier distributions and mean premiums using demographic parity as a distributional diagnostic.
Method. Divide each tier count by its group total. At $600, both groups have \(30/120=0.25\), or 25.00%. At $900, Group A has \(30/120=0.25\), or 25.00%, while Group B has \(40/120=0.3333\), or 33.33%.
At $1,200, Group A has \(60/120=0.50\), or 50.00%. Group B has \(50/120=0.4167\), or 41.67%. Thus, the correct high-tier proportions are 50.00% and 41.67%, not proportions calculated from different counts or denominators. The three proportions sum to 100% within each group: Group A has \(25.00\%+25.00\%+50.00\%=100.00\%\); Group B has \(25.00\%+33.33\%+41.67\%=100.00\%\).
The mean premium for Group A is \( (30(600)+30(900)+60(1{,}200))/120 =(18{,}000+27{,}000+72{,}000)/120 =117{,}000/120=\$975.00\). For Group B it is \( (30(600)+40(900)+50(1{,}200))/120 =(18{,}000+36{,}000+60{,}000)/120 =114{,}000/120=\$950.00\).
Conclusion and check. The tier distributions are not equal: in particular, the $1,200 tier differs by \(50.00\%-41.67\%=8.33\) percentage points. The mean premiums also differ by \(\$975-\$950=\$25\). The means alone do not describe the full distribution, so checking only the mean would omit the tier-level differences. Common mistake: using a tier count from one group over the other group’s total, or treating 60 policies out of 120 as 25%. The correct calculation is \(60/120=50.00\%\), and \(50/120=41.67\%\).
Equalized Error Rates: Comparing Mistakes by Actual Outcome
To calculate error rates, first define a binary actual outcome \(Y\) and binary model decision \(D\). For example, \(Y=1\) might indicate that a policy’s loss exceeds a specified threshold during a defined period, and \(D=1\) might indicate that the model predicts an exceedance. A true positive has \(Y=1,D=1\); a false negative has \(Y=1,D=0\); a false positive has \(Y=0,D=1\); and a true negative has \(Y=0,D=0\).
The denominators follow directly from the conditional probabilities. Among actual negatives, the model either predicts positive (a false positive) or negative (a true negative), so the false-positive rate is \(FP/(FP+TN)\). Among actual positives, it either predicts positive (a true positive) or negative (a false negative), so the false-negative rate is \(FN/(TP+FN)\). The denominator is not the number of predicted positives or predicted negatives.
Worked Example: Checking Equalized Error Rates
Hypothetical scenario. An insurer tests a binary claim-severity flag on two groups. The actual positive outcome is a claim above a defined severity threshold; the model predicts whether that threshold will be exceeded. Group A has 60 actual positives and 90 actual negatives. Of the positives, 36 are predicted positive; of the negatives, 18 are predicted positive. Group B has 50 actual positives and 120 actual negatives. Of the positives, 30 are predicted positive; of the negatives, 24 are predicted positive. Calculate both error rates by group.
Group A. False negatives are \(60-36=24\), and true negatives are \(90-18=72\). The false-positive rate is \(18/(18+72)=18/90=0.2000\), or 20.00%. The false-negative rate is \(24/(36+24)=24/60=0.4000\), or 40.00%.
Group B. False negatives are \(50-30=20\), and true negatives are \(120-24=96\). The false-positive rate is \(24/(24+96)=24/120=0.2000\), or 20.00%. The false-negative rate is \(20/(30+20)=20/50=0.4000\), or 40.00%.
Both error rates match, so equalized error rates hold for these observed counts. This does not mean the groups have equal outcome prevalence: Group A has \(60/150=40.00\%\) actual positives, whereas Group B has \(50/170=29.41\%\). Nor does it mean they have equal shares flagged: Group A has \((36+18)/150=36.00\%\) flagged, while Group B has \((30+24)/170=31.76\%\) flagged.
Check and common mistake. In each group, false positives plus true negatives equal all actual negatives, and false negatives plus true positives equal all actual positives. Dividing Group A’s 24 false negatives by its 90 actual negatives would be wrong: the false-negative denominator is 60 actual positives, giving 40.00%. Equal error rates describe conditional error frequencies, not equal overall flagging rates or equal accuracy.
Calibration: Comparing Predictions With Outcomes
A prediction is calibrated for a group when, among policies assigned the same predicted value, the average observed outcome matches that value. For an expected loss-cost prediction \(S\), the population condition is \(E[L\mid S=s,G=g]=s\), where \(L\) is the outcome measured on the same basis as \(S\). In a finite data set, an analyst estimates this condition by comparing average observed outcomes with predicted values within score bands.
The basis must be consistent. A pure premium prediction should be checked against loss costs, not unadjusted total premium if the latter includes expenses and other components. Exposure, development, trend, and the observation period must also be handled consistently. An empirical average is subject to sampling variation; a small difference in one sparse band is not automatically evidence of a persistent model defect.
Worked Example: Checking Calibration Within Rate Bands
Hypothetical scenario. A model assigns expected annual loss costs of either $500 or $1,000 per exposure unit. In each of two groups, there are 100 exposure units in each band. For Group A, observed aggregate losses are $50,000 in the $500 band and $100,000 in the $1,000 band. For Group B, observed aggregate losses are $50,000 in the $500 band and $120,000 in the $1,000 band. Assess calibration by group and then compare aggregate averages.
Group A by band. The observed average is \(\$50{,}000/100=\$500\) in the lower band and \(\$100{,}000/100=\$1{,}000\) in the higher band. Both match the predicted loss costs.
Group B by band. The observed average is \(\$50{,}000/100=\$500\) in the lower band, which matches its prediction. In the higher band it is \(\$120{,}000/100=\$1{,}200\), which exceeds the predicted \(\$1{,}000\) by \(\$200\) per exposure unit, or 20.00% of the prediction. Group B is therefore underpredicted in that observed band.
Each group has 200 exposure units and a predicted aggregate loss cost of \(100(\$500)+100(\$1{,}000)=\$150{,}000\), so its predicted overall average is \(\$150{,}000/200=\$750\). Group A’s observed total is \(\$50{,}000+\$100{,}000=\$150{,}000\), also \(\$750\) per unit. Group B’s observed total is \(\$50{,}000+\$120{,}000=\$170{,}000\), or \(\$850\) per unit. Here the aggregate comparison also identifies underprediction for Group B, but the band calculations show exactly where it occurs.
Check and common mistake. Dividing each band’s total losses by its 100 exposure units gives the appropriate observed average; dividing by the group’s 200 total units would mix the bands and conceal the conditional comparison. Calibration is about predicted versus observed outcome at a given score, not about whether groups have the same average loss or premium.
What Each Metric Can—and Cannot—Show
The three metrics answer different questions. Demographic parity asks whether model outputs are distributed similarly across groups, without conditioning on the actual outcome. Equalized error rates ask whether false-positive and false-negative probabilities are similar among actual negatives and positives, respectively. Calibration asks whether predictions correspond to observed outcomes within score levels and groups.
| Metric | Comparison | Common interpretation trap |
|---|---|---|
| Demographic parity | Decision or rate distribution by group | Equal average rates do not establish equal full distributions. |
| Equalized error rates | False-positive and false-negative rates conditional on actual outcome | Equal error rates do not imply equal shares receiving a decision. |
| Calibration | Observed outcomes versus predicted values within score levels and groups | Aggregate agreement can conceal errors in individual score bands. |
A measure can also be misleading if the outcome, group definition, or time period is poorly chosen. A premium tier may be a convenient diagnostic category but may hide variation within the tier. A binary claim threshold discards information about claim size. Calibration can appear satisfactory in a broad band even when underprediction at one end offsets overprediction at the other. Accordingly, a defensible analysis states the metric and its denominator, examines the underlying counts or exposure, and interprets the result alongside the purpose and limitations of the model.
None of these calculations alone establishes that a rating factor is permissible, actuarially justified, or compliant with applicable requirements. Those questions require the broader context of the rating plan, data, regulatory constraints, and the evidence supporting the proposed use. The metrics are diagnostic tools to make specific comparisons transparent.
Check Your Understanding
For each hypothetical case, identify the appropriate calculation and check the interpretation.
- A rating model assigns a selected tier to 48 of 240 policies in Group X and 63 of 210 policies in Group Y. Calculate each group’s selected-tier percentage and the percentage-point difference. Does this binary demographic-parity check show equality? (Answer: Group X \(48/240=20.00\%\); Group Y \(63/210=30.00\%\); the difference is 10.00 percentage points, so the rates are not equal.)
- A claim flag has 80 actual negatives in Group A, of which 16 are flagged, and 100 actual negatives in Group B, of which 25 are flagged. Calculate each false-positive rate. Are they equal? (Answer: Group A \(16/80=20.00\%\); Group B \(25/100=25.00\%\); no, the rates differ by 5.00 percentage points.)
- Among actual positive claims, Group M has 50 cases, including 10 false negatives. Group N has 80 cases, including 20 false negatives. Calculate both false-negative rates and state whether this part of equalized error rates holds. (Answer: Group M \(10/50=20.00\%\); Group N \(20/80=25.00\%\); they are not equal.)
- A model predicts a $900 average annual loss cost for a band containing 60 exposure units. The observed aggregate loss cost is $57,600. Calculate the observed average and the dollar and percentage underprediction per unit. (Answer: Observed average \(=\$57{,}600/60=\$960\); underprediction is \(\$960-\$900=\$60\) per unit, or \(\$60/\$900=6.67\%\) of the prediction.)
- Two groups each have 100 policies with a predicted loss cost of $700. Group A’s observed average is $700; Group B’s is $770. State what the calibration comparison shows, and whether that result establishes demographic parity. (Answer: Group A matches its prediction; Group B is underpredicted by $70 per policy, or 10.00%. This is a calibration comparison and does not establish demographic parity, which concerns the distribution of assigned decisions or rates.)