Calibration Tradeoffs Across Risk Groups

Why Fairness Metrics Can Disagree

A rating or underwriting rule can be evaluated in several ways: does its predicted loss match observed loss within each group, does it make errors at equal rates, and does it produce similar outcomes across groups? These are different questions. When groups have different claim frequencies or loss distributions, satisfying one criterion does not generally ensure the others are satisfied.

This tutorial uses simplified hypothetical insurance examples to isolate the arithmetic. A claim indicator is treated as a binary outcome, and a flagged policy is treated as receiving a higher-risk classification. Real ratemaking also involves claim severity, expenses, coverage terms, uncertainty, and legal constraints. The calculations here clarify metric tradeoffs; they do not determine which rating practice is permissible or appropriate. The distinctions are aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.

Definition: Group-specific calibration means that, within each group and for each stated prediction level or score band, the average predicted outcome agrees with the observed average outcome. For a binary claim outcome, if a group of policies is assigned predicted claim probability $q$, calibration requires the observed claim proportion in that group to be $q$, in the population or in a suitable validation sample.

Calibration is conditional on the prediction: among policies given a particular predicted probability, how often does the event occur? Error-rate parity instead compares error rates across groups. For a binary flag, common measures are the false-positive rate $P(\text{flag}\mid\text{no claim})$ and the true-positive rate $P(\text{flag}\mid\text{claim})$. Equal true-positive rates are also called equal opportunity; requiring both true- and false-positive rates to match is often called equalized odds.

Outcome parity is a broad label for requiring a selected outcome to be equal across groups. Depending on the decision being audited, that could mean the same proportion is flagged, the same share receives a surcharge, or the same average premium. Those definitions are not interchangeable. In this tutorial, each example states the particular outcome being compared.

How Base Rates Affect Error-Rate Comparisons

Let $p_g$ be the claim prevalence in group $g$, $t_g$ its true-positive rate, and $f_g$ its false-positive rate. Among $N_g$ policies, the expected counts are $N_gp_gt_g$ true positives and $N_g(1-p_g)f_g$ false positives. Therefore, the claim proportion among flagged policies, also called the positive predictive value, is

P(\text{claim}\mid\text{flag},g) = \frac{p_gt_g}{p_gt_g+(1-p_g)f_g}.

The denominator is the proportion of all policies that are flagged: flagged claimants plus flagged nonclaimants. This formula shows the central tradeoff. If groups have different prevalences, identical true- and false-positive rates generally lead to different claim proportions among flagged policies.

Worked Example: Equal Error Rates, Different Flagged-Group Claim Rates

A hypothetical insurer evaluates a claim-risk flag on 1,000 policies in each of two groups. Group A has a 10% claim prevalence, and Group B has a 30% prevalence. In both groups, the flag has an 80% true-positive rate and a 20% false-positive rate. Calculate the confusion counts and the observed claim proportions among flagged policies. Does a common predicted claim probability for flagged policies satisfy calibration in both groups?

Method. Calculate claimants and nonclaimants first, then apply the conditional error rates within each category. This avoids treating the false-positive rate as a proportion of all policies.

Group A. There are $1{,}000(0.10)=100$ claimants and $1{,}000-100=900$ nonclaimants. The true-positive count is $100(0.80)=80$, leaving $100-80=20$ false negatives. The false-positive count is $900(0.20)=180$, leaving $900-180=720$ true negatives. Thus, 260 policies are flagged, of whom 80 have claims. The flagged-group claim proportion is $80/260=0.3077$, to four decimals.

Group B. There are $1{,}000(0.30)=300$ claimants and 700 nonclaimants. The true-positive count is $300(0.80)=240$, leaving 60 false negatives. The false-positive count is $700(0.20)=140$, leaving 560 true negatives. Thus, 380 policies are flagged, of whom 240 have claims. The flagged-group claim proportion is $240/380=0.6316$, to four decimals.

Both groups have equalized error rates: the true-positive rate is 0.80 and the false-positive rate is 0.20 in each. But the flagged claim proportions are 0.3077 and 0.6316. If the same predicted probability were assigned to every flagged policy in both groups, it could not equal both observed proportions. The flag may be calibrated separately by group, but the same flag value is not calibrated across groups.

Check. Group A has $80+180=260$ flagged policies; Group B has $240+140=380$. The computed proportions use flagged claimants divided by all flagged policies, as required. The common mistake is to use the true-positive rate as the flagged-group claim probability.

A Short Derivation of the Tradeoff

Suppose the true-positive rate $t$ and false-positive rate $f$ are identical across groups, while prevalence $p$ varies. The flagged-group claim proportion is

r(p)=\frac{pt}{pt+(1-p)f} =\frac{pt}{f+p(t-f)}.

For $f+p(t-f)>0$, differentiating with respect to prevalence gives

r'(p) = \frac{t\{f+p(t-f)\}-pt(t-f)} {\{f+p(t-f)\}^2} = \frac{tf}{\{f+p(t-f)\}^2}.

When $t>0$ and $f>0$, this derivative is positive. Thus, with both types of classifications occurring and nonzero false positives, higher prevalence leads to a higher claim proportion among flagged policies, even though the two error rates are held fixed. This is a mathematical consequence of conditioning on being flagged, not evidence by itself that either group’s model is miscalibrated within that group.

The result has limits. If there are no false positives ($f=0$) and there are flagged claimants, every flagged policy is a claimant, so the flagged-group claim proportion is 1 in both groups. If a group has no flagged policies, that proportion is undefined. State such boundary conditions rather than applying the ratio mechanically.

Calibration and Equal Average Premiums

For a pure-premium illustration, assume the only loss is a claim payment of fixed amount $c$, claim frequency in group $g$ is $p_g$, and expenses, investment income, and profit provision are excluded. The expected loss per policy is $p_gc$. If group premiums are calibrated to these expected losses, groups with different frequencies have different indicated pure premiums.

Worked Example: Calibrated Pure Premiums Versus Equal Rate Outcomes

A hypothetical insurer covers two groups. Each claim pays exactly $2,000. Group A has claim frequency 0.12 and Group B has claim frequency 0.28. Calculate the calibrated pure premium for each group. Then compare those indications with a constraint requiring the same average pure premium of $400 in both groups.

Method. Under the stated fixed claim amount and frequency assumptions, expected loss per policy is claim frequency multiplied by payment per claim.

For Group A, the expected loss is $0.12\times \$2{,}000=\$240$ per policy. For Group B, it is $0.28\times \$2{,}000=\$560$ per policy. These are the group-calibrated pure premiums under the simplified assumptions.

If both groups instead receive an average pure premium of $400, the difference between premium and expected loss is $ \$400-\$240=+\$160$ per Group A policy and $ \$400-\$560=-\$160$ per Group B policy. The equal-premium constraint creates an average shortfall relative to expected loss in Group B and an average excess in Group A.

This does not establish that an equal-premium rule is required, permitted, or fair in a particular setting. It shows that, with these data and this definition of calibration, the common $400 outcome cannot also equal both group expected losses. The common mistake is to call the $400 premium calibrated merely because it lies between the two indications. Calibration is assessed against each group’s observed expected loss, not against the pooled midpoint.

Check. The two frequencies average 0.20 only if the groups have equal exposure, giving a pooled expected loss of $0.20\times \$2{,}000=\$400$. That pooled value does not make the premiums calibrated within either group.

Equal Outcome Shares and Group-Specific Calibration

An equal share receiving a high-risk label is a different outcome criterion from equal error rates or equal average premiums. The share can be held constant while the actual claim proportions within the score bands differ. A calibrated model may then assign different risk estimates to the same band label across groups.

Worked Example: Equal High-Risk Shares With Different Calibrated Risks

A hypothetical portfolio has 500 policies in each of two groups. Group A has 100 claims; 200 policies receive a high-risk label, including 80 claimants. Group B has 200 claims; also 200 policies receive a high-risk label, including 120 claimants. Calculate each group’s high-risk share and the observed claim proportions in its high- and low-risk bands. Assume a $1,000 payment for every claim and no other premium components; find the corresponding band pure premiums.

Method. For each group, divide high-risk policies by all policies to assess outcome parity. Then divide claims in each band by the number of policies in that band to assess band-level calibration.

In both groups, the high-risk share is $200/500=0.40$. The outcome criterion of equal high-risk shares is therefore met.

For Group A, the high-risk claim proportion is $80/200=0.40$. The low-risk band has $500-200=300$ policies and $100-80=20$ claims, so its claim proportion is $20/300=0.0667$, to four decimals. Multiplying by the $1,000 claim payment gives pure premiums of $400.00 for the high-risk band and $0.0667\times \$1{,}000=\$66.67$ for the low-risk band, with the latter calculated before rounding as $(20/300)\times \$1{,}000$.

For Group B, the high-risk claim proportion is $120/200=0.60$. The low-risk band has 300 policies and $200-120=80$ claims, so its claim proportion is $80/300=0.2667$, to four decimals. The corresponding pure premiums are $600.00 for the high-risk band and $(80/300)\times \$1{,}000=\$266.67$ for the low-risk band.

The same share receives the high-risk label, yet the calibrated pure premiums differ by band between groups. The group average expected losses also differ: Group A has $100/500=0.20$ claims per policy, or $200.00 expected loss per policy; Group B has $200/500=0.40$, or $400.00. The common mistake is to infer equal risk from equal classification shares. Equal shares describe how often a label is assigned, not the losses represented by that label.

Check. In each group, the high- and low-band claim counts sum to the group total: $80+20=100$ for A and $120+80=200$ for B. Each overall expected loss also equals the weighted average of the band pure premiums.

Interpreting the Metrics Together

A fairness review should identify the prediction, decision, population, and comparison before calculating a metric. For example, a claim probability, a high-risk label, and a premium are distinct objects. Calibration of claim probabilities does not automatically imply equal premium outcomes, because premiums can include additional components; equal high-risk shares do not establish equal error rates; and equal error rates do not establish equal flagged-group claim proportions.

A disciplined comparison makes the tradeoff explicit:

Define the outcome and unit.
Specify whether the event is a claim, a high-risk flag, or a particular premium outcome, and whether the unit is a policy, exposure, or claim.

State the group metric precisely.
For calibration, name the score or band. For error parity, state whether comparing false-positive rates, true-positive rates, or both. For outcome parity, state exactly which share or average is being compared.

Calculate the relevant denominators.
False-positive rates use nonclaimants as their denominator; true-positive rates use claimants; flagged-group claim proportions use flagged policies.

Check compatibility with observed risk.
Compare group frequencies or expected losses and explain any metric that cannot be met simultaneously with the others under the stated data and model.

A metric difference is a diagnostic, not a complete actuarial or regulatory conclusion. Estimates from finite samples are uncertain; the examples above use exact counts to keep the arithmetic transparent. In practice, conclusions also depend on data quality, exposure definitions, model purpose, applicable restrictions, and the consequences of alternative decisions. Avoid claiming a universal impossibility without naming the metrics and assumptions: the derivation above, for example, concerns equal true- and false-positive rates and the flagged-group claim proportion when prevalence differs and both $t$ and $f$ are positive.

Key takeaway: Calibration asks whether predictions match observed outcomes within score groups. Error-rate parity compares conditional error frequencies. Outcome parity compares a chosen decision or rate outcome. When underlying risk differs, these criteria can point in different directions; a defensible analysis defines each metric, calculates it with the correct denominator, and explains the tradeoff rather than treating one metric as a substitute for another.

Check Your Understanding

Use the stated definitions and show the denominator for each rate before calculating.

A hypothetical group has claim prevalence 0.08. A flag has true-positive rate 0.75 and false-positive rate 0.10. Calculate the claim proportion among flagged policies. (Answer: $0.06/(0.06+0.092)=0.3947$, to four decimals.)
Two groups have claim frequencies 0.12 and 0.28, and every claim pays $2,500. Under a pure-premium model with no other components, what are their calibrated pure premiums, and what signed premium-minus-expected-loss differences result from charging both groups $500? (Answer: calibrated premiums $300.00 and $700.00; differences +$200.00 and $-\$200.00$ per policy, respectively.)
In each of two groups of 400 policies, 120 receive a high-risk label. Group A has 80 claims total, including 60 in the high-risk band. Group B has 160 claims total, including 90 in the high-risk band. Find each high-risk share and each group’s high- and low-band claim proportions. (Answer: both high-risk shares are 0.30; A’s high and low proportions are $60/120=0.5000$ and $20/280=0.0714$; B’s are $90/120=0.7500$ and $70/280=0.2500$.)
Two groups have prevalences 0.25 and 0.50, and both have true-positive rate 0.80 and false-positive rate 0.20. Calculate the flagged-group claim proportion for each. (Answer: $0.25(0.80)/[0.25(0.80)+0.75(0.20)]=0.5714$; $0.50(0.80)/[0.50(0.80)+0.50(0.20)]=0.8000$.)
In one group, 25% of policies are claimants. A rule flags 100 policies out of 500; 50 flagged policies have claims. What are the flagged share, the positive predictive value, and the false-positive rate if the group has 125 claimants total? (Answer: flagged share $100/500=0.20$; positive predictive value $50/100=0.5000$; false-positive rate $50/375=0.1333$, since there are 375 nonclaimants.)

What You'll Learn