Common Errors in Insurance Fairness Analysis

Why Fairness Analysis Needs More Than One Calculation

A rate plan can predict expected losses accurately and still raise a fairness concern. Conversely, a difference in rates between groups does not, by itself, show that a plan is unfair. Sound analysis separates what the data establish from the judgment being made: prediction, association, impact, and fairness are related questions, but they are not interchangeable.

This tutorial addresses four recurring errors: equating prediction with fairness, treating correlation as causation, choosing a metric without a defined objective, and interpreting noisy subgroup differences as proof. The earlier material on multivariable fairness diagnostics and sparse groups provides useful context, but the principles here apply whenever an analyst evaluates a proposed insurance rate plan. The distinctions are aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.

Definition: Predictive performance describes how closely model predictions match a stated target, such as expected loss. Fairness analysis evaluates a rate plan against an explicitly stated fairness objective and relevant constraints. Predictive performance can inform that evaluation, but it does not define the objective or settle the fairness question.

Error 1: Treating Accurate Prediction as Proof of Fairness

A model can have good overall predictive performance while making systematically high or low predictions for particular groups. Even if its group-level predictions are well calibrated, calibration alone does not establish that the selected variables, the resulting premiums, or the distribution of costs meet a particular fairness standard.

For a group with predicted total losses and observed total losses, a simple diagnostic is the observed-to-predicted ratio:

\text{O/P ratio}=\frac{\text{observed losses}}{\text{predicted losses}}.

A ratio above 1 indicates that observed losses exceeded predicted losses for the group and period measured; a ratio below 1 indicates the opposite. The ratio is meaningful only when the predictions and observations use consistent definitions, exposure bases, and time periods. A ratio near 1 is evidence about aggregate calibration, not a universal fairness verdict.

Worked Example: Separating Calibration From a Fairness Conclusion

A hypothetical insurer evaluates a model for two groups, each with 100 policies. The model predicts an average annual loss of $600 per policy for Group A and $800 for Group B. Observed average losses are $620 and $790, respectively. Calculate each group’s observed-to-predicted ratio and the combined ratio. Then state what these results do and do not establish.

Method. Compare observed and predicted losses on the same per-policy basis, then aggregate the totals. The common mistake avoided is treating calibration as a complete measure of fairness.

Group A. Predicted total losses are $100 \times \$600=\$60{,}000$. Observed total losses are $100 \times \$620=\$62{,}000$. Thus:

\text{O/P}_A=\frac{\$62{,}000}{\$60{,}000}=1.0333.

Group B. Predicted total losses are $100 \times \$800=\$80{,}000$. Observed total losses are $100 \times \$790=\$79{,}000$. Thus:

\text{O/P}_B=\frac{\$79{,}000}{\$80{,}000}=0.9875.

Combined result. Total predicted losses are $\$60{,}000+\$80{,}000=\$140{,}000$. Total observed losses are $\$62{,}000+\$79{,}000=\$141{,}000$. Therefore the combined ratio is $\$141{,}000/\$140{,}000=1.0071$.

The combined model is close to aggregate calibration, but Group A’s observed losses exceed its predictions while Group B’s are slightly below. These results do not establish why the differences occurred, whether they are stable, or whether the model meets a stated fairness objective. A defensible conclusion needs the objective and context, not just the ratios.

Error 2: Treating Association as Causation

An observed relationship between a rating variable and losses does not, on its own, show that changing the variable would change losses. Other factors may influence both the variable and the loss outcome. Exposure, geography, eligibility, and the way the data were collected can all complicate an apparent relationship.

A useful first check is to ask what the observation actually supports. For example, a loss rate calculated per mile describes an association between a recorded behavior and losses per mile in the observed data. It does not establish that the behavior caused the rate difference. A causal claim requires a credible explanation for how other relevant influences have been addressed; a simple group comparison generally does not provide that explanation.

Worked Example: A Behavioral Measure and Loss Frequency

A hypothetical telematics study records two groups of drivers. Drivers with frequent hard-braking events have 8 claims over 80,000 miles. Drivers with infrequent events have 12 claims over 240,000 miles. Calculate the claim frequency per 100,000 miles and interpret the comparison.

Method. Use mileage as the exposure base, rather than comparing raw claim counts. The common mistake avoided is concluding that the group with fewer claims has lower claim frequency without accounting for its exposure.

Frequent hard braking. The rate per 100,000 miles is:

\frac{8}{80{,}000}\times100{,}000=10 \quad\text{claims per 100,000 miles.}

Infrequent hard braking. The corresponding rate is:

\frac{12}{240{,}000}\times100{,}000=5 \quad\text{claims per 100,000 miles.}

The raw counts are 8 versus 12, but the exposure-adjusted rates are 10 versus 5 claims per 100,000 miles. In this observed sample, the frequent-hard-braking group has twice the measured claim frequency. This is an association, not proof that hard braking caused the difference. For example, route conditions or other characteristics could differ between the groups. The calculation alone does not isolate those influences.

Error 3: Choosing a Metric Before Defining the Objective

No single statistic answers every fairness question. A metric can be appropriate for one objective and uninformative for another. Before calculating, state what decision the analysis is intended to support: for example, evaluating alignment with expected losses, comparing premium changes, or examining affordability impacts. The objective determines which outcomes and comparisons are relevant.

Different objectives can point to different comparisons. Equal premiums would produce no premium difference between two groups, but could be poorly aligned with their expected losses. A plan aligned with expected losses could produce different premiums and different premium-to-income burdens. Neither comparison alone supplies a universal fairness rule. Practical and regulatory constraints also matter, but should not be substituted for a clearly stated analytical objective.

Worked Example: How the Objective Changes the Comparison

A hypothetical insurer compares two premium options for two groups. Group X has an expected loss of $450 per policy and annual income of $20,000; Group Y has an expected loss of $650 and annual income of $100,000. Option A charges each group $600. Option B charges $450 to X and $650 to Y. Compare the options using (1) premium relative to expected loss and (2) premium as a percentage of income.

Method. Calculate both measures for both options, then identify what each comparison says. The common mistake avoided is presenting one chosen metric as if it answered every fairness question.

Option A, premium relative to expected loss. For X, $\$600/\$450=1.3333$, so the premium is 133.33% of expected loss. For Y, $\$600/\$650=0.9231$, or 92.31% of expected loss.

Option B, premium relative to expected loss. For X, $\$450/\$450=1.0000$. For Y, $\$650/\$650=1.0000$. On this comparison, both premiums equal the stated expected loss.

Option A, premium as a percentage of income. For X, $\$600/\$20{,}000=0.03=3.00\%$. For Y, $\$600/\$100{,}000=0.006=0.60\%$. The difference is $3.00\%-0.60\%=2.40$ percentage points.

Option B, premium as a percentage of income. For X, $\$450/\$20{,}000=2.25\%$. For Y, $\$650/\$100{,}000=0.65\%$. The difference is $2.25\%-0.65\%=1.60$ percentage points.

Option B aligns more closely with the stated expected losses and has a smaller premium-to-income gap in this example. Option A charges the same dollar premium to both groups, but its premiums differ substantially relative to expected loss and income. Which comparison matters depends on the defined objective; these calculations alone do not establish a complete fairness conclusion.

Error 4: Treating Noisy Subgroup Differences as Proof

Observed subgroup results vary partly because of sampling. A difference in a small or noisy sample can be substantial even when the underlying rates are similar; a modest observed difference can also be uncertain. Report the estimate and its uncertainty, and avoid interpreting a lack of strong evidence as proof that groups are identical.

For independent binomial samples, an approximate standard error for the difference between two observed proportions is obtained from the variances of the two sample proportions. If the observed proportions are $\hat p_1$ and $\hat p_2$, based on sample sizes $n_1$ and $n_2$, the unpooled standard error is:

SE(\hat p_1-\hat p_2) = \sqrt{\frac{\hat p_1(1-\hat p_1)}{n_1} +\frac{\hat p_2(1-\hat p_2)}{n_2}}.

Justification. Under the binomial model, the variance of a sample proportion is $p(1-p)/n$. If the two samples are independent, the variance of their difference is the sum of their variances. Replacing the unknown proportions $p_1$ and $p_2$ with their observed estimates gives the displayed estimated standard error. This approximation is most useful when the samples are large enough for a normal approximation to be reasonable; it can be unreliable for very small samples or very rare outcomes.

Worked Example: Interpreting an Uncertain Difference

A hypothetical review observes claims for 100 policies in each of two groups. Group A has 12 claims and Group B has 8. Calculate the observed difference in claim proportions and an approximate 95% interval using 1.96 standard errors. Explain whether the difference proves a disparity.

Method. Estimate each proportion, calculate the unpooled standard error, and form the approximate interval. The common mistake avoided is treating a point estimate as proof while ignoring sampling variability.

The observed proportions are $\hat p_A=12/100=0.12$ and $\hat p_B=8/100=0.08$. The estimated difference is $0.12-0.08=0.04$, or 4 percentage points.

The estimated standard error is:

SE= \sqrt{\frac{0.12(0.88)}{100}+\frac{0.08(0.92)}{100}} = \sqrt{0.001056+0.000736} = \sqrt{0.001792} \approx 0.0423.

The approximate margin is $1.96(0.0423)\approx0.0829$. The 95% interval for the difference is $0.04\pm0.0829$, or approximately $[-0.0429,\,0.1229]$. Expressed as percentage points, this is about $-4.29$ to $12.29$.

The interval includes zero, so these data do not provide strong evidence of a nonzero difference under this approximate analysis. They also do not prove equality: the interval includes differences in either direction that could matter in context. Further evidence, an appropriate objective, and attention to the study design are needed.

A Disciplined Review Sequence

A useful fairness review makes the reasoning visible, rather than jumping from a metric to a verdict. The sequence below helps separate what is measured from what is concluded.

State the question.
Specify the outcome, population, comparison groups, and decision the analysis is intended to inform.

Define the objective and metric.
Explain why the selected measure is relevant to the stated question; do not assume that calibration, premium parity, or another single metric is universally decisive.

Check the evidence and uncertainty.
Use consistent definitions and exposure bases, assess the study design, and quantify uncertainty where appropriate.

Limit the conclusion to what the evidence supports.
Distinguish a predictive association from a causal explanation, and an inconclusive result from evidence of equality.

Common Mistakes to Avoid

“The model is accurate, so it is fair.” Accuracy describes performance against a target. State the fairness objective and assess the relevant impacts separately.
“The variable is associated with losses, so it causes losses.” Association in observed data does not rule out other explanations or establish a causal effect.
“This metric is the fairness answer.” A metric has meaning in relation to a specified objective. Different valid objectives can call for different comparisons.
“The subgroup rates differ, so the difference is established.” Show the estimates and their uncertainty. A noisy difference is not proof, and an inconclusive result is not proof of equality.

Key takeaway: A defensible fairness analysis states its objective, uses a metric suited to that objective, distinguishes association from causation, and interprets subgroup estimates in light of uncertainty. Predictive accuracy is important evidence, but it is not a complete fairness conclusion.

Check Your Understanding

For each problem, calculate the requested quantity before reading the answer.

A hypothetical model predicts losses of $48,000 for Group R and $72,000 for Group S. Observed losses are $50,400 and $68,400. Calculate each observed-to-predicted ratio and the combined ratio. What can the ratios establish? (Answer: R = 1.0500; S = 0.9500; combined = $118,800/$120,000 = 0.9900. They describe calibration for these totals, not a complete fairness conclusion.)
A hypothetical driver group has 6 claims over 120,000 miles; another has 9 claims over 150,000 miles. Calculate claims per 100,000 miles and state why the comparison is not causal proof. (Answer: 5 and 6 claims per 100,000 miles, respectively; the observed rate difference is an association and does not establish causation.)
Two groups have expected losses of $500 and $700, and premiums of $550 and $700. Calculate premium-to-expected-loss ratios. What additional information is needed before calling the plan fair or unfair? (Answer: 1.1000 and 1.0000; a defined fairness objective, relevant context, and the applicable constraints are needed.)
Two independent groups each have 400 policies. One has 44 claims and the other 32. Calculate the observed difference in claim proportions and approximate unpooled standard error. Does the point estimate alone prove a difference? (Answer: 11% − 8% = 3 percentage points; $SE=\sqrt{0.11(0.89)/400+0.08(0.92)/400}\approx0.0207$, or 2.07 percentage points; no, the point estimate alone does not prove a difference.)
A small subgroup has 3 claims among 20 policies, while a comparison group has 1 claim among 20. Calculate the observed difference and give one reason not to treat it as proof of a stable disparity. (Answer: 15% − 5% = 10 percentage points; with only 20 policies per group, sampling uncertainty is substantial, so the observed difference alone does not establish a stable disparity.)

What You'll Learn