Measuring Disparate Impact Across Rating Groups

What a Group Comparison Measures

A measured difference across rating groups can be an important signal for review, but the calculation is only meaningful when the outcome and its denominator are clearly defined. A premium comparison might use dollars per policy-year; an approval comparison might use approved applications divided by eligible applications; a renewal comparison might use renewal offers divided by policies eligible for renewal. Changing the denominator changes the question.

This tutorial focuses on describing group-level differences and assessing their uncertainty. A difference by itself does not explain why it arose, establish that a rating variable caused it, or determine whether the difference is legally or actuarially acceptable. Those conclusions require additional evidence and applicable standards. A sound measurement is a disciplined starting point, not a verdict.

For example, a difference of \(0.04\) between approval rates is a four-percentage-point difference. It is not a four-percent relative difference. If the rates are \(0.80\) and \(0.76\), the absolute difference is \(0.04\), while the rate ratio is \(0.80/0.76 \approx 1.053\), or about a 5.3% relative difference. Both can be useful descriptions, but neither replaces the other.

Define the Outcome and Comparison Population First

A fair comparison begins by specifying who is included, what event counts, and the period of observation. For premium comparisons, decide whether the target is quoted premium, written premium, or earned premium, and identify the exposure basis. For approval comparisons, the denominator should generally be applications that were eligible for a decision, not all people who visited a website. For renewal comparisons, define which policies were actually eligible to renew. Exclusions such as withdrawals or incomplete applications should be handled consistently and disclosed.

The groups should also be compared on a meaningful basis. Premiums can differ because policies have different coverage limits, deductibles, territories, or terms, even when the underlying rating relationship is identical. Where possible, compare like coverage and use a consistent exposure measure, such as earned policy-years. If groups have different distributions of relevant characteristics, report what population is being used as the comparison. A crude group average describes each group as observed; a standardized comparison describes both groups using the same chosen weights.

Using a Common Comparison Population

Suppose group rates vary across categories of business, and the groups contain different proportions of those categories. A crude rate combines each category-specific rate with that group’s own mix. To make a like-for-like descriptive comparison, select a common set of weights in advance and apply those weights to each group’s category-specific rates.

Let \(r_{gk}\) be the outcome rate for group \(g\) in category \(k\), and let \(w_k\) be the selected comparison weight for that category. The weights must be nonnegative and sum to one. The directly standardized rate is the weighted average of the group’s category-specific rates. This answers: what rate would each group have if both were summarized using this same category distribution? The choice of weights matters, so it should be justified and reported.

This formula follows from a weighted-average interpretation. Imagine a reference population in which category \(k\) represents fraction \(w_k\). If group \(g\) had category-specific outcome rate \(r_{gk}\), its expected overall outcome proportion under that reference mix would be the sum of each category’s contribution, \(w_k r_{gk}\). Summing across categories gives \(r_g^{\mathrm{std}}\). Because both groups use identical weights, their difference is not driven simply by using different category proportions in the two overall averages. Standardization does not remove every possible difference between groups, and it does not establish a causal effect.

Quantifying Uncertainty in a Difference

When outcomes are observed for a sample and the goal is to learn about an underlying process, a point estimate alone does not show how much it could vary. For a binary outcome such as whether a policy received a renewal offer, a common introductory model treats the outcomes within each group as independent Bernoulli observations with a constant group-specific probability. Under that model, the sample proportion \(\hat p_g=x_g/n_g\) estimates the group probability, where \(x_g\) is the number of outcomes and \(n_g\) is the number of eligible observations.

For two independent groups, the estimated standard error of the difference in proportions is the square root of the sum of the estimated variances. A normal-approximation 95% interval is the estimated difference plus or minus 1.96 standard errors. This approximation is most suitable when each group has enough observed outcomes and non-outcomes; as a practical check, examine whether \(n_g\hat p_g\) and \(n_g(1-\hat p_g)\) are each at least about 10. For sparse outcomes, use a method designed for small counts rather than relying on this approximation.

The variance addition is justified because the two estimated proportions are independent under the model: the variance of their difference is the sum of their variances. For a Bernoulli outcome with probability \(p_g\), the variance of one observation is \(p_g(1-p_g)\); averaging \(n_g\) independent observations divides that variance by \(n_g\). Replacing the unknown \(p_g\) with \(\hat p_g\) gives the estimated standard error above.

Interpreting Results Carefully

An uncertainty interval describes sampling variability under its assumptions. It does not account automatically for inconsistent eligibility rules, missing observations, measurement error, changing business practices, or a poorly chosen comparison population. If the data cover a complete portfolio for the period of interest, the observed rate difference is an exact description of that portfolio; an interval may still be relevant if the aim is to generalize to future experience, but that target and model must be stated.

A confidence interval that includes zero does not prove that the groups are equivalent. It indicates that the data and method do not clearly distinguish the difference from zero at the corresponding confidence level. Conversely, a narrow interval excluding zero does not establish that a difference is practically important, caused by a particular rating factor, or unacceptable under applicable requirements. Magnitude, context, data quality, and the purpose of the review remain essential.

For a defensible group comparison, retain both the unadjusted and standardized results when each answers a useful question, and explain why the reference population was selected. Show counts and denominators alongside percentages. When the comparison is intended to assess a rating outcome, do not confuse an observed group disparity with the effect of a rating variable: differences may reflect exposure, coverage, eligibility, or other characteristics. The earlier material on causal pathways is relevant to evaluating those explanations, but causal conclusions require more than a group-level rate calculation.