Tutorials › CAS Exam 8 - Advanced Ratemaking › Simpson’s Paradox in Fairness Audits

Home  /  Tutorials  /  CAS Exam 8 / Fairness and Actuarial Standards

CAS Exam 8 · Fairness and Actuarial Standards · Tutorial 924 of 989

Simpson's Paradox in Fairness Audits

Use segment-specific rates and a common reference mix to distinguish differences within risk segments from differences caused by group composition.

Advanced 12 min read

What You'll Learn

  • Define Simpson’s paradox for aggregate rate comparisons across segments
  • Calculate crude rates using each group’s own exposure weights
  • Directly standardize rates to a common segment distribution
  • Explain how the choice of reference population affects standardized results
  • Avoid interpreting an aggregate reversal as proof of fairness or unfairness

Why Aggregate Comparisons Can Mislead

A fairness audit may compare an observed outcome rate across two groups, such as claim frequency per policy-year. The aggregate comparison is useful, but it combines two features: the rate within each segment and the proportion of each group in that segment. If the segment mixes differ substantially, the aggregate comparison can point in the opposite direction from every within-segment comparison. This reversal is known as Simpson’s paradox.

The effect is not a mathematical error. Each aggregate rate can be calculated correctly from its own observations, while still answering a different question from a comparison made within segments. An audit should therefore report the denominators, segment-level rates, and group composition alongside the aggregate result.

Definition: Let \(r_{gs}\) be the observed rate for group \(g\) in segment \(s\), calculated as the number of events divided by the relevant exposure in that group-segment cell. The crude rate for group \(g\) is the total number of events divided by total exposure across its segments. A Simpson’s-paradox reversal occurs when the direction of a crude-rate comparison differs from the direction of the comparisons within the relevant segments. In the strongest form, one group has a lower rate in every segment but a higher crude rate overall.

For claim frequency, the event count might be claims and the exposure might be earned vehicle-years. Other audits may use a different outcome and its appropriate denominator. The denominator must be defined consistently: comparing claim counts without accounting for exposure, for example, does not compare claim frequencies.

How Segment Composition Enters the Crude Rate

Write \(E_{gs}\) for the exposure of group \(g\) in segment \(s\), and \(N_{gs}\) for its event count. Then \(r_{gs}=N_{gs}/E_{gs}\). The group’s crude rate is:

$$ r_g^{\mathrm{crude}} = \frac{\sum_s N_{gs}}{\sum_s E_{gs}} = \frac{\sum_s E_{gs}r_{gs}}{\sum_s E_{gs}} = \sum_s w_{gs}r_{gs}, \qquad w_{gs}=\frac{E_{gs}}{\sum_j E_{gj}}. $$

This identity follows by substituting \(N_{gs}=E_{gs}r_{gs}\) into the total event count. The crude rate is therefore an exposure-weighted average of the segment rates, with weights specific to that group. If two groups have different weights, their crude-rate comparison combines within-segment rate differences with different segment compositions. Even if one group’s rate is lower in every segment, that group can have the higher crude rate if it has sufficiently more exposure in segments with high rates.

Key distinction: A crude rate uses the observed segment weights for each group separately. A directly standardized rate uses the same chosen segment weights for every group. Standardization makes the comparisons use a common mix; it does not identify a single uniquely correct mix for every audit purpose.

Worked Example: A Reversal in Claim Frequency

Worked Example: A Reversal in Claim Frequency

A hypothetical insurer audits claim frequency for two groups, A and B, within low- and high-frequency territories. Low-frequency territory has 10 claims per 1,000 vehicle-years in group A and 20 per 1,000 in group B. High-frequency territory has 180 claims per 900 vehicle-years in A and 21 per 100 vehicle-years in B. Calculate the segment and crude rates, then interpret them. Treat the counts as observed data; no parametric probability distribution, interest rate, or mortality assumption is used.

Method. Divide claims by vehicle-year exposure in each cell, then pool the counts and exposures separately for each group to obtain crude rates.

Low-frequency territory. For A, \(10/1{,}000=0.010=1.00\%\). For B, \(20/1{,}000=0.020=2.00\%\). A’s rate is lower by 1.00 percentage point.

High-frequency territory. For A, \(180/900=0.200=20.00\%\). For B, \(21/100=0.210=21.00\%\). A’s rate is lower by 1.00 percentage point here as well.

Crude rate for A. Total claims are \(10+180=190\), and total exposure is \(1{,}000+900=1{,}900\) vehicle-years. Thus \(190/1{,}900=0.100=10.00\%\).

Crude rate for B. Total claims are \(20+21=41\), and total exposure is \(1{,}000+100=1{,}100\) vehicle-years. Thus \(41/1{,}100=0.0372727\ldots=3.73\%\), rounded to two decimal places.

A has a lower rate in both territories but a higher crude rate overall: \(10.00\%\) versus \(3.73\%\). The exposure mixes explain the reversal. High-frequency territory accounts for \(900/1{,}900=47.37\%\) of A’s exposure, compared with \(100/1{,}100=9.09\%\) of B’s exposure.

Check. Weighting A’s segment rates by its exposure shares gives \(1{,}000/1{,}900 \times 1.00\%+900/1{,}900 \times 20.00\%=10.00\%\). For B, \(1{,}000/1{,}100 \times 2.00\%+100/1{,}100 \times 21.00\%=3.73\%\), matching the pooled calculations. The common mistake avoided is to describe A as lower overall merely because it is lower in each segment.

Direct Standardization: Comparing Groups on a Common Mix

Direct standardization answers a specific comparison question: what would each group’s weighted average segment rate be if both groups were evaluated using the same segment proportions? Choose reference weights \(w_s\), where each weight is nonnegative and \(\sum_s w_s=1\). The standardized rate for group \(g\) is:

$$ r_g^{\mathrm{std}}=\sum_s w_s r_{gs}. $$

Because each group uses the same weights, differences in these standardized rates are not due to one group having more exposure in a particular segment than the other. The rates still depend on the segment definitions, the observed cell rates, and the selected reference weights. A relevant reference could be a fixed portfolio mix or another explicitly chosen target population. The weights should be stated; standardized rates based on different reference mixes need not be equal.

1
Define the comparison.
Specify the outcome, denominator, groups, and segments before calculating rates.
2
Calculate cell rates.
For each group and segment, divide the event count by its appropriate exposure.
3
Show the crude comparison.
Pool event counts and exposures within each group, and report the groups’ segment exposure shares.
4
Choose and apply common weights.
State the reference mix and apply those same weights to each group’s cell rates.
5
Interpret both views.
Describe what the crude and standardized comparisons show, without treating either one alone as a complete fairness determination.

Worked Example: Standardizing to a Fixed Reference Mix

In a hypothetical audit of home-claim frequency, two groups have the following results by construction type. Group A has 24 claims in 1,200 policy-years for Type 1 homes and 100 claims in 500 policy-years for Type 2 homes. Group B has 40 claims in 1,600 policy-years for Type 1 and 110 claims in 500 policy-years for Type 2. Calculate crude rates and standardized rates using a reference mix of 70% Type 1 and 30% Type 2. These are empirical rates; no probability distribution, interest rate, or mortality assumption is specified or needed.

Cell rates. For A, Type 1 is \(24/1{,}200=2.00\%\), and Type 2 is \(100/500=20.00\%\). For B, Type 1 is \(40/1{,}600=2.50\%\), and Type 2 is \(110/500=22.00\%\). A is lower in both types.

Crude rate for A. A has \(24+100=124\) claims and \(1{,}200+500=1{,}700\) policy-years, giving \(124/1{,}700=0.072941\ldots=7.29\%\), rounded to two decimal places.

Crude rate for B. B has \(40+110=150\) claims and \(1{,}600+500=2{,}100\) policy-years, giving \(150/2{,}100=0.071428\ldots=7.14\%\), rounded to two decimal places. The crude comparison slightly favors B, reversing the within-type comparisons.

Standardized rate for A. Apply the common weights: \(0.70(2.00\%)+0.30(20.00\%)=1.40\%+6.00\%=7.40\%\).

Standardized rate for B. Apply the same weights: \(0.70(2.50\%)+0.30(22.00\%)=1.75\%+6.60\%=8.35\%\). Under this reference mix, A’s standardized rate is lower by \(8.35\%-7.40\%=0.95\) percentage points.

Check. Type 2 accounts for \(500/1{,}700=29.41\%\) of A’s exposure and \(500/2{,}100=23.81\%\) of B’s. A has more exposure in the higher-rate type, helping explain why its crude rate is higher despite lower rates within both types. The standardized calculations use 70% and 30% for each group, not either group’s observed mix.

Worked Example: A Different Reference Mix, Same Cell Ordering

Worked Example: A Different Reference Mix, Same Cell Ordering

A hypothetical audit compares two service plans across low- and high-usage policy segments. Plan R has 14 events in 700 exposures in the low-usage segment and 72 events in 360 exposures in the high-usage segment. Plan S has 30 events in 1,200 exposures in the low-usage segment and 84 events in 400 exposures in the high-usage segment. Calculate the crude and standardized rates using a reference mix of 60% low-usage and 40% high-usage exposure. The figures are observed proportions, so no parametric distribution, interest rate, or mortality assumption is used.

Cell rates. For R, the low-usage rate is \(14/700=2.00\%\), and the high-usage rate is \(72/360=20.00\%\). For S, the corresponding rates are \(30/1{,}200=2.50\%\) and \(84/400=21.00\%\). R is lower in both segments.

Crude rate for R. The total is \(14+72=86\) events in \(700+360=1{,}060\) exposures. Its crude rate is \(86/1{,}060=0.081132\ldots=8.11\%\), rounded to two decimal places.

Crude rate for S. The total is \(30+84=114\) events in \(1{,}200+400=1{,}600\) exposures. Its crude rate is \(114/1{,}600=0.07125=7.13\%\), rounded to two decimal places. Thus the crude comparison favors S, unlike both segment comparisons.

Standardized rate for R. Using the chosen reference mix gives \(0.60(2.00\%)+0.40(20.00\%)=1.20\%+8.00\%=9.20\%\).

Standardized rate for S. The corresponding rate is \(0.60(2.50\%)+0.40(21.00\%)=1.50\%+8.40\%=9.90\%\). Under this reference mix, R’s standardized rate is lower by \(0.70\) percentage points.

Check. The high-usage segment accounts for \(360/1{,}060=33.96\%\) of R’s exposure and \(400/1{,}600=25.00\%\) of S’s exposure. R’s greater high-usage share contributes to the crude reversal. The common reference weights deliberately replace those different observed shares. The calculation avoids the mistake of assuming that a standardized rate is an intrinsic, reference-free property of a group.

Interpretation and Common Exam Trap

A reversal is a signal to examine composition and segment-specific results; it is not, by itself, proof that either group is being treated fairly or unfairly. The crude comparison describes the observed portfolio mixes. A standardized comparison describes rates recalculated under the stated common mix. Neither calculation establishes why rates differ, whether the segment definitions are appropriate, or which comparison is decisive for a particular decision.

The choice of segment matters as well. Segments should be relevant to the audit question and defined consistently across groups. If a segment contains no exposure for one group, its cell rate cannot be calculated from those data, and direct standardization over that segment requires additional methodological care. Do not silently substitute a rate or omit a segment without explaining the consequence.

A common exam trap is to average segment rates equally and call the result the crude rate. In Worked Example 1, the unweighted averages are \( (1.00\%+20.00\%)/2=10.50\%\) for A and \((2.00\%+21.00\%)/2=11.50\%\) for B. Those figures are not the crude rates, because the groups did not have equal exposure in the two territories. They are standardized rates under an explicitly equal 50%/50% reference mix. The actual crude rates are 10.00% for A and 3.73% for B, calculated by pooling each group’s counts and exposures. The correction is to identify the question first, then use group-specific exposure weights for crude rates or a stated common set of weights for standardized rates.

Key takeaway: Always distinguish the rate within a segment, the crude rate using each group’s actual mix, and the standardized rate using a stated common mix. A reversal can arise because those calculations answer different questions.

Check Your Understanding

For each problem, use the stated denominators and keep crude and standardized comparisons distinct.

  1. Group M has 12 events in 800 low-risk exposures and 60 in 300 high-risk exposures. Group N has 20 in 1,000 low-risk exposures and 84 in 400 high-risk exposures. Calculate both crude rates and state whether the aggregate comparison reverses the segment comparisons. (Answer: M is 1.50% and 20.00% by segment, with a crude rate of \(72/1{,}100=6.55\%\); N is 2.00% and 21.00%, with a crude rate of \(104/1{,}400=7.43\%\). M is lower in both segments and in aggregate; there is no reversal.)
  2. Group X has rates of 1.00% and 20.00% in two segments; group Y has rates of 2.00% and 21.00%. Calculate standardized rates using weights of 75% for the first segment and 25% for the second. (Answer: X is \(0.75(1\%)+0.25(20\%)=5.75\%\); Y is \(0.75(2\%)+0.25(21\%)=6.75\%\).)
  3. Group C has 10 events in 1,000 low-risk exposures and 90 in 500 high-risk exposures. Group D has 20 in 1,000 low-risk exposures and 40 in 200 high-risk exposures. Calculate crude rates and identify whether the within-segment ordering reverses. (Answer: C is lower in both segments, at 1.00% and 18.00%, but its crude rate is \(100/1{,}500=6.67\%\). D’s segment rates are 2.00% and 20.00%, and its crude rate is \(60/1{,}200=5.00\%\). The crude comparison reverses.)
  4. Group P has rates of 2.00% and 20.00% in low- and high-risk segments; group Q has rates of 2.50% and 25.00%. Find each standardized rate with weights of 80% low-risk and 20% high-risk. (Answer: P is \(0.80(2\%)+0.20(20\%)=5.60\%\); Q is \(0.80(2.5\%)+0.20(25\%)=7.00\%\).)
  5. Can a crude rate, calculated from positive exposures in all segments, be greater than the highest segment rate for that group? Explain using the weighted-average identity. (Answer: No. The crude rate is a weighted average of the segment rates with nonnegative exposure shares summing to one, so it lies between the minimum and maximum segment rates.)

Previous

Next

Back to Tutorials
Clinical Biostats