Why Portfolio Averages Can Conceal Group Effects
A price test can produce an acceptable portfolio average while imposing a substantial retention loss on a particular customer group. The reverse can also happen: the overall result may look poor because the treatment arm contains more customers from a group with lower baseline retention, even when the within-group effects are modest. For ratemaking decisions, the overall result is not a substitute for examining outcomes across groups that were specified before the test.
This analysis is about diagnosing differences, not about selecting a rollout threshold. Groups might be defined by a pre-existing risk tier, customer tenure, coverage type, or another relevant characteristic. The definition should be set before examining the outcomes; otherwise, searching across many possible segments can produce apparently striking patterns by chance.
Use the same eligibility, renewal, and observation-window definitions in each arm. A group must be defined using information available before treatment, not a variable that the price change itself could alter. If a customer contributes multiple policies or observations, those observations may not be independent; standard errors then need to reflect the assignment unit or clustering rather than treating every policy record as independent.
Measure Effects Within Groups First
For a binary renewal outcome, let \(n_{ag}\) be the number of eligible customers in arm \(a\) and group \(g\), and \(r_{ag}\) the number who renewed. The observed retention rate is \(\hat p_{ag}=r_{ag}/n_{ag}\). With treatment and control arms, the group-specific effect is \(\hat\Delta_g=\hat p_{Tg}-\hat p_{Cg}\). Report the rates and the difference in percentage points; a percentage-point difference is not the same as a relative percent change.
If customers are independently randomized within each group and each outcome is a binary renewal indicator, an estimated standard error for the difference in rates is:
This formula estimates sampling variation under independent Bernoulli outcomes within each arm and group. It is not appropriate without adjustment when there is material clustering, repeated observations, or a different assignment design. A confidence interval can be formed as the estimated difference plus or minus a critical value times its standard error; for a large-sample 95% interval, the usual normal critical value is approximately 1.96.
Worked Example: Locating a Disproportionate Retention Loss
A hypothetical insurer tests a renewal price change in two pre-specified risk tiers. Within each tier, 500 eligible customers are assigned to control and 500 to treatment. In the lower-risk tier, 450 control customers and 440 treatment customers renew. In the higher-risk tier, 350 control customers and 300 treatment customers renew. Calculate each group’s treatment effect, the portfolio effect, and the estimated difference between the two group effects. Use the independent-Bernoulli standard error.
Method. Calculate retention separately in each tier before aggregating. This avoids relying on the portfolio average to describe an effect that may differ by risk tier.
For the lower-risk tier, control retention is \(450/500=0.900\), and treatment retention is \(440/500=0.880\). Therefore, \(\hat\Delta_L=0.880-0.900=-0.020\), or a 2-percentage-point decrease.
For the higher-risk tier, control retention is \(350/500=0.700\), and treatment retention is \(300/500=0.600\). Therefore, \(\hat\Delta_H=0.600-0.700=-0.100\), or a 10-percentage-point decrease.
Across both tiers, control has \(450+350=800\) renewals among 1,000 customers, for retention of \(0.800\). Treatment has \(440+300=740\) renewals among 1,000 customers, for retention of \(0.740\). The portfolio effect is \(0.740-0.800=-0.060\), a 6-percentage-point decrease.
The higher-risk tier accounts for \(350-300=50\) of the 60 fewer renewals observed in treatment, or \(50/60=0.8333\), approximately 83.33%. This contribution is descriptive; it does not replace the group-specific rate comparison.
The estimated standard error for the lower-risk effect is:
For the higher-risk effect, it is:
The estimated difference between group effects is \(\hat\Delta_H-\hat\Delta_L=-0.100-(-0.020)=-0.080\). Under independent assignment and outcomes across the four cells, its standard error is \(\sqrt{0.03000^2+0.01978^2}=\sqrt{0.0012912}=0.03593\). The standardized contrast is \(-0.080/0.03593=-2.23\). This is evidence that the estimated effect differs by tier under the stated assumptions, although the size and practical importance of the effect still require actuarial judgment.
Common mistake avoided: Reporting only the 6-percentage-point portfolio decrease would miss that the higher-risk tier’s observed decrease is five times the lower-risk tier’s decrease.
Separate Treatment Effects from Customer-Mix Shifts
A pooled rate is a weighted average of group rates, and the weights are the groups’ shares of the eligible customers in that arm. If those shares differ across arms, the difference in pooled rates reflects both within-group treatment effects and a change in composition. One way to make the distinction is to calculate standardized rates using the same target group weights for both arms.
The reason standardization removes the change in composition is visible by writing each crude rate using that arm’s own weights. Let \(w_{Tg}\) and \(w_{Cg}\) be treatment and control group shares. Then:
The first term uses control weights to average within-group differences; the second is a composition term. This identity follows by adding and subtracting \(\sum_g w_{Cg}\hat p_{Tg}\). A different reference weighting scheme gives a different decomposition, so state the chosen weights. Standardization describes a common-composition comparison; it does not repair confounding or make a nonrandomized comparison causal.
Worked Example: Standardizing Unequal Group Shares
A hypothetical renewal test has two pre-specified groups. In control, 800 lower-risk customers have retention \(720/800=0.900\), while 200 higher-risk customers have retention \(120/200=0.600\). In treatment, 200 lower-risk customers have retention \(176/200=0.880\), while 800 higher-risk customers have retention \(440/800=0.550\). Calculate the crude and standardized effects, using equal weights for the standardized comparison, and reconcile the crude effect.
Method. First calculate the arm-specific pooled rates using actual group shares. Then calculate both standardized rates using weights of 0.500 for each group.
The crude control retention rate is \((720+120)/(800+200)=840/1{,}000=0.840\). The crude treatment retention rate is \((176+440)/(200+800)=616/1{,}000=0.616\). Thus, the crude effect is \(0.616-0.840=-0.224\), or a 22.4-percentage-point decrease.
With equal target weights, standardized control retention is \(0.500(0.900)+0.500(0.600)=0.750\). Standardized treatment retention is \(0.500(0.880)+0.500(0.550)=0.715\). The standardized effect is \(0.715-0.750=-0.035\), or a 3.5-percentage-point decrease.
The control group shares are 0.800 lower-risk and 0.200 higher-risk; treatment shares are 0.200 and 0.800. Using control weights, the within-group term is \(0.800(-0.020)+0.200(-0.050)=-0.026\). The composition term is \((0.200-0.800)(0.880)+(0.800-0.200)(0.550)=-0.528+0.330=-0.198\). Together, \(-0.026-0.198=-0.224\), which matches the crude effect.
The large difference between the crude and standardized effects is driven by the arms’ very different group shares. In an experiment, such a difference may arise from the assignment or sampling design; in observational data, it may also reflect selection. The standardized estimate answers a different question from the crude estimate and should be reported with its target weights.
Common mistake avoided: Treating the crude 22.4-point decrease as a pure within-group price effect would ignore the large composition term.
Check Whether the Retained Risk Mix Changes
Retention is not the only outcome to examine. If groups have different expected loss costs, a treatment that changes their renewal rates also changes the risk composition of the retained portfolio. This can occur even when the expected cost within each group is assumed unchanged. The resulting average expected cost per retained customer is a composition measure, not evidence that the treatment changed an individual customer’s underlying risk.
For group \(g\), let \(N_g\) be the number offered renewal and \(r_g\) the number retained. If the expected cost per retained customer in that group is \(c_g\), the mix-weighted average expected cost among retained customers is \(\sum_g (r_g/\sum_j r_j)c_g\). This calculation assumes the supplied group costs are applicable to the retained customers in each group. It does not account for within-group selection unless the costs already reflect it.
Worked Example: Measuring an Unfavorable Shift in Retained Risk
A hypothetical insurer offers renewal to 600 lower-risk and 400 higher-risk customers in each test arm. In control, 540 lower-risk and 320 higher-risk customers renew. In treatment, 450 lower-risk and 312 higher-risk customers renew. Assume expected annual claim costs are $400.00 per lower-risk retained customer and $1,200.00 per higher-risk retained customer in both arms. Calculate retention by group and the average expected claim cost per retained customer.
Method. Compare retention within each risk group, then weight the group costs by each arm’s actual retained mix.
In control, lower-risk retention is \(540/600=0.900\), and higher-risk retention is \(320/400=0.800\). There are \(540+320=860\) retained customers. Expected claim cost is \(540($400.00)+320($1{,}200.00)=$216{,}000.00+$384{,}000.00=$600{,}000.00\). Average expected cost is \($600{,}000.00/860=$697.67\), rounded to the nearest cent.
In treatment, lower-risk retention is \(450/600=0.750\), and higher-risk retention is \(312/400=0.780\). There are \(450+312=762\) retained customers. Expected claim cost is \(450($400.00)+312($1{,}200.00)=$180{,}000.00+$374{,}400.00=$554{,}400.00\). Average expected cost is \($554{,}400.00/762=$727.56\), rounded to the nearest cent.
The average expected cost among retained customers increases by \($727.56-$697.67=$29.89\) per customer. The high-risk share of retained customers increases from \(320/860=0.3721\) to \(312/762=0.4094\), or from approximately 37.21% to 40.94%. These figures are consistent with a more costly retained mix: the lower-risk retention rate fell by 15 percentage points, while the higher-risk rate fell by 2 percentage points.
Total expected cost among retained customers falls from $600,000.00 to $554,400.00 because fewer customers renew. That does not contradict the increase in average expected cost: total cost and average cost per retained customer answer different questions.
Common mistake avoided: Comparing only total expected claim costs would suggest a lower cost under treatment, while missing that the average expected cost per retained customer rose as the retained mix became more heavily weighted toward the higher-risk group.
Interpret Group Differences with Care
A group-specific estimate is useful even when it is imprecise, but uncertainty matters. A difference between two point estimates is not, by itself, proof that the treatment effect differs between groups. The relevant contrast is the difference between their estimated treatment effects, with uncertainty calculated for that contrast. For independent groups and arms, its variance is the sum of the estimated variances of the group-specific differences; a shared control or clustered assignment can require a different covariance calculation.
Pre-specification helps control the risk of overinterpreting random variation. If many segments are examined, some may show large differences by chance. Report which group definitions and outcomes were planned in advance, show the group sample sizes and estimates, and distinguish confirmatory comparisons from exploratory patterns. A small subgroup may have an unstable estimate even when its observed effect is large.
- Use comparable denominators. Keep eligibility and outcome windows consistent across arms and groups.
- Show group counts and rates. Percentages without denominators can hide how uncertain a subgroup estimate is.
- Separate outcomes. Retention, expected cost per retained customer, and total expected cost are distinct measures.
- Interpret group definitions carefully. A group-level pattern does not establish that every customer in that group responds in the same way.
- State the comparison basis. Identify whether an effect is crude, standardized, or adjusted for the test design.
The central diagnostic is to compare pre-specified group-specific effects and the composition of retained business, rather than assuming that a portfolio average describes every customer group. This framework is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.
Check Your Understanding
For each problem, calculate the requested group comparison before drawing a portfolio-level conclusion.
- A test has 300 control and 300 treatment customers in each of two groups. In group A, 270 control and 255 treatment customers renew. In group B, 210 control and 180 treatment customers renew. Calculate both group-specific effects, the pooled effect, and the proportion of the total renewal-count decrease attributable to group B. (Answer: Group A: \(0.850-0.900=-0.050\); group B: \(0.600-0.700=-0.100\); pooled effect \(435/600-480/600=-0.075\), or -7.5 percentage points; group B accounts for \(30/45=66.67\%\) of the count decrease.)
- Control retention is 0.920 in group A and 0.650 in group B; treatment retention is 0.890 and 0.600, respectively. Standardize both arms using weights 0.600 for A and 0.400 for B. (Answer: Standardized control \(=0.600(0.920)+0.400(0.650)=0.812\); treatment \(=0.600(0.890)+0.400(0.600)=0.774\); effect \(=-0.038\), or -3.8 percentage points.)
- Among retained customers, group A has expected cost $300.00 and group B $900.00. Control retains 320 A and 480 B customers; treatment retains 280 A and 480 B customers. Calculate average expected cost per retained customer in each arm and the change. (Answer: Control \(=(320($300.00)+480($900.00))/800=$660.00\); treatment \(=(280($300.00)+480($900.00))/760=$678.95\); increase \(=$18.95\) per retained customer.)
- In group A, retention is 0.900 in control and 0.870 in treatment. In group B, it is 0.700 in control and 0.610 in treatment. What is the estimated difference between the group effects? State its direction. (Answer: Group A effect \(=-0.030\); group B effect \(=-0.090\); group B’s effect is 0.060, or 6 percentage points, more negative.)
- Control has 100 group-A customers with 90 renewals and 900 group-B customers with 90 renewals. Treatment has 900 group-A customers with 801 renewals and 100 group-B customers with 9 renewals. Calculate the crude pooled effect and the equal-weight standardized effect. (Answer: Crude control \(=180/1{,}000=0.180\), treatment \(=810/1{,}000=0.810\), effect \(=+0.630\); group effects are \(-0.010\) in A and \(-0.010\) in B, so the equal-weight standardized effect is \(-0.010\), or -1 percentage point.)