Checking Calibration Across Retention Segments

Why Check Calibration by Segment?

A renewal model can be well calibrated overall and still systematically overpredict renewal for one group of policyholders while underpredicting it for another. Those errors can cancel in the portfolio total. If product, tenure, distribution channel, or the size of a proposed price change is relevant to a decision, calibration should be examined within those segments as well as across the full validation sample.

A segment-level check compares predicted renewal probabilities with observed renewal outcomes for a defined group. It does not ask whether the model ranks policyholders correctly; it asks whether the probabilities are numerically consistent with renewal frequencies. A segment with few policies can show a large observed-versus-predicted gap by chance, so the size of the gap must be considered alongside its sampling uncertainty.

Definition: For a segment containing \(n\) policies, let \(Y_i=1\) if policy \(i\) renews and \(Y_i=0\) otherwise, and let \(p_i\) be its predicted renewal probability. The expected number of renewals is \(E=\sum_{i=1}^{n}p_i\), and the observed number is \(O=\sum_{i=1}^{n}Y_i\). The mean predicted probability is \(E/n\), the observed renewal rate is \(O/n\), and the calibration difference is \(O/n-E/n=(O-E)/n\). A positive difference means the segment renewed more often than predicted; a negative difference means it renewed less often.

This comparison is meaningful when predictions are evaluated against outcomes defined consistently—for example, renewal at the scheduled expiration date, rather than a mixture of renewal and midterm continuation. The validation sample should also reflect the population and time period for which the predictions are being assessed. These checks describe model performance; they do not, by themselves, establish why a discrepancy occurred.

Quantifying Variation Within a Segment

Even a correctly calibrated model will not predict the exact number of renewals in every finite sample. Under the standard validation assumption that policy outcomes are independent Bernoulli variables conditional on their predicted probabilities, \(E[Y_i]=p_i\) and \(\operatorname{Var}(Y_i)=p_i(1-p_i)\). Therefore, the expected count is \(\sum_i p_i\), while the variance of the count is the sum of the individual variances.

\operatorname{Var}(O)=\sum_{i=1}^{n}p_i(1-p_i), \qquad \operatorname{SD}(O)=\sqrt{\sum_{i=1}^{n}p_i(1-p_i)}.

A useful descriptive diagnostic is the standardized calibration residual \((O-E)/\operatorname{SD}(O)\). Its magnitude expresses the count discrepancy in standard deviation units. It is not a universal pass/fail rule: the distribution may not be well approximated by a normal distribution when expected counts are small, and examining many segments creates opportunities for chance extremes.

Condition: The variance formula above assumes independent outcomes conditional on the predictions. If policies have correlated outcomes, such as shared exposure to a common event, this calculation can understate uncertainty because covariance terms may be present. When all policies in a segment have the same probability \(p\), the formula simplifies to \(\operatorname{Var}(O)=np(1-p)\). With heterogeneous probabilities, use \(\sum_i p_i(1-p_i)\), not automatically \(n\bar p(1-\bar p)\).

Worked Examples

Worked Example: Comparing Two Product Segments

A hypothetical insurer validates renewal predictions in two product segments. In Segment A, 200 policies have predicted renewal probabilities of 0.80 and 154 renew. In Segment B, 100 policies have predicted probabilities of 0.60 and 66 renew. Assume independent Bernoulli outcomes and a common predicted probability within each segment. Calculate the calibration differences and standardized residuals, then interpret them.

Method. For each segment, calculate \(E=np\), \(O/n\), \(E/n\), and \((O-E)/\sqrt{np(1-p)}\). This uses the binomial variance because each segment has a common probability.

Segment A. The expected renewal count is \(E=200(0.80)=160\). The observed rate is \(154/200=0.7700\), and the mean predicted probability is \(160/200=0.8000\). Thus the calibration difference is \(0.7700-0.8000=-0.0300\), or \(154-160=-6\) policies relative to expectation. The count variance is \(200(0.80)(0.20)=32\), so the standard deviation is \(\sqrt{32}=5.657\). The standardized residual is \(-6/5.657=-1.061\).

Segment B. The expected count is \(E=100(0.60)=60\). The observed rate is \(66/100=0.6600\), compared with a predicted rate of \(60/100=0.6000\), giving a difference of \(0.0600\). The count variance is \(100(0.60)(0.40)=24\), and the standard deviation is \(\sqrt{24}=4.899\). The standardized residual is \((66-60)/4.899=1.225\).

Both segments show discrepancies in opposite directions, but neither residual is large in standard deviation units. These results do not provide strong evidence of a substantial segment-specific mismatch on their own. The calculation is checked by recovering each count difference from its rate difference: Segment A has \(-0.0300(200)=-6\), and Segment B has \(0.0600(100)=6\). The common mistake to avoid is interpreting a three-percentage-point rate gap as automatically important without considering the number of policies and the expected sampling variation.

Worked Example: Allowing for Different Policy Probabilities

A hypothetical tenure segment contains 100 policies. Fifty have predicted renewal probability 0.40, and the other 50 have probability 0.80. There are 67 observed renewals. Assuming conditional independence, calculate the mean predicted probability, calibration difference, count standard deviation, and standardized residual.

Method. Sum the policy probabilities to obtain the expected count, and sum the Bernoulli variances to obtain the count variance. This accounts for the different probabilities within the segment.

The expected count is \(E=50(0.40)+50(0.80)=20+40=60\). The mean predicted probability is \(60/100=0.6000\), while the observed rate is \(67/100=0.6700\). The calibration difference is \(0.6700-0.6000=0.0700\), corresponding to 7 more renewals than expected.

The variance contribution from the first group is \(50(0.40)(0.60)=12\). The contribution from the second group is \(50(0.80)(0.20)=8\). Thus \(\operatorname{Var}(O)=12+8=20\), and \(\operatorname{SD}(O)=\sqrt{20}=4.472\). The standardized residual is \((67-60)/4.472=1.565\).

The arithmetic checks: the two groups contribute \(20+40=60\) expected renewals, and the seven-count excess divided by 100 equals the 0.0700 rate difference. A tempting but incorrect shortcut is to use \(100(0.60)(0.40)=24\), treating every probability as 0.60. That produces a standard deviation of \(\sqrt{24}=4.899\), which ignores the individual probability mix. The correct heterogeneous-probability variance is 20, so the residual is 1.565, not \(7/4.899=1.429\). The gap is not especially large, but it should be interpreted in context rather than treated as proof of a segment error.

Worked Example: Overall Calibration Hiding Segment Errors

A hypothetical insurer checks renewal predictions by price-change band. In a lower-change band, 200 policies each have predicted probability 0.90 and 160 renew. In a higher-change band, 200 policies each have predicted probability 0.50 and 120 renew. Assume independent outcomes within each band and a common probability within each band. Compare calibration by band with calibration overall.

Lower-change band. The expected count is \(200(0.90)=180\), while the observed rate is \(160/200=0.8000\) and the predicted rate is 0.9000. The difference is \(-0.1000\), or \(-20\) renewals. The variance is \(200(0.90)(0.10)=18\), so the standard deviation is \(\sqrt{18}=4.243\). The standardized residual is \(-20/4.243=-4.714\).

Higher-change band. The expected count is \(200(0.50)=100\); the observed rate is \(120/200=0.6000\), giving a difference of \(0.1000\), or \(20\) additional renewals. The variance is \(200(0.50)(0.50)=50\), so the standard deviation is \(\sqrt{50}=7.071\). The standardized residual is \(20/7.071=2.828\).

Overall. Expected renewals total \(180+100=280\), and observed renewals total \(160+120=280\). The overall predicted and observed rates are both \(280/400=0.7000\). Overall calibration is exact by this measure, even though the two bands have substantial discrepancies in opposite directions.

The totals check: the lower band is short by 20 renewals and the higher band exceeds expectation by 20, so the errors cancel. The common mistake is to stop at the overall rate and conclude that predictions are calibrated for every policyholder group. The correction is to compare within decision-relevant segments: aggregate calibration cannot reveal offsetting subgroup errors. These residuals are diagnostic evidence, not a claim that one particular cause explains the discrepancies.

Choosing and Interpreting Segments

Segmentation should reflect plausible differences in model performance and decisions that depend on renewal predictions. Common candidates include product, tenure, channel, price-change band, and combinations of those variables. For example, a model may be accurate overall but miss differences in response to a price change for newer versus longer-tenured policyholders. That possibility motivates a check; it does not establish that a difference exists.

A disciplined review distinguishes three questions:

Is the group relevant?
Define the segment using information available for the decision and give it a clear operational meaning. Avoid searching an unlimited number of arbitrary cuts and treating every apparent extreme as a discovery.

How large is the discrepancy?
Report the observed rate, mean predicted probability, difference, and observed and expected counts. A rate gap alone does not show how much portfolio experience it represents.

How uncertain is the discrepancy?
Use the predicted probabilities to estimate count variation under stated assumptions. For very small samples or sparse outcomes, do not rely uncritically on a normal approximation; use an appropriate interval or probability calculation for the count model.

A nominal rule such as flagging a residual whose absolute value exceeds about 1.96 can be a rough screen under suitable large-sample assumptions. It is not a guarantee of a real or practically important error. If many segments are checked, some will appear unusual by chance; related segments may also overlap, so their results are not necessarily independent. Predefining key checks, reporting the number of comparisons, and seeking support in additional data help prevent overinterpretation.

Small segments deserve special care. Their observed rates can move sharply when only a few policies change outcome, and their estimated uncertainty is correspondingly large. Combining groups may improve stability, but only when the combined policies form a meaningful segment; averaging unlike groups can conceal precisely the pattern the analysis is meant to detect. It is useful to show counts alongside rates so readers can see whether a striking percentage is based on a robust sample or only a handful of policies.

Common Calibration Traps

Checking only the portfolio total. Opposing segment errors can cancel, as in the price-change example. Calculate segment-level summaries for groups relevant to the decision.
Comparing counts without exposure size. A difference of ten renewals means something different in a segment of 50 policies than in one of 5,000. Report rates and counts together.
Using a single-probability variance for mixed predictions. For independent outcomes with different \(p_i\), use \(\sum_i p_i(1-p_i)\), not a binomial shortcut without justification.
Treating a noisy extreme as conclusive. A large percentage-point gap in a small group may be consistent with chance. Inspect uncertainty, sample size, and the number of groups examined.
Reading a diagnostic as a cause. A segment discrepancy identifies where predictions and outcomes differ; it does not by itself identify the reason for that difference.

Key takeaway: Check calibration at both portfolio and decision-relevant segment levels. Compare observed rates with mean predicted probabilities, quantify count variation using the individual probabilities, and interpret apparent subgroup errors in light of sample size, assumptions, and the number of comparisons.

Check Your Understanding

For each problem, calculate the requested calibration quantities and consider whether the discrepancy alone supports a firm conclusion.

A hypothetical channel segment has 80 policies, each with predicted renewal probability 0.75; 54 renew. Find the expected count, observed rate, calibration difference, and count standard deviation. (Answer: Expected count 60; observed rate 0.6750; difference \(-0.0750\); standard deviation \(\sqrt{15}=3.873\).)
A tenure segment has 40 policies with probability 0.30 and 60 with probability 0.70. There are 57 renewals. Find the mean predicted probability, count variance, and standardized residual. (Answer: Expected count \(12+42=54\); mean predicted probability 0.5400; variance \(40(0.30)(0.70)+60(0.70)(0.30)=8.4+12.6=21\); residual \(3/\sqrt{21}=0.655\).)
In a product segment, 25 policies each have predicted probability 0.80 and 15 renew. Find the calibration difference and standardized residual. State one reason not to overinterpret the result. (Answer: Expected count 20; observed rate 0.6000; difference \(-0.2000\); standard deviation \(\sqrt{25(0.80)(0.20)}=2\); residual \(-5/2=-2.500\); the small sample and multiple segment checks can make an apparent extreme uncertain.)
Two equally sized price-change bands each contain 100 policies. Their common predicted probabilities are 0.80 and 0.40, and observed renewal counts are 70 and 50. Calculate overall observed and predicted rates and explain whether overall calibration reveals both segment gaps. (Answer: Expected counts are 80 and 40, totaling 120; observed counts total 120; both overall rates are 0.6000. Segment differences are \(-0.1000\) and \(0.1000\), so the overall result hides offsetting gaps.)
A segment has 50 policies: 25 have predicted probability 0.20 and 25 have probability 0.80. There are 28 renewals. Compare the correct heterogeneous-probability variance with the variance obtained by incorrectly treating every policy as having probability 0.50, and find the correct residual. (Answer: Expected count \(5+20=25\); correct variance \(25(0.20)(0.80)+25(0.80)(0.20)=8\); incorrect variance \(50(0.50)(0.50)=12.5\); correct residual \(3/\sqrt{8}=1.061\).)

What You'll Learn