Why Diagnose Renewal Probability Calibration?
A renewal model assigns a probability to each policy, but a plausible ranking of policies does not guarantee that those probabilities are accurate. If a model assigns a renewal probability of 0.70 to many policies, calibration asks whether approximately 70% of those policies actually renew. This matters when probabilities are used as forecasts of renewal counts or as inputs to decisions: systematic overprediction or underprediction can distort those uses.
Calibration is assessed by comparing predicted probabilities with observed outcomes. This tutorial focuses on three complementary tools: a calibration plot across probability ranges, the observed-to-expected renewal ratio, and the Brier score. Each uses the same binary outcome but summarizes a different feature of prediction error.
The prediction unit and observation window must be consistent: each record should represent one eligible policy at the defined prediction point, and renewal must have the same operational definition across records. Assume below that each \(p_i\) lies between 0 and 1, each observed outcome is recorded once, and the evaluation cohort was not used to fit the predictions. No discounting or mortality assumption is relevant to these binary renewal diagnostics.
Grouped Calibration Plots and Observed-to-Expected Ratios
A calibration plot groups policies into ranges of predicted probabilities. For group \(g\), calculate the mean prediction and the observed renewal rate. Plot the mean prediction on the horizontal axis and the observed rate on the vertical axis. Points near the 45-degree line indicate agreement; a point above the line means the observed renewal rate exceeds the mean prediction, while a point below it means the predictions exceed the observed rate.
Here \(n_g\) is the number of policies in the group, \(O_g\) is the observed number of renewals, and \(E_g\) is the expected number of renewals under the predictions. The group observed-to-expected ratio is \(O_g/E_g\), provided \(E_g>0\). The overall ratio is calculated from pooled totals, not by giving every group ratio equal weight.
For example, an overall ratio can be close to 1 even when the model overpredicts in one probability range and underpredicts in another. A calibration plot helps expose that offsetting pattern. The plot is also affected by how ranges are chosen: wide bins can conceal local differences, while narrow bins can contain too few policies for a stable observed rate. Report the number of policies in each range and interpret small groups cautiously.
Worked Example: Finding Opposing Errors Across Probability Ranges
A hypothetical insurer evaluates 100 renewal predictions in three probability ranges. In the high range, 40 policies have mean predicted probability 0.90 and 34 renew. In the middle range, 30 policies have mean prediction 0.70 and 24 renew. In the low range, 30 policies have mean prediction 0.40 and 15 renew. Calculate each range’s observed rate and observed-to-expected ratio, then calculate the overall ratio and interpret the pattern.
Method and assumptions. Use the group mean prediction to calculate expected renewals, and compare with the observed count. The forecasts and outcomes refer to the same hypothetical cohort and fixed renewal window.
High range. The expected count is \(40(0.90)=36\). The observed rate is \(34/40=0.85\), and the ratio is \(34/36=0.9444\). Because the ratio is below 1, this range is overpredicted.
Middle range. The expected count is \(30(0.70)=21\). The observed rate is \(24/30=0.80\), and the ratio is \(24/21=1.1429\). This range is underpredicted.
Low range. The expected count is \(30(0.40)=12\). The observed rate is \(15/30=0.50\), and the ratio is \(15/12=1.2500\). This range is also underpredicted.
Overall. Observed renewals total \(34+24+15=73\). Expected renewals total \(36+21+12=69\). Thus the overall ratio is \(73/69=1.0580\), or 1.058 to three decimals. The model underpredicts in aggregate, but the overall ratio hides the high-range overprediction. A plot would show the high-range point below the diagonal and the other two points above it.
Check and common mistake. The group counts sum to \(40+30+30=100\), and expected counts sum to 69. Do not average the three group ratios as if they had equal importance: that would give a different quantity, not the pooled ratio \(73/69\).
The Brier Score: A Policy-Level Measure
The Brier score is the mean squared difference between each predicted probability and its binary outcome. It uses every policy-level prediction rather than only group totals. For renewal, a perfect prediction of 1 for a renewal or 0 for a nonrenewal contributes zero; a confident prediction contradicted by the outcome contributes a large error.
For a binary outcome and a probability between 0 and 1, each squared error is between 0 and 1, so the Brier score is also between 0 and 1. Lower is better for the evaluated observations, but the score has no universal cutoff that by itself declares a model adequate. It depends on the prevalence of renewal and on the difficulty of the prediction task. The score also does not indicate the direction of error: inspect residuals or calibration summaries to determine overprediction versus underprediction.
The term \(q(1-q)\) does not depend on the forecast. The remaining term, \((p-q)^2\), is minimized at \(p=q\). Thus, in expectation, the squared-error criterion favors reporting the true probability. This justification concerns expected loss; a finite-sample Brier score can still be high or low because outcomes are random.
Worked Example: Calculating a Brier Score and Reading the Errors
For eight hypothetical policies, predictions are \(0.80, 0.80, 0.60, 0.60, 0.40, 0.40, 0.20, 0.20\). The corresponding renewal outcomes are \(1,0,1,1,0,0,0,0\). Calculate the Brier score and describe what the observed outcomes suggest across the displayed prediction values.
Method. Calculate \((p_i-Y_i)^2\) for each policy, sum the eight losses, and divide by eight.
- For \(p=0.80\), the losses are \((0.80-1)^2=0.04\) and \((0.80-0)^2=0.64\).
- For \(p=0.60\), the losses are \((0.60-1)^2=0.16\) and \((0.60-1)^2=0.16\).
- For \(p=0.40\), the losses are \((0.40-0)^2=0.16\) and \((0.40-0)^2=0.16\).
- For \(p=0.20\), the losses are \((0.20-0)^2=0.04\) and \((0.20-0)^2=0.04\).
The total is \(0.04+0.64+0.16+0.16+0.16+0.16+0.04+0.04=1.40\). Therefore, the Brier score is \(1.40/8=0.1750\), to four decimal places.
At \(p=0.80\), one of two policies renewed, an observed rate of 0.50, below the prediction. At \(p=0.60\), both renewed, above the prediction. At \(p=0.40\), neither renewed, below the prediction; at \(p=0.20\), neither renewed, also below the prediction. These groups are extremely small, so the rates are descriptive, not stable estimates. The Brier score summarizes the individual squared errors but does not itself label these directions.
Check and common mistake. The score must divide the sum of eight losses by eight, not by the number of distinct probability values. Also, do not square an average residual in place of averaging the individual squared residuals; those operations generally differ.
Why Pooling and Weighting Matter
For an overall observed-to-expected ratio, the correct calculation is \(\sum_g O_g/\sum_g E_g\). This is equivalent to weighting each group’s observed rate and expected rate by its policy count before comparing them. An unweighted average of group rates gives a small range the same influence as a large one, even though they contain different numbers of policies.
The same care applies when a calibration plot is summarized numerically. A point represents a group, but its reliability depends partly on group size. The visual distance from the diagonal is not a formal significance test, and a ratio modestly different from 1 may reflect ordinary sample variation. These diagnostics describe performance in the evaluation cohort; they do not by themselves prove why a model is miscalibrated or establish its future performance.
Worked Example: Avoiding an Unweighted Calibration Summary
A hypothetical evaluation cohort has two prediction groups. Twenty policies each have predicted renewal probability 0.80, and 12 renew. Eighty policies each have predicted probability 0.50, and 48 renew. Calculate the group ratios, overall observed-to-expected ratio, and Brier score. Explain why averaging the two observed rates without weights is misleading for the cohort total.
Group calculations. In the first group, expected renewals are \(20(0.80)=16\), so the observed-to-expected ratio is \(12/16=0.7500\), indicating overprediction. Its observed rate is \(12/20=0.60\). In the second group, expected renewals are \(80(0.50)=40\), so the ratio is \(48/40=1.2000\), indicating underprediction. Its observed rate is \(48/80=0.60\).
Overall ratio. Observed renewals are \(12+48=60\); expected renewals are \(16+40=56\). The overall ratio is \(60/56=1.0714\), or 1.071 to three decimals. Although the two observed rates happen to be equal, the first group is overpredicted and the second underpredicted.
Brier score. In the first group, 12 renewals each contribute \((0.80-1)^2=0.04\), for \(12(0.04)=0.48\). Eight nonrenewals each contribute \((0.80-0)^2=0.64\), for \(8(0.64)=5.12\). The first group’s loss total is \(0.48+5.12=5.60\). In the second group, every outcome contributes \((0.50-1)^2=0.25\) or \((0.50-0)^2=0.25\). Across 80 policies the loss total is \(80(0.25)=20.00\). The full total is \(25.60\), so the Brier score is \(25.60/100=0.2560\).
Check and common mistake. The pooled observed rate is \(60/100=0.60\), and the pooled mean prediction is \(56/100=0.56\), consistent with the overall ratio above 1. An unweighted average of group predictions would be \((0.80+0.50)/2=0.65\), which is not the cohort’s mean prediction because the group sizes differ.
Practical Interpretation and Common Traps
Use the measures together rather than treating any one as a complete diagnosis. A calibration plot localizes discrepancies by probability range. The observed-to-expected ratio gives a concise comparison of total observed and predicted renewals. The Brier score summarizes the magnitude of policy-level probability errors. None substitutes for the others: an overall ratio of 1 can conceal offsetting range errors, and a Brier score alone cannot show whether probabilities are systematically too high or too low.
- Reversing the ratio interpretation: \(O/E>1\) means observed renewals exceed expected renewals, so predictions are too low on average. \(O/E<1\) means predictions are too high on average.
- Confusing count and rate: \(O/E\) compares counts. The calibration plot compares rates with probabilities. They are related, but should not be interchanged without accounting for group size.
- Ignoring group size: A rate based on a handful of policies is more variable than the same rate based on hundreds. Show group counts and avoid overinterpreting sparse ranges.
- Assuming a good overall result proves local calibration: Overprediction and underprediction can offset when pooled. Inspect the range-level plot and ratios as well as the overall summary.
- Using the Brier score as a directional diagnostic: Squared errors are nonnegative and do not retain the sign of each error. Pair the score with observed-versus-predicted summaries to locate the direction.
A sound report states the target definition and evaluation cohort, presents range counts with mean predictions and observed rates, gives the pooled observed and expected counts, and reports the Brier score with its sample size. This makes the evidence interpretable without implying more certainty than the data support. These diagnostics are aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.
Check Your Understanding
For each hypothetical evaluation, use the stated renewal outcome definition and calculate the requested diagnostic. Probabilities are forecasts for the same fixed observation window.
- Ten policies have predicted probability 0.80 and six renew; 20 policies have predicted probability 0.50 and 12 renew. Find the pooled observed-to-expected ratio and state its overall direction. (Answer: \(18/(8+10)=1.0000\); aggregate totals match.)
- Six policies have forecasts \(0.70,0.70,0.40,0.40,0.20,0.20\) and outcomes \(1,0,1,0,0,0\). Find the Brier score to four decimals. (Answer: \(1.18/6=0.1967\).)
- Twenty-five policies have mean prediction 0.60 and 18 renew; 75 have mean prediction 0.40 and 33 renew. Find both group observed rates and the pooled observed-to-expected ratio. (Answer: rates \(0.7200\) and \(0.4400\); ratio \(51/(15+30)=1.1333\), indicating overall underprediction.)
- Thirty policies all receive probability 0.40, and 12 renew. Calculate the Brier score and observed-to-expected ratio. (Answer: Brier score \([12(0.60^2)+18(0.40^2)]/30=0.2400\); \(O/E=12/12=1.0000\).)
- Four policies receive probability 0.90, with outcomes \(1,0,0,0\). Four receive probability 0.30, with outcomes \(1,1,0,0\). Calculate the pooled observed-to-expected ratio and Brier score; identify the overall direction. (Answer: observed \(=3\), expected \(=4(0.90)+4(0.30)=4.80\), so \(O/E=0.6250\), indicating overprediction. The loss total is \(2.44+1.16=3.60\), giving Brier score \(3.60/8=0.4500\).)