Why Validate Across Renewal Periods?
A retention model can perform well on a random test sample and still perform poorly when used for a later renewal season. Random splitting can place policies from the same calendar period, pricing environment, or campaign on both sides of the split. It may therefore test interpolation among familiar conditions rather than the model’s ability to predict future renewals. For deployment, the key question is whether predictions made using information available at a particular date remain useful for a later, unseen renewal cohort.
Out-of-time validation answers that question by training on earlier renewal periods and evaluating on one or more later periods. Repeating the evaluation across successive cohorts shows whether a single poor result is an isolated fluctuation or part of a persistent change. This tutorial is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.
The time ordering must reflect actual operations. If a model predicts renewal before an offer is sent, its predictors must be values available by that prediction date. A recorded final price, a post-offer contact, or a status updated after the renewal decision is not a valid predictor for that use, even if the field is present in a historical data extract. Data leakage of this kind can make validation scores look unrealistically strong.
Designing an Out-of-Time Validation
First define the prediction point, outcome window, cohort boundaries, and eligible policies. A policy should enter the evaluation only when its outcome can be observed under the chosen definition. For example, a policy whose renewal decision is still pending at the data extraction date should not be coded as a nonrenewal. The training data should end before the validation cohort begins, and any feature processing or model selection should use training-period data only.
Specify when predictions are made, what counts as renewal, and how long the outcome window remains open.
Use only policies and predictor values that would have been available before the validation cohort’s prediction dates. Fit transformations and select model settings without using validation outcomes.
Generate and retain predictions for each later cohort, then observe outcomes after the specified window has closed.
Review outcome rates, mean predicted probabilities, probability scores, calibration by band, and predictor distributions for each cohort.
A single random holdout can still be useful for some development questions, but it does not replace a temporal test when the intended application is future renewal decisions. Successive time-based cohorts also require care: if the model is refit between periods, record which version produced each cohort’s predictions. Otherwise a performance change may reflect a model update rather than changing conditions.
Metrics for Cohort Comparisons
For a cohort of \(N\) policies, the observed renewal rate is \(\bar{Y}=\sum_iY_i/N\), and the mean predicted renewal probability is \(\bar{p}=\sum_ip_i/N\). Their difference, \(\bar{Y}-\bar{p}\), is a calibration-in-the-large diagnostic: a negative value means the model predicted more renewals on average than occurred. These two averages should be reported together; a model can have a plausible portfolio average while making offsetting errors among policy groups.
This identity follows by subtracting the mean predicted probability from the mean observed outcome and combining the sums over the same \(N\) policies. Thus the cohort-level discrepancy is exactly the average prediction residual. It diagnoses an overall shift, but not whether the model ranks policies well or whether errors are concentrated in particular segments.
The Brier score is the mean squared error for binary probability forecasts. It is zero for perfect predictions and lower is better. Unlike an observed-versus-predicted rate comparison, it penalizes individual probability errors, including predictions that are far from the eventual outcome.
Calibration checks compare predicted and observed renewal rates within probability bands or meaningful segments. Ranking measures, such as a statistic based on how well scores order renewers above nonrenewers, answer a different question: a model may rank policies usefully but systematically overpredict renewal. Earlier material on renewal probability calibration and retention ranking distinguishes these questions; temporal validation asks whether each property persists in future cohorts.
Worked Example: A Later Cohort’s Average Forecast and Brier Score
A hypothetical insurer evaluates 200 renewal offers in a later quarter. For 100 policies, the model predicted renewal probability \(0.80\), and 72 renewed. For the other 100, it predicted \(0.60\), and 56 renewed. Calculate the cohort’s observed and predicted rates, calibration-in-the-large difference, and Brier score. Assume the stated predictions apply to every policy in their respective groups.
Method. Calculate the two cohort averages, then calculate the Brier score separately within each group and weight by group size. This avoids treating a group’s observed rate as if it were the prediction for each individual.
Predicted renewal count: the first group contributes \(100(0.80)=80\), and the second contributes \(100(0.60)=60\). Thus the total is \(80+60=140\), giving \(\bar{p}=140/200=0.700\).
Observed renewal count: \(72+56=128\), so \(\bar{Y}=128/200=0.640\). Therefore \(\bar{Y}-\bar{p}=0.640-0.700=-0.060\). The model overpredicted the cohort renewal rate by 6 percentage points.
Brier score, group 1: 72 outcomes were \(1\), each with squared error \((0.80-1)^2=0.04\); 28 outcomes were \(0\), each with squared error \((0.80-0)^2=0.64\). The group’s mean score is \([72(0.04)+28(0.64)]/100=(2.88+17.92)/100=0.208\).
Brier score, group 2: 56 outcomes were \(1\), with squared error \((0.60-1)^2=0.16\); 44 outcomes were \(0\), with squared error \((0.60-0)^2=0.36\). The group’s mean score is \([56(0.16)+44(0.36)]/100=(8.96+15.84)/100=0.248\).
The total Brier score is \([100(0.208)+100(0.248)]/200=0.228\). The check is that the overall observed rate, \(0.640\), equals \(128/200\), while the forecast rate, \(0.700\), equals \(140/200\). The average residual is consequently \(-12/200=-0.060\), matching the rate difference. The common mistake is to report the rate difference as if it were a probability score; it is a calibration diagnostic, while \(0.228\) measures squared individual forecast errors.
Diagnosing Drift Across Cohorts
A change in cohort performance can have several causes. The overall renewal rate may change because the portfolio contains a different mix of policy types, because renewal behavior changed within otherwise comparable groups, or because the relationship between predictors and renewal changed. Predictor distributions may also shift, including shifts that create greater exposure to parts of the predictor space where the model has little support.
| Diagnostic | Question it helps answer | Interpretive caution |
|---|---|---|
| Observed and predicted cohort rates | Is the model over- or underpredicting overall? | Good aggregate agreement can conceal offsetting segment errors. |
| Calibration by probability band or segment | Where do predicted and observed renewal rates diverge? | Small groups have noisy observed rates; show group sizes. |
| Brier score by cohort | Are individual probability errors worsening? | A score change can reflect changes in outcome prevalence or difficulty as well as model quality. |
| Predictor distributions | Has the input population changed? | A distribution shift flags a need to investigate; it does not by itself prove that predictions are biased. |
A useful decomposition holds predicted segment rates fixed and changes only the segment weights. If the segment-specific renewal behavior is stable, a changed mix can change the portfolio rate even when the model remains calibrated within each segment. Compare the rate implied by the new mix at the earlier segment rates with the new cohort’s actual rate. A remaining discrepancy points toward within-segment change or other model error, subject to sampling variation.
Worked Example: Separating Mix Shift From Within-Segment Drift
A hypothetical renewal portfolio has two segments, lower-risk and higher-risk. In an earlier cohort of 300 policies, 180 are lower-risk and 120 are higher-risk; observed renewal counts are 162 and 48, respectively. In a later cohort of 300, there are 90 lower-risk and 210 higher-risk policies; observed renewals are 79 and 77. The model’s predictions for the later cohort remain \(0.90\) for lower-risk policies and \(0.40\) for higher-risk policies. Compare the overall change with the mix-only expectation.
Earlier segment rates: lower-risk renewal rate is \(162/180=0.90\); higher-risk rate is \(48/120=0.40\). The earlier overall rate is \((162+48)/300=210/300=0.700\).
Later model forecast: predicted renewals are \(90(0.90)=81\) in the lower-risk segment and \(210(0.40)=84\) in the higher-risk segment. The total prediction is \(81+84=165\), or \(165/300=0.550\).
Mix-only calculation: applying the earlier segment rates to the later cohort’s mix also gives \(90(0.90)+210(0.40)=165\) expected renewals, or \(0.550\). The predicted decline from \(0.700\) to \(0.550\) is therefore explained by the changed segment weights under the assumption that the earlier segment rates continue to hold.
Later observed results: actual renewals total \(79+77=156\), so the observed rate is \(156/300=0.520\). Within the lower-risk segment, the rate is \(79/90\approx0.8778\); within the higher-risk segment, it is \(77/210\approx0.3667\). Both are below their earlier rates and model predictions. The aggregate residual is \(0.520-0.550=-0.030\), or 9 fewer renewals than predicted.
The check is that \(79+77=156\) and \(81+84=165\), so the difference is \(-9\) renewals, equal to \(-9/300=-0.030\). The changed mix explains the decline to \(0.550\); the additional gap to \(0.520\) is consistent with within-segment drift or random variation. The common mistake is attributing the full decline from \(0.700\) to \(0.520\) to model failure without accounting for the lower-risk segment’s smaller share.
Worked Example: Choosing a Time Split and Interpreting a Worse Score
A hypothetical insurer has complete renewal outcomes for quarters 1 through 8. It wants to assess a model intended for quarter 9. The prediction is made 30 days before each policy’s renewal date; one candidate predictor records a service contact made 10 days before renewal. The insurer also observes a higher Brier score in quarter 9 than in quarter 8. Specify a defensible validation setup and the first checks needed before concluding that the model has deteriorated.
Validation setup. Use quarters 1 through 8 for model fitting and development, then score quarter 9 without using its outcomes to select features, transformations, or tuning settings. For each quarter 9 policy, include only predictor values available at least 30 days before that policy’s renewal date. The contact made 10 days before renewal is unavailable at the prediction point and must be excluded from this forecast model or replaced with a version defined using information available by the 30-day cutoff.
Outcome completeness. Apply the same renewal definition and outcome window in quarters 8 and 9. Compare only policies whose outcome windows have closed; do not classify unresolved decisions as nonrenewals. Confirm that eligibility rules and treatment of cancellations are consistent across periods.
Interpret the score change. A higher Brier score means larger average squared probability errors in quarter 9, but it does not identify the cause. Compare observed and predicted renewal rates, calibration by probability band and material segment, cohort sizes, and predictor distributions. If the overall rate changed while calibration within stable segments remains similar, mix shift may explain some of the score change. If predicted rates exceed observed rates within many bands, broad overprediction is more plausible. A smaller cohort can also produce a less stable comparison.
The conclusion is not that the model deteriorated solely because one score increased. The defensible conclusion is that quarter 9 shows worse Brier performance and requires checks for leakage, outcome-definition changes, mix and predictor drift, calibration changes, and sampling variation. The common mistake is to use a future-updated predictor or tune the model after inspecting quarter 9 outcomes, which turns the supposed out-of-time test into part of model development.
Practical Interpretation and Common Traps
Temporal validation is most informative when the same definitions and reporting structure are applied to every cohort. Present cohort sizes alongside rates and scores, because a difference based on a small cohort may be less stable. Where appropriate, uncertainty intervals for observed rates or score differences help distinguish a persistent signal from ordinary sampling variation. They do not correct leakage, inconsistent outcome definitions, or an unrepresentative validation period.
- Do not shuffle time indiscriminately. Random splits can mix periods and conceal degradation under future conditions.
- Do not use information from after the prediction date. Availability in a database is not the same as availability to the model when a decision is made.
- Do not confuse an aggregate rate with full calibration. Different segments can overpredict and underpredict by amounts that cancel in the portfolio total.
- Do not label all drift as model failure. A composition change may alter the overall outcome rate even when segment relationships remain stable.
- Do not treat a shifted predictor distribution as proof of bias. It is a diagnostic signal; evaluate whether the shift changes observed outcomes or calibration.
Check Your Understanding
For each problem, use the stated cohort definitions and treat the supplied predictions as available before the renewal decision.
- A later cohort has 80 policies, each with predicted renewal probability \(0.75\), and 54 actual renewals. Find the predicted rate, observed rate, and calibration-in-the-large difference. (Answer: predicted rate \(0.750\), observed rate \(0.675\), difference \(-0.075\), or 6 fewer renewals than predicted.)
- In a 200-policy cohort, 100 policies have prediction \(0.80\), of whom 70 renew, and 100 have prediction \(0.50\), of whom 60 renew. Calculate the Brier score. (Answer: \(0.235\).)
- Earlier segment renewal rates are \(0.95\) for lower-risk policies and \(0.50\) for higher-risk policies. A later cohort has 40 lower-risk and 60 higher-risk policies, with 65 actual renewals. Find the mix-only expected rate and the observed-minus-expected difference. (Answer: expected rate \(0.680\), difference \(-0.030\), or 3 percentage points below the mix-only expectation.)
- A model is scored 20 days before renewal, but a predictor is recorded 5 days before renewal. Should that predictor be included in an out-of-time validation of the 20-day forecast? State why. (Answer: No; it was unavailable at the prediction point and would create information leakage.)
- Across two later cohorts, overall predicted and observed renewal rates match in both, but calibration bands show overprediction in one segment and underprediction in another. What does the aggregate match establish, and what does it fail to establish? (Answer: It establishes aggregate calibration-in-the-large for each cohort, but not calibration within segments or accurate individual probabilities.)