Why Validate a Rate Model on Later Experience?
A rate model is intended to support decisions about risks that will be written or renewed in the future. A random holdout can measure predictive performance on observations resembling the data used to fit the model, but it may not reveal whether the model works when the business environment, mix of risks, or claims experience changes over time. Out-of-time validation addresses that question by training on earlier experience and evaluating predictions on a later, chronologically separate period.
The central requirement is that the model be evaluated as it could actually have been used at the prediction date. The fitted coefficients, rating variables, and any preprocessing rules must be determined using information available before the validation period. For a policy-level prediction, predictors must be values known when the prediction would have been made. Claims in the validation period are outcomes to compare with predictions, not information to use in constructing them.
Set the Time Boundary and Prediction Unit
A chronological split is only meaningful after specifying what an observation represents and which date assigns it to a period. For a policy-term frequency model, the prediction unit might be one policy term and the period might be assigned by the term’s effective date. For a claim-frequency model, the outcome might count claims occurring during the covered exposure. The choice must match the intended use of the model; mixing assignment rules can put nominally later business into training or earlier business into validation.
A basic single-cutoff design has a cutoff date \(T\). Observations assigned to dates before \(T\) form the training data, and observations assigned to dates on or after \(T\) form the validation data. The cutoff and assignment rule should be stated before examining validation outcomes. The model is then fitted on training data, and predictions are generated for validation observations using only predictors that would have been available at the prediction time.
For a frequency model with a log link and an exposure offset, a standard specification is \(\log E[N_i\mid x_i,e_i]=\log(e_i)+x_i^\mathsf{T}\beta\), where \(N_i\) is the claim count for observation \(i\), \(e_i\) is its exposure, \(x_i\) is its predictor vector, and \(\beta\) is the coefficient vector. Under the usual Poisson frequency model, conditional claim counts are independent Poisson variables with these means. The fitted model therefore predicts the count \(e_i\exp(x_i^\mathsf{T}\hat{\beta})\). In validation, \(\hat{\beta}\) and the model specification remain fixed; only the eligible validation predictors and exposure are supplied.
A validation-period prediction should not use information that arrived later than the date at which that prediction is meant to represent. Examples include a revised risk classification made after a claim, or a data transformation whose levels or rules were learned using the validation outcomes. Prior-period information may be used when it would genuinely have been available at the time of prediction, but its timing and role need to be explicit.
What an Aggregate Calibration Check Shows
One direct summary for a frequency model is to compare the total observed claim count with the total predicted claim count. For validation observations \(i=1,\ldots,m\), let \(N_i\) denote observed claims and \(\hat{\lambda}_i\) the predicted claim frequency per unit of exposure. The predicted count is \(e_i\hat{\lambda}_i\), and the observed-to-predicted count ratio is
A ratio of 1 indicates that observed and predicted counts agree in aggregate. A ratio above 1 means more claims occurred than predicted in total; a ratio below 1 means fewer occurred. This is an aggregate calibration check, not a complete measure of predictive quality. Overprediction in one segment can offset underprediction in another, and the ratio alone says nothing about the accuracy of individual predictions.
A Practical Out-of-Time Validation Process
State the prediction unit, outcome period, exposure measure, and date used to assign each observation to training or validation.
Use earlier experience for training and later experience for validation. Do not choose the boundary after inspecting which split makes a candidate model look best.
Fit coefficients and determine the data preparation and prediction rules using training information only. Record them before calculating validation results.
Apply the frozen model to validation exposures and predictors that would have been available at the relevant prediction time.
Summarize observed and predicted experience, including an aggregate calibration check. Investigate important segment differences rather than relying on a single overall number.
Worked Examples
Worked Example: Aggregate Frequency Calibration in a Later Period
A hypothetical insurer fits a frequency model using policy terms effective during 2021–2023. It freezes the model before examining 2024 outcomes. Assume the 2024 validation records are complete for the intended claim-count comparison, and the model predicts the following annual frequencies. Calculate predicted counts, the total observed-to-predicted ratio, and the overall observed and predicted frequencies.
| Risk class | 2024 exposure | Predicted frequency | Observed claims |
|---|---|---|---|
| A | 500 | 0.060 | 33 |
| B | 250 | 0.100 | 20 |
| C | 150 | 0.160 | 31 |
Step 1: Calculate predicted counts by class. Class A has \(500(0.060)=30\) predicted claims. Class B has \(250(0.100)=25\). Class C has \(150(0.160)=24\).
Step 2: Sum predicted and observed counts. Predicted claims total \(30+25+24=79\). Observed claims total \(33+20+31=84\).
Step 3: Calculate the aggregate ratio. The observed-to-predicted ratio is \(84/79=1.063291\ldots\), or \(1.063\) to three decimals. Thus, actual validation-period claims were about 6.3% above the model’s aggregate prediction.
Step 4: Check using overall frequencies. Total exposure is \(500+250+150=900\). The observed frequency is \(84/900=0.093333\ldots\), and the predicted frequency is \(79/900=0.087777\ldots\). Their ratio is \(0.093333\ldots/0.087777\ldots=1.063291\ldots\), agreeing with the count ratio.
The method is an exposure-weighted aggregate comparison. A common mistake is to average the three class frequencies equally; that would give each class the same weight despite their different exposures. The overall frequency must use total claims divided by total exposure. The check confirms the calculated ratio.
Worked Example: Assigning Policy Terms to the Correct Period
A hypothetical commercial insurer trains a frequency model using policy terms effective in 2021–2023 and validates it on terms effective in 2024. The rating variables are available at each term’s effective date. Assume the 2024 claim counts are complete for this exercise. The 2024 validation data are summarized below. Calculate aggregate predicted claims and the observed-to-predicted ratio.
| 2024 policy-term class | Exposure | Predicted frequency | Observed claims |
|---|---|---|---|
| Urban | 600 | 0.050 | 27 |
| Rural | 400 | 0.080 | 38 |
Step 1: Apply the model to the validation exposures. Urban predicted claims are \(600(0.050)=30\). Rural predicted claims are \(400(0.080)=32\).
Step 2: Aggregate. Total predicted claims are \(30+32=62\). Total observed claims are \(27+38=65\). Total exposure is \(600+400=1{,}000\).
Step 3: Calculate the ratio and rates. The ratio is \(65/62=1.048387\ldots\), or \(1.048\) to three decimals. Observed frequency is \(65/1{,}000=0.065\); predicted frequency is \(62/1{,}000=0.062\). Their ratio is \(0.065/0.062=1.048387\ldots\), the same result.
The validation assignment is by policy-term effective date, so a 2024 term belongs to the later period even if the same account had an earlier term in training. A common mistake is to assign the 2024 term to training because the account was previously insured. That would let later-period experience influence model fitting. The account’s prior history may be a valid predictor if it was available at the 2024 effective date, but the 2024 term’s outcomes remain validation outcomes. The arithmetic check confirms the aggregate ratio.
Worked Example: Distinguishing Historical Predictors From Future Outcomes
A hypothetical insurer predicts claim frequency for policy terms effective in 2025. Its model was fitted using terms effective through 2024. At a 2025 effective date, a policyholder has 2022–2024 prior claims history available. A new claim occurs during the 2025 term, but an internal claims system records a revised risk category after that claim. For this example, assume the frozen model predicts frequency \(0.080\) for a 2025 exposure of \(250\), and the final 2025 count is two claims. Determine the valid prediction, the observed-to-predicted ratio, and whether the revised category may be used.
Step 1: Use information available at prediction time. The model was fitted before 2025. Prior claims history through 2024 is available at the term’s effective date, so it may be used if it is a defined model predictor and is supplied in the same way as during training. The 2025 claims are not available then.
Step 2: Calculate the predicted count. The expected count is \(250(0.080)=20\).
Step 3: Compare with the observed outcome. The observed count is 2, so the observed-to-predicted ratio is \(2/20=0.100\). The 2025 observed frequency is \(2/250=0.008\), while the predicted frequency is \(0.080\); their ratio is also \(0.008/0.080=0.100\).
The revised category recorded after the 2025 claim must not be used to create the original 2025 prediction: it is future information relative to the prediction time and may reflect the outcome itself. Using it would contaminate the validation. The proper prediction remains 20 expected claims, and the check confirms that the frequency ratio matches the count ratio. This example is intentionally stark; the low ratio is not, by itself, evidence about why the difference occurred.
Common Traps and Interpretation
The most serious error is to preserve a date column while still allowing future information into the model. Examples include fitting a transformation using all years, choosing model terms after repeatedly checking the validation outcome, or using a predictor updated after the policy effective date. These practices make the validation result more optimistic or otherwise less representative of real future use. The essential check is not merely “Was the record dated later?” but “Could every part of this prediction have been produced with information available at the intended prediction time?”
A second error is to treat every record associated with a previously observed customer or account as training data. The correct assignment depends on the prediction unit and time rule. A later policy term can be validation data even when that account appeared earlier, while its genuinely available prior history may remain a permissible predictor. Conversely, splitting claim rows at random when the intended prediction unit is a policy can create an invalid comparison if related records cross the boundary.
Out-of-time validation provides evidence about performance in one later period. It does not prove that future results will be identical, nor does an overall ratio establish good performance for every class. If several models are repeatedly modified in response to the same validation outcomes, that period gradually becomes part of model development rather than an independent check. Report the cutoff, prediction unit, information timing, and relevant aggregate results so the evidence can be interpreted in context. This method is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.
Check Your Understanding
For each problem, keep the time assignment and exposure weighting explicit.
- A frozen model predicts frequency \(0.030\) for 200 validation exposure units and \(0.090\) for 100 units. The observed counts are 8 and 10, respectively. Find the total predicted count and observed-to-predicted ratio. (Answer: 15 predicted claims; ratio \(18/15=1.200\))
- A later-period validation set has 80 exposure units at predicted frequency \(0.040\) and 120 units at \(0.070\). There are 10 observed claims in total. Calculate the predicted count and aggregate ratio. (Answer: 11.6 predicted claims; ratio \(10/11.6=0.8621\))
- A model predicts policy-term frequency at each term’s effective date. A risk characteristic is revised after a claim during the term. May the revised value be used for the prediction made at the effective date? State the reason. (Answer: No; it was not available at the effective date and may also reflect information arising after the prediction.)
- For a later period, two segments have predicted counts of 20 and 30 and observed counts of 22 and 24. Calculate the aggregate ratio and interpret it. (Answer: \(46/50=0.920\); observed claims are 8% below aggregate prediction.)
- A frequency model is fitted on policy terms effective in 2021–2023 and evaluated on terms effective in 2024. A claim from a 2022 accident is reported in 2024. Under a validation rule assigning policy-term frequency observations by effective date and claim outcomes by accident period, does that claim make the policy term a 2024 validation observation? (Answer: No; the policy term remains assigned by its 2022 effective date, and the claim’s later report date does not change that assignment.)