Why Validate at More Than One Forecast Origin?
A rate model can perform well on one future period and poorly on another. A single out-of-time holdout answers how the model predicted one particular period; it does not show whether that result was typical, whether performance changed as experience accumulated, or whether a short-lived shift drove the outcome. Rolling-origin validation addresses this by repeating the forecast exercise at successive points in time.
At each origin, the model is fitted using historical data available through that point, then used to predict the next period. The next origin moves forward one period and includes the newly observed period in the training history. Thus the training window expands, while each one-period-ahead forecast is made without using the period being predicted. This sequential design is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.
For \(T\) observed periods and an initial training history through period \(t_0\), one-period-ahead validation produces forecasts for periods \(t_0+1,\ldots,T\). There are \(T-t_0\) origins and therefore \(T-t_0\) distinct model fits. The first origin uses periods \(1\) through \(t_0\); the next uses periods \(1\) through \(t_0+1\), and so on.
The phrase expanding window distinguishes this design from one that keeps a fixed-length training period and drops older data as the origin advances. Here, the full available history remains in the fit unless the validation plan explicitly defines a different design. The forecast horizon also matters: the examples below use one period ahead. A design that predicts multiple periods from a single origin answers a different forecasting question and should be specified as such.
Set Up the Validation Sequence
Before fitting, specify the ordered periods, the initial training history, the prediction target, and the forecast horizon. For example, the target may be claim frequency for the next accident year, with policy exposure serving as the denominator. Keep the target definition consistent from origin to origin. A change in what counts as a claim, how exposure is measured, or which period receives the loss can make forecast comparisons difficult to interpret.
At each origin, refit the model using only the data then available. This includes estimating model parameters again; carrying forward estimates from the first fit does not implement expanding-window refitting. For a GLM, the model form and predictor definitions are generally held fixed when evaluating a candidate specification, while its fitted parameters are estimated anew for each training window. If predictor values themselves use information observed after the origin, the resulting forecast leaks future information.
Establish a genuine time sequence and the same prediction unit and target for each forecast window.
Ensure the initial training period is sufficient to fit the model, and state how many periods ahead each forecast will be.
Use only information through the origin, then predict the next period.
Use the same target definition and preselected performance summary for every window, then examine whether results are stable or changing over time.
Frequency Forecasts With an Exposure Offset
A simple illustration uses an intercept-only Poisson frequency GLM. Suppose period \(j\) has claim count \(N_j\) and exposure \(e_j\), and assume \(N_j\) is Poisson with mean \(e_j\lambda\), where the common frequency rate \(\lambda\) is positive. The logarithm of exposure is an offset, so the fitted mean count is exposure multiplied by the fitted rate. This illustration assumes a common underlying rate across the training periods; it is not appropriate if that assumption is unsuitable for the application.
The fitted rate at origin \(t\) is estimated from the training periods only. A forecast for the next period is that fitted rate multiplied by the next period’s exposure. Exposure in the forecast period is used to scale the prediction; it is not claim experience and does not belong in the training estimate of the rate.
Worked Rolling-Origin Examples
Worked Example: One-Step-Ahead Frequency Forecasts
A hypothetical portfolio has the following claim counts and exposures. Use an intercept-only Poisson frequency model with an exposure offset. Start with period 1 as the training history, then forecast each next period in turn. Calculate each forecast count and its absolute count error.
| Period | Claims | Exposure |
|---|---|---|
| 1 | 8 | 1,000 |
| 2 | 12 | 1,200 |
| 3 | 9 | 900 |
| 4 | 15 | 1,500 |
Origin 1: forecast period 2. The training rate is \(8/1{,}000=0.008\). Multiplying by period 2 exposure gives a predicted count of \(0.008(1{,}200)=9.6\). The actual count is 12, so the absolute count error is \(|12-9.6|=2.4\).
Origin 2: forecast period 3. Add period 2 to the training history. The refitted rate is \((8+12)/(1{,}000+1{,}200)=20/2{,}200=0.0090909\ldots\). The predicted period 3 count is \((20/2{,}200)(900)=8.1818\ldots\). Its absolute error is \(|9-8.1818\ldots|=0.8182\ldots\).
Origin 3: forecast period 4. The training history now has 29 claims over 3,100 exposure, so the rate is \(29/3{,}100=0.0093548\ldots\). The predicted count is \((29/3{,}100)(1{,}500)=14.0323\ldots\), and the absolute error is \(|15-14.0323\ldots|=0.9677\ldots\).
The three forecasts are 9.60, 8.18, and 14.03 claims, with absolute count errors 2.40, 0.82, and 0.97, respectively (rounded to two decimals). The method is expanding-window Poisson refitting. A common mistake is to use the period being forecast in its own rate estimate; doing so would use future claims and make the apparent forecast error artificially small.
Worked Example: Compare Two Candidate Models on the Same Windows
Two hypothetical frequency models have been refitted at each origin and produce the following predicted rates. Compare them using total absolute count error over the same three forecast periods. The actual experience is 10 claims on 1,000 exposure in period 2, 15 on 1,500 in period 3, and 8 on 800 in period 4.
| Forecast period | Exposure | Actual claims | Model A rate | Model B rate |
|---|---|---|---|---|
| 2 | 1,000 | 10 | 0.0090 | 0.0105 |
| 3 | 1,500 | 15 | 0.0110 | 0.0095 |
| 4 | 800 | 8 | 0.0125 | 0.0090 |
For each window, multiply the predicted rate by exposure to obtain the predicted claim count. For Model A, the predicted counts are \(0.0090(1{,}000)=9\), \(0.0110(1{,}500)=16.5\), and \(0.0125(800)=10\). The absolute errors are \(|10-9|=1\), \(|15-16.5|=1.5\), and \(|8-10|=2\). Total absolute count error is \(1+1.5+2=4.5\).
For Model B, predicted counts are \(0.0105(1{,}000)=10.5\), \(0.0095(1{,}500)=14.25\), and \(0.0090(800)=7.2\). The absolute errors are \(0.5\), \(0.75\), and \(0.8\), respectively. Total absolute count error is \(0.5+0.75+0.8=2.05\).
Under this specified summary and these windows, Model B has the smaller total absolute count error. This does not prove that it will perform better on future periods; it describes performance on these particular forecasts. The method avoids a common comparison trap: scoring candidate models on different forecast periods, which confounds model differences with differences in the periods being predicted.
Worked Example: Spot a Change in Forecast Performance Over Time
A hypothetical book has experience as follows: periods 1, 2, and 3 each have a frequency of 0.010; period 4 has 18 claims on 1,200 exposure; period 5 has 15 claims on 1,000 exposure. Apply the same expanding-window intercept-only Poisson model, beginning with period 1, to forecast periods 2 through 5. Show how the errors change.
Period 2 is forecast from period 1: the estimated rate is \(10/1{,}000=0.010\), giving \(0.010(1{,}200)=12\) predicted claims, exactly the observed 12. Period 3 is forecast from periods 1 and 2: the rate is \(22/2{,}200=0.010\), giving \(0.010(800)=8\), again exactly observed.
Period 4 is forecast from periods 1 through 3: the rate is \(30/3{,}000=0.010\), giving \(0.010(1{,}200)=12\) predicted claims against 18 actual, an underprediction of 6. Period 5 is forecast from periods 1 through 4: the rate is \((30+18)/(3{,}000+1{,}200)=48/4{,}200=0.0114286\ldots\). The predicted count is \((48/4{,}200)(1{,}000)=11.4286\ldots\), versus 15 actual, an underprediction of \(3.5714\ldots\).
Absolute count errors by forecast period are 0, 0, 6, and 3.57 (rounded to two decimals), for a total of 9.57. The late underpredictions suggest that performance changed over the sequence. Rolling-origin results reveal that the model’s forecast errors were not evenly distributed across time; they do not, by themselves, establish the cause. The common mistake is to report only the combined total and miss that nearly all the error arose in the later windows.
Interpreting the Sequence and Avoiding Traps
Rolling-origin validation is useful because it preserves temporal order and tests repeated use of a model in a forecasting role. Comparing the windows can show whether performance is broadly steady, concentrated in particular periods, or changing as the origin advances. This evidence informs model assessment, but a small number of windows can be noisy. It is not a guarantee that future results will match the observed sequence.
Several design choices must remain explicit:
- Do not randomize the time order. A random split can train on later periods while testing on earlier ones, which does not reproduce the intended forecasting task.
- Do not reuse one fit for every origin. The model must be refitted as the training history expands. Otherwise the exercise does not assess the stated rolling procedure.
- Do not let the forecast period influence its own prediction. That includes claim experience and any predictor information unavailable at the origin.
- Do not change the target or comparison basis silently. Use consistent definitions and the same forecast windows when comparing candidate models.
- Do not treat an error summary as an explanation. A change in errors flags a pattern to investigate; it does not identify whether the cause was a rate shift, a mix change, random variation, or another issue.
The examples used one-step-ahead frequency forecasts and an absolute count-error summary for transparent arithmetic. Other targets and forecast summaries may be appropriate in practice, but the validation design still requires correctly ordered origins, repeated fitting, and forecasts made from information available at each origin. Choices about period alignment, exposure and loss maturity are important to the validity of the target data and are treated separately.
Check Your Understanding
For each problem, follow the stated rolling-origin setup and keep the forecast period out of its own training data.
- A frequency model is trained on period 1, with 6 claims on 600 exposure. What is its forecast count for period 2 if exposure is 900? (Answer: 9 claims.)
- There are seven ordered periods, and the first three form the initial training history. The design makes one-period-ahead forecasts through period 7. How many origins and model fits are required? (Answer: 4.)
- An intercept-only Poisson model has 24 claims on 3,000 exposure in its training history. What rate does it estimate, and what count does it predict for the next period with 500 exposure? (Answer: Rate 0.008; 4 claims.)
- A validation period has 14 actual claims on 1,200 exposure. A model predicts a rate of 0.0105. Find the predicted count and absolute count error. (Answer: 12.6 predicted claims; absolute error 1.4 claims.)
- A model starts with periods 1 and 2 as its initial history and makes one-period-ahead forecasts through period 6. State the training periods for the forecast of period 5 and the number of total forecast windows. (Answer: Train on periods 1–4 for the period 5 forecast; 4 forecast windows.)