Why Validation Must Respect Time
When a model will be used to predict a future outcome, its performance should be assessed using only information that would have been available when that prediction was made. This is essential for time-dependent data: outcomes close together in time may be related, and later observations can contain information about earlier conditions. A random split can put a future observation in the training set while an earlier observation is being assessed. The resulting estimate may not represent the task the model will face in practice.
Time-ordered validation addresses this by setting each training sample before its validation sample. The model is fitted using the past, then evaluated on a later period. The same principle applies to observations collected over time, such as monthly claim counts, daily demand, or successive measurements of a hypothetical insurance portfolio. This tutorial is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
A split is valid only if each training observation precedes the outcomes it is used to predict. In practice, this requirement applies to predictor availability as well as outcome dates. A predictor labelled with time \(t\) cannot be used at origin \(o\) if it was not actually recorded or released until after \(o\). The date attached to a value and the date on which it became available need not be the same.
Expanding-Window Validation
In an expanding-window design, the first training sample ends at an initial origin, and each later origin adds the newly observed data to the training sample. The training period grows over the folds, while validation always occurs after the corresponding origin. This resembles repeated model updates as new observations arrive.
Worked Example: Building Expanding-Window Folds
A hypothetical insurer has 11 consecutive monthly observations. The first five months are the minimum training period, and each forecast assesses the next two months. Construct non-overlapping validation blocks by advancing the origin two months at a time. State the training and validation indices for every fold.
Method. Use an expanding window, horizon \(h=2\), and initial origin \(o_1=5\). Advancing by two months gives origins \(5,7,9\).
Fold 1, origin 5. The training indices are \(1,2,3,4,5\). The next two observations, \(6,7\), form the validation block.
Fold 2, origin 7. The training indices are \(1,2,3,4,5,6,7\). The next two observations, \(8,9\), form the validation block.
Fold 3, origin 9. The training indices are \(1,2,3,4,5,6,7,8,9\). The next two observations, \(10,11\), form the validation block.
The training sizes are therefore 5, 7, and 9, and the validation blocks cover all six months from 6 through 11 exactly once. At each origin, every training observation occurs before both validation observations. This avoids the common mistake of keeping a fixed training sample while calling the design “expanding-window” validation.
In an actual evaluation, fit the model separately at each origin and record predictions for that fold’s validation block. The prediction for month 7 in Fold 1 must be made from the model fitted at origin 5; month 6’s realized outcome is not yet available at that origin. By contrast, when the origin advances to 7, the realized values through month 7 may be used for the next fit.
Rolling-Window Validation
An expanding window retains all earlier observations. A rolling window keeps only a fixed number of the most recent observations. Rolling windows can be appropriate when older data are no longer representative of current conditions, but choosing the window length is a modeling decision. The window must still end at the forecast origin.
where \(w\) is the fixed training-window length. As origins advance, old observations leave the training window while new ones enter. A rolling window does not mean that validation is random; it still follows the training period in time.
Worked Example: Constructing a Rolling Window
A hypothetical claims-monitoring model is assessed on nine consecutive weeks. Use the most recent four weeks to predict the next week. Begin at origin 4, then advance one week at a time through origin 8. List every training index and validation index.
Method. Use a rolling window of length \(w=4\) and a one-week horizon \(h=1\).
Origin 4. Training is weeks \(1,2,3,4\); validation is week \(5\).
Origin 5. Training is weeks \(2,3,4,5\); validation is week \(6\).
Origin 6. Training is weeks \(3,4,5,6\); validation is week \(7\).
Origin 7. Training is weeks \(4,5,6,7\); validation is week \(8\).
Origin 8. Training is weeks \(5,6,7,8\); validation is week \(9\).
Each training sample has four observations, and each validation observation comes after the training window at its origin. The common mistake is to keep weeks 1–4 in every fold: that would be a fixed-window assessment, not a rolling window, and it would not use the most recent four observations.
Forecast Horizons and Repeated Origins
A validation horizon specifies what is being assessed. A one-step-ahead forecast at origin \(o\) predicts \(o+1\). A fixed-origin forecast of horizon \(h\) predicts \(o+1,\ldots,o+h\) using information available at \(o\). The model cannot use the actual value at \(o+1\) to produce its forecast for \(o+2\) if both forecasts are intended to be made at the same origin.
Repeated one-step forecasting is different. After the outcome at \(o+1\) is observed, the origin can advance to \(o+1\), the model can be refitted or updated, and a new forecast can be made for \(o+2\). Thus, a long validation period may represent either multiple forecasts updated as data arrive or one multi-step forecast issued in advance. The evaluation design should match the intended use.
Worked Example: Fixed-Origin Versus Repeated One-Step Forecasts
A hypothetical monthly model is evaluated at origin 6 with horizon 3. The observed outcomes for months 7–9 are 20, 24, and 22. A fixed-origin model predicts 19, 21, and 23, respectively. A second evaluation instead makes a new one-step prediction at each successive origin: its predictions for months 7, 8, and 9 are 19, 23, and 21. Compute the mean absolute error for each design, and explain which predictions may use newly observed outcomes.
Method. For predictions \(\hat y_t\) and observations \(y_t\), the mean absolute error over three points is the sum of the three absolute errors divided by 3.
Fixed-origin calculation. The absolute errors are \(|20-19|=1\), \(|24-21|=3\), and \(|22-23|=1\). Their sum is \(1+3+1=5\), so the mean absolute error is \(5/3=1.6667\) to four decimal places.
Repeated one-step calculation. The absolute errors are \(|20-19|=1\), \(|24-23|=1\), and \(|22-21|=1\). Their sum is \(1+1+1=3\), so the mean absolute error is \(3/3=1.0000\) to four decimal places.
In the fixed-origin design, all three forecasts must be based only on information through month 6. In the repeated one-step design, the forecast for month 8 can use the observed month 7 outcome, and the forecast for month 9 can use observed outcomes through month 8. The two error values are correctly calculated, but they assess different forecasting tasks; comparing them as though they came from the same task is the mistake to avoid.
Why Random Shuffling Can Leak Future Information
Random splitting assigns observations to training and validation without regard to their time order. For independent and identically distributed observations, this can be a reasonable way to create representative subsets. For forecasting, however, a randomly selected training set can contain observations from after a validation time. The model is then assessed using information that would not have been available when that earlier forecast was made.
Worked Example: Quantifying the Effect of a Future Training Observation
Consider a hypothetical series with values \(y_t=t\) for \(t=1,\ldots,6\). Use the simple prediction rule that predicts a validation outcome with the mean of the training outcomes. Compare a random split that validates month 5 and trains on months 1, 2, 3, 4, and 6 with a chronological forecast at origin 4 that validates month 5.
Method. Calculate the training mean for each split and the absolute prediction error for month 5. The random split is intentionally shown to reveal whether it uses information unavailable at the chronological forecast origin.
Random split. The training outcomes are \(1,2,3,4,6\). Their sum is \(1+2+3+4+6=16\), so the training mean is \(16/5=3.2\). The prediction for month 5 is 3.2, and the absolute error is \(|5-3.2|=1.8\).
Chronological split. At origin 4, the training outcomes are \(1,2,3,4\). Their sum is \(1+2+3+4=10\), so the training mean is \(10/4=2.5\). The prediction for month 5 is 2.5, and the absolute error is \(|5-2.5|=2.5\).
The random-split error is lower by \(2.5-1.8=0.7\), but its training set includes \(y_6=6\), which occurs after the month 5 outcome being assessed. That information could not have been available at origin 4. The random split therefore gives an invalid estimate for this forecasting task, even though the arithmetic is correct. This example demonstrates a possible optimistic effect; random shuffling need not make every particular error smaller.
Practical Checks and Common Traps
Before interpreting a time-ordered validation result, check the full information timeline, not just the row indices. Confirm the initial training length, the origin of each fold, the forecast horizon, and whether the origin advances by one time point or by a whole validation block. Also check that the data used as predictors would have been observable at the stated origin.
- Randomizing time-dependent rows: a future outcome may enter the training set for an earlier validation observation.
- Mixing up the forecast task: a fixed-origin multi-step forecast cannot use outcomes that arrive during its validation block; repeated one-step forecasts may use them after each origin advances.
- Mislabeling the window: expanding training retains all earlier observations, while rolling training retains only a fixed number of recent observations.
- Ignoring release timing: a predictor’s timestamp alone does not establish that it was available at the forecast origin.
- Assuming folds are independent: successive training sets often overlap, and nearby validation outcomes may be dependent. The fold results should not automatically be treated as independent replicates.
Time-ordered validation is not a claim that dependence has disappeared. Rather, it preserves the direction of information flow while measuring performance in a setting that resembles future use. The split design should reflect how often the model is updated, how far ahead it must forecast, and whether older observations remain relevant.
Check Your Understanding
For each problem, write down the time-ordered split or identify the information issue before checking the final answer.
- A series has 12 observations. Use an expanding window with initial training observations 1–6, a two-observation validation horizon, and non-overlapping validation blocks. List the origins, training ranges, and validation ranges if the evaluation continues through observation 10. (Answer: Origins 6 and 8; training 1–6 with validation 7–8, then training 1–8 with validation 9–10.)
- A series has nine observations. Use a rolling training window of length 5 and one-step validation, starting at origin 5 and advancing through origin 8. Give the final fold’s training and validation indices. (Answer: Training 4–8; validation 9.)
- A model is assessed on observation 9, but its randomly assigned training set contains observation 11. Is this a valid split for a forecast made at origin 8, and why? (Answer: No; training includes a future observation, time 11, that was unavailable at origin 8.)
- A predictor labelled \(x_7\) is not released until the end of time 9. Can it be used to forecast \(y_8\) at origin 7? (Answer: No; it is not available until after origin 7.)
- At one origin, actual outcomes for three validation times are 10, 13, and 12; the corresponding predictions are 9, 15, and 10. Calculate the mean absolute error for this validation block. (Answer: \((1+2+2)/3=5/3\), or 1.6667 to four decimal places.)