Why Evaluate at More Than One Origin?
A single time-ordered train/test split answers a limited question: how did a model perform on one later block of observations after being trained on an earlier block? That can be useful, but the result may depend heavily on which period was selected for testing. Repeating the evaluation at several points in time shows how performance behaves across different prediction environments.
A rolling-origin design does this by moving the point at which a model is trained forward through the ordered data. At each origin, the model is fit using the permitted history and evaluated on subsequent observations. The training history can either accumulate as the origin advances or remain a fixed width. Those alternatives answer related but not identical questions: should older observations continue to inform the model, or should the training sample emphasize a more recent period?
The design is appropriate when the intended use is to predict future observations from past information, such as monthly claim counts, quarterly policy retention, or daily service demand. The examples below use invented data for illustration. A rolling-origin evaluation is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
The origin is a cutoff in time, not a randomly selected row. If the observations are indexed \(1,\ldots,T\), then an evaluation at origin \(t\) must not train on periods later than \(t\). The model is fitted again at each origin, using the training window defined by the design. The observations after the origin are used to assess predictions, not to fit that origin's model.
Expanding and Fixed-Width Windows
An expanding window retains all available observations from the start of the series through the current origin. If the first training period is 1, its training set at origin \(t\) is periods \(1,\ldots,t\). As the origin advances, the training set grows.
A fixed-width rolling window uses only the most recent \(w\) observations, where \(w\) is a chosen positive integer and \(t\geq w\). At origin \(t\), it uses periods \(t-w+1,\ldots,t\). Advancing the origin by one period removes the oldest training observation and adds the newly available one.
In either design, the training window is paired with the same forecast horizon if the evaluation goal is to compare models under the same prediction task. For a horizon of \(h\), the target block after origin \(t\) is \(t+1,\ldots,t+h\). The next origin need not be one period later. For example, advancing by \(h\) periods creates consecutive, nonoverlapping target blocks; advancing by one period creates overlapping target blocks when \(h>1\).
| Design choice | What enters the training set as the origin advances? | What question it emphasizes |
|---|---|---|
| Expanding | New observations are added; earlier observations remain | How well does learning from the full accumulated history work? |
| Fixed-width rolling | New observations enter and the oldest observations leave | How well does a recent history of length \(w\) work? |
Neither choice is automatically superior. An expanding window may be attractive when historical data remain relevant and the process is reasonably stable. A fixed-width window may be more representative when the relationship between predictors and outcomes changes over time, because it excludes older experience. But it also discards information and may leave fewer observations for estimating a model. These are design trade-offs, not guarantees about future performance.
Worked Example: Building an Expanding-Window Evaluation
A hypothetical insurer has 12 quarters of ordered claim-frequency data. An analyst chooses an initial training history of six quarters, a two-quarter forecast horizon, and origins at quarters 6, 8, and 10. The fitted model produces the predictions below. Construct the windows, compute the mean absolute error (MAE), and explain what was evaluated.
| Target quarter | Observed value | Prediction | Absolute error |
|---|---|---|---|
| 7 | 18 | 17 | 1 |
| 8 | 20 | 19 | 1 |
| 9 | 21 | 22 | 1 |
| 10 | 23 | 24 | 1 |
| 11 | 25 | 24 | 1 |
| 12 | 28 | 27 | 1 |
Step 1: List the training and target periods. At origin 6, train on quarters 1–6 and evaluate quarters 7–8. At origin 8, train on quarters 1–8 and evaluate quarters 9–10. At origin 10, train on quarters 1–10 and evaluate quarters 11–12. The training windows expand because each later fit retains the earlier quarters.
Step 2: Calculate absolute errors. The six errors are \(|18-17|=1\), \(|20-19|=1\), \(|21-22|=1\), \(|23-24|=1\), \(|25-24|=1\), and \(|28-27|=1\).
Step 3: Calculate the MAE. The MAE is the sum of absolute errors divided by the number of predictions: \((1+1+1+1+1+1)/6=6/6=1\) claim-frequency unit.
Check. There are three origins, each producing two predictions, so there are \(3\times2=6\) errors, as used in the denominator. The evaluation describes two-quarter-ahead forecasts made from growing histories; it does not describe one model fitted once to all 12 quarters.
What Changes When the Window Is Fixed?
A fixed-width window sets a limit on the amount of history used at every origin. For width \(w=4\), the training periods at origin 5 are 2–5; at origin 6 they are 3–6. Period 1 has left the window by the second origin. This makes the window's recency rule explicit: observations older than the most recent four periods do not contribute to that fit.
The choice of width should reflect the prediction task and the data available. A window that is too short may make estimates unstable or prevent a model from being fit reliably. A very long window may retain observations that describe a substantially different environment. A width is therefore a modeling decision to be assessed with an evaluation design, rather than a universal constant.
Worked Example: Moving a Four-Period Window
A hypothetical service center records demand in periods 1–9. An analyst uses a fixed-width window of four periods and makes one-period-ahead predictions at origins 5, 6, 7, and 8. The observed demands in periods 6–9 are 27, 31, 30, and 34. The corresponding predictions from models fit at those origins are 23, 26, 27, and 32. Identify each training window and calculate the MAE.
Step 1: Construct each training window. At origin 5, the window is periods \(5-4+1=2\) through 5, or 2–5; the target is period 6. At origin 6, it is periods 3–6, with target 7. At origin 7, it is periods 4–7, with target 8. At origin 8, it is periods 5–8, with target 9. Each window has exactly four observations.
Step 2: Calculate each error. For period 6, \(|27-23|=4\). For period 7, \(|31-26|=5\). For period 8, \(|30-27|=3\). For period 9, \(|34-32|=2\).
Step 3: Average the errors. The total is \(4+5+3+2=14\), and there are four predictions. Therefore, the MAE is \(14/4=3.5\) demand units.
Check. The four targets are the one-period-ahead observations 6, 7, 8, and 9, one from each origin. The training window at origin 8 ends at period 8, so period 9 is not used to fit the prediction for period 9.
Choosing Origins, Horizons, and Gaps
The origin schedule determines which points in time are represented in the evaluation. Origins spaced one period apart give many evaluation points, but for a horizon greater than one, their target blocks overlap. For example, with horizon 3, origins 5 and 6 evaluate periods 6–8 and 7–9. Periods 7 and 8 contribute to both blocks. Origins spaced three periods apart instead give nonoverlapping target blocks, though they provide fewer evaluation origins.
Overlapping target blocks are not inherently wrong. They may match an operational task in which a new forecast is made every period, even when earlier forecasts still cover some of the same future periods. The analyst should state the origin spacing and horizon so readers know what the set of predictions represents. An aggregate score over overlapping forecasts counts forecast instances, not necessarily distinct calendar periods.
A gap can be inserted between the origin and the forecast targets when predictions cannot use the immediately following period or when the operational process has a delay. Let \(g\) be the number of periods in the gap. With horizon \(h\), the gap periods are \(t+1,\ldots,t+g\), and the targets are \(t+g+1,\ldots,t+g+h\). If \(g=0\), this reduces to the usual targets beginning at \(t+1\). The gap must reflect the real information and timing constraints of the intended use.
The score also needs a clear aggregation rule. If every origin produces the same number \(h\) of predictions and every forecast instance is given equal weight, the overall MAE is the total absolute error across all origins and targets divided by the total number of forecast instances. If origins produce different numbers of predictions, pooling all errors weights origins with more predictions more heavily than origins with fewer predictions. Averaging origin-specific MAEs instead gives each origin equal weight. These summaries can differ, so the chosen rule should match the evaluation question.
Why the Training Window Defines the Evaluation
A rolling-origin score is conditional on its design. The window determines which historical outcomes are available for fitting; the horizon determines how far ahead the predictions must reach; and the origins determine which future environments are represented. A low error for a fixed-width window, for example, supports performance for that particular width and origin schedule. It does not by itself establish that every recent-history window will perform well.
The distinction between expanding and fixed-width windows is especially meaningful when the data-generating environment may change. An expanding window assumes that the older history remains eligible to inform the current fit. A fixed-width window deliberately excludes older observations once they fall outside its width. If the underlying relationship is stable, additional history can be useful. If it changes, older data can pull the fit toward a no-longer-representative pattern. The evaluation compares these choices only to the extent that its origins and targets reflect environments relevant to future use.
A short justification helps explain why the evaluation score depends on all these decisions. Suppose absolute error is the chosen metric and there are \(J\) origins, each with \(h\) targets. Write \(\hat{y}_{j,k}\) for the prediction at origin \(j\) for its \(k\)-th target, and \(y_{j,k}\) for the corresponding observed value. Equal weight per forecast instance gives:
This expression follows directly from the definition of an arithmetic mean: sum the \(Jh\) absolute errors and divide by their count. If each origin's MAE is averaged with equal weight instead, the expression is:
When every origin has the same \(h\), these two formulas are algebraically equal, because each origin contributes exactly \(h\) errors. If the number of targets varies across origins, they need not be equal: pooled MAE weights each forecast instance equally, while the second formula weights each origin equally. Overlapping target blocks may also cause the same observed period to appear in more than one forecast instance, which is appropriate only if that matches the intended evaluation.
A Common Exam Trap
A frequent error is to treat a fixed-width window as if it were expanding, or to let a training window include observations after its origin. For a width of 4 at origin 7, the correct training periods are 4–7. Using periods 1–7 is an expanding window, not a fixed-width window; using periods 5–8 is invalid for an origin at 7 because period 8 lies in the future.
The correction is to write down the origin, width, and target periods before calculating any score. For a fixed-width window, calculate its first training index as \(t-w+1\); then verify that the last training index is \(t\). For an expanding window starting at period 1, verify that the training indices are 1 through \(t\). Finally, list targets after the origin, with any stated gap applied. This simple index check catches window-label and time-order mistakes before they affect the evaluation.
Specify the outcome timing, forecast horizon, and whether any gap is needed.
Set the first origin after enough initial training history is available, then state how origins advance.
Use all observations through the origin for an expanding window, or only the most recent \(w\) observations for a fixed-width window.
Use observed outcomes after each origin and state whether the metric weights forecast instances or origins equally.
Check Your Understanding
For each problem, identify the relevant windows or compute the requested evaluation quantity. Assume observations are indexed by consecutive integers.
- An expanding design starts with periods 1–5, uses a one-period horizon, and evaluates at origins 5, 6, and 7. List the three training windows and their targets. (Answer: Training windows 1–5, 1–6, and 1–7; targets 6, 7, and 8, respectively.)
- A fixed-width design has width 4, horizon 2, and origins 6 and 8 with no gap. List the training periods and targets at each origin. (Answer: At origin 6, train on 3–6 and target 7–8; at origin 8, train on 5–8 and target 9–10.)
- At three one-period-ahead origins, the observed values are 12, 15, and 19, and predictions are 10, 18, and 17. Calculate the MAE. (Answer: Absolute errors are 2, 3, and 2; MAE \(=7/3=2.333\) to three decimals.)
- A fixed-width design uses width 5 at origin 9, with a two-period gap and a three-period horizon. State the training, gap, and target periods. (Answer: Training periods 5–9; gap periods 10–11; target periods 12–14.)
- Two origins each produce two forecasts. The absolute errors are 1 and 3 at the first origin, and 2 and 6 at the second. Compute the pooled MAE and the equal-origin average MAE, then explain why they agree here. (Answer: Each origin MAE is 2 and 4, so their average is 3; pooled MAE \(=(1+3+2+6)/4=3\). They agree because both origins contribute the same number of forecast instances.)