Rolling-Origin Forecast Evaluation

Why the Forecasting Timeline Matters

A forecasting model can be estimated correctly and still be evaluated incorrectly. The central design question is whether each forecast uses only information that would actually have been available when it was made. A random split of a time series can put later observations into the training sample for a forecast of an earlier period; the resulting evaluation then benefits from information unavailable in real use.

A rolling-origin evaluation avoids that particular problem by moving the forecast origin forward through time. At each origin, the model is fitted or updated using an explicitly defined history, and its forecast is recorded for a future target. The design must specify the training window, forecast horizons, origins, and re-estimation schedule. These choices determine what forecasting procedure is being evaluated.

Definition: Let the observed series be \(y_1,\ldots,y_N\), ordered in time. A forecast origin \(t\) is the last time point whose information is permitted to inform a forecast. A horizon \(h\) is the number of time steps from the origin to the target. Write the forecast as \(\hat{y}_{t+h\mid t}\): the forecast of \(y_{t+h}\) made using information available through time \(t\). The forecast error, once the target is observed, is \(e_{t+h\mid t}=y_{t+h}-\hat{y}_{t+h\mid t}\).

The notation makes the timing explicit: the forecast at origin \(t\) is paired with the observation at \(t+h\), not with \(y_t\) or \(y_{t+h-1}\). In a monthly series, for example, \(h=1\) is the next month and \(h=12\) is twelve months ahead. Any fitted model, parameter estimate, filter state, transformation, or other input used at origin \(t\) must be obtainable without using observations later than \(t\).

Expanding and Fixed Training Windows

In an expanding-window evaluation, the training sample starts at the beginning of the available series and grows as the origin advances. If the first forecast origin is \(m\), the fit at origin \(t\) uses \(y_1,\ldots,y_t\). This design retains all prior observations and represents a procedure that continues to accumulate historical data.

In a fixed-window evaluation, the training sample has a constant length \(w\). At origin \(t\), it uses the most recent \(w\) observations, \(y_{t-w+1},\ldots,y_t\). Older observations are dropped as the origin moves. This design can represent a forecasting procedure intended to adapt to changing conditions by placing a limit on how far back its estimation sample reaches.

\begin{aligned} \text{Expanding window at origin }t &: \quad y_1,\ldots,y_t,\\ \text{Fixed window of length }w\text{ at origin }t &: \quad y_{t-w+1},\ldots,y_t. \end{aligned}

Both designs can be implemented with the same model specification, such as an ARMA model, while using different estimation samples. The window choice is part of the forecasting procedure, not just a bookkeeping detail. A fixed window can reduce the influence of remote history, but it also discards observations that may help estimate parameters. An expanding window preserves those observations, but can be less responsive if the series’ behavior changes. The evaluation describes performance for the chosen design; it does not by itself establish which window will be best in every future period.

Worked Example: Building an Expanding-Window Schedule

Suppose a hypothetical quarterly series contains \(N=18\) observations. A forecasting team requires an initial estimation sample of \(m=10\) quarters and wants to evaluate three-step-ahead forecasts. At each origin, it re-estimates the model using every observation available through that origin. Determine the origins, training samples, and targets.

A three-step forecast from origin \(t\) targets \(y_{t+3}\). To observe that target within the data, the origin must satisfy \(t+3\leq18\), or \(t\leq15\). The eligible origins are therefore \(t=10,11,12,13,14,15\).

At origin 10, the training sample is \(y_1,\ldots,y_{10}\) and the target is \(y_{13}\). At origin 11, the sample expands to \(y_1,\ldots,y_{11}\) and the target is \(y_{14}\). At origin 15, the sample is \(y_1,\ldots,y_{15}\) and the target is \(y_{18}\). The design produces \(18-3-10+1=6\) observed three-step forecasts.

For example, suppose the fitted model at origin 10 produces \(\hat{y}_{13\mid10}=52.4\) index points and the observed value is \(y_{13}=54.1\) index points. The recorded forecast error is \(54.1-52.4=1.7\) index points. The origin, horizon, forecast, target, and units are all explicit; the same pairing is recorded at each other eligible origin.

This schedule assumes the quarterly observations through each origin were available when the forecast was made, the initial sample is sufficient for the selected fitting procedure, and the same three-step horizon is used throughout. If observations are released with a delay, the origin must instead reflect the latest data actually available on the forecast date.

Choosing Origins and Horizons

For a single horizon \(h\), let \(m\) be the first forecast origin. If every target must lie in the observed series, origins run from \(m\) through \(N-h\). The number of forecast-target pairs is

$$ (N-h)-m+1=N-h-m+1, $$

provided \(m\leq N-h\). The \(+1\) counts both endpoint origins. For a set of horizons from 1 through \(H\), one useful comparison design uses the common origins \(m,\ldots,N-H\). Every origin in this set then has an observed target at every horizon being compared. Alternatively, a study can use all available origins separately for each horizon, but the number and calendar dates of the evaluation pairs will differ by horizon. The choice should be stated rather than left implicit.

Using a common origin set makes the forecasts across horizons refer to the same forecast dates, which can simplify comparisons. It also sacrifices some short-horizon forecasts near the end of the sample. Whatever the choice, a forecast should only enter the evaluation once its target is observed. Do not silently fill a missing target with a later value or include forecasts whose target falls outside the evaluation period.

Worked Example: Comparing Window Designs at the Same Origins

Consider a hypothetical monthly series with \(N=20\) observations. Use an initial origin \(m=8\), a fixed-window length \(w=6\), and evaluate forecasts at horizons \(h=1\) and \(h=2\). The team wants both horizons available at each origin, so it uses common origins through \(N-H=20-2=18\). The origins are \(8,\ldots,18\), giving \(18-8+1=11\) origins and 22 forecast-target pairs.

At origin 8, an expanding fit uses \(y_1,\ldots,y_8\); the fixed-window fit uses \(y_3,\ldots,y_8\), because \(8-6+1=3\). At origin 9, the respective samples are \(y_1,\ldots,y_9\) and \(y_4,\ldots,y_9\). At origin 18, they are \(y_1,\ldots,y_{18}\) and \(y_{13},\ldots,y_{18}\). Both methods forecast \(y_{t+1}\) and \(y_{t+2}\) at each common origin.

Suppose, for illustration, the expanding fit at origin 8 gives a one-step forecast of 101.2 units and the observed \(y_9\) is 100.7 units. Its recorded one-step error is \(100.7-101.2=-0.5\) units. If the fixed-window fit at the same origin gives 100.9 units, its error for that same target is \(100.7-100.9=-0.2\) units. The values are illustrative outputs of two fitted procedures; the essential design feature is that both forecasts use only data through origin 8 and are paired with the same observed target \(y_9\).

The common-origin rule is satisfied because the latest target is \(y_{18+2}=y_{20}\), which is observed. The sample sizes and window endpoints are also feasible: the fixed window always contains six observations, and the expanding window has at least eight. This comparison evaluates two different window rules while holding the origins and horizons fixed.

Forecast Frequency and Re-estimation Frequency

A forecast can be produced at every origin even if model parameters are not re-estimated at every origin. These are distinct design choices. For example, a team might generate monthly forecasts but re-estimate model parameters quarterly. Between estimation dates, it can retain the latest parameter estimates while updating the model’s conditional information using observations that have become available. For an ARMA forecast, this means the forecast at a later origin can condition on newly observed outcomes without pretending that the parameters were re-estimated.

The schedule must say which procedure is being evaluated. One option is to fit the model anew at every origin. Another is to re-estimate at specified origins and carry the latest parameter estimates forward in between. For the second option, record both the forecast origin and the parameter-estimation date. If parameters are estimated at origins 12, 15, and 18, for example, forecasts made at 13 and 14 use the estimates from 12, while the information set for each forecast can still include observations through its own origin.

Worked Example: Monthly Forecasts With Quarterly Re-estimation

Suppose a hypothetical monthly series has observations \(y_1,\ldots,y_{24}\). A team begins forecasting at origin 12, issues a one-step forecast each month, and re-estimates an ARMA model every three months, at origins 12, 15, 18, and 21. Assume monthly observations are available before each month’s forecast is issued. The parameter fit at origin 12 uses the specified training window through \(y_{12}\); forecasts at origins 13 and 14 retain those parameters but use the observations available through \(y_{13}\) and \(y_{14}\), respectively, when forming conditional forecasts.

At origin 13, for example, the forecast is \(\hat{y}_{14\mid13}\), not a forecast of \(y_{13}\). If the forecast is 48.6 units and the observed \(y_{14}\) is 49.0 units, the recorded error is \(49.0-48.6=0.4\) units. At origin 15 the model is re-estimated using data through \(y_{15}\), and its one-step forecast is paired with \(y_{16}\).

With origins 12 through 23, all 12 one-step targets are observed, since \(23+1=24\). The example assumes the monthly updates to the conditional information are possible and that the re-estimation schedule is part of the intended procedure. If the actual forecasting operation instead keeps the entire fitted model state unchanged between quarterly updates, that different procedure must be reproduced in the evaluation.

Safeguards Against Look-Ahead Information

A rolling origin alone does not guarantee a valid evaluation. Every step that produces a forecast must respect the information boundary at that origin. In particular, model estimation, data cleaning, transformations, and any choice among candidate specifications must not use future observations. A transformation whose parameters are estimated from the full series, for example, can leak future information even if the final ARMA fit uses only the training window.

Practical safeguards include:

For each origin, fit parameters using only the designated training window and any permitted past information.
Estimate data-dependent preprocessing choices, such as centering or scaling values, from that training data only, then apply them to the forecast input.
If a forecasting procedure chooses among model specifications, make that choice using information available at the origin, not the complete evaluation sample.
Respect real release dates, publication lags, and data revisions. Use the data vintage that would have been available at the forecast date when that distinction matters.
Record the origin, estimation window, model specification, re-estimation date, horizon, forecast, and target date for every forecast.

These safeguards do not require a series to be stationary in every possible application. They do require the historical forecasting exercise to represent a procedure that could actually have been implemented with the information then available. Whether past forecast performance is informative about future performance also depends on how relevant the evaluation period is to the conditions in which forecasts will be used.

A Short Justification for the Evaluation Count

For a fixed horizon \(h\), the target at origin \(t\) is \(t+h\). The target is observed precisely when \(t+h\leq N\), which rearranges to \(t\leq N-h\). If the first eligible origin is \(m\), the integer origins are \(m,m+1,\ldots,N-h\). Counting the endpoints gives

\#\{\text{origins}\}=(N-h)-m+1.

This count is a useful design check: if a schedule claims more forecasts than this for a single horizon and evaluation series, then it must be using a different target range, counting forecasts whose outcomes are not yet observed, or making an indexing error.

Why the Design Must Be Reported

An evaluation result is meaningful only in relation to the procedure that generated it. A model re-estimated at every origin is not the same forecasting system as a model updated annually. A fixed window of 60 months is not the same as an expanding history, even if the same ARMA form is used. Similarly, changing the initial origin changes the amount of estimation data and the calendar periods being evaluated.

A clear design report states the series and observation timing, initial estimation period, window type and length, forecast origins, horizons, re-estimation schedule, and any rules for handling data availability or model selection. When comparing procedures, keeping their origins, horizons, and target observations aligned helps ensure that differences reflect the procedures rather than mismatched evaluation samples. Forecast errors at different origins can share target periods or reflect common shocks; their dependence is one reason not to treat the resulting list as automatically independent observations.

Common pitfall: A researcher fits a model using the full series, then describes forecasts for earlier dates as “out of sample” because each forecast target was omitted from the final prediction calculation. This is still look-ahead: the fitted parameters or preprocessing may have used observations after those forecast origins. The correction is to reconstruct the entire forecasting procedure at each origin, restricting every information-dependent step to data available by that date.

Check Your Understanding

Use the definitions and design rules above to answer the following questions.

A series has \(N=30\) observations, an initial origin \(m=12\), and a forecast horizon \(h=4\). What are the first and last eligible origins, and how many forecast-target pairs are available?
At origin \(t=25\), what observations enter an expanding window? What observations enter a fixed window of length \(w=8\)?
A team forecasts every month but refits its model every six months. What dates should it record to distinguish forecast timing from parameter-estimation timing?
Why can estimating a centering constant from the full series create leakage, even if model parameters are fitted using only observations through the forecast origin?
For a comparison of horizons \(h=1,2,3\), what is one advantage of using the same origins for all three horizons, and what is the corresponding cost near the sample end?