Why Evaluate Forecasts at Multiple Origins?
A forecasting procedure is used repeatedly: information accumulates, a forecast is made, the outcome arrives, and the next forecast is produced. Evaluating only one forecast gives little evidence about how the procedure performs across those repeated decisions. Rolling-origin evaluation recreates that sequence at several points in the observed time series.
The key is to preserve the timing of an actual forecast. At each origin, use only observations that would have been available then, forecast a specified number of periods ahead, and compare that forecast with the outcome at the corresponding future time. To make the results comparable, this tutorial focuses on a fixed horizon: the same number of periods ahead at every origin.
With this sign convention, a positive error means the forecast was too low, and a negative error means it was too high. The observed value at \(T_j+h\) must be available to the evaluator, but it must not be used to produce the forecast made at \(T_j\).
Constructing a Fixed-Horizon Evaluation
First specify the forecasting procedure being evaluated. This includes how the model is fitted and how much historical data it uses. At each origin, apply that procedure using information available through the origin; generate the \(h\)-step forecast; and, once the outcome is observed, record its error. Depending on the procedure, the model may be refitted at each origin.
Two common ways to choose the observations used for fitting are:
- Expanding window: the first training observation stays fixed, while each successive origin adds the newly available observations.
- Fixed-length rolling window: each fit uses a set number of the most recent observations, so older observations leave the window as the origin advances.
These are different forecasting procedures and can perform differently. The evaluation should use the window rule that represents the procedure of interest, rather than changing the rule partway through without explanation.
For fixed horizon \(h\), the evaluation set can be represented as the errors \(e_{T_1,h},\ldots,e_{T_m,h}\). Common summaries include mean error, mean absolute error, mean squared error, and root mean squared error:
Mean error retains direction: positive values indicate average underforecasting under the stated error convention, and negative values indicate average overforecasting. MAE and RMSE are nonnegative summaries of error magnitude, in the units of \(y_t\). MSE is in squared units. Squaring gives relatively more weight to large errors, so RMSE can respond more strongly than MAE to an unusually large miss.
Specify the horizon \(h\), the origins, and the training-window and refitting rules.
For origin \(T_j\), use data available through \(T_j\) to produce \(\hat{y}_{T_j+h\mid T_j}\).
When observed, compare the forecast with \(y_{T_j+h}\), not with the observation at the origin or at another date.
Calculate the selected summary over the \(m\) matched errors and interpret it for this horizon and design.
Why the Average Error Measures Repeated Performance
For a fixed forecasting procedure, horizon, and evaluation design, each origin produces a loss. For example, absolute-error loss at origin \(T_j\) is \(|e_{T_j,h}|\). The average of the observed losses is the empirical average performance across those forecast occasions:
This is not merely a convenient calculation. If the origins represent the kinds of occasions on which the procedure will be used, and the forecasting environment is sufficiently stable, their average loss provides an estimate of the procedure's expected loss on a future occasion of the same kind. The reasoning is that expected loss is an average over possible forecasting occasions; the evaluation approximates that average by observing losses at a set of past occasions.
That interpretation has conditions. If the series changes substantially over time, earlier origins may not represent the future. If forecast errors are dependent, the \(m\) errors are not equivalent to \(m\) independent observations; overlapping target periods can contribute to that dependence. Consequently, a small average error is evidence about performance under the evaluated design, not a guarantee of future performance or proof that the procedure is best in every setting.
Worked Example: Summarizing One-Step Errors
A hypothetical service team evaluates one forecasting procedure at four successive origins, using a one-period horizon. The forecasts and later observed values are shown below. Calculate each error, the mean error, MAE, MSE, and RMSE.
| Origin \(T_j\) | Forecast \(\hat{y}_{T_j+1\mid T_j}\) | Actual \(y_{T_j+1}\) |
|---|---|---|
| 20 | 104 | 106 |
| 21 | 108 | 105 |
| 22 | 107 | 109 |
| 23 | 111 | 110 |
Method. Use actual minus forecast at every origin, then aggregate the four errors. The horizon is \(h=1\) throughout.
At origin 20, \(e_{20,1}=106-104=2\). At origin 21, \(e_{21,1}=105-108=-3\). At origin 22, \(e_{22,1}=109-107=2\). At origin 23, \(e_{23,1}=110-111=-1\). Thus the errors are \(2,-3,2,-1\).
The mean error is \(\operatorname{ME}_1=(2-3+2-1)/4=0/4=0\). The absolute errors are \(2,3,2,1\), so \(\operatorname{MAE}_1=(2+3+2+1)/4=8/4=2\).
The squared errors are \(2^2=4\), \((-3)^2=9\), \(2^2=4\), and \((-1)^2=1\). Therefore, \(\operatorname{MSE}_1=(4+9+4+1)/4=18/4=4.5\), and \(\operatorname{RMSE}_1=\sqrt{4.5}\approx2.1213\).
Check. The positive and negative errors cancel in the mean, while the MAE remains positive. The RMSE, approximately \(2.1213\), is in the same units as the observations; the MSE, \(4.5\), is in squared units. The mistake avoided is treating zero mean error as zero forecast error: the four forecasts still have nonzero errors.
Keeping the Horizon and Comparison Fair
A fixed-horizon summary answers a specific question: how did this procedure perform \(h\) periods ahead across the chosen origins? An error at horizon one and an error at horizon four refer to different forecasting tasks. They should not be silently pooled and described as the performance of one fixed-horizon task.
When comparing two procedures, use the same horizon and the same origins so that each is evaluated against the same target observations. At each origin, both procedures must be given the information available at that origin and evaluated against \(y_{T_j+h}\). If one procedure is evaluated on easier dates or a different set of targets, the difference in aggregate errors cannot be attributed cleanly to the forecasting procedures.
Window choices also affect the meaning of a comparison. An expanding-window model and a fixed-length-window model may receive different training data. That is legitimate if the goal is to compare those complete procedures, but the window design should be stated. A comparison of model forms alone requires holding the rest of the forecasting procedure constant.
Worked Example: Evaluating Two-Step Forecasts with a Rolling Window
A hypothetical analyst evaluates a two-step-ahead forecasting procedure that uses the most recent eight observations at each origin. At origins 30, 32, and 34, the forecasts are 49, 50, and 53. The corresponding observations two periods later are 52, 47, and 55. Calculate the three errors and the aggregate measures.
Method. The horizon is \(h=2\) at all three origins. At each origin, subtract the forecast from the observation two periods later. The stated eight-observation window describes how each forecast is fitted; it does not change which target is matched to that forecast.
For origin 30, the target is time 32 and the error is \(52-49=3\). For origin 32, the target is time 34 and the error is \(47-50=-3\). For origin 34, the target is time 36 and the error is \(55-53=2\). The errors are therefore \(3,-3,2\).
The mean error is \(\operatorname{ME}_2=(3-3+2)/3=2/3\approx0.6667\). The absolute errors sum to \(3+3+2=8\), so \(\operatorname{MAE}_2=8/3\approx2.6667\). The squared errors sum to \(3^2+(-3)^2+2^2=9+9+4=22\), so \(\operatorname{MSE}_2=22/3\approx7.3333\), and \(\operatorname{RMSE}_2=\sqrt{22/3}\approx2.7080\).
Check. Each error uses the value two periods after its own origin. The observations at times 32, 34, and 36 are the targets, not the values at times 30, 32, and 34 paired by position in the origin list. The mistake avoided is shifting a multi-step forecast onto the wrong target date.
Worked Example: Comparing Procedures with the Same Targets
Two hypothetical forecasting procedures are evaluated at the same four origins and at the same fixed horizon. Their actual target values and forecasts are below. Compare their MAE and RMSE.
| Origin | Actual target | Procedure A forecast | Procedure B forecast |
|---|---|---|---|
| 40 | 20 | 19 | 20 |
| 41 | 18 | 20 | 18 |
| 42 | 23 | 21 | 19 |
| 43 | 25 | 26 | 25 |
Method. Calculate actual minus forecast for each procedure against each shared target. Then calculate MAE and RMSE separately.
For Procedure A, the errors are \(20-19=1\), \(18-20=-2\), \(23-21=2\), and \(25-26=-1\). Their absolute values sum to \(1+2+2+1=6\), giving \(\operatorname{MAE}_A=6/4=1.5\). Their squares sum to \(1+4+4+1=10\), giving \(\operatorname{MSE}_A=10/4=2.5\) and \(\operatorname{RMSE}_A=\sqrt{2.5}\approx1.5811\).
For Procedure B, the errors are \(20-20=0\), \(18-18=0\), \(23-19=4\), and \(25-25=0\). Their absolute values sum to \(4\), giving \(\operatorname{MAE}_B=4/4=1\). Their squares sum to \(16\), giving \(\operatorname{MSE}_B=16/4=4\) and \(\operatorname{RMSE}_B=\sqrt{4}=2\).
Check. Procedure B has the lower MAE, \(1\) versus \(1.5\), but the higher RMSE, \(2\) versus approximately \(1.5811\). Its single error of 4 is heavily reflected in the squared-error measure. There is no contradiction: the two summaries weight error patterns differently. The mistake avoided is claiming one procedure is better under every error measure merely because it has the lower MAE.
Common Traps and Interpretation Limits
- Forecasting with future information. A forecast at origin \(T\) must be based on information available through \(T\), not on the outcome at \(T+h\). Otherwise, the evaluation does not represent a real-time forecast.
- Pairing a forecast with the wrong date. For horizon \(h\), the target is \(y_{T+h}\), not \(y_T\). This becomes especially easy to misread when the horizon exceeds one period.
- Mixing horizons without saying so. An average combining several horizons does not describe performance at one fixed horizon. State the horizon associated with the reported errors.
- Comparing different evaluation sets. A metric difference is not a fair procedure comparison if the procedures were tested on different origins or targets.
- Reading a small average as a guarantee. The observed origins may not represent future conditions, and successive errors may be dependent. Rolling-origin results estimate performance under the evaluated conditions; they do not remove uncertainty about change over time.
Rolling-origin evaluation is useful because it respects the sequence in which forecasts are actually made. Its conclusions are strongest when the origin schedule, horizon, fitting rule, and error summary match the intended repeated use. When reporting results, state these design choices alongside the metric so that the reader knows what performance was measured.
Check Your Understanding
For each problem, use actual minus forecast and keep the stated horizon fixed.
- A procedure makes one-step forecasts of 12, 13, and 14; the corresponding next-period observations are 14, 11, and 15. Find the errors, MAE, and RMSE. (Answer: Errors \(2,-2,1\); MAE \(=5/3\approx1.6667\); RMSE \(=\sqrt{3}\approx1.7321\).)
- At three origins, a two-step procedure has errors \(4,-1,2\). Find its mean error and MAE, and state what the sign of the mean error indicates. (Answer: Mean error \(=5/3\approx1.6667\); MAE \(=7/3\approx2.3333\); positive mean error indicates average underforecasting.)
- At four common origins and one fixed horizon, a procedure has errors \(1,1,-4,0\). Calculate MSE and RMSE. (Answer: MSE \(=(1+1+16+0)/4=4.5\); RMSE \(=\sqrt{4.5}\approx2.1213\).)
- At origins 10, 11, and 12, an analyst reports one-step forecasts. Which actual observations must be used to score them, and what information may be used to form the forecast at origin 11? (Answer: Score against \(y_{11},y_{12},y_{13}\), respectively; the forecast at origin 11 may use observations available through time 11, but not \(y_{12}\) or \(y_{13}\).)
- Two procedures are tested at the same three origins and horizon. Procedure A has errors \(0,0,3\), and Procedure B has errors \(1,1,1\). Compare their MAE and RMSE. (Answer: A has MAE \(=1\) and RMSE \(=\sqrt{3}\approx1.7321\); B has MAE \(=1\) and RMSE \(=1\). Their MAEs tie, while B has the lower RMSE.)