Time-Ordered Evaluation Of Smoothing Methods

Why Forecast Evaluation Must Respect Time

A smoothing method is useful only if its forecasts perform well on observations that were not available when the forecasts were made. For time series, this requirement has a direct consequence: training data must precede the observations used to evaluate a forecast. A random split can place a later observation in the training set while an earlier observation is being treated as a forecast target. That no longer represents the task of forecasting forward in time.

A time-ordered evaluation compares methods at successive forecast origins. At each origin, the method is fitted or updated using the observations available up to that time, forecasts later values, and is scored only after those later values are observed. The same chronological procedure is applied to every candidate method.

Definition: Let $y_t$ denote the observed value at time $t$. A forecast origin $T$ is the last time point whose observation is available when a forecast is made. The forecast of $y_{T+h}$, made at origin $T$, is written $\hat{y}_{T+h|T}$, where $h$ is the forecast horizon. Its forecast error is $$ e_{T+h|T}=y_{T+h}-\hat{y}_{T+h|T}. $$ The error is calculated only after $y_{T+h}$ is observed.

The information set at origin $T$ consists of observations through $T$, together with any information that would genuinely have been available at that time. It does not include $y_{T+1}, y_{T+2}, \ldots$. A forecast that uses one of those later observations to estimate a level, trend, seasonal component, or smoothing parameter is not a genuine forecast from origin $T$.

An expanding-window evaluation increases the training sample as the origin moves forward: the first origin uses observations $1,\ldots,T_1$, the next uses $1,\ldots,T_2$, with $T_2>T_1$. A fixed-width rolling-window evaluation instead keeps the training window at a fixed length and moves both its beginning and end forward. Either approach can be appropriate. The choice should reflect the forecasting situation, and the candidate methods must be evaluated on the same origins and target times.

Scoring Forecasts at Comparable Origins

For a collection of $m$ forecast errors $e_1,\ldots,e_m$, two common summaries are the mean absolute error (MAE) and root mean squared error (RMSE):

\operatorname{MAE}=\frac{1}{m}\sum_{j=1}^{m}|e_j|, \qquad \operatorname{RMSE}=\sqrt{\frac{1}{m}\sum_{j=1}^{m}e_j^2}.

Both measures are expressed in the units of the observations. MAE gives the average absolute miss. RMSE gives extra weight to larger misses because it squares each error before averaging. Lower values indicate smaller errors on the evaluated forecasts, but do not by themselves establish that a method will perform best in every future period.

The errors must be paired with the forecasts that were made before their target observations became available. For a one-step-ahead comparison, every forecast should target the next observation after its origin. For a two-step-ahead comparison, every forecast should target the second observation after its origin. Comparing different horizons without separating them can obscure how a method behaves as the forecast lead increases.

Justification: At origin $T$, the forecast is a function of information available through $T$, while the realized value $y_{T+h}$ is not yet known. After it is observed, the difference $y_{T+h}-\hat{y}_{T+h|T}$ records the miss on a genuinely future target. Averaging a loss such as $|e_{T+h|T}|$ over successive origins therefore summarizes performance on the sequence of forecast tasks actually being evaluated. If a later observation is allowed into the training information for an earlier target, this logic fails: the score no longer measures forecasting from the stated origin.

One-Step-Ahead Evaluation Across Successive Origins

In a rolling-origin comparison, the forecast model is updated or refitted at each cutoff using only the data available by that cutoff. The resulting forecasts may come from simple exponential smoothing, Holt’s linear trend method, or another smoothing method. For this evaluation, what matters is that the methods forecast the same target observations under the same timing rules.

Worked Example: Comparing Two Smoothing Methods at Three Origins

A hypothetical portfolio team compares simple exponential smoothing (SES) with Holt’s linear trend method for a monthly claim-count index. The observed values at months 5, 6, and 7 are 108, 111, and 109. Each method has already been fitted using only information available at each listed origin. The table gives its one-step-ahead forecasts; calculate MAE and RMSE for each method.

Origin	Target month	Actual	SES forecast	Holt forecast
4	5	108	106	107
5	6	111	109	112
6	7	109	110	108

Method. Calculate each error as actual minus forecast, then calculate the absolute errors and squared errors. The three targets are the same for both methods, so the comparison is paired and time-aligned.

For SES, the errors are $108-106=2$, $111-109=2$, and $109-110=-1$. The absolute errors are $2,2,1$, with sum $5$, so $ \operatorname{MAE}=5/3=1.6667$ to four decimal places. The squared errors are $4,4,1$, with sum $9$, so $ \operatorname{RMSE}=\sqrt{9/3}=\sqrt{3}=1.7321$ to four decimal places.

For Holt’s method, the errors are $108-107=1$, $111-112=-1$, and $109-108=1$. The absolute errors sum to $1+1+1=3$, giving $ \operatorname{MAE}=3/3=1.0000$. The squared errors also sum to $1+1+1=3$, giving $ \operatorname{RMSE}=\sqrt{3/3}=1.0000$.

Check. Holt’s method has the smaller MAE and RMSE on these three one-step forecasts. This is a conclusion about this evaluation window, not a guarantee about all future months. The main mistake avoided is scoring one method on a different set of target months than the other.

Evaluate Each Forecast Horizon Deliberately

A method’s one-step-ahead errors need not describe its longer-horizon performance. Trend and seasonal components can have substantial effects on forecasts several periods ahead. When a decision depends on a specific lead time, calculate and report errors at that horizon. If several horizons matter, report them separately as well as, if appropriate, a clearly defined combined score.

A combined score should specify exactly which origin-horizon pairs it includes. For example, an average over all one-step and two-step forecasts gives equal weight to each included forecast, not necessarily equal weight to each origin or each horizon. Reporting horizon-specific results makes that weighting visible.

Worked Example: Scoring Two Horizons Separately and Together

A hypothetical insurer compares two smoothing procedures for a quarterly service-demand index. Actual values for quarters 5 through 8 are 50, 54, 53, and 57. Each row below is a forecast made before its target quarter was observed. Calculate the MAE for each method by horizon, and then calculate the combined MAE over all six forecasts.

Origin	Horizon	Target actual	Method A forecast	Method B forecast
4	1	50	48	51
5	1	54	52	55
6	1	53	55	52
4	2	54	51	55
5	2	53	54	52
6	2	57	56	58

Method. For horizon 1, the actual targets are quarters 5, 6, and 7. For horizon 2, they are quarters 6, 7, and 8. Keep these target sets distinct while calculating the horizon-specific scores.

For Method A at horizon 1, the errors are $50-48=2$, $54-52=2$, and $53-55=-2$. The absolute errors sum to $2+2+2=6$, so the MAE is $6/3=2$. At horizon 2, the errors are $54-51=3$, $53-54=-1$, and $57-56=1$. The absolute errors sum to $3+1+1=5$, so the MAE is $5/3=1.6667$.

For Method B at horizon 1, the errors are $50-51=-1$, $54-55=-1$, and $53-52=1$. The absolute errors sum to $3$, so the MAE is $3/3=1$. At horizon 2, the errors are $54-55=-1$, $53-52=1$, and $57-58=-1$. Again the absolute errors sum to $3$, so the MAE is $3/3=1$.

For the combined score, Method A has total absolute error $6+5=11$ across six forecasts, giving $11/6=1.8333$. Method B has total absolute error $3+3=6$, giving $6/6=1$. These combined values give equal weight to each of the six forecasts. In this example the horizon-specific results agree with the combined comparison: Method B has the lower MAE at both horizons.

Check. The target for the horizon-2 forecast from origin 4 is quarter 6, not quarter 5. Treating a two-step forecast as if it were one-step would pair it with the wrong actual and invalidate the score.

Why Random Shuffling Can Mislead

Random shuffling is often used when observations can reasonably be treated as exchangeable. A time series generally has a different structure: the order carries information, and the forecasting task is to predict later values from earlier ones. A random split can let a model learn from observations that occur after its test targets. Even if no target value is directly copied into a prediction, later data can influence estimated smoothing states, fitted parameters, or choices made during model development.

The following example isolates the timing problem with a simple level-only benchmark. It is not a claim that a constant level is the best model for a changing series; it shows why a shuffled test can answer a different question from a chronological forecast evaluation.

Worked Example: Detecting Future-Data Leakage in a Shuffled Split

A hypothetical quarterly index has values $10,12,14,16,18,20$ at times 1 through 6. A level-only benchmark forecasts every target with the mean of its training observations. Compare a chronological evaluation that trains on times 1 and 2 and tests on times 3 and 4 with a shuffled split that trains on times 1, 2, 5, and 6 and tests on times 3 and 4.

Chronological calculation. The training mean is $(10+12)/2=11$. The two forecasts are both 11. The errors are $14-11=3$ at time 3 and $16-11=5$ at time 4. Thus the MAE is $(3+5)/2=4$.

Shuffled calculation. The training mean is $(10+12+18+20)/4=60/4=15$. The two forecasts are both 15. The errors are $14-15=-1$ at time 3 and $16-15=1$ at time 4. Thus the MAE is $(1+1)/2=1$.

The shuffled split has a much smaller score here because observations at times 5 and 6, which occur after both test targets, helped determine its forecast level. That score does not describe what would have been forecast at times 2 or earlier. It rewards the split for access to future information, rather than showing that the method can predict forward more accurately.

Check. For a forecasting comparison, the training information for a target at time $t$ must not include observations after $t$. The common mistake is to interpret a lower score from a randomly shuffled split as evidence of better future forecasting, without checking whether later observations entered training.

A Practical Evaluation Sequence

A disciplined evaluation makes the timing and scoring choices explicit before methods are compared. The following sequence works for smoothing forecasts at one or several horizons.

Choose origins and horizons.
Specify the cutoffs and the lead times that match the intended forecast task. Each target must occur after its origin.

Limit information at each origin.
Fit or update each candidate using only observations and other information available by that cutoff. Keep any tuning decisions from using held-out future targets.

Record forecasts before scoring.
Store the origin, horizon, target time, and forecast for every method. This makes it possible to check that the comparisons are aligned.

Calculate forecast errors.
After each target is observed, subtract its forecast from the actual value. Keep signed errors if direction matters, and use absolute or squared errors for the stated summary.

Compare like with like.
Report scores on the same origins and targets. Separate horizons when their forecasting tasks differ, and describe the evaluation window when drawing a conclusion.

The forecasts themselves may arise from smoothing updates that use a level, trend, or seasonal component. Those components must be estimated using the training information at each origin. The evaluation does not require a particular probability distribution for the errors: MAE and RMSE are descriptive point-forecast scores. If the goal is to assess forecast uncertainty rather than point accuracy, interval coverage and width require separate evaluation.

Common Evaluation Traps

Scoring training fit instead of future forecasts. In-sample residuals measure how closely a fitted method follows observations it has already seen. They are not a substitute for errors on later observations.
Pairing a forecast with the wrong target. A forecast at origin $T$ for horizon 2 is scored against $y_{T+2}$, not $y_{T+1}$.
Comparing methods on different dates. A method can appear better simply because it was evaluated on easier targets. Use common origins and targets for a direct comparison.
Combining horizons without stating the weighting. A pooled MAE over all forecasts weights each included forecast equally. It does not necessarily give equal weight to each horizon.
Allowing future observations to tune the method. If a parameter or method choice is made after inspecting the evaluation targets, the reported score can be optimistic. Keep those targets separate from decisions made at the forecast origin.

Key takeaway: A forecast score is meaningful only in relation to when the forecast was made, what information was available then, which future value it targeted, and how errors were summarized. For time series, preserve chronology from training through evaluation.

Check Your Understanding

For each problem, use only the stated actuals and forecasts; assume each forecast was produced using information available at its stated origin.

A monthly series has actual values 42, 46, and 43 at three successive targets. One method forecasts 41, 44, and 45. Calculate its MAE and RMSE, to four decimal places. (Answer: MAE $=1.6667$; RMSE $=1.7321$.)
At two one-step targets, the actual values are 30 and 33. Method A forecasts 28 and 34; Method B forecasts 32 and 31. Calculate each MAE and RMSE and identify which has the lower MAE. (Answer: A has MAE $1.5$ and RMSE $1.5811$; B has MAE $2$ and RMSE $2$; A has the lower MAE.)
At three common one-step origins, actuals are 20, 19, and 23. Method A forecasts 18, 21, and 21; Method B forecasts 19, 20, and 22. Calculate both MAEs and identify the lower one. (Answer: A has MAE $2$; B has MAE $1$; B is lower.)
A validation split withholds times 3 and 4, but includes times 5 and 6 in training. Why is its score not a time-ordered forecast score for times 3 and 4? (Answer: Training uses observations from after both targets, so future information can influence the fitted forecast.)
Actuals at times 5 and 6 are 70 and 74. Two forecasts made at horizons 2 from earlier origins are 68 for time 5 and 71 for time 6. Calculate the MAE for this horizon. (Answer: Errors are $2$ and $3$, so MAE $=(2+3)/2=2.5$.)

What You'll Learn