Pseudo-Out-of-Sample Forecast Evaluation

Why Evaluate Forecasts Pseudo-Out-of-Sample?

A VAR can fit its estimation data well and still forecast poorly. A credible forecast comparison therefore asks what each model would have predicted using only information available at the time. In a pseudo-out-of-sample exercise, the analyst recreates this sequence of information sets using historical data: estimate the model at an origin, forecast a future value, advance the origin, and repeat.

This exercise is pseudo-out-of-sample because the historical outcomes are already known to the analyst, even though each forecast is constructed as if they were not yet observed. Its value depends on preserving that information boundary. Model choices, transformations, and variable definitions should be fixed in advance or selected using only information available at each forecast origin.

Definition: A forecast origin \(t\) is the last date whose observations are available when a forecast is made. For target variable \(y\), the \(h\)-step forecast is \(\hat{y}_{t+h\mid t}\), and its forecast error is \(e_{t+h\mid t}=y_{t+h}-\hat{y}_{t+h\mid t}\). The notation \(t+h\mid t\) identifies the target date and the information available at the origin.

Designing the Forecast Exercise

Suppose a VAR has been specified for a vector \(y_t\), and the goal is to forecast one component at horizon \(h\). Choose an initial estimation sample ending at \(T_0\), a final forecast origin \(T_1\), and a window rule. At each eligible origin \(t\), estimate the VAR using observations available by \(t\), then use the recursive VAR forecasting procedure to obtain \(\hat{y}_{t+h\mid t}\). The realized target \(y_{t+h}\) is used only afterward to calculate the error.

In an expanding window, the estimation sample starts at the same initial date and grows as the origin advances. In a rolling window, the estimation sample has a fixed length and moves forward, dropping its oldest observations as new ones arrive. Both approaches re-estimate the VAR at each origin; a rolling window is not simply a single fit to the most recent data.

Design choice	Expanding window	Rolling window
Observations used at origin \(t\)	All observations from the initial date through \(t\)	The most recent \(R\) observations through \(t\)
Sample size as origins advance	Increases	Remains fixed at \(R\)
Potential advantage	Uses a growing amount of historical data	Can adapt more quickly if older observations are less representative
Potential limitation	Older data may be less relevant after change	Fewer observations may make parameter estimates less precise

The comparison should use the same forecast origins, target dates, horizon, and outcome definition for every method. If the models require different starting sample sizes, begin the evaluation only when every model can produce a forecast, or report clearly why the evaluation samples differ. Do not use observations after \(t\) to estimate, select, or transform the model that generates the forecast at \(t\). A horizon-\(h\) evaluation has target dates \(t+h\); ensure each such outcome is available and counted once for its corresponding origin.

Worked Example: Setting Rolling and Expanding Windows

Suppose a hypothetical quarterly data set runs from 2000Q1 through 2020Q4. The first eligible origin is 2010Q4, the last is 2020Q3, the forecast horizon is one quarter, and the rolling-window length is 40 quarters. Describe the estimation samples for the first two origins under each rule.

At origin 2010Q4, an expanding-window VAR uses 2000Q1–2010Q4, or 44 quarters. At origin 2011Q1, it uses 2000Q1–2011Q1, or 45 quarters. A rolling-window VAR uses the most recent 40 quarters at each origin: 2001Q1–2010Q4 at the first origin, then 2001Q2–2011Q1 at the second.

Both designs forecast one quarter ahead. Their first target is 2011Q1 and their second target is 2011Q2. The later outcomes are used to score forecasts, not to estimate either VAR at the corresponding origin. To compare these designs fairly, retain the same origins and targets for both.

Summarizing Forecast Errors

For \(P\) matched forecast origins, a positive error means the forecast was below the realized value under the convention \(e_{t+h\mid t}=y_{t+h}-\hat{y}_{t+h\mid t}\). Two common summaries are root mean squared error (RMSE) and mean absolute error (MAE). Both are nonnegative and have the same units as the forecasted variable. RMSE gives relatively more weight to large errors; MAE is less sensitive to them.

\operatorname{RMSE} = \sqrt{\frac{1}{P}\sum_{t\in\mathcal{T}} e_{t+h\mid t}^{\,2}}, \qquad \operatorname{MAE} = \frac{1}{P}\sum_{t\in\mathcal{T}} |e_{t+h\mid t}|

Here \(\mathcal{T}\) is the set of evaluation origins and \(P\) is its size. Calculate these measures separately for each target variable and forecast horizon. For a VAR with several outcomes, a single combined score requires an explicit scaling and weighting rule; adding squared errors measured in unrelated units does not provide a meaningful comparison by itself.

Worked Example: Calculating RMSE and MAE

At four hypothetical one-quarter-ahead forecast origins, the actual quarterly growth rates are 1.0%, 1.5%, 0.5%, and 2.0%. A VAR forecasts 0.8%, 1.9%, 0.2%, and 1.6%. Calculate the forecast errors, RMSE, and MAE, expressing errors in percentage points.

Using actual minus forecast, the errors are \(0.2,-0.4,0.3,0.4\) percentage points. Thus,

\operatorname{RMSE} = \sqrt{\frac{0.2^2+(-0.4)^2+0.3^2+0.4^2}{4}} = \sqrt{\frac{0.45}{4}} = 0.3354 \text{ percentage points}.

For the absolute errors,

\operatorname{MAE} = \frac{|0.2|+|-0.4|+|0.3|+|0.4|}{4} = \frac{1.3}{4} = 0.3250 \text{ percentage points}.

These summaries describe errors at these four origins; neither alone establishes that the VAR forecasts better than another method. That requires a matched comparison and, if inference is wanted, an assessment of uncertainty in the difference in predictive performance.

Benchmarks and Matched Forecast Comparisons

A benchmark gives the forecast comparison a reference point. Examples include a no-change forecast, a mean forecast estimated from the available training sample, or a smaller VAR. Specify the benchmark before comparing performance, and produce its forecasts at the same origins and horizon as the candidate VAR. Comparing a VAR’s RMSE over one period with a benchmark’s RMSE over another confounds model performance with differences in evaluation dates.

For a chosen loss function \(L\), define the loss differential as benchmark loss minus VAR loss. With squared-error loss, for example, \(d_t=e_{b,t+h\mid t}^{\,2}-e_{v,t+h\mid t}^{\,2}\), where subscripts \(b\) and \(v\) identify the benchmark and VAR. A positive \(d_t\) means the VAR had lower squared error at that origin. The sample mean of the differentials estimates the difference in average loss over the evaluation period.

Definition: For matched forecasts, the null hypothesis of equal predictive accuracy under a specified loss is \(H_0:E(d_t)=0\), where \(d_t=L_{b,t}-L_{v,t}\). The two-sided alternative is \(H_a:E(d_t)\ne 0\). A positive average differential favors the VAR under this sign convention; a negative one favors the benchmark.

Forecast errors and loss differentials across origins need not be independent. In particular, multi-step forecast targets can overlap, and successive estimates use many of the same observations. A test that treats all differentials as independent can therefore understate or overstate uncertainty. A standard approach estimates the long-run variance of \(d_t\) with a heteroskedasticity-and-autocorrelation-consistent (HAC) estimator, then forms a Diebold–Mariano statistic. A bandwidth appropriate to the horizon and dependence pattern is needed; choosing it is part of the analysis, not a guarantee of exact finite-sample inference.

\bar d=\frac{1}{P}\sum_{t\in\mathcal{T}}d_t, \qquad DM=\frac{\bar d}{\sqrt{\widehat{\Omega}/P}}

Here \(\widehat{\Omega}\) estimates the long-run variance of the loss-differential sequence, including its serial covariances. Under the null and suitable regularity conditions, the statistic is asymptotically standard normal. Thus a small two-sided p-value is evidence against equal expected loss for this comparison; it is not the probability that the null is true. With short evaluation samples, the normal approximation may be poor, so the result should be interpreted cautiously.

Testing a Forecast Comparison

The following calculation shows how the dependence adjustment enters. It is deliberately small to make the arithmetic visible; eight origins are generally too few for dependable asymptotic inference. Consider a one-step-ahead comparison using squared-error loss. Let \(d_t\) be benchmark loss minus VAR loss at each of eight hypothetical origins.

Worked Example: A HAC Test of Equal Predictive Accuracy

Suppose the loss differentials, in squared units of the outcome, are \(0,0,0,0,1,1,1,1\). Test \(H_0:E(d_t)=0\) against \(H_a:E(d_t)\ne0\). For illustration, use a Bartlett HAC estimator with lag \(q=1\), and the convention \(\hat{\gamma}_k=P^{-1}\sum_{t=k+1}^{P}(d_t-\bar d)(d_{t-k}-\bar d)\).

The average differential is \(\bar d=4/8=0.5\). The centered values are four \(-0.5\) values followed by four \(0.5\) values. Therefore, \(\hat{\gamma}_0=8(0.5^2)/8=0.25\). At lag one, three adjacent pairs within each block have product \(0.25\), and the pair crossing the blocks has product \(-0.25\). Hence \(\hat{\gamma}_1=(6(0.25)-0.25)/8=0.15625\).

With the Bartlett weight \(1-1/(q+1)=1/2\), the lag-one estimate is

\widehat{\Omega} = \hat{\gamma}_0+ 2\left(\frac{1}{2}\right)\hat{\gamma}_1 = 0.25+0.15625 = 0.40625.

Thus the HAC standard error of the mean differential is \(\sqrt{0.40625/8}=0.2254\), and \(DM=0.5/0.2254=2.219\). The asymptotic two-sided normal p-value is about 0.0265. The positive mean favors the VAR, and this illustrative asymptotic calculation would reject equal predictive accuracy at the 5% level. Because the evaluation has only eight origins, that p-value should not be treated as reliable evidence in an actual application.

When the Benchmark Is Nested

A further issue arises when the benchmark is nested within the VAR being evaluated. For example, a smaller model may be obtained by setting some coefficients of a larger model to zero. Even if the larger model has no genuine forecasting advantage, estimating its additional parameters can make its raw squared errors appear worse on average. The ordinary equal-loss comparison may then be biased against the larger model under the null.

For nested forecasts and squared-error loss, the Clark–West adjustment is commonly used. If model \(b\) is the restricted benchmark and model \(v\) is the larger model, define the adjusted differential as \(d^{CW}_t=e_{b,t+h\mid t}^2-\{e_{v,t+h\mid t}^2-(\hat{y}_{t+h\mid t,v}-\hat{y}_{t+h\mid t,b})^2\}\). A positive mean favors the larger model. Test it with dependence-robust standard errors when the forecast design induces serial dependence. This adjustment is specific to the nested-model, squared-error setting; it is not a universal correction for every source of estimation uncertainty.

Worked Example: Applying the Nested-Model Adjustment

At one hypothetical origin, the benchmark forecast is 10 and the larger VAR forecast is 11. The realized value is 12. The benchmark error is \(12-10=2\), while the VAR error is \(12-11=1\). The raw squared-loss differential is \(2^2-1^2=3\), favoring the VAR.

The forecast difference is \(11-10=1\), so the Clark–West adjusted differential is \(2^2-\{1^2-1^2\}=4\). This origin contributes 4 to the adjusted comparison. It does not by itself establish an advantage: calculate adjusted differentials over all matched origins, average them, and test their mean using a suitable dependence-robust standard error. The adjustment addresses the nested-model comparison, while the HAC calculation addresses serial dependence; they serve distinct roles.

Common Pitfalls and a Practical Sequence

A lower RMSE or MAE is a descriptive result for the chosen sample, target, horizon, loss, and benchmark. It does not guarantee better forecasts in a future period, nor does a forecast comparison establish a causal relationship. A fair evaluation requires a clearly stated protocol as well as correct arithmetic.

Fix the target and horizon.
Name the forecasted variable, its units, and the lead time \(h\).

Set origins and window rules.
Record the initial sample, evaluation period, and whether estimation expands or rolls.

Generate matched forecasts.
At each origin, estimate using only available observations and forecast every method for the same target date.

Score and compare.
Report RMSE and MAE, state the benchmark and loss, and use an appropriate predictive-accuracy test if inference is needed.

Using future information. A forecast at \(t\) is not genuinely pseudo-out-of-sample if estimation or model selection uses data after \(t\). Recreate the information set at every origin.
Comparing unmatched samples. Different origins or horizons can change losses independently of the models. Use common targets and dates wherever possible.
Treating origins as independent by default. Repeated estimation samples and overlapping multi-step forecasts can create dependent loss differentials. Use a suitable HAC variance estimate and acknowledge finite-sample limitations.
Choosing a nested model comparison without adjustment. A larger nested model can be penalized by parameter-estimation noise in a raw squared-error comparison. Consider the Clark–West adjustment in its appropriate setting.
Calling a lower score proof of general superiority. The score summarizes one evaluation design. State the sample, horizon, target, benchmark, and loss so readers can judge what the result supports.

Key takeaway: Forecast evaluation is a sequence of matched, historically feasible forecasts—not merely a comparison of fitted values. Re-estimate at each origin using the declared rolling or expanding window, score identical targets, and account for dependence when testing differences in predictive accuracy.

Check Your Understanding

Answer these questions using the definitions and procedures above.

At a forecast origin \(t\), which observations may be used to estimate an expanding-window VAR, and how does that set change at \(t+1\)?
Why can two models’ RMSE values not be fairly compared if they were calculated over different forecast origins?
A benchmark loss is 5 and a VAR loss is 7 at one origin. What is the loss differential under the benchmark-minus-VAR convention, and which forecast has lower loss there?
Why can overlapping multi-step forecasts make a standard error based on independent differentials inappropriate?
What problem does the Clark–West adjustment address, and why is it separate from using a HAC standard error?