Uncertainty in Forecast Model Comparisons

Why Forecast Comparisons Need Uncertainty

Two forecasting models can be evaluated on the same sequence of future observations. Their average scores may differ, but that difference is itself uncertain: it depends on which observations happened to occur during evaluation. A small score advantage does not establish that one model will generally forecast better.

The comparison should be paired. At each forecast origin, score both models against the same target, using the same forecast horizon and the same loss function. This pairing recognizes that both models face the same realized outcome and often respond to the same underlying time-series shocks.

For squared-error loss, \(L(e)=e^2\); for absolute-error loss, \(L(e)=|e|\). The comparison concerns expected loss under the chosen scoring rule, not whether the models’ forecasts are identical. A conclusion based on squared-error loss need not match one based on absolute-error loss, because the two rules weight forecast errors differently.

Why Forecast Errors and Scores Are Dependent

At a given origin, errors from two models are generally related because both forecasts are made before the same outcome is observed. One model may be more responsive to a shared shock than the other, but the outcome itself is not separately generated for each model. Across origins, dependence can arise as well. In particular, successive \(h\)-step forecasts can cover overlapping periods: the forecast made at origin \(t\) targets \(Y_{t+h}\), while the next targets \(Y_{t+h+1}\). Their forecast errors may therefore share future shocks.

Even if the model-specific errors at one origin have variances \(\operatorname{Var}(L_{A,t})\) and \(\operatorname{Var}(L_{B,t})\), the variance of their difference also depends on their covariance. For any two random variables with finite variances,

The same principle applies over time: the uncertainty in the average differential depends on the serial autocovariances of the differentials. Thus, the number of evaluated forecasts alone is not enough to calculate a valid standard error.

Deriving the Uncertainty of the Mean Differential

Let \(d_1,\ldots,d_T\) be the observed loss differentials from \(T\) evaluation origins. Suppose the differential process is stationary with finite second moments, and write \(\gamma_k=\operatorname{Cov}(d_t,d_{t-k})\), with \(\gamma_0=\operatorname{Var}(d_t)\). The sample mean is \(\bar d=T^{-1}\sum_{t=1}^T d_t\).

To see why autocovariances enter its variance, expand the variance of the sum. There are \(T\) terms with lag zero. For each lag \(k>0\), there are \(T-k\) pairs in each direction, and stationarity gives the same covariance \(\gamma_k\) for each pair.

This identity explains why the independent-observation standard error, based only on \(\gamma_0\), can be wrong: it omits the terms involving \(\gamma_1,\gamma_2,\ldots\). Under standard weak-dependence conditions, the large-sample variance is approximated using the long-run variance \(\Omega=\gamma_0+2\sum_{k=1}^{\infty}\gamma_k\), so that \(\operatorname{Var}(\bar d)\) is approximately \(\Omega/T\).

In practice, the long-run variance is estimated from the observed differentials using a heteroskedasticity-and-autocorrelation-consistent (HAC) estimator. The estimator combines the sample variance with weighted estimates of autocovariances. A bandwidth and weighting rule must be chosen; for overlapping \(h\)-step forecast errors, dependence through at least the first \(h-1\) lags is a standard concern. More persistent dependence may call for additional lags. The appropriate choice depends on the forecast setup and sample size.

The Diebold–Mariano Comparison at a Specified Horizon

The null hypothesis is that the two forecasts have equal expected loss at the chosen horizon: \(H_0:E[d_t]=0\). The alternative may be two-sided, \(E[d_t]\ne0\), or one-sided if a direction was specified in advance. With a consistent estimate \(\widehat{\Omega}\) of the long-run variance and suitable stationarity, weak-dependence, and finite-moment conditions, the Diebold–Mariano statistic has an approximate standard normal reference distribution under the null.

A positive statistic points toward lower expected loss for model B under the sign convention above; a negative statistic points toward model A. The sign indicates direction, but the magnitude relative to its estimated standard error indicates how much statistical evidence the data provide. The normal approximation can be unreliable in short samples, especially when multi-step forecasts overlap. A test result should therefore be interpreted with attention to sample size and dependence, not as a mechanical verdict.

Horizon, Practical Importance, and Common Traps

Forecast accuracy is horizon-specific. A comparison at horizon \(h=1\) uses forecasts of \(Y_{t+1}\); one at \(h=3\) uses forecasts of \(Y_{t+3}\). Each model pair must be evaluated on the same target dates at that horizon, with the same loss function. An average across horizons can conceal different performance patterns and is not the same test as a comparison at one specified horizon.

A statistically detectable difference may be too small to matter for a decision, while a potentially meaningful difference may be estimated too imprecisely to establish. Report the estimated average loss difference and its uncertainty, and relate the size of the difference to the units and consequences of the forecasting task. A failure to reject equal expected loss is not proof that the models are identical.

The standard test compares a specified pair of forecasting procedures on the evaluation differentials. It does not automatically turn a small difference into a robust finding, nor does it remove every uncertainty created by the evaluation design. In particular, the calculated uncertainty is only as credible as the pairing, horizon alignment, and dependence estimate.

Another frequent error is reversing the sign interpretation. With \(d_t=L_{A,t}-L_{B,t}\), a positive mean favors B, not A. State the differential definition before reporting a test statistic, and keep that convention through the calculation and conclusion.