Diebold-Mariano Tests for Forecast Accuracy

Why Compare Forecasts With a Test?

Forecast accuracy measures summarize how well forecasts perform, but differences in a finite evaluation sample do not by themselves establish that one forecasting method has lower expected loss. The Diebold–Mariano (DM) test turns a sequence of paired forecast losses into a test of whether the forecasts have equal expected loss. Pairing matters: both forecasts must be evaluated against the same realized outcomes at the same forecast origins and horizon.

The central quantity is the loss differential, the loss from one forecast minus the loss from the other. Its average describes the observed performance difference. Because consecutive loss differentials can be dependent—especially when forecasts have overlapping horizons—the uncertainty in that average must be based on a long-run variance, not automatically on an independent-observation standard error.

Definition: Let $y_t$ be the outcome observed at time $t$, and let $\hat{y}_{1,t|t-h}$ and $\hat{y}_{2,t|t-h}$ be two forecasts of that outcome made $h$ periods earlier. For a specified loss function $L(y,\hat{y})$, define $$ d_t=L(y_t,\hat{y}_{1,t|t-h})-L(y_t,\hat{y}_{2,t|t-h}). $$ The null hypothesis is $H_0:E(d_t)=0$, meaning equal expected loss. With this sign convention, $E(d_t)<0$ favors forecast 1, and $E(d_t)>0$ favors forecast 2.

The loss function must be chosen for the forecasting objective before examining which forecast it favors. Squared-error loss is $L(y,\hat{y})=(y-\hat{y})^2$; absolute-error loss is $L(y,\hat{y})=|y-\hat{y}|$. Other losses may be appropriate when overprediction and underprediction have different consequences. A DM test compares expected values of the chosen loss, not every possible notion of forecast quality.

The Test Statistic and Its Assumptions

Suppose there are $n$ paired evaluation observations, indexed by $t=1,\ldots,n$. Write $\bar{d}=n^{-1}\sum_{t=1}^n d_t$. If the loss-differential process is stationary and weakly dependent, has suitable finite moments, and satisfies a central limit theorem, then its sample mean is asymptotically normal:

\sqrt{n}(\bar{d}-E(d_t))\ \xrightarrow{d}\ N(0,\Omega), \qquad \Omega=\sum_{k=-\infty}^{\infty}\gamma_k =\gamma_0+2\sum_{k=1}^{\infty}\gamma_k,

where $\gamma_k=\operatorname{Cov}(d_t,d_{t-k})$, and the equality uses the symmetry $\gamma_{-k}=\gamma_k$. The quantity $\Omega$ is the long-run variance: it incorporates both the variance of an individual differential and its serial covariances. It must be positive for the usual normal approximation to be useful.

A common estimate is the Bartlett-kernel heteroskedasticity-and-autocorrelation-consistent (HAC) estimate. For a chosen maximum lag $q$, estimate $\gamma_k$ by $\hat{\gamma}_k=n^{-1}\sum_{t=k+1}^{n}(d_t-\bar{d})(d_{t-k}-\bar{d})$, and set:

\widehat{\Omega} =\hat{\gamma}_0+ 2\sum_{k=1}^{q}\left(1-\frac{k}{q+1}\right)\hat{\gamma}_k, \qquad DM=\frac{\bar{d}}{\sqrt{\widehat{\Omega}/n}}.

Under the null and the stated regularity conditions, $DM$ is approximately standard normal in large samples. A two-sided test uses $H_a:E(d_t)\ne0$; a directional test uses $H_a:E(d_t)<0$ or $H_a:E(d_t)>0$, as specified in advance. The p-value is calculated under the null distribution and measures how unusual a statistic at least as extreme as the observed one would be if the null held. It is not the probability that the null is true.

Dependence and Forecast Horizon

For forecasts $h$ steps ahead, evaluation outcomes and losses can overlap across adjacent origins. For example, two successive two-step forecasts share part of the period over which their forecasts are assessed. That overlap can induce serial correlation in $d_t$, so treating the differentials as independent can understate or overstate uncertainty.

A conventional horizon-based choice for overlapping $h$-step forecasts is a Bartlett HAC estimate with $q=h-1$. This includes the lags associated with the overlap and downweights higher-lag autocovariances. It is a useful benchmark, not a guarantee that all dependence ends at lag $h-1$: rolling estimation, persistent data, or other features may leave additional dependence. The dependence structure and sample size should inform the bandwidth, and sensitivity to reasonable bandwidth choices can be reported. For a one-step forecast, $q=0$ is appropriate only when the differentials can reasonably be treated as serially uncorrelated; the one-step horizon alone does not establish independence.

Worked Example: A Two-Sided Test for One-Step Loss Differences

Suppose two hypothetical one-step forecasting procedures are evaluated on six common origins using squared-error losses. The observed differentials, in squared outcome units, are $0.4,-0.2,0.1,0.5,-0.1,0.3$. Assume these one-step differentials are uncorrelated and that the large-sample normal approximation is being used. Test equal expected loss against unequal expected loss at $\alpha=0.05$.

The sample mean is $\bar{d}=(0.4-0.2+0.1+0.5-0.1+0.3)/6=1.0/6=0.1667$ squared outcome units. Using the stated uncorrelated-differential assumption, estimate the long-run variance by $\hat{\gamma}_0$, with the divisor $n$:

\hat{\gamma}_0 =\frac{1}{6}\sum_{t=1}^{6}(d_t-\bar{d})^2 =\frac{0.393333}{6} =0.0655556.

Thus $SE(\bar{d})=\sqrt{0.0655556/6}=0.1045$ squared outcome units, and $DM=0.1667/0.1045=1.595$. The two-sided normal p-value is approximately $0.111$, which exceeds $0.05$. The test does not reject equal expected loss. The positive observed mean favors forecast 2 in this sample, but the test does not provide sufficient evidence at the 5% level that their expected losses differ.

Worked Example: Correcting for Overlapping Two-Step Forecasts

For a hypothetical evaluation of 16 paired two-step forecasts using squared-error loss, suppose the computed differentials have mean $\bar{d}=0.25$ squared outcome units. Their centered autocovariance estimates, calculated with divisor $n$, are $\hat{\gamma}_0=0.984375$ and $\hat{\gamma}_1=-0.111328125$. Use the conventional $q=h-1=1$ Bartlett correction and test equal expected loss against unequal expected loss.

At lag 1 the Bartlett weight is $1-1/(1+1)=0.5$. Retaining full precision in the substitution gives:

\widehat{\Omega} =0.984375+2(0.5)(-0.111328125) =0.873046875 \approx 0.8730.

The HAC standard error is $\sqrt{0.873046875/16}=0.2336$ squared outcome units, so $DM=0.25/0.2336=1.070$. The two-sided standard-normal p-value is approximately $0.284$. There is not sufficient evidence at the 5% level to reject equal expected loss. The positive mean is in the direction favoring forecast 2, but its size relative to the dependence-adjusted uncertainty is not large enough to establish a difference.

This calculation illustrates why displayed precision matters: substituting the displayed autocovariances must reproduce the displayed long-run variance. Rounding intermediate values can create an apparent arithmetic discrepancy even when a result computed from unrounded values is correct.

A Small-Sample Adjustment

The normal approximation can be rough when the evaluation sample is short. A commonly reported refinement is the Harvey–Leybourne–Newbold (HLN) adjustment. For sample size $n$ and forecast horizon $h$, multiply the DM statistic by the following factor and compare the result with a $t$ distribution having $n-1$ degrees of freedom:

DM_{\mathrm{HLN}} =DM\sqrt{\frac{n+1-2h+h(h-1)/n}{n}}.

This correction is a finite-sample refinement under its intended conditions; it is not a replacement for an appropriate long-run variance estimate, nor does it make an invalid dependence assumption valid. As with any reference distribution, its use should be stated clearly so readers can distinguish it from the asymptotic standard-normal DM test.

Worked Example: A Directional Test With Absolute Loss

Consider six hypothetical paired forecasts evaluated with absolute-error loss. The differentials $d_t=|e_{1,t}|-|e_{2,t}|$, in outcome units, are $-0.6,-0.2,0.1,-0.4,0,0.1$. Assume the differentials are uncorrelated, use the large-sample normal reference distribution, and test at $\alpha=0.05$ whether forecast 1 has lower expected absolute loss: $H_0:E(d_t)=0$ against $H_a:E(d_t)<0$.

Their sum is $-1.0$, so $\bar{d}=-1.0/6=-0.1667$ outcome units. The centered squared deviations sum to $0.413333$, giving $\hat{\gamma}_0=0.413333/6=0.0688889$ and $SE(\bar{d})=\sqrt{0.0688889/6}=0.1072$ outcome units. Therefore $DM=-0.1667/0.1072=-1.556$.

The lower-tail standard-normal p-value is approximately $0.060$, just above $0.05$. The test does not reject equal expected loss at the 5% level. The negative sample mean favors forecast 1 descriptively, but it is not enough to conclude that forecast 1 has lower expected absolute loss at this significance level.

Interpretation and Common Pitfalls

The DM test addresses equality of expected loss for a particular loss function, forecast horizon, evaluation period, and forecasting setup. A failure to reject is not proof that the forecasts are equally accurate; it may reflect limited precision or a small evaluation sample. Conversely, rejection does not show that one forecast is better under every loss function or horizon. Report the sample mean loss differential alongside the statistic, p-value, loss definition, horizon, and variance correction so that statistical evidence remains connected to practical forecast performance.

A common mistake is to calculate the standard error of $\bar{d}$ as though the $d_t$ values were independent, even when multi-step forecast losses overlap. The reasoning “there are $n$ origins, so the usual independent-observation standard error applies” ignores serial covariance. The correction is to estimate the long-run variance with a horizon-appropriate HAC method, explain the bandwidth choice, and check whether the evaluation design supports the assumptions behind the test. The same paired origins and realized outcomes must be used for both forecasts; otherwise the differential no longer isolates their paired loss contrast.

Key takeaway: The DM test compares the mean paired loss differential with its dependence-adjusted standard error. The sign identifies which forecast has lower observed loss, while the p-value quantifies evidence against equal expected loss under the specified assumptions and test procedure.

Check Your Understanding

Answer each question using the definitions and assumptions in this tutorial.

If $d_t=L_{1,t}-L_{2,t}$, what does a negative sample mean indicate about the observed losses?
Why can overlapping two-step forecast losses make an independent-observation standard error inappropriate?
For a four-step forecast, what is the conventional Bartlett HAC lag choice associated with horizon overlap, and why might additional lags still merit consideration?
In a two-sided test, what does a p-value quantify, and what does it not quantify?
Why is failing to reject equal expected loss not the same as proving the forecasts have equal expected loss?