Forecast Accuracy Measures Across Horizons

Why Forecast Accuracy Must Be Indexed by Horizon

A forecast made one period ahead and a forecast made several periods ahead answer different predictive questions. Their errors may have different magnitudes and patterns, so a single accuracy score can conceal important differences. Forecast accuracy measures summarize errors, but they are meaningful only when the forecasts and outcomes being summarized are properly matched.

For an observed series \(y_t\), let \(\hat{y}_{t\mid t-h}\) denote the forecast of \(y_t\) made at time \(t-h\). Its forecast horizon is \(h\): the number of periods between the forecast origin and the outcome. The horizon-specific evaluation uses forecasts and realized outcomes for the same target variable and the same \(h\).

This sign convention is useful for interpreting average errors. Squared and absolute errors do not depend on the sign convention, but the error itself and its average do. Forecast records from rolling-origin evaluation can be grouped by horizon and summarized using the definitions below. No probability distribution for the errors is needed to calculate these descriptive summaries. Their interpretation does require the forecasts to be paired with the correct realized outcomes and horizons.

Three Core Accuracy Measures

For the \(m_h\) errors available at horizon \(h\), the mean squared error (MSE) averages squared errors, the root mean squared error (RMSE) takes the square root of that average, and the mean absolute error (MAE) averages absolute errors.

MSE is expressed in squared units of the target, while RMSE and MAE are expressed in the target's original units. For example, if the target is a price measured in dollars, MSE is in dollars squared, and RMSE and MAE are in dollars. RMSE gives relatively greater influence to large errors because squaring magnifies their contribution. MAE increases in direct proportion to each error's absolute size. Neither measure indicates whether forecasts tend systematically to be too high or too low; the signed errors or their average are needed for that.

Summarize Each Horizon Separately

The subscript \(h\) in each measure matters. An evaluation should calculate the errors, MSE, RMSE, and MAE separately for each horizon of interest. A table by horizon shows whether a model's accuracy changes as the forecast lead increases, rather than hiding that pattern in a pooled number.

For example, suppose a hypothetical four-origin evaluation produces the following errors in units of an index:

What MSE Adds: A Decomposition

MSE reflects both the average signed error and how much errors vary around that average. To make this precise for one horizon, write the errors as \(e_1,\ldots,e_m\), where \(m=m_h\), and define their average as \(\bar e=m^{-1}\sum_{i=1}^{m}e_i\). Define their mean squared dispersion around that average as \(v=m^{-1}\sum_{i=1}^{m}(e_i-\bar e)^2\). Then MSE decomposes exactly as follows.

To verify the identity, write each error as \(e_i=(e_i-\bar e)+\bar e\), square, and average:

The middle term is zero because \(\sum_{i=1}^{m}(e_i-\bar e)=\sum e_i-m\bar e=0\). Thus a large MSE can reflect a persistent average underprediction or overprediction, substantial variation in errors, or both. This is an exact description of the observed errors, not a claim about the population distribution of future forecast errors.

Scale, Units, and Fair Comparisons

Accuracy measures expressed in the target's units depend on the target's scale. If all values and forecasts are converted by multiplying by a positive constant \(c\), each error becomes \(c e_{o,h}\). Consequently, MAE and RMSE are multiplied by \(c\), while MSE is multiplied by \(c^2\). If a series is recorded in thousands rather than single units, the numerical MAE and RMSE are one-thousandth as large, even though the forecasts have not become more accurate in a substantive sense.

Even when units match, raw accuracy scores can be misleading across different target series. An MAE of 2 dollars for a low-priced item and an MAE of 2 dollars for a high-priced item have the same units, but not necessarily the same importance relative to the typical values of those targets. Scores for different variables—for example, temperature and sales—are not directly comparable merely because both use MAE. A fair raw-score comparison ordinarily uses the same target, units, horizon, and evaluation period, with outcomes and forecasts matched consistently.

If different targets or scales must be compared, a scale-adjusted measure may be considered, but that changes what is being summarized and requires an explicitly chosen reference scale. A score should not be called “better” solely because its raw numerical value is smaller when the targets have different scales or units.

Practical Checks Before Reporting

Accuracy summaries are only as useful as the comparisons behind them. Before interpreting a score, check that the forecast and outcome refer to the same target and date, that each horizon is defined consistently, and that missing forecasts have not silently changed the evaluated set. When comparing two forecasting methods, use the same origins and realized outcomes at each horizon where possible. Otherwise, a score difference may reflect different evaluation cases as well as different forecast performance.

Report the number of evaluated errors alongside the measures, especially if it differs by horizon. If \(m_h\) is smaller for one horizon because fewer outcomes have become available, that score is based on a different amount of information. Also report units and enough context to identify the target and horizon. These measures describe realized forecast errors; their calculation alone does not establish whether an observed difference between forecasting methods is statistically distinguishable from chance.