Economic Series Forecasting with ARMA and a Holdout

Why a Holdout Changes the Forecasting Question

A fitted time-series model describes patterns in observations already available. Forecasting asks a different question: how well would a procedure predict values it had not yet seen? A holdout sample provides a direct, if necessarily limited, check. Its usefulness depends on preserving the time order: the model is selected and estimated using earlier months, and the later months are reserved for evaluation.

The examples use hypothetical monthly observations of an economic series measured in percentage points. The numbers are illustrative rather than real study results. The analysis assumes the series is suitable for a stationary ARMA model over the period considered. If the observed behavior instead indicates a trend or other nonstationarity, a stationary ARMA forecast should not be treated as adequate merely because it can be fitted.

Set Up the Forecasting Exercise

The training and holdout periods must be fixed before examining holdout outcomes. Candidate models, transformations, and tuning decisions are made using training data alone. Once the specification has been chosen, it is estimated on that training sample and used to forecast forward. Looking at holdout values to decide which model to report would make the holdout partly part of model selection and would generally give an overly favorable impression of performance.

An ARMA(\(p,q\)) model for a stationary series can be written as

Here \(\mu\) is the series mean, \(\varepsilon_t\) is a mean-zero innovation with variance \(\sigma^2\), and the autoregressive and moving-average terms describe dependence over time. For the usual Gaussian prediction intervals, assume the innovations are independent and normally distributed, and that the fitted model is an adequate representation over the forecast period. Stable, invertible specifications and residual diagnostics matter: a low information criterion does not by itself establish that the model is suitable.

A practical selection procedure compares a prespecified set of plausible candidate orders on the same training observations. AICc, for example, is AIC with a small-sample correction, \(AICc=AIC+2k(k+1)/(n-k-1)\), where \(k\) is the number of estimated parameters and \(n\) is the estimation sample size. Smaller values favor a better penalized fit among the candidates, but should be considered alongside residual autocorrelation and innovation checks. Those diagnostics should not show a systematic pattern left unexplained by the model.

Forecasts and Prediction Intervals

Forecasts are conditional expectations under the fitted model. For an AR(1) model with mean \(\mu\), the forecast recursively approaches \(\mu\) as the horizon grows. Prediction intervals describe uncertainty in a future observation, not uncertainty about the unknown mean alone. In a plug-in interval, estimated parameters are treated as fixed; such intervals therefore do not fully account for parameter-estimation uncertainty.

Evaluate Holdout Forecasts at Their Horizons

After the holdout outcomes become available, compute each forecast error as actual minus forecast. The sign indicates direction: a positive error means the forecast was too low, and a negative error means it was too high. Common summaries include mean absolute error (MAE) and root mean squared error (RMSE). For \(m\) forecast errors at a specified horizon, \(MAE=m^{-1}\sum |e_i|\) and \(RMSE=\sqrt{m^{-1}\sum e_i^2}\). Both are in the units of the series; RMSE gives relatively more weight to larger errors.

Accuracy should be reported by forecast horizon when the practical question distinguishes, for example, next-month forecasts from forecasts three months ahead. If there is just one holdout origin, there is only one error at each horizon. A collection of errors pooled across different horizons is not the same thing as an estimate of accuracy at any one horizon. Multiple forecast origins allow repeated errors to be summarized separately at each horizon.

Common Pitfalls and Reporting

A forecast evaluation is only as informative as its design. The training period should be long enough to estimate the candidate models, while the holdout should represent the future period the user cares about. If the economic process may change substantially over time, a single old holdout may be a poor guide to current performance. In that situation, rolling-origin evaluation can provide a more informative record, while still ensuring each forecast uses only information available at its origin.

• Using the holdout to select the winner. This turns evaluation data into training information. Choose the model on training data, then report its holdout performance, or explicitly describe a separate validation scheme.
• Reporting only an overall error. A pooled MAE or RMSE can hide that performance varies with the forecast horizon. Report horizon-specific measures when horizons matter.
• Treating a wide prediction interval as a poor point forecast. Point accuracy and uncertainty are different summaries. Assess forecast errors and interval coverage or width for their distinct purposes.
• Reading a diagnostic p-value as proof. A nonsignificant residual test does not establish that every assumption holds; it only fails to find evidence against the tested feature at the available sample size and lags.
• Changing the analysis after seeing the outcome. Any such change should be disclosed, since the reported performance is no longer an untouched holdout assessment.