How ARMA Order Selection Changes Forecasts

Why Order Selection Changes Forecasts

An ARMA forecast depends on more than the observations available at the forecast origin. It also depends on the chosen autoregressive and moving-average orders, the estimated parameters, and the observations included in estimation. Consequently, two forecasting procedures applied to the same series can produce different forecasts even when both use a standard information criterion.

The key distinction is whether the order is fixed across forecast origins or selected again as the data window changes. A fixed order makes the structure of the forecasting model consistent over time, while repeated selection allows the fitted structure to respond to the current sample. Neither strategy is automatically superior. Their forecast accuracy must be compared out of sample, using the same forecast origins, horizons, and loss measure.

Definition: A fixed-order forecasting strategy chooses an ARMA order \((p,q)\) using a prespecified training sample and keeps that order at later forecast origins, while re-estimating its parameters as specified. A repeatedly selected strategy applies an order-selection rule to the available training data at each origin, then estimates and forecasts using the order selected at that origin.

For an ARMA\((p,q)\) model, \(p\) and \(q\) are the autoregressive and moving-average orders. At origin \(t\), a one-step-ahead forecast is formed using observations available through time \(t\), and its error is \(e_{t+1}=y_{t+1}-\hat{y}_{t+1\mid t}\). Here \(y_{t+1}\) is the subsequently observed value and \(\hat{y}_{t+1\mid t}\) is the forecast made at \(t\). The sign convention means a positive error indicates underprediction.

In both strategies, parameters are usually re-estimated at each origin using that origin’s estimation window. Thus, “fixed order” does not mean “fixed coefficients.” The comparison concerns whether the orders are held constant or allowed to change. An order-selection criterion such as AICc or BIC is calculated from the training sample; it must not use the future observation whose forecast is being evaluated.

Making the Comparison Fair

A useful evaluation starts by defining both procedures precisely. For example, one might choose a fixed order using the first training window, then retain that order while re-estimating its parameters at every later origin. The repeated-selection strategy might minimize AICc over the same prespecified candidate set at each origin. Both procedures should use the same deterministic terms, estimation method, candidate models where relevant, and available information. The selected models must satisfy the constraints required for the ARMA estimation and forecasting procedure being used.

The forecasts should then be evaluated on common origins and at the same horizon. If one strategy is evaluated at origins where the other has no forecast, a direct comparison of average losses may be misleading. For one-step forecasts, a common summary is the root mean squared forecast error:

\operatorname{RMSE}=\sqrt{\frac{1}{T}\sum_{t=1}^{T} e_{t+1}^{\,2}},

where \(T\) is the number of common forecast origins. A lower RMSE means smaller squared forecast errors on those origins; it does not establish that a strategy will be more accurate in every future sample. Other loss measures, such as mean absolute error, answer a different question and can rank strategies differently.

Worked Example: Comparing Fixed and Repeated Order Selection

Suppose a hypothetical forecasting exercise evaluates four common one-step forecast origins. An initial training sample selected ARMA\((1,0)\) by AICc, so that order is retained in the fixed-order procedure. At each origin, the repeated-selection procedure chooses the lowest-AICc model from the same candidate set. The selected models and forecasts are:

Origin	Repeated procedure’s selected order	Selected AICc	Fixed forecast	Repeated-selection forecast	Observed value
1	ARMA(1,0)	48.2	11	11	12
2	ARMA(1,1)	51.0	11	8	10
3	ARMA(2,0)	49.6	12	15	13
4	ARMA(0,1)	50.1	12	10	11

The AICc values are shown for the model selected at each origin; each is the minimum within that origin’s candidate set. The fixed-order errors, computed as observed minus forecast, are \(1,-1,1,-1\). Therefore,

\operatorname{RMSE}_{\mathrm{fixed}} =\sqrt{\frac{1^2+(-1)^2+1^2+(-1)^2}{4}} =\sqrt{1} =1.

The repeated-selection errors are \(1,2,-2,1\), giving

\operatorname{RMSE}_{\mathrm{reselected}} =\sqrt{\frac{1^2+2^2+(-2)^2+1^2}{4}} =\sqrt{\frac{10}{4}} \approx 1.581.

On these four origins, the fixed-order procedure has the lower RMSE: 1.000 versus 1.581 units of the series. This small hypothetical comparison does not show that fixed orders are generally better. It shows that selecting the best in-sample criterion at each origin need not select the procedure with the lowest subsequent forecast error.

Selection Uncertainty and In-Sample Fit

Repeated selection makes the chosen order a feature of the realized training sample. If a different sample would select a different order, then the forecast has an additional source of variation: uncertainty about which model structure the selection rule would choose. This is distinct from uncertainty in the estimated parameters within a chosen model. A fixed-order procedure removes repeated order changes by design, but it can still have parameter uncertainty and can be poorly specified if its chosen order is unsuitable.

An information criterion ranks candidate models according to a training-sample trade-off between fit and complexity. It is useful for choosing among the specified candidates, but its minimum is not a direct measurement of future forecast accuracy. A brief argument explains why the best observed in-sample criterion can look especially favorable.

Justification: Suppose a criterion for candidate \(j\) can be represented as \(C_j=c_j+\eta_j\), where \(c_j\) is its systematic component and \(\eta_j\) is sample-specific variation. The selected criterion is \(\min_j C_j\), which cannot exceed \(C_k\) for any particular candidate \(k\). When candidates have similar systematic components, selecting the minimum tends to select a negative realization of \(\eta_j\). That favorable selection noise does not have to recur in a new sample. Therefore, a low selected training criterion alone does not guarantee low out-of-sample forecast loss. This argument describes selection effects; it does not imply a universal direction or size of forecast-performance differences.

Order instability can be assessed by recording the selected \((p,q)\) at each origin, alongside the criterion differences between the leading candidates. Small differences and frequent switches indicate that the selected order is sensitive to the particular training sample or estimation window. They do not mean that the orders are equally plausible in a probabilistic sense, and the fraction of origins selecting an order is not the probability that it is the true order.

Worked Example: Reading Order Instability

Suppose a hypothetical rolling evaluation compares ARMA\((1,0)\) and ARMA\((1,1)\). The following table records their AICc values at three training origins. The difference is defined as AICc for ARMA\((1,1)\) minus AICc for ARMA\((1,0)\).

Origin	AICc ARMA(1,0)	AICc ARMA(1,1)	Difference	Selected order
1	100.0	100.4	0.4	ARMA(1,0)
2	101.2	100.9	-0.3	ARMA(1,1)
3	98.7	99.0	0.3	ARMA(1,0)

At origin 1, \(100.0<100.4\), so ARMA\((1,0)\) is selected. At origin 2, \(100.9<101.2\), so ARMA\((1,1)\) is selected. At origin 3, \(98.7<99.0\), so ARMA\((1,0)\) is selected again. The order changes even though the criterion differences are small in this example. A careful report would describe the observed instability and give the criterion values; it would not interpret two selections out of three as a 2/3 probability that ARMA\((1,0)\) is correct.

Estimation Windows Also Change Forecasts

Order selection and estimation windows are separate choices, but their effects can interact. With an expanding window, the estimation sample retains its starting observation and adds new observations as the origin advances. With a rolling window, the sample contains a fixed number of the most recent observations, so older observations leave the window. In either case, the data used for selection and estimation at an origin must end at that origin.

A rolling window can respond more quickly when recent observations differ from earlier ones, while an expanding window retains more information when the underlying process is stable. These are trade-offs, not general guarantees. A short rolling window can make estimates and selected orders more variable; an expanding window can give older observations influence even when recent behavior differs. The forecasting window should therefore be treated as part of the procedure being evaluated, not as a neutral implementation detail.

Worked Example: Expanding Versus Rolling Estimation

Consider a hypothetical series with observed values \(8,12,9,11,10,12\), followed by an observed value of \(11\). For illustration, use an ARMA\((0,0)\) model with a constant mean and forecast the next value by the sample mean in the estimation window. Assume the stable-mean white-noise specification is the intended model; these few observations alone cannot establish that assumption. Compare an expanding window using all six observations with a rolling window using only the most recent four.

The expanding-window forecast is

\hat{y}_{7\mid 6}^{\,\mathrm{expanding}} =\frac{8+12+9+11+10+12}{6} =\frac{62}{6} \approx 10.33.

The rolling-window forecast is

\hat{y}_{7\mid 6}^{\,\mathrm{rolling}} =\frac{9+11+10+12}{4} =\frac{42}{4} =10.50.

When the observed value \(11\) becomes available, the expanding-window error is \(11-10.33\approx0.667\), and the rolling-window error is \(11-10.50=0.500\). The rolling forecast is closer at this one origin, but one observation cannot establish a forecast-accuracy advantage. The example isolates how the window changes a forecast even when the order is held at ARMA\((0,0)\). If the model is reselected as well, both the candidate scores and the selected order can change with the window.

Interpreting Results and Avoiding Pitfalls

A defensible comparison states the entire forecasting strategy: candidate orders, selection criterion, selection frequency, estimation window, parameter-estimation method, forecast horizon, and evaluation loss. This makes clear whether a reported difference reflects order selection, parameter re-estimation, window choice, or several changes at once. When the goal is to isolate one factor, keep the others the same.

The accuracy result is conditional on the evaluated period and procedure. A lower average loss on a test period is evidence about performance on those evaluated origins, not proof that the selected order is the true data-generating model or that the ranking will persist. If formal comparisons of forecast loss are used, their assumptions and dependence conditions matter; an observed RMSE difference alone is not a significance test.

A frequent mistake is to conclude that repeated selection must forecast better because it chooses the lowest AICc or BIC at every origin. That reasoning confuses a training-sample ranking with an out-of-sample result. The correction is to treat the selection rule as part of the forecasting procedure, generate forecasts without using future outcomes, and compare its realized forecast losses with those from the fixed-order procedure on common origins.

Do not confuse fixed order with fixed parameters. A fixed-order strategy can re-estimate its coefficients at each origin.
Do not use future observations for selection. A forecast evaluation is invalid if the order or parameters were chosen using the outcome being forecast.
Do not compare unlike evaluation samples. Use common origins and horizons, or explain why the samples differ.
Do not treat selection frequency as model probability. It describes the behavior of the rule across the evaluated origins.

Key takeaway: Order selection, parameter estimation, and the estimation window jointly define an ARMA forecasting strategy. Whether an order is fixed or repeatedly selected matters because it changes the forecasts; only a fair out-of-sample comparison can show how those changes performed in the evaluated period.

Check Your Understanding

Use the definitions and comparisons in this tutorial to answer the following questions.

In a fixed-order procedure, what may change from one forecast origin to the next, and what is held constant?
Why can the model with the lowest AICc at an origin still produce a less accurate forecast than a fixed-order model?
What does frequent switching between two selected orders indicate, and what does it not establish?
Why should fixed-order and repeated-selection forecasts be evaluated on common origins and at the same horizon?
How does a rolling estimation window differ from an expanding window, and what trade-off can that create?