Why ARIMA Selection Requires Combined Evidence
Several nonseasonal ARIMA models can describe the same differenced time series plausibly. One may have the largest likelihood because it uses more parameters, while another may leave residual autocorrelation or produce worse forecasts. A defensible selection therefore checks whether the candidates are comparable, evaluates their residuals, and assesses forecasts at the horizons that matter.
The focus here is a worked comparison, not an automatic rule that picks the smallest number in one column. We use likelihood and AICc to compare in-sample fit with complexity, residual ACF and Ljung–Box results to check remaining dependence, and rolling-origin errors to evaluate forecasts. These forms of evidence answer related but distinct questions.
For the likelihood calculations below, assume Gaussian innovations with mean zero and constant variance, and use the same likelihood method and the same 36 differenced observations for every candidate. The drift is fixed at zero rather than estimated. The innovation variance is estimated and is included in the parameter count. These details matter: likelihoods or information criteria computed on different observations, or with inconsistent parameter counts, are not directly comparable. If the innovations are not reasonably represented by the assumed distribution, a Gaussian likelihood comparison may be less informative.
AICc and the Cost of Complexity
Let \(\ell\) be the maximized log likelihood, \(k\) the number of estimated parameters, and \(n\) the number of observations used in the likelihood. The small-sample corrected Akaike information criterion is
The first term rewards fit: a larger log likelihood makes \(-2\ell\) smaller. The \(2k\) term penalizes the number of estimated parameters, and the final term adds a finite-sample correction. Among candidates fitted comparably, smaller AICc is preferred, but the value is relative to the candidate set; it is not a test of whether a model is true.
The correction follows the AIC penalty by adding an amount that grows with \(k\) and becomes more important when \(n\) is not large relative to \(k\). For example, with \(k=2\), its size is \(12/(n-3)\). Thus a parameter costs more when the effective sample is small. If \(n\) is too small for the denominator to be positive, this AICc formula is not usable for that candidate. Implementations can differ in how they define \(n\) after differencing and how they count parameters, so apply one stated convention consistently.
Worked Example: Comparing AICc for Four Candidates
A hypothetical monthly service-cost index is differenced once. Four nonseasonal candidates are fitted to the same \(n=36\) observations with Gaussian likelihood and zero drift fixed. The table gives their maximized log likelihoods. For ARIMA\((p,1,q)\), count the estimated innovation variance and the AR and MA coefficients, but not the fixed drift.
| Candidate | \(\ell\) | \(k\) |
|---|---|---|
| ARIMA\((0,1,0)\) | \(-54.0\) | 1 |
| ARIMA\((0,1,1)\) | \(-47.8\) | 2 |
| ARIMA\((1,1,1)\) | \(-47.2\) | 3 |
| ARIMA\((2,1,0)\) | \(-46.9\) | 3 |
Method. Substitute each candidate’s likelihood and parameter count into the AICc formula, keeping \(n=36\) fixed.
For ARIMA\((0,1,0)\), \(k=1\):
For ARIMA\((0,1,1)\), \(k=2\):
For ARIMA\((1,1,1)\), \(k=3\):
For ARIMA\((2,1,0)\), also \(k=3\):
Check. The highest likelihood is \(-46.9\), for ARIMA\((2,1,0)\), but its AICc is not the smallest. ARIMA\((0,1,1)\) has the lowest AICc, \(99.9636\), because its improvement in fit is balanced against a smaller parameter count. The common mistake is to select the candidate with the largest likelihood without accounting for complexity.
Residual ACF and the Ljung–Box Test
After fitting a model, inspect its residuals: the differences between observed values and fitted values on the modeled scale. A satisfactory model should leave residuals with no evident systematic autocorrelation. The residual ACF is a useful visual check. With \(n=36\), the rough reference limits for individual residual autocorrelations are \(\pm1.96/\sqrt{36}=\pm0.3267\), under the usual white-noise approximation. These are approximate individual-lag limits, not a guarantee that all residual dependence has been eliminated.
The Ljung–Box statistic tests a joint null hypothesis that residual autocorrelations through lag \(m\) are zero. If \(r_j\) is the residual autocorrelation at lag \(j\), its form is
For a fitted ARMA model, a common reference approximation is a chi-square distribution with \(m-p-q\) degrees of freedom, when this is positive. The test adjusts for the estimated AR and MA terms; differencing order \(d\) is not subtracted in this convention. The approximation can be unreliable with small samples, very large \(m\), or model conditions that make the reference distribution poor. A large p-value means the test does not find strong evidence against the joint null; it does not prove the residuals are independent or normally distributed.
Worked Example: Reading Residual Diagnostics Together
For the same hypothetical data, residual diagnostics are calculated through lag \(m=10\). The reported ACF maxima summarize the largest absolute sample residual autocorrelation among the first ten lags.
| Candidate | Largest absolute residual ACF | \(Q(10)\) | Degrees of freedom | p-value |
|---|---|---|---|---|
| ARIMA\((0,1,0)\) | 0.41 | 19.0 | 10 | 0.0403 |
| ARIMA\((0,1,1)\) | 0.23 | 10.8 | 9 | 0.2897 |
| ARIMA\((1,1,1)\) | 0.21 | 9.7 | 8 | 0.2867 |
| ARIMA\((2,1,0)\) | 0.24 | 8.4 | 8 | 0.3954 |
Step 1: Set the reference distribution. For ARIMA\((0,1,1)\), \(m-p-q=10-0-1=9\), so compare \(Q=10.8\) with a chi-square distribution having 9 degrees of freedom. Its upper-tail probability is 0.2897. For ARIMA\((1,1,1)\), the degrees of freedom are \(10-1-1=8\), giving an upper-tail probability of 0.2867 for \(Q=9.7\).
Step 2: Compare the null evidence. The two p-values are both well above 0.05, so neither test gives strong evidence of residual autocorrelation through lag 10. ARIMA\((2,1,0)\) also has a large p-value. By contrast, the \((0,1,0)\) candidate has p-value 0.0403, and its largest residual ACF, 0.41, exceeds the rough reference limit of 0.3267. Both checks raise concern about that candidate.
Step 3: State the limitation. The evidence does not establish that the first three acceptable candidates have perfect residuals. The ACF checks individual lags, while Ljung–Box checks a group of lags; neither substitutes for examining the residual series and the modeling context.
Common mistake. Using 10 degrees of freedom for every row ignores the AR and MA terms estimated in each model. For ARIMA\((1,1,1)\), using 10 instead of 8 would give the wrong reference distribution and p-value. Here the reported p-values are calculated with the stated adjusted degrees of freedom.
Rolling-Origin Forecast Errors
Residual checks are in-sample diagnostics. Rolling-origin validation asks how well a model forecasts observations not used to fit that particular forecast. At each origin, fit or update the model using only observations available up to that time, then forecast the next observation or a specified longer horizon. Evaluate all candidates on the same origins and the same target observations. If the validation design or model search uses future information, its error estimates can be optimistic.
For errors \(e_1,\ldots,e_R\) at one forecast horizon, the root mean squared error is
An error is the observed value minus its forecast; its sign does not affect RMSE because it is squared. RMSE is expressed in the units of the response, and it penalizes large errors more than small ones. A separate RMSE should be calculated for each horizon when performance at different horizons matters.
Worked Example: Combining Fit, Diagnostics, and Forecast Scores
Eight common rolling origins are used to assess the four candidates above. At each origin, forecasts are evaluated one month ahead and three months ahead. The table gives the sum of squared errors across the eight origins for each horizon. All errors are measured in index points.
| Candidate | One-month sum of squared errors | Three-month sum of squared errors |
|---|---|---|
| ARIMA\((0,1,0)\) | 36.64 | 67.76 |
| ARIMA\((0,1,1)\) | 23.68 | 45.32 |
| ARIMA\((1,1,1)\) | 25.92 | 50.40 |
| ARIMA\((2,1,0)\) | 22.85 | 52.02 |
Step 1: Calculate the one-month RMSE for ARIMA\((0,1,1)\). Divide the sum of squared errors by the eight origins and take the square root:
Step 2: Calculate its three-month RMSE.
Step 3: Compare all candidates. Applying the same calculation to every row gives one-month RMSEs of 2.1401, 1.7205, 1.8000, and 1.6900, respectively. The corresponding three-month RMSEs are 2.9103, 2.3801, 2.5100, and 2.5500. Values are rounded to four decimal places only at this final reporting step.
Step 4: Make a qualified choice. ARIMA\((0,1,1)\) has the lowest AICc, has no strong Ljung–Box evidence of residual autocorrelation, and has the lowest three-month RMSE. ARIMA\((2,1,0)\) has a slightly lower one-month RMSE, 1.6900 rather than 1.7205, and also passes the reported residual check. The AICc difference between them is \(100.5500-99.9636=0.5864\), so the information criterion alone does not create a decisive separation. If three-month forecasting is the priority, the evidence supports ARIMA\((0,1,1)\); if one-month forecasts dominate, the small one-step advantage of ARIMA\((2,1,0)\) merits attention. In this stated setting, select ARIMA\((0,1,1)\), while reporting the close comparison rather than presenting the choice as certain.
Check. The three-month errors use the same eight origins for every candidate, so the comparison is aligned by horizon and evaluation sample. The lowest one-month RMSE belongs to a different candidate than the lowest three-month RMSE, which is why a single undifferentiated forecast score would conceal a relevant trade-off.
A Practical Selection Sequence
A consistent selection is a reasoned conclusion from evidence, not a mechanical vote among columns. The following sequence keeps the main checks distinct while making them comparable.
Check that candidates use the same observations, response scale, likelihood convention, and stated parameter-count rule. Verify the differencing choice and any fixed or estimated drift terms.
Calculate the criterion from the reported likelihood and count. Use the criterion to weigh fit against complexity, not as a standalone goodness-of-fit test.
Review the residual ACF and a Ljung–Box test at a defensible lag. Apply the stated degrees-of-freedom convention and investigate visible patterns even when the test p-value is not small.
Use common rolling origins and compare the horizons relevant to the decision. State the error metric and its units; do not confuse an in-sample fit measure with an out-of-sample score.
Explain which evidence is strongest for the intended use, note meaningful trade-offs, and avoid claiming certainty when candidate scores are close.
AICc, residual diagnostics, and forecast errors are not interchangeable. A small AICc does not guarantee white-noise residuals; a large Ljung–Box p-value does not guarantee accurate forecasts; and a low rolling-origin RMSE does not erase a systematic residual pattern. The combined assessment is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Check Your Understanding
For each problem, state the calculation or reasoning that supports your answer before checking the final result.
- A nonseasonal candidate has \(\ell=-30\), \(k=2\), and \(n=25\). Using the stated AICc formula, calculate its AICc to four decimal places. (Answer: \(64.5455\))
- Two models are fitted to the same 30 observations. Model A has \(\ell=-40\) and \(k=2\); Model B has \(\ell=-39\) and \(k=3\). Calculate both AICc values and identify the smaller one. (Answer: Model A, \(84.4444\); Model B, \(84.9231\); Model A has the smaller AICc.)
- A fitted ARIMA\((1,1,1)\) is checked with a Ljung–Box statistic through lag \(m=12\). Under the \(m-p-q\) convention, how many degrees of freedom are used? (Answer: \(12-1-1=10\))
- At one forecast horizon, the sum of squared errors over eight common origins is 50. Calculate RMSE in the response’s units. (Answer: \(\sqrt{50/8}=2.5000\))
- Candidate A has the lowest AICc and passes a residual autocorrelation check, but candidate B has a lower three-month rolling-origin RMSE. The application specifically prioritizes three-month forecasts. What is the appropriate conclusion? (Answer: The evidence favors candidate B for the stated forecasting objective, but the comparison should report candidate A’s AICc advantage and verify that both candidates were evaluated on identical origins and three-month targets.)