Comparing ARIMA Candidates Under Parsimony

Why ARIMA Candidates Need a Fair Comparison

After identifying plausible ARIMA specifications, the next question is not simply which model fits the observed series most closely. A more flexible model will often achieve a higher likelihood or a smaller in-sample error because it has more parameters to adjust. The comparison must account for that flexibility and must be relevant to the forecasts the model is intended to produce.

Likelihood-based information criteria combine a measure of fit with a penalty for model complexity. They help compare candidate models, but they do not certify that a model is adequate or guarantee that it will forecast best. In particular, model rankings are meaningful only when the candidates are fitted to the same target using comparable likelihood calculations and observations.

Definition: For a candidate model with maximized log-likelihood $\ell$, $k$ estimated parameters, and $n$ observations used in the likelihood, the Akaike information criterion and Bayesian information criterion are $$ \mathrm{AIC}=-2\ell+2k,\qquad \mathrm{BIC}=-2\ell+k\ln(n). $$ Smaller values are preferred within a valid comparison. The small-sample corrected Akaike information criterion is $$ \mathrm{AICc}=\mathrm{AIC}+\frac{2k(k+1)}{n-k-1}, $$ provided $n-k-1>0$.

Here, $\ell$ is the maximized log-likelihood, not the likelihood itself. Since log-likelihoods for fitted models are usually negative, a value closer to zero indicates a better likelihood fit for the same observations and likelihood definition. The parameter count $k$ includes estimated AR and MA coefficients, seasonal coefficients, an estimated mean or drift when present, and the innovation variance when it is estimated. It does not include a fixed order such as $d=1$ as though it were an estimated coefficient. If a parameter is fixed rather than estimated, it is not counted as an estimated parameter.

The formulas are commonly applied to likelihoods under a specified innovation distribution, often Gaussian innovations. For ARIMA candidates, the likelihood may be calculated exactly or conditionally. The fitting method and treatment of initial observations must be consistent across candidates. The effective $n$ is the number of observations contributing to the likelihood under that calculation, not necessarily the raw number of rows in the original data.

Why the Criteria Penalize Complexity

The likelihood term rewards fit: maximizing $\ell$ minimizes $-2\ell$. The penalty recognizes that fitting parameters to a finite sample makes in-sample fit optimistic. Under regularity conditions—such as an identifiable, correctly specified, finite-dimensional model with a well-behaved likelihood—the expected optimism in fitted log-likelihood is approximately $k$ units. Because the criteria use deviance scale, which is $-2$ times log-likelihood, the corresponding correction is approximately $2k$. This gives the AIC penalty.

\text{in-sample deviance}=-2\ell \quad\longrightarrow\quad \text{optimism-adjusted comparison}\ \approx -2\ell+2k.

This argument explains the purpose of the AIC penalty; it is not an exact guarantee for every finite-sample time series. AICc adds a further correction that matters particularly when $n$ is not large relative to $k$. BIC uses a stronger penalty for many sample sizes because $\ln(n)$ eventually exceeds 2. Under regular likelihood assumptions, BIC is also motivated as an approximation to a model's marginal likelihood. These criteria answer related, but not identical, model-selection questions, so they need not select the same candidate.

Worked Comparisons

Worked Example: Ranking Three Candidates by AIC and AICc

A hypothetical monthly exposure index has 60 usable observations. Three Gaussian-likelihood ARIMA candidates were fitted to the same transformed target, the same 60 observations, and the same exact-likelihood convention. Candidate A is ARIMA(1,1,0) with no drift; Candidate B is ARIMA(2,1,1) with drift; Candidate C is a seasonal ARIMA(1,1,1)(1,0,1)₁₂ with drift. Their maximized log-likelihoods are $-83$, $-79$, and $-78.5$, respectively. Their parameter counts, including innovation variance, are 2, 5, and 6. Compute AIC and AICc and choose the lowest.

Method. Use the same formulas for all candidates because they share the target, observations, and likelihood convention. For Candidate A,

\mathrm{AIC}_A=-2(-83)+2(2)=166+4=170.

For Candidate B, $-2\ell=158$, so

\mathrm{AIC}_B=158+2(5)=168.

For Candidate C, $-2\ell=157$, giving

\mathrm{AIC}_C=157+2(6)=169.

AIC selects Candidate B: its value of 168 is the smallest. Candidate C has the best raw log-likelihood, but its larger parameter count offsets that improvement in AIC.

Now calculate the AICc corrections. For A, the correction is $2(2)(3)/(60-2-1)=12/57=0.2105$, so $\mathrm{AICc}_A=170.2105$. For B, it is $2(5)(6)/(60-5-1)=60/54=1.1111$, so $\mathrm{AICc}_B=169.1111$. For C, it is $2(6)(7)/(60-6-1)=84/53=1.5849$, so $\mathrm{AICc}_C=170.5849$. AICc also selects Candidate B.

Check. The corrections are positive, so each AICc value exceeds its AIC value. Candidate B remains the winner under both criteria. The common mistake this avoids is selecting Candidate C solely because its maximized likelihood is highest, without accounting for its six estimated parameters.

Worked Example: Correcting a Comparison With Unequal Samples

Two hypothetical specifications for quarterly benefit counts were initially fitted using different effective samples because their differencing choices left different numbers of usable observations. Model M used 36 observations, had $k=4$, and gave $\ell=-50$. Model N used 35 observations, had $k=6$, and gave $\ell=-47.4$. Explain why directly ranking these results is questionable, then compare them after both are refitted on the same 35-observation interval. On that common interval, M has $\ell=-48.8$; N retains $\ell=-47.4$.

Method. Directly computing AIC from the first pair would give M: $-2(-50)+2(4)=108$, and N: $-2(-47.4)+2(6)=106.8$. But the likelihoods summarize different observations, so their fit terms do not measure performance on the same data. Refit both models on the common interval before comparing.

On 35 observations, M has $-2\ell=97.6$, so $\mathrm{AIC}_M=97.6+8=105.6$. N has $-2\ell=94.8$, so $\mathrm{AIC}_N=94.8+12=106.8$. AIC now favors M.

For BIC, $\ln(35)\approx3.5553$. Therefore,

\mathrm{BIC}_M=97.6+4(3.5553)=111.8212, \qquad \mathrm{BIC}_N=94.8+6(3.5553)=116.1318.

BIC also favors M. For AICc, M's correction is $2(4)(5)/(35-4-1)=40/30=1.3333$, giving $106.9333$. N's correction is $2(6)(7)/(35-6-1)=84/28=3$, giving $109.8$. AICc favors M as well.

Check. All three criteria select M after refitting on the same 35 observations. The direct AIC ranking differed because the initially reported log-likelihoods used different data. A common mistake is to treat an information criterion as if it automatically makes likelihoods from different samples comparable; the penalty does not repair that mismatch.

Worked Example: Separating Fit Ranking From Forecast Purpose

A hypothetical monthly portfolio measure is to be forecast six months ahead. Both candidate models use the same 48 observations and likelihood convention. Model R has $k=4$, $\ell=-40$; Model S has $k=7$, $\ell=-39$. At four earlier forecast origins, each model also produced a six-month-ahead forecast. Define error as actual value minus forecast. Model R's errors were $2,-3,1,-2$; Model S's were $1,-1,2,-1$. Compare AIC and the validation errors, then state what the evidence supports.

Method. Model R has $\mathrm{AIC}_R=-2(-40)+2(4)=80+8=88$. Model S has $\mathrm{AIC}_S=-2(-39)+2(7)=78+14=92$. Thus, AIC favors R on the fitting sample.

For the six-month errors, R's absolute errors sum to $2+3+1+2=8$, so its mean absolute error is $8/4=2$. Its squared errors sum to $2^2+(-3)^2+1^2+(-2)^2=4+9+1+4=18$, so its root mean squared error is $\sqrt{18/4}=\sqrt{4.5}\approx2.121$.

S's absolute errors sum to $1+1+2+1=5$, giving mean absolute error $5/4=1.25$. Its squared errors sum to $1+1+4+1=7$, so its root mean squared error is $\sqrt{7/4}=\sqrt{1.75}\approx1.323$. The validation errors favor S for this six-month forecasting purpose.

Conclusion and check. The criteria and the forecast-error comparison answer different questions: AIC favors R on likelihood fit adjusted for complexity, while the small set of past six-month forecasts favors S on both reported error measures. This is not enough evidence to claim S will always forecast better, but it shows why a lower in-sample criterion or error alone does not settle a forecast decision. The common mistake is to treat the AIC winner as necessarily the best model at every forecast horizon.

Using Rankings With Care

Information criteria are comparative: the numerical size of an AIC or BIC value has no universal interpretation by itself. A difference is useful only among models fitted to the same response series, on a comparable observation interval, with compatible likelihood calculations. A transformation, differencing choice, missing-value treatment, or likelihood convention can alter which observations contribute. If those differ, first make the comparison fair—for example, by refitting on a common interval when that is appropriate.

A lower criterion is a relative preference among the candidates considered, not proof that the model is correctly specified. If AIC and BIC disagree, report the disagreement and recognize the different penalties rather than declaring one universally correct. AIC's penalty is $2k$; BIC's penalty is $k\ln(n)$. For $n>e^2$, $\ln(n)>2$, so BIC penalizes each additional parameter more heavily than AIC does.

The forecast target also matters. A model intended for one-step prediction need not be the best choice for a longer horizon or for forecasts at a particular season. Information criteria summarize likelihood fit over the fitted sample; they do not directly assess performance at each intended horizon. Forecast errors evaluated on observations not used to estimate the candidate models can provide relevant evidence, provided the evaluation represents the intended use. A modest evaluation set is still uncertain evidence, not a guarantee.

Finally, parsimony does not mean choosing the model with the fewest terms regardless of fit. It means balancing fit and complexity, then checking whether the remaining candidate serves the forecasting purpose. Information criteria narrow the comparison; they do not replace sound modeling judgment.

Key takeaway: Compare ARIMA candidates using a consistent likelihood and sample, count all estimated parameters, and interpret AIC, AICc, and BIC as relative fit-complexity measures. A lower in-sample error or criterion does not by itself establish the best forecast for the intended horizon.

Check Your Understanding

For each problem, state the comparison or calculation requested. Treat all stated likelihoods as maximized log-likelihoods.

Two models use the same $n=40$ observations. Model A has $k=3$, $\ell=-52$; Model B has $k=5$, $\ell=-49$. Calculate AIC and BIC for both, using $\ln(40)\approx3.6889$, and identify each criterion's preferred model. (Answer: AICs are 110 and 108, so AIC prefers B; BICs are approximately 115.0667 and 116.4445, so BIC prefers A.)
For $n=25$, Model C has $k=4,\ell=-30$, and Model D has $k=6,\ell=-28.5$; both use the same observations. Calculate AICc and select the lower value. (Answer: $\mathrm{AICc}_C=70.0000$, $\mathrm{AICc}_D\approx73.6667$; select C.)
Two ARIMA fits have reported log-likelihoods $-61$ and $-58$, but the first uses 52 observations and the second 50 because of different differencing. What must be addressed before using their AIC values to rank them? (Answer: Refit or otherwise construct a comparison with a common target, comparable likelihood treatment, and common observation interval; the reported likelihoods are not directly comparable as stated.)
A model with more parameters has lower training sum of squared errors, but a simpler model has lower AIC. Give the reason this can happen and state whether the training-error result alone determines the preferred model. (Answer: The more complex model may reduce in-sample error, while AIC penalizes its extra estimated parameters; training error alone does not determine the preferred model.)
Two models have the same sample size $n=30$. Model E has $k=3,\ell=-35$; Model F has $k=6,\ell=-32$. Calculate their AICs and say which has the lower AIC. (Answer: $\mathrm{AIC}_E=76$, $\mathrm{AIC}_F=76$; they tie under AIC.)

What You'll Learn