AIC, BIC, and AICc for ARMA Order Selection

Why ARMA Order Selection Needs a Complexity Penalty

A more flexible ARMA model can often achieve a higher maximized likelihood than a simpler one because it has more parameters available to fit the observed series. That improvement in fit does not by itself show that the more complex order will represent the underlying process better. AIC, BIC, and AICc combine the fitted likelihood with a penalty for model complexity, giving a disciplined way to rank a specified set of candidate orders.

These criteria answer a comparative question: among the models fitted to the same series under comparable likelihood calculations, which offers the preferred balance of fit and complexity according to a particular criterion? They do not establish that the winning model is true, guarantee adequate residuals, or provide a hypothesis test of one ARMA order against another.

Definition: For a fitted candidate model with maximized log-likelihood $\ell(\hat{\theta})$, let $k$ be the number of freely estimated parameters included in that likelihood and let $n$ be the number of observations used in the likelihood calculation. The Akaike information criterion (AIC), Bayesian information criterion (BIC), and small-sample corrected AIC (AICc) are $$ \mathrm{AIC}=-2\ell(\hat{\theta})+2k, \qquad \mathrm{BIC}=-2\ell(\hat{\theta})+k\log(n), \qquad \mathrm{AICc}=\mathrm{AIC}+\frac{2k(k+1)}{n-k-1}, $$ where the displayed AICc formula requires $n>k+1$. For each criterion, the preferred candidate is the one with the smallest value.

For a usual Gaussian ARMA($p,q$) fit with an estimated mean and innovation variance, $k$ includes the $p$ autoregressive coefficients, the $q$ moving-average coefficients, the mean, and the variance parameter: $k=p+q+2$. If the mean is fixed at zero, it is not counted as estimated, so $k=p+q+1$. Count each distinct free parameter actually estimated, including any other estimated terms in the specification, but do not count a fixed value. An alternative parameterization may use an intercept rather than a mean; count the one estimated location parameter, not both.

What the Penalties Do

All three scores start with $-2\ell(\hat{\theta})$, a measure of lack of fit on the likelihood scale: a larger maximized likelihood gives a smaller value of this term. Each then adds a complexity penalty. AIC adds $2k$, regardless of sample size. BIC adds $k\log(n)$, so the cost per extra parameter increases with $n$. AICc adds a positive finite-sample correction to AIC, and that correction grows as the available sample becomes small relative to the parameter count.

The different penalties reflect different aims. AIC is commonly motivated as an estimate of expected predictive information loss, with its penalty correcting for the optimism that arises when parameters are fitted and assessed on the same data. BIC penalizes additional parameters more heavily as the sample grows and is commonly used for model identification under its regularity conditions. AICc adjusts AIC for finite-sample bias in settings where the usual AIC approximation can be too optimistic. These motivations do not make any criterion universally best for every purpose.

Justification for the AIC penalty: Under regular likelihood conditions, fitting $k$ parameters makes the in-sample fit optimistic: it tends to look better than the fit those estimates would give to an independent replicate. The expected difference between the replicate deviance and the fitted-sample deviance is approximately $2k$. Thus $-2\ell(\hat{\theta})+2k$ is a first-order estimate of predictive deviance, which motivates AIC. This is an approximation, not an exact identity for every finite-sample time-series model; AICc adds a further correction when finite-sample effects matter.

For one extra parameter, the AIC penalty increases by $2$, while the BIC penalty increases by $\log(n)$. For example, at $n=20$, $\log(20)\approx2.996$, so BIC charges about $2.996$ per additional parameter; at $n=100$, it charges about $4.605$. BIC's penalty per parameter exceeds AIC's once $n>e^2$, although the ranking in any particular comparison also depends on the likelihood improvement. AICc's correction tends toward zero as $n$ grows, but can be substantial when $n$ is only moderately larger than $k$.

Worked Comparisons

Worked Example: A Close Three-Way Ranking

Suppose a hypothetical series has $n=20$ usable observations. Two converged, admissible Gaussian fits use the same observations and likelihood convention. The smaller ARMA(1,0) has $k=3$ estimated parameters (one AR coefficient, mean, and variance) and $-2\ell(\hat{\theta})=60$. The larger ARMA(2,1) has $k=5$ parameters and $-2\ell(\hat{\theta})=54$. Compute all three scores.

For ARMA(1,0), $$ \mathrm{AIC}=60+2(3)=66, \qquad \mathrm{BIC}=60+3\log(20)\approx68.9872, $$ and $$ \mathrm{AICc}=66+\frac{2(3)(4)}{20-3-1} =66+\frac{24}{16}=67.5. $$

For ARMA(2,1), $$ \mathrm{AIC}=54+2(5)=64, \qquad \mathrm{BIC}=54+5\log(20)\approx68.9787, $$ and $$ \mathrm{AICc}=64+\frac{2(5)(6)}{20-5-1} =64+\frac{60}{14}\approx68.2857. $$

AIC favors the larger model, $64<66$. BIC also very slightly favors the larger model, $68.9787<68.9872$, a difference of about $0.0085$. AICc favors the smaller model, $67.5<68.2857$. The ranking is not contradictory: each criterion applies a different penalty to the same improvement in fit, and BIC's margin here is negligible.

Worked Example: How a Larger Sample Changes BIC's Penalty

Consider two hypothetical fits to the same series with $n=100$, using the same likelihood method and observations. The smaller model has $k=3$ and $-2\ell=200$; the larger has $k=5$ and $-2\ell=194$. The larger model improves $-2\ell$ by 6, at the cost of two more parameters.

For AIC, the scores are $200+2(3)=206$ for the smaller model and $194+2(5)=204$ for the larger. AIC therefore favors the larger model. For BIC, $$ \mathrm{BIC}_{\text{small}}=200+3\log(100)\approx213.8155, \qquad \mathrm{BIC}_{\text{large}}=194+5\log(100)\approx217.0259. $$

BIC favors the smaller model by approximately $217.0259-213.8155=3.2104$. The likelihood gain is 6, but the two extra parameters add $2\log(100)\approx9.2103$ to the larger model's BIC penalty. For AICc, the correction terms are $24/(100-3-1)=0.25$ and $60/(100-5-1)\approx0.6383$, respectively. The scores are therefore $206.25$ and $204.6383$, so AICc favors the larger model in this comparison.

This calculation isolates the sample-size role of the penalty: BIC charges more per added parameter at $n=100$ than at $n=20$. In real applications, likelihood gains will also vary with the data and sample size; this example holds the gain fixed to make the penalty arithmetic clear.

Worked Example: Comparing Several ARMA Orders on One Basis

Suppose a hypothetical series has $n=80$ observations, and three candidate Gaussian ARMA fits have all converged under the same constraints and likelihood convention. Each includes an estimated mean and innovation variance. The fitted results are:

Candidate	$-2\ell(\hat{\theta})$	$k$
ARMA(1,0)	224	3
ARMA(0,1)	222	3
ARMA(1,1)	219	4

The AIC values are $224+2(3)=230$, $222+2(3)=228$, and $219+2(4)=227$. Thus AIC selects ARMA(1,1). For BIC, $\log(80)\approx4.3820$, giving $224+3(4.3820)\approx237.1461$, $222+3(4.3820)\approx235.1461$, and $219+4(4.3820)\approx236.5281$. BIC selects ARMA(0,1).

For AICc, the correction for $k=3$ is $24/(80-3-1)=24/76\approx0.3158$, and for $k=4$ it is $40/(80-4-1)=40/75\approx0.5333$. The AICc values are therefore approximately $230.3158$, $228.3158$, and $227.5333$. AICc selects ARMA(1,1). This is a ranking of the three candidates, not evidence that the selected model is adequate; the residual and innovation checks remain relevant.

Making Likelihood Comparisons Fair

The information criteria are only meaningful as a comparison when the likelihood values refer to comparable data and fitting conventions. In particular, candidate models should use the same observed series and the same effective sample. If one likelihood conditions on initial observations while another uses an exact likelihood for all observations, their $-2\ell$ values can differ partly because of that setup rather than because one order fits better. Likewise, changing the treatment of the mean, missing values, or initial values can undermine a direct ranking.

For a practical comparison, fit each candidate using a consistent likelihood method and record the number of observations actually contributing to that likelihood. Use the corresponding $n$ and count the estimated parameters consistently. AICc must be calculated with that parameter count and is undefined under the displayed formula when $n\leq k+1$. Software may use different conventions for parameter counts or conditional likelihoods, so verify what its reported criterion includes before combining results from different sources.

Set the candidate set.
Specify the ARMA orders and the treatment of the mean and variance before comparing scores.

Fit comparably.
Use the same observations, likelihood convention, and relevant constraints for every candidate; check that each optimization has converged.

Count and calculate.
For each fit, record $-2\ell(\hat{\theta})$, $k$, and $n$, then calculate the chosen criterion with the same conventions.

Rank, then assess adequacy.
Identify the lowest score for each criterion, note close rankings, and assess the selected fit using residual diagnostics and the modeling goal.

Interpreting Differences and Avoiding Pitfalls

The absolute value of an information criterion is generally not interpretable on its own. Its main use is the difference between candidate scores computed on a comparable basis. A lower value indicates preference under that criterion, not certainty that the model is correct. A small difference means the criterion gives little separation between those candidates; it does not turn the difference into a formal significance test. If AIC, BIC, and AICc select different orders, report that sensitivity and consider whether prediction, identification, or finite-sample performance is the main objective.

Information criteria also do not replace residual diagnostics. A low score among a set of candidates can belong to a model whose residuals still show serial dependence, unusual observations, or other problems. Conversely, passing residual checks does not show that a model is the best-ranked candidate. Use the criteria to compare fit-complexity tradeoffs, and use diagnostics to assess whether the fitted model leaves important structure unexplained.

Common mistake: “The model with the larger likelihood must be preferred because it fits better.” This ignores that adding parameters can improve the maximized likelihood even when the extra complexity is not worthwhile. The correction is to compute the relevant information criterion for every candidate on a comparable likelihood basis, include all estimated parameters in $k$, and select the lowest score for the chosen criterion. Do not describe that ranking as proof, a p-value, or a test result.

Key takeaway: AIC applies a fixed $2k$ penalty, BIC applies a sample-size-dependent $k\log(n)$ penalty, and AICc adds a finite-sample correction to AIC. Their rankings can differ; compare like with like and interpret each as a model-selection criterion rather than a verdict on model truth or adequacy.

Check Your Understanding

Try these questions before reviewing the worked calculations.

A Gaussian ARMA(2,1) model estimates its AR and MA coefficients, a mean, and an innovation variance. What value of $k$ should be used?
Two comparable models have the same $-2\ell$, but one has more estimated parameters. Which model has the lower AIC, and why?
For a fixed number of additional parameters, how does BIC's penalty change as $n$ increases?
Why is it risky to compare one model's exact likelihood with another model's conditional likelihood when their likelihoods use different effective samples?
If AICc is lower for one candidate but BIC is lower for another, what can be concluded, and what cannot be concluded, from that disagreement?