Deriving AIC for Time-Series Model Selection

Why AIC Balances Fit and Complexity

A fitted time-series model can improve its likelihood by using additional parameters to follow features in the observed series. But a more flexible model may fit random variation that will not recur. AIC, the Akaike information criterion, addresses this tension by combining a model’s maximized likelihood with a penalty for the number of parameters used to obtain that fit.

AIC is a selection criterion: among candidates fitted to the same observations using comparable likelihoods, the smaller value is preferred as an estimate of relative predictive information loss. It is not a test that declares a model true, and it does not establish that the selected model is adequate. Residual diagnostics and the purpose of the analysis still matter.

For a time-series model, \(k\) will commonly include estimated autoregressive and moving-average coefficients, an estimated intercept or mean when present, and the innovation variance when it is estimated. Include other estimated parameters that enter the fitted likelihood, such as seasonal coefficients where applicable. A quantity fixed in advance rather than estimated is not counted as an estimated parameter. The exact count depends on the fitted model and likelihood specification.

For a sound ranking, candidates must be fitted to the same observed series using likelihoods that are comparable: the same response scale and compatible treatment of observations, such as initial values or conditional likelihood contributions. A likelihood calculated from a different number of observations or on an incompatible scale should not be put into the same AIC comparison without appropriate justification. This is why establishing likelihood comparability is part of constructing a useful candidate set.

Why the Penalty Is \(2k\)

The penalty is not an arbitrary preference for short models. The maximized likelihood is evaluated on data that were also used to estimate the parameters, so it tends to look better than the model’s expected performance on new data. Under regularity conditions, the \(2k\) term approximately corrects that optimism on the deviance scale.

A short derivation shows the source of the correction. Suppose a regular parametric model is correctly specified, with true parameter \(\theta_0\) of dimension \(k\), and the likelihood is based on \(n\) observations. Write \(U_n\) for the score vector at \(\theta_0\), and \(I\) for the Fisher information per observation. Under the usual large-sample approximation, the maximum-likelihood estimation error is approximately \(I^{-1}U_n/n\), and the observed information is approximately \(nI\). A second-order expansion of the log likelihood around \(\theta_0\) gives the following approximate in-sample gain from fitting rather than using the true parameter:

Because the score has approximate mean zero and covariance \(nI\), the expected value of the quadratic expression on the right is \(k/2\). For an independent future sample of the same size, the fitted parameter is held fixed while the new observations are evaluated. A second-order expansion then gives an expected log-likelihood change of approximately \(-k/2\) relative to evaluation at \(\theta_0\): estimation error costs expected future fit. Thus, the expected gap between the fitted in-sample log likelihood and future log likelihood is approximately \(k\). Multiplying a log-likelihood difference by \(-2\) converts that optimism to a deviance-scale penalty of \(2k\).

The same principle motivates AIC for regular time-series likelihoods under suitable large-sample conditions, including appropriate handling of dependence. The derivation is an approximation, not a guarantee that AIC selects the true model or forecasts best in every finite sample. In particular, it presumes a well-behaved likelihood and parameter estimation; unusual boundaries, non-identifiability, or seriously misspecified models can undermine the simple parameter-count correction.

Calculating AIC for Fitted Time-Series Candidates

To rank fitted candidates, record each maximized log likelihood and its parameter count, calculate \(-2\ell(\hat{\theta})\), add \(2k\), and compare the results. Keep full precision while calculating and round only the reported final values. The likelihood values are often negative, so the sign in \(-2\ell(\hat{\theta})\) deserves particular attention.

Using the Gaussian Likelihood When Only the Residual Sum Is Given

Sometimes a comparison reports the sum of squared innovations rather than the maximized log likelihood. For a conditional Gaussian innovations likelihood with \(n\) contributions and residual sum of squares \(S\), assume the innovations are independent normal terms with common unknown variance \(\sigma^2\), conditional on the model’s fitted values and the specified initial conditions. The log likelihood is \(\ell=-\frac{n}{2}\log(2\pi\sigma^2)-\frac{S}{2\sigma^2}\). Maximizing with respect to the variance gives \(\hat{\sigma}^2=S/n\). Substitution yields the maximized log likelihood used below. This calculation applies to that Gaussian likelihood setup; do not use it as though it were the exact likelihood for a different error distribution or a different treatment of initial observations.

The variance is estimated from the residual sum, so it contributes one parameter to \(k\). If the candidate models use the same \(n\) and likelihood setup, the common \(\log(2\pi)\) term does not change their ordering. It should nevertheless be included when calculating the complete AIC values from this formula.

Interpreting Relative AIC Scores

AIC differences are usually more useful than the raw values. Let \(\operatorname{AIC}_{\min}\) be the smallest value in a candidate set and define \(\Delta_j=\operatorname{AIC}_j-\operatorname{AIC}_{\min}\). The candidate with the minimum has \(\Delta=0\); positive differences show how far the other candidates are from it on the AIC scale.

A common relative-score calculation is \(\exp(-\Delta_j/2)\). It expresses a candidate’s relative support compared with the minimum-AIC candidate within the specified set. It is not the probability that the candidate is true, nor is it a probability of forecasting correctly. Normalizing these relative scores to sum to one gives Akaike weights, which can be used as a measure of relative support across the candidate set, but they remain dependent on which candidates were included.

Common AIC Mistakes

AIC rewards fit and penalizes estimated complexity, but correct arithmetic cannot repair an inconsistent comparison. Before choosing a candidate, verify both the likelihood basis and the parameter count.

• Using the wrong sign. If \(\ell(\hat{\theta})=-40\), then \(-2\ell(\hat{\theta})=80\), not \(-80\).
• Leaving out estimated parameters. An estimated innovation variance, mean, or intercept belongs in \(k\), as do estimated dynamic coefficients. Count each parameter once.
• Comparing incompatible likelihoods. Candidate likelihoods must refer to comparable observations and scales. A smaller AIC is not meaningful if one value is based on a different set of likelihood contributions without a defensible adjustment.
• Treating the minimum as proof of adequacy. AIC ranks the candidates supplied; every candidate may still be misspecified. Residual checks and the intended forecasting or modeling task remain important.
• Calling an AIC difference a probability. A difference can be converted to a relative score, but that score is not, by itself, the probability that a model is true.