Deriving AIC From Predictive Optimism

Why Fitted Likelihood Needs a Correction

A fitted model is evaluated on the same observations that helped determine its parameter estimates. That reuse makes the fitted log-likelihood look better than the log-likelihood the model is expected to achieve on new observations. AIC corrects for this optimism: in a regular, correctly specified model, the expected difference in total log-likelihood is approximately the number of estimated parameters. Since the deviance scale is negative twice the log-likelihood, the corresponding correction is twice that number.

The correction is an estimate of predictive performance for a fixed candidate model. It is not a guarantee that the model is correct, and its usual derivation does not remove extra optimism caused by searching over many models and then reporting only the selected one.

Predictive Optimism and the AIC Definition

For independent observations with a common density or mass function \(f(y;\theta)\), the log-likelihood for a data set \(y=(y_1,\ldots,y_n)\) is \(\ell_y(\theta)=\sum_{i=1}^n \log f(y_i;\theta)\). If the model has \(k\) freely estimated parameters and satisfies the regularity conditions below, the expected optimism is approximately \(k\) log-likelihood units. Thus an estimate of expected future negative twice log-likelihood is

Here \(k\) counts the free parameters estimated by maximum likelihood, including an estimated dispersion parameter when it is part of the likelihood. AIC values are comparable only when they come from likelihoods for the same response data under a consistent likelihood convention. In particular, likelihood constants may be omitted when they are common across candidate models, but then the reported AIC values are shifted by the same constant.

Deriving the Parameter-Count Correction

Write \(s(Y_i;\theta_0)=\partial \log f(Y_i;\theta)/\partial\theta\vert_{\theta=\theta_0}\), and define the total score \(U=\sum_{i=1}^n s(Y_i;\theta_0)\). Under the stated assumptions, \(E[U]=0\), \(E[UU^T]=nI\), and \(\hat{\theta}-\theta_0=I^{-1}U/n+r_n\), where the remainder \(r_n\) is small enough in mean square that \(E[U^Tr_n]=o(1)\).

Let \(Q(\theta)=E_{\theta_0}[\log f(Y;\theta)]\) be the expected log density of one future observation. Correct specification and the information identity give \(\nabla Q(\theta_0)=0\) and \(\nabla^2Q(\theta_0)=-I\). Taylor expansion of the training log-likelihood and the expected future log-likelihood therefore gives

Conditionally on the training data, \(nQ(\hat{\theta})\) is the expected log-likelihood on an independent future sample. Subtracting the second expansion from the first cancels the quadratic terms. Also, \(E[\ell_Y(\theta_0)-nQ(\theta_0)]=0\), since both terms have the same expectation. Hence

For the last equality, \(E[U^TI^{-1}U]=\operatorname{tr}(I^{-1}E[UU^T])=\operatorname{tr}(I^{-1}nI)=nk\). This proves the optimism result. Multiplying the log-likelihood optimism by two gives the \(2k\) correction on the negative-twice-log-likelihood scale used by AIC. \(\square\)

The same argument applies to regular GLMs with independent responses and a fixed design when the likelihood is correctly specified and the information matrix is nonsingular, with the information matrices summed over observations. The trace calculation then yields the dimension of the fitted parameter vector. The fixed-dimension and regularity conditions matter: the derivation does not assert an exact finite-sample correction for every GLM.

Worked Examples

Assumptions, Interpretation, and Common Pitfalls

AIC’s penalty is a derived approximation, not a universal finite-sample law. Its standard parameter-count form relies on a regular likelihood model: parameters should be identifiable and interior, the information should be nonsingular, and the likelihood should be sufficiently smooth. The usual derivation also assumes the candidate model is correctly specified and that the number of parameters remains fixed as sample size grows. For fitted GLMs, separation, boundary estimates, sparse data, or a poorly identified parameter can undermine the regular approximation.

If the likelihood is misspecified, the information identity used in the derivation need not hold, and the optimism correction is not generally \(k\); a correction involving both score variability and curvature may be needed. Likewise, quasi-likelihood alone does not define an ordinary likelihood-based AIC. The likelihood and its parameter count must match the fitted model and the data representation.

A frequent mistake is to say that AIC adds \(k\) to the negative log-likelihood. The derived optimism is approximately \(k\) on the log-likelihood scale, but AIC uses negative twice log-likelihood, so the addition is \(2k\). Another mistake is to treat AIC as a test of whether a model is true. It is instead an estimated predictive comparison under its assumptions; a small AIC difference can indicate that the candidates have similar estimated predictive performance.