AIC and Expected Predictive Information Loss

Why Compare Models by More Than Fit?

A more flexible regression model can often fit the observed data at least as well as a simpler one. That improved fit does not guarantee better predictions for new observations: extra flexibility can capture chance features of the sample. Model comparison therefore needs to balance how well a model fits with the optimism created by estimating its parameters from the same data.

Akaike’s Information Criterion, usually abbreviated AIC, is one way to make this comparison. It starts with the maximized likelihood, which measures fit, and adds a penalty for the number of estimated parameters. Its purpose is not to identify a model that is certainly true. Under suitable conditions, it estimates which candidate model has the smaller expected out-of-sample information loss.

Likelihood and Information Loss

Suppose a candidate model \(M\) assigns a probability density or probability mass function \(f(y\mid x;\theta)\) to a response \(Y\) given predictors \(x\). The parameter vector \(\theta\) contains the model’s unknown parameters. For observed data, the likelihood is the probability model viewed as a function of \(\theta\), with the observed responses held fixed. Its maximized log-likelihood is written \(\hat{\ell}\). Larger values of \(\hat{\ell}\) indicate better fit to those observations, when the models use the same likelihood basis.

To describe predictive information loss, imagine that the actual distribution of responses given predictors is \(g(y\mid x)\). If a candidate model assigns very low probability to outcomes that occur under \(g\), it incurs a large log loss. The Kullback–Leibler divergence from \(g\) to a candidate distribution \(f\), for a fixed predictor value \(x\), is

This quantity is nonnegative and is zero only when the two distributions agree, subject to the usual conditions for defining the divergence. In model comparison, the candidate parameters are estimated from a training sample, and the fitted model is judged by its expected log loss on new observations. The expectation therefore concerns both the new response and the variation in the training sample that produced the fitted parameters.

The first term rewards fit: a larger maximized log-likelihood makes \(-2\hat{\ell}\) smaller. The second term penalizes complexity: estimating more parameters increases \(2k\). The penalty is not an arbitrary preference for simple models. It is intended to correct, approximately, for the fact that a fitted model tends to look better on the data used to estimate it than it will on new data.

AIC is an estimate of relative predictive information loss. The true distribution \(g\) is not known, and the criterion generally omits a term shared by all candidate models that depends on \(g\). Thus, an AIC value by itself is not an absolute measure of predictive quality. Its useful information comes from comparing values calculated on a suitable common basis.

Calculating and Comparing AIC

The following examples use hypothetical data and likelihood summaries. In each valid comparison, candidate models are fitted to the same response observations using comparable likelihoods. The reported parameter count \(k\) includes every free parameter estimated as part of that likelihood, not just the regression coefficients.

The arithmetic also shows a useful general comparison formula. For candidate models \(M_1\) and \(M_2\), subtract the first AIC from the second:

The first difference records the change in fit, and the second records the change in the complexity penalty. If model 2 has \(q\) more parameters, its AIC is lower only when its log-likelihood improvement exceeds \(q\): the condition is \(\hat{\ell}_2-\hat{\ell}_1>q\). This is an algebraic consequence of the AIC formula, not a significance test.

What AIC Does and Does Not Say

AIC ranks a specified set of candidate models by estimated predictive information loss. A lower value is preferred by this criterion; it is not a proof that the model is correct. AIC does not report the probability that a model is true, the probability that its predictions will be accurate, or the probability that a particular coefficient is nonzero. It is also not a hypothesis test: it does not supply an \(H_0\), an \(H_a\), or a p-value.

AIC values are relative, so the numerical size of a single value has no universal interpretation. For instance, an AIC of 230 does not mean that a model is twice as informative as one with AIC 460. What matters is the difference among candidates assessed on the same basis. Even then, a smaller value is a ranking, not a promise about realized prediction error in a particular future sample.

Conditions for a Useful Comparison

The interpretation of AIC depends on the comparison being well formed and on the approximation behind the criterion being reasonable. In particular:

• Use a common data basis. Candidate models should ordinarily be fitted to the same response observations with comparable likelihoods. A change in the data or likelihood constants can make raw AIC values incomparable.
• Compare models aimed at the same prediction task. The response and intended target of prediction should match. A numerical ranking cannot make a scientifically irrelevant candidate useful.
• Use a defensible likelihood model. The likelihood must represent the assumed distribution of the observations appropriately. AIC cannot correct a misspecified response distribution or an invalid dependence assumption.
• Count the parameters estimated by the likelihood. The penalty uses \(k\), the number of free parameters, including nuisance parameters estimated as part of the model. It is not automatically equal to the number of named predictors.
• Recognize the approximation’s setting. The usual AIC justification relies on regularity conditions for likelihood estimation, such as identifiable, interior parameters and suitable large-sample behavior. Its estimate is not an exact finite-sample guarantee.

AIC can still rank candidates when none is exactly the true data-generating model. In that setting, the goal is to select the candidate with the least estimated information loss among those considered, not to certify any candidate as correct. The ranking is conditional on the candidate set: an omitted model might have lower expected loss.

Why the Penalty Matters

The maximized log-likelihood is measured on the training observations. Since the model parameters were chosen to fit those observations, its in-sample fit is optimistically favorable. More flexible candidates can exploit random variation as well as genuine structure. The complexity term \(2k\) adjusts the fit score to estimate expected performance on new observations.

This explains why simply choosing the largest maximized log-likelihood is not the same as minimizing expected predictive information loss. It also explains the tradeoff in AIC: a more complex model can be preferred, but only when its improvement in fit is large enough to compensate for the penalty. The precise form and derivation of that penalty require additional likelihood theory; for using AIC here, its role is to provide an approximate optimism correction.

Common Mistakes

• Choosing the largest log-likelihood alone. That rewards fit but ignores the parameters used to achieve it. Compare AIC values, which combine fit and a complexity penalty.
• Treating a low AIC as proof. AIC estimates relative predictive information loss within the candidates; it does not show that a model is true or that its assumptions hold.
• Interpreting an AIC difference as a probability. A difference of 2 units does not mean a 2% chance, nor does it give the probability that either model is correct.
• Comparing values from incompatible analyses. If models use different observations or noncomparable likelihoods, a difference in AIC may reflect the changed basis rather than the models’ relative predictive performance.