Using AIC to Balance Fit and Complexity

Why AIC Balances Fit and Complexity

A more flexible generalized linear model (GLM) can often fit the observed data better simply because it has more parameters to estimate. AIC, the Akaike information criterion, puts a cost on that added flexibility: it rewards a higher maximized likelihood but penalizes the number of estimated parameters. The result is a single quantity for comparing candidate models fitted to the same response data under comparable likelihood conventions.

AIC is a relative selection criterion, not a test that a model is correct and not a direct measure of predictive accuracy on new observations. Its practical question is narrower: among the candidates being compared, which model offers the more favorable balance of in-sample fit and estimated complexity?

The first term measures lack of fit on a deviance-like scale: a larger maximized log-likelihood (less negative, in common GLM examples) makes \(-2\ell(\hat{\theta})\) smaller. The second term is the complexity penalty. Every additional estimated parameter adds 2 to AIC, so a more complex model needs enough likelihood improvement to offset that increase.

Counting Parameters and Comparing Like with Like

The value \(k\) is the number of freely estimated parameters in the likelihood being used. In a GLM, this normally includes the regression coefficients. If a dispersion parameter is estimated as part of the likelihood, it counts too. For standard binomial and Poisson likelihoods, dispersion is usually fixed by the distributional specification rather than estimated as an additional parameter. A known offset is not an estimated parameter; it is a fixed, known part of the model.

Parameter counting must match the likelihood calculation. Do not count an intercept twice, or count a fixed quantity as if it were estimated. Conversely, do not omit an estimated dispersion parameter merely because it is not a regression coefficient. If software reports the degrees of freedom of the fitted likelihood, check what that count includes before using it.

AIC comparisons require comparable likelihoods. In practice, candidate GLMs should be fitted to the same response observations, and the log-likelihoods must use consistent conventions for the response distribution and likelihood constants. A likelihood calculated on one subset of observations cannot be fairly compared with one calculated on a different subset. Nor should values based on incompatible likelihood definitions be treated as directly comparable.

Why the Penalty Is \(2k\)

AIC is motivated by expected information loss. Suppose the data arise from an unknown distribution \(f\), while a candidate model uses a fitted distribution \(g_{\hat{\theta}}\). The Kullback–Leibler divergence from \(f\) to that fitted model is the expected value, under \(f\), of the log-density difference. The term involving \(f\) does not depend on which candidate is selected, so comparing expected information loss amounts to comparing the expected negative log-likelihood of future observations under each fitted model.

The maximized likelihood is evaluated on the same data used to estimate the parameters. That makes it optimistic as an estimate of performance on new data: the fitted parameters have, in effect, adapted to random features of the training sample. Under regularity conditions for maximum-likelihood estimation, the expected in-sample log-likelihood optimism is approximately \(k\) log-likelihood units. Since AIC uses negative twice the log-likelihood, the corresponding correction is approximately \(2k\) deviance units.

This gives the AIC formula and explains its trade-off: improving fit lowers the first term, while estimating more parameters increases the correction for optimism. This is an asymptotic justification, not a guarantee that AIC identifies the true model in every finite sample. The usual interpretation relies on a regular likelihood-based setting, with identifiable parameters and a sufficiently informative sample.

Interpreting AIC Differences

AIC values are most useful as differences within a specified set of candidates. The absolute value alone does not say that a model fits well or poorly. For two models, define the difference as the second model’s AIC minus the smallest AIC in the candidate set. A positive difference means that model has the higher AIC; a difference of zero means a tie on this criterion.

A smaller value identifies the AIC-preferred candidate, but the size and context of the difference matter. A small gap indicates that the criterion gives little separation between the candidates; it does not establish that their practical predictions are identical. AIC also does not establish causation, validate the model assumptions, or account automatically for the consequences of a particular decision. Diagnostics and the purpose of the analysis remain important.

A Practical AIC Comparison

A careful calculation is straightforward, but the comparison is only meaningful if the inputs and interpretation are sound. Use the following sequence:

The common errors are mechanical but consequential: dropping the negative sign in \(-2\ell\), using \(k\) instead of \(2k\), counting parameters inconsistently, or comparing log-likelihoods from different observation sets. A further interpretive error is to call the lowest-AIC model “the true model.” AIC selects relative to the candidates and the likelihood setup; it does not certify assumptions or replace judgment about the modeling objective.