AIC as a Predictive Model Selection Criterion

Why Penalize In-Sample Fit?

A more flexible ratemaking model can usually fit its observed data at least as well as a simpler model. For example, adding a territory interaction to a frequency GLM cannot reduce the maximized training likelihood when the simpler model is nested within it. But some of that improved fit may reflect random features of the training data rather than patterns that will recur in future policies. A predictive selection criterion should therefore reward fit while accounting for the optimism created by estimating parameters.

The Akaike information criterion, or AIC, makes this adjustment using the number of freely estimated parameters. It is a relative criterion: among comparable candidate models, the model with the smaller AIC is preferred as an estimate of predictive performance. The criterion does not establish that a model is correct, and it does not replace validation on future or held-out data.

The \(-2\ell(\hat{\theta})\) term rewards fit: a larger maximized log likelihood gives a smaller fit term. The \(2k\) term penalizes complexity. A model with more parameters can still have a lower AIC, but only if its improvement in likelihood is large enough to offset that penalty. When both models use the same observations and the same likelihood definition, AIC differences can be calculated directly; common constants in the likelihood then cancel in the difference.

Why the Penalty Is \(2k\)

AIC is motivated by estimating expected predictive information loss, commonly expressed through expected log likelihood or, equivalently, expected deviance. The fitted model has an advantage on the data used to estimate it: its parameters were chosen to fit that particular sample. The derivation below explains why the usual correction is \(2k\), rather than \(k\).

Assume a regular, correctly specified likelihood model with \(k\) identifiable free parameters, an interior true parameter value \(\theta_0\), and finite, nonsingular Fisher information. Assume independent observations and a sufficiently large sample for a local quadratic approximation to the log likelihood. Write \(\ell(\theta)\) for the training log likelihood, \(U\) for its score at \(\theta_0\), and \(I\) for its expected information matrix. Near \(\theta_0\), the log likelihood is approximately

Maximizing this quadratic approximation gives \(\hat{h}\approx I^{-1}U\). Substituting it into the approximation, the training log-likelihood gain over the true parameter is

Under the stated conditions, the score has mean zero and covariance approximately \(I\). Thus the expected quadratic form is approximately its dimension, \(k\), and the expected training log-likelihood gain is approximately \(k/2\):

Now consider an independent test sample of the same size, drawn from the same process. Conditional on the fitted training estimate, its expected log likelihood is maximized at \(\theta_0\). The local expected curvature of the test log likelihood is approximately \(-I\), so using \(\hat{\theta}\) rather than \(\theta_0\) reduces expected test log likelihood by approximately \(\frac{1}{2}E[\hat{h}^\mathsf{T}I\hat{h}]\). Since \(\hat{h}\approx I^{-1}U\), that expected reduction is also approximately \(k/2\). The training fit gains about \(k/2\), while its expected test log likelihood loses about \(k/2\), relative to the true-parameter benchmark. Therefore, for independent test data of the same size, the expected difference between training and test log likelihood is approximately \(k\).

An estimate of expected test log likelihood is consequently \(\ell(\hat{\theta})-k\). Multiplying its negative by two gives AIC:

This is an asymptotic justification, not an exact finite-sample identity. The standard correction can be unreliable for small samples, boundary estimates, nonidentifiable or singular models, or settings where the likelihood assumptions do not hold. AIC is also tied to the likelihood and predictive target being used; it should not be treated as a universal score for any model output.

Calculating and Comparing AIC

For each candidate, obtain the maximized log likelihood from the fitted likelihood model and count its estimated parameters. Use the full log likelihood, including terms that may be constant with respect to the parameters, when comparing different likelihood families for the same response data. For a Poisson count model, for example, the likelihood includes the \(-\log(y!)\) terms. Omitting such terms may not change parameter estimates within one model, but can make AIC comparisons across likelihood specifications invalid.

In a GLM, a fitted intercept and each independently estimated regression coefficient count toward \(k\). An exposure offset is specified rather than estimated, so it does not count. A dispersion parameter estimated as part of a likelihood does count. A model’s coefficient count is determined by its actual fitted design matrix and constraints, not simply by the number of variable names in its formula.

Interpreting AIC Differences Carefully

AIC is useful because it puts the fit-complexity tradeoff on a single scale. It is not a test of whether an added term is statistically significant, a measure of calibration, or a guarantee of stable indicated rates. A low AIC says that a model has a better estimated predictive information tradeoff than the particular alternatives considered. It does not say that predictions are well calibrated in every segment or that the chosen specification is acceptable for implementation.

The penalty depends on how many parameters are estimated, not on how many variables appear in a model description. A categorical factor with several levels may require multiple coefficients, while an offset contributes none if fixed. Constraints such as reference levels affect the number of free coefficients. When fitting software reports likelihood and parameter counts, confirm that the count includes every estimated likelihood parameter and that the likelihoods are on a comparable basis.

AIC comparisons require the same observed response data and a common likelihood basis. If one model is fit to a different set of records, a different response definition, or a different aggregation without a likelihood that supports a valid comparison, the numerical AIC ranking may not answer the intended question. In particular, likelihood constants can matter when comparing different distributions, so use full likelihoods rather than arbitrary software summaries. A quasi-likelihood fit without a fully specified likelihood does not automatically supply an ordinary AIC.

AIC is designed to estimate relative predictive information loss, not to reward simplicity for its own sake. A more complex candidate can be preferable when it produces enough improvement in likelihood. Conversely, a modestly higher likelihood does not necessarily justify extra coefficients. Since the selection itself can add uncertainty, use AIC alongside sound model diagnostics and genuinely out-of-sample assessment where available. This interpretation is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.