Akaike Information Criterion in NLME

Why AIC Balances Fit and Complexity

A more flexible NLME model can often fit the observed data at least as well as a simpler one. But a model that fits its own data unusually well may predict new data less well: some of its apparent fit reflects the particular sample. The Akaike Information Criterion (AIC) addresses this trade-off by subtracting a complexity penalty from the reward for better likelihood fit.

For NLME models, the likelihood ordinarily describes the observed data after the subject-specific random effects have been integrated out. AIC compares maximized likelihoods on a common basis and penalizes the number of estimated model parameters. Its purpose is to compare estimated predictive information loss, not to test whether one model is statistically significantly better.

In an NLME model, population fixed effects and estimated random-effects distribution parameters contribute to \(K\). In the usual marginal likelihood, the individual random effects have been integrated out; their estimated subject-specific values are not treated as additional free parameters in this AIC formula. The details of counting parameters in particular NLME models require care, but the central principle here is that \(K\) counts the free parameters estimated in the likelihood being used.

Why the Penalty Is \(2K\)

AIC's penalty corrects, approximately, for the optimism created by fitting parameters to the same data used to assess fit. The following derivation gives the standard justification under a correctly specified, regular model. It is an asymptotic argument: it describes the expected difference over repeated datasets, rather than an exact correction for every finite sample.

Suppose independent observational units have a correctly specified parametric density \(f(y\mid\theta)\), with true parameter \(\theta_0\) in the interior of the parameter space. Assume the model is identifiable and sufficiently smooth, the maximum likelihood estimator is consistent and asymptotically normal, and the Fisher information is finite and nonsingular. For NLME data, independent units are typically subjects; dependence among observations within a subject is represented in that subject's likelihood. Let \(U\) be the score vector at \(\theta_0\), and let \(I\) be the expected information matrix for the full dataset, that is, the negative expected Hessian of the log likelihood at \(\theta_0\).

A second-order expansion of the log likelihood around \(\theta_0\), together with the likelihood approximation \(\hat{\theta}-\theta_0\approx I^{-1}U\), shows that fitting the parameters raises the training log likelihood above its value at the true parameter by approximately \(\frac{1}{2}U^\mathsf{T}I^{-1}U\). Under correct specification, the information identity gives \(\operatorname{Cov}(U)=I\). Therefore,

The fitted model also loses approximately \(K/2\) units of expected log likelihood when evaluated on independent new data rather than at the true parameter. To see the latter term, average the new-data log likelihood over data generated at \(\theta_0\). Its expected curvature near \(\theta_0\) is \(-I\), so the expected change at \(\hat{\theta}\) is approximately \(-\frac{1}{2}E[(\hat{\theta}-\theta_0)^\mathsf{T}I(\hat{\theta}-\theta_0)]\), which is \(-K/2\). Thus the expected training-to-new-data optimism is approximately \(K/2-(-K/2)=K\) in log-likelihood units. Multiplying the log-likelihood scale by \(-2\) gives the penalty \(2K\) in AIC.

Correct specification matters in this derivation. If the candidate model is misspecified, the score covariance need not equal the expected information. In that case, with \(J=\operatorname{Cov}(U)\), the corresponding asymptotic penalty generally involves \(2\operatorname{tr}(I^{-1}J)\), not \(2K\). The ordinary AIC formula remains widely used for model comparison, including when candidates are approximations, but its standard \(2K\) penalty does not have the same unbiased-optimism justification under misspecification.

Interpreting AIC Differences

AIC values are most useful as differences within a set of candidate models. If \(A_{\min}\) is the smallest AIC in that set, define \(\Delta_i=\mathrm{AIC}_i-A_{\min}\). The model with \(\Delta_i=0\) has the smallest AIC in the set; other models have larger estimated relative information loss according to AIC. Because AIC is on a deviance-like scale, a difference of 6 means a difference of 6 AIC units, not a 6% change in predictive accuracy.

The information-loss interpretation comes from estimating expected Kullback–Leibler discrepancy between a candidate model's predictive distribution and the data-generating distribution. For a given set of candidates, AIC estimates that discrepancy up to terms common to the candidates. It does not directly report an absolute prediction error, and the difference should be interpreted as approximate and relative. AIC alone also does not provide the probability that a particular model is true.

Conditions, Comparability, and Diagnostics

The arithmetic in the AIC formula is simple; establishing that the inputs are meaningful is the harder part. Compare models fitted to the same observed data using likelihoods defined on the same basis. In particular, the likelihood must represent the same outcome and observations, and the treatment of the random effects and likelihood approximation must be consistent. When fixed-effect structures differ, use maximum-likelihood fits for an AIC comparison rather than comparing restricted maximum-likelihood likelihoods, which are not generally comparable across different fixed-effect structures.

A maximized likelihood is only useful if it has actually been maximized. Check convergence, gradients, and whether alternative initial values lead to the same solution. Boundary estimates, flat likelihood profiles, or poor parameter identifiability can make the reported likelihood and parameter count difficult to interpret. AIC cannot repair a failed fit or distinguish a useful improvement from an artifact of numerical optimization. If an approximate likelihood is used, the AIC values inherit that approximation and should be compared consistently.

The standard \(2K\) penalty is an asymptotic, regular-model correction. It can be inadequate when information is limited relative to the number of estimated parameters, and the standard derivation does not cover every boundary or nonidentifiable case. AIC is also not a guarantee of good absolute fit: inspect whether the candidate models adequately describe the concentration-time patterns and variability relevant to the analysis. Choosing the smallest AIC only ranks the candidates supplied.

What AIC Is Not

AIC is not a likelihood ratio test. A likelihood ratio test compares nested models using a likelihood-ratio statistic and a reference distribution under specified null hypotheses and conditions. AIC instead combines maximized log likelihood with a parameter penalty to estimate relative information loss. Consequently, an AIC difference does not have a null distribution or a p-value by itself. Nor should a small difference be described as proof that two models are equivalent: it says only that their AIC values are close on this criterion.

A common mistake is to say, “Model B has a lower AIC, so its extra random effect is statistically significant.” That conclusion confuses a selection criterion with a hypothesis test. The correction is to say, “Model B has the lower AIC among the candidates, indicating lower estimated relative information loss under this comparison.” If the scientific question is whether a specified parameter or structure differs from a null value, use an appropriate inferential procedure and its assumptions rather than interpreting AIC as a test.