Using AIC for Non-Nested NLME Models

Why Compare Non-Nested Models With AIC?

In pharmacometrics, two plausible NLME models may describe the same concentration data using different structural assumptions. For example, one model might describe disposition with one compartment and first-order absorption, while another uses two disposition compartments and transit absorption. If neither model is obtained from the other by fixing or removing parameters, they are non-nested. A likelihood ratio test based on a nested-model comparison is not the appropriate way to rank them.

Akaike’s information criterion (AIC) provides a different comparison. It balances fit to the observed data against a penalty for the number of estimated parameters. It can compare non-nested models, provided that their maximized likelihoods refer to the same response data and are calculated on a common likelihood basis. A lower AIC indicates a preferred trade-off within the candidate set; it does not establish that the model is true or guarantee better predictions in every setting.

Conditions for a Fair Comparison

Before calculating differences, check what likelihood each model used. In an NLME model, observations from the same subject are generally dependent through shared random effects. The likelihood should reflect the model for the observed data and that dependence. A common basis is the marginal likelihood: the subject-level likelihood contributions account for the random effects by integrating them out. The particular likelihood convention matters less than applying a comparable convention to every model being compared.

• Same response data: Models should be fit to the same response observations, with the same handling of missing observations and censoring. A model fit after excluding difficult observations is not directly comparable to one fit using them.
• Same likelihood basis: The likelihoods must refer to the same response scale and probability construction. For example, do not compare a marginal likelihood for one model with a conditional likelihood for another, or likelihoods based on different response transformations as if their values were directly comparable.
• Comparable estimation: Each likelihood should be maximized successfully under the stated model. Different numerical approximations can affect the reported likelihood, so use the same suitable estimation and likelihood-approximation approach across candidates when possible.
• Correct parameter count: \(K\) includes all estimated parameters contributing to the likelihood, not just the structural fixed effects. Depending on the model, this includes random-effect variance and covariance parameters and residual-error parameters.

AIC comparisons are usually based on maximum likelihood. Restricted maximum likelihood (REML) likelihoods are generally not suitable for comparing models that differ in their fixed-effects structure, because the restricted likelihood depends on that structure. The underlying principle is to compare like with like: a common label such as “likelihood” does not make two differently constructed likelihood values interchangeable.

Why the Penalty Is \(2K\)

AIC is motivated by the expected information loss when a fitted model is used to represent data generated from an unknown process. One way to understand its penalty is to consider the tendency of a fitted model to look better on the data used to estimate it than on new data. Estimation uses the observed data to choose parameter values, so the maximized log likelihood is optimistic as an estimate of future fit.

Under regularity conditions—such as identifiable parameters, an interior maximum, and a sufficiently informative sample—the fitted parameters' estimation error creates an approximate optimism of \(K\) units on the log-likelihood scale. The deviance scale is \(-2\) times the log likelihood, so the corresponding correction is approximately \(2K\). Adding that correction to the in-sample deviance gives the AIC formula. This is an asymptotic justification, not an exact correction for every finite or irregular NLME problem.

For NLME data, repeated measurements within a subject do not automatically provide the same amount of independent information as the same number of observations from unrelated subjects. The likelihood must represent the dependence induced by the random effects. AIC uses the likelihood and parameter count supplied by the fitted model; it does not repair a misspecified dependence structure or poor likelihood approximation.

Differences, Not Just Rankings

AIC values have meaning comparatively, not as a standalone measure of model adequacy. For candidates fit on a common basis, let \(\mathrm{AIC}_{\min}\) be the smallest AIC and define \(\Delta_m=\mathrm{AIC}_m-\mathrm{AIC}_{\min}\). The candidate with \(\Delta_m=0\) is the best-ranked under AIC among those considered. The size of other differences describes their relative AIC support, but no universal cutoff transforms a difference into a test result.

A useful relative-likelihood summary is \(\exp(-\Delta_m/2)\). It compares a candidate's AIC support with that of the minimum-AIC candidate. A normalized set of these values is often called the set of Akaike weights. Both summaries depend on which candidates are included: adding or removing candidates can change normalized weights. Neither is a posterior probability, a p-value, or a measure of the probability that a model is correct.

AIC Is Not a Likelihood Ratio Test

A likelihood ratio test uses the difference in maximized log likelihoods between a larger model and a nested smaller model, with a reference distribution that depends on the test's assumptions. For non-nested models, there is no general nested-model chi-square reference distribution for that difference. AIC instead applies a complexity penalty to each model's maximized likelihood and compares the resulting criteria. It does not require the models to be nested, and it does not produce a null hypothesis, a test statistic with a p-value, or a significance decision.

For two models A and B fit on the same basis, the difference can be written as $$ \mathrm{AIC}_A-\mathrm{AIC}_B = -2(\ell_A-\ell_B)+2(K_A-K_B). $$ The first term reflects relative fit; the second reflects the difference in parameter counts. This identity explains the trade-off but does not turn the AIC difference into a likelihood ratio test.

Diagnostics, Pitfalls, and Limitations

A numerical ranking is useful only after its inputs are checked. Confirm convergence and inspect whether the fitted parameters are plausible, identifiable, and consistent with the intended model. A nominally smaller AIC from a fit at a boundary, a failed optimization, or an unreliable likelihood approximation should not be accepted without investigation. Also examine whether the candidate models describe the data adequately: AIC ranks the candidates supplied, even if all of them fit poorly.

The most common mistake is to calculate AIC for two fits that do not have comparable likelihoods and then interpret the smaller number as a fair model preference. For example, changing the analyzed observations, response scale, or marginal-versus-conditional likelihood construction can change the likelihood value in ways that the AIC penalty does not correct. Verify comparability before ranking. If the likelihood bases differ, the arithmetic may be correct but the comparison is not justified.

AIC is an estimate-based selection criterion, not a guarantee of scientific plausibility, parameter precision, or successful prediction for a new population. It is sensitive to the candidate set and relies on an asymptotic approximation whose justification can be weaker in small or irregular samples. It should be read alongside model diagnostics and the scientific question, rather than treated as a substitute for them. Out-of-sample validation is a distinct way to assess predictive performance, and the validation design must match the prediction target.