Common Misinterpretations of NLME Selection Criteria

What Selection Criteria Can and Cannot Tell Us

Information criteria and likelihood ratio tests answer limited questions. AIC compares the relative fit and complexity of candidate models; a likelihood ratio test assesses whether a specified restriction is compatible with the data under a set of assumptions. Neither result, by itself, shows that a model is the true data-generating mechanism. Nor does either establish that a model will predict well in a new setting.

These distinctions matter in nonlinear mixed-effects (NLME) analysis, where models can differ in their structural assumptions, random-effects distributions, residual error models, and number of estimated parameters. A numerical ranking can be useful, but its meaning depends on how the models were fitted and on what conclusion is being drawn from the ranking.

Definition: For a fitted model \(i\), the Akaike information criterion is \(AIC_i=-2\ell_i+2K_i\), where \(\ell_i\) is the maximized log likelihood and \(K_i\) is the number of estimated parameters counted under the stated likelihood formulation. Within a comparable candidate set, a smaller AIC indicates a more favorable estimated trade-off between fit and complexity. It is a relative selection criterion, not a test of whether a model is true.

AIC comparisons require care. The models should be fitted to the same response data, with likelihoods defined on a comparable basis. In NLME work, the likelihood method and any approximation used to obtain the likelihood should also be comparable. AIC values from different response data, incompatible likelihood definitions, or materially different likelihood approximations should not be treated as a straightforward ranking.

Worked Examples: Ranking Is Not Truth

Worked Example: A Lower AIC Does Not Prove a Model Is True

Suppose two candidate NLME models are fitted by maximum likelihood to the same hypothetical concentration observations. Model A has maximized log likelihood \(\ell_A=-120.4\) and \(K_A=6\) estimated parameters. Model B has \(\ell_B=-116.9\) and \(K_B=8\). Assume the likelihoods are comparable and the parameter counts are appropriate for these fits.

For Model A,

$$ AIC_A=-2(-120.4)+2(6)=240.8+12=252.8. $$

For Model B,

$$ AIC_B=-2(-116.9)+2(8)=233.8+16=249.8. $$

Model B has the smaller AIC by \(252.8-249.8=3.0\). The appropriate conclusion is that Model B is preferred by AIC among these two candidates, given these data and this candidate set. The result does not show that Model B is true, that its assumptions are correct, or that it will predict accurately in new data.

AIC is motivated by an estimate of relative expected information loss when a fitted model is used to represent the data-generating process. That motivation makes it useful for ranking candidate models, including non-nested ones, but does not turn the ranking into a hypothesis test. AIC does not provide a sampling distribution for the difference that can be used as a p-value.

An AIC Difference Is Not a P-Value

For a candidate model \(i\), define its AIC difference from the best-ranked model as \(\Delta_i=AIC_i-AIC_{\min}\). A common transformation of this difference is its relative Akaike support, \(\exp(-\Delta_i/2)\). The transformation is useful for describing relative support within the specified candidate set; it is not a tail probability under a null hypothesis.

Key distinction: A p-value is calculated from a test statistic and its null distribution, under a specified null hypothesis. An AIC difference is a difference between penalized fit criteria. It has no p-value interpretation simply because it is numerical or can be transformed using an exponential.

The relationship between AIC differences and relative support follows directly from the criterion. Let \(m\) denote the model with the smallest AIC. Then

\Delta_i=AIC_i-AIC_m =-2(\ell_i-\ell_m)+2(K_i-K_m).

Rearranging and exponentiating gives

\exp(-\Delta_i/2) =\exp\big((\ell_i-\ell_m)-(K_i-K_m)\big).

This is a relative measure derived from the AIC ranking and its complexity penalty. It is not calculated from a null distribution, so it cannot be reported as a p-value. If normalized across the candidate set, the resulting Akaike weights sum to one, but they remain relative weights conditional on that set and its AIC values. They are not posterior probabilities that the models are true.

Worked Example: Interpreting a Small AIC Difference

Consider two hypothetical NLME models fitted to the same observations with comparable likelihoods. Model C has \(AIC_C=180.4\), and Model D has \(AIC_D=179.6\). Model D is the AIC minimum.

The difference for Model C is

\Delta_C=180.4-179.6=0.8.

Its relative Akaike support compared with Model D is

\exp(-\Delta_C/2)=\exp(-0.8/2)=\exp(-0.4)\approx0.6703.

Thus, under this AIC-based relative-support calculation, Model C has about \(0.6703\) times the support of Model D in this two-model comparison. If normalized into weights, Model D's weight is \(1/(1+0.6703)\approx0.5987\), and Model C's is \(0.6703/(1+0.6703)\approx0.4013\). These are not probabilities that either model is true, and \(0.8\) is not a p-value. The weights depend on which candidate models were included; adding or removing candidates can change them.

A Significant Likelihood Ratio Test Is Not a Prediction Check

For nested models, a likelihood ratio test (LRT) compares a restricted model with a less restricted model that contains it. The test statistic is twice the difference in maximized log likelihoods. Under regularity conditions and the null hypothesis, it has an approximate chi-square distribution, with degrees of freedom equal to the number of independent restrictions imposed by the null.

Definition: For nested models fitted to the same data by maximum likelihood, the likelihood ratio statistic is \(LR=2(\ell_{\mathrm{larger}}-\ell_{\mathrm{smaller}})\). Under the null and standard regularity conditions, \(LR\) is approximately distributed as \(\chi^2_{df}\), where \(df\) is the difference in the number of freely estimated parameters. A small p-value is evidence against the specified restrictions; it is not a measure of predictive accuracy.

The usual chi-square reference requires, among other conditions, nested models, a correctly specified likelihood for the test, and regular parameters under the null. Parameters on a boundary, such as a random-effect variance tested against zero, can require a different reference distribution. The earlier material on boundary tests explains why the usual chi-square calibration can fail in that case. The LRT also tests a particular restriction; it does not compare every possible model or verify all assumptions of the larger model.

Worked Example: Strong Evidence Against a Restricted Model

In a hypothetical analysis, a smaller NLME model has maximized log likelihood \(\ell_0=-205\). A nested model adds two fixed-effect parameters, has \(\ell_1=-190\), and is fitted to the same data by maximum likelihood. Assume the two added parameters are interior parameters and the other regularity conditions for the usual LRT are reasonable.

The likelihood ratio statistic is

LR=2(\ell_1-\ell_0)=2[-190-(-205)]=2(15)=30.

There are two added parameters, so \(df=2\). For this reference distribution, the upper-tail probability is

P(\chi^2_2\geq30)=\exp(-30/2)=\exp(-15)\approx3.06\times10^{-7}.

At conventional significance levels, this is strong evidence against the null restrictions represented by the smaller model. It does not establish that the larger model predicts well. For example, the larger model could still have biased predictions, an inadequate residual error model, or poor performance in a population unlike the one represented by these observations. Those questions require relevant predictive assessment; they are not answered by the LRT p-value.

Statistical significance can also depend on information in the sample: with sufficiently informative data, a modest departure from a restriction may produce a small p-value. Conversely, failure to reject a restriction does not prove it is correct. The test is evidence about the restrictions under its assumptions, not a certificate of model adequacy.

Keep the Questions Separate

When interpreting a selection result, first identify the question the method actually answers. AIC ranks candidate models using fit and a complexity penalty. An LRT evaluates a specified nested restriction under a null distribution. Predictive adequacy concerns how well model predictions perform for the intended use. These questions can inform one another, but their answers are not interchangeable.

Result	What it supports	What it does not establish
Lower AIC	Relative preference within a comparable candidate set	Truth, correct assumptions, or good prediction
Small AIC difference	Similar AIC ranking, with relative support summarized if desired	A p-value or the probability a model is true
Small LRT p-value	Evidence against the tested restrictions, given the test assumptions	Predictive adequacy or correctness of every feature in the larger model

Other approaches answer different questions. BIC applies a different complexity penalty and can rank the same candidate set differently; it does not resolve whether any candidate is true. An LRT is available only for an appropriate nested comparison and tests the restrictions represented by that comparison. Predictive diagnostics or evaluation on data not used to fit the model are needed to assess predictive performance. Each approach has its own assumptions and limitations, so a numerical result should be reported with the question it addresses.

Diagnostics and Common Pitfalls

Before drawing a conclusion, check that the candidate set and comparison are coherent. Verify that models use the same observations and comparable likelihoods, and that the parameter counts and fitting method match the intended criterion. For an LRT, verify nesting, the null restrictions, the degrees of freedom, and whether any tested parameter lies on a boundary. Inspect convergence and whether the fitted estimates are plausible; a numerical criterion from an unreliable fit is not a sound basis for selection.

Wrong: “Model B has lower AIC, so it is the true model.” Correction: Say that B is preferred by AIC among the stated candidates; assess model assumptions and other evidence separately.
Wrong: “The AIC difference is 3, so the p-value is 0.03.” Correction: An AIC difference is not a test statistic with a null distribution. Report the difference as an AIC difference, not as a p-value.
Wrong: “The LRT is significant, so the larger model predicts adequately.” Correction: The LRT provides evidence against the tested restrictions under its assumptions. Prediction requires its own assessment.
Wrong: “A non-significant LRT proves the smaller model is correct.” Correction: Failure to reject is not proof of the null; the test may have limited information or may not detect the departure of interest.

Key takeaway: Treat AIC as a relative ranking, an AIC difference as a criterion difference rather than a p-value, and an LRT as evidence about specified restrictions rather than a prediction check. State the scope and assumptions of each conclusion.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

Two comparable candidate models have AIC values 310 and 314. What is justified by this comparison, and what is not?
A model has an AIC difference of 2 from the best model. Is 2 its p-value? Explain.
A regular LRT compares nested models differing by three freely estimated parameters and gives \(LR=12\). What is the reference distribution under the null, and what does a small p-value address?
Why does a significant LRT not establish that the larger model predicts adequately?

Solutions. (1) The model with AIC 310 is preferred by AIC among these candidates, assuming comparable likelihoods and fitting data. This does not prove it is true or predictively adequate. (2) No. The value 2 is an AIC difference; it is not generated from a null distribution and is not a p-value. (3) Under the stated regularity conditions, the reference distribution is \(\chi^2_3\). A small p-value is evidence against the three restrictions imposed by the smaller model, not evidence about prediction quality. (4) The LRT assesses specified restrictions using the observed-data likelihood. Predictive adequacy concerns prediction performance and must be evaluated separately.