Information Criteria From Approximate NLME Likelihoods

Why the reported likelihood needs scrutiny

AIC, BIC, and likelihood-ratio comparisons are calculated from fitted likelihoods. In an NLME model, the likelihood for the observed data involves integrating over each subject’s random effects. That integral is often not available in closed form, so software may report a likelihood obtained through an approximation. As a result, an objective value is not interpretable for model comparison merely because it is labelled “-2 log likelihood” or “objective function value.”

Two separate issues matter. Likelihood approximation concerns how the marginal likelihood integral is evaluated. Optimization concerns whether the fitting algorithm has found a satisfactory optimum for the chosen likelihood approximation. A model can be optimized well under a rough approximation, or poorly under a more accurate one. These are different problems and require different checks.

Definition: For subject $i$, let $y_i$ denote that subject’s observed data, $b_i$ the random effects, $f(y_i\mid b_i,\theta)$ the conditional data density, and $g(b_i\mid\theta)$ the random-effects density. The subject-specific marginal likelihood contribution is $$ L_i(\theta)=\int f(y_i\mid b_i,\theta)g(b_i\mid\theta)\,db_i. $$ For independent subjects, the marginal log likelihood is $\ell(\theta)=\sum_{i=1}^{N}\log L_i(\theta)$. In practice, a reported value may instead use an approximation $\tilde{\ell}(\theta)$ to this log likelihood. An objective value on the usual deviance scale is $Q=-2\tilde{\ell}(\hat{\theta})$, where $\hat{\theta}$ is the fitted parameter estimate under that procedure.

What must match for comparisons?

The familiar formulas use the maximized log likelihood and the number of estimated parameters $k$:

\mathrm{AIC}=-2\hat{\ell}+2k,\qquad \mathrm{BIC}=-2\hat{\ell}+k\log(n).

For calculations based on an approximate likelihood, $\hat{\ell}$ is replaced by the reported approximate maximized value. Smaller AIC or BIC indicates the preferred relative trade-off among the candidate models, provided the values were constructed comparably. The likelihood-ratio statistic for a nested pair is $2(\hat{\ell}_{1}-\hat{\ell}_{0})$, with model 1 the larger model. If the software reports $Q=-2\hat{\ell}$, the same statistic is $Q_0-Q_1$.

A meaningful comparison generally requires the same observed data and response definition, compatible likelihood targets and constants, the same estimation framework, and a consistent treatment of missing records and likelihood contributions. The models should use the same likelihood approximation and settings, and should each be adequately optimized. For BIC, use a consistent definition of $n$ for the comparison; the earlier material on defining sample size for BIC in NLME is relevant here. For an ordinary likelihood-ratio test, the models must additionally be nested and the test’s regularity conditions must be appropriate. Boundary cases require the special treatment discussed in the earlier material on boundary tests.

Common approximation approaches include first-order linearization, conditional first-order methods such as FOCE, Laplace approximation, and Gaussian quadrature. They can differ in how they represent or integrate the subject-level random-effects contributions. Their relative accuracy depends on the model and data; no single label guarantees that all models have negligible approximation error. The particular algorithm used to estimate parameters is another matter: for example, a stochastic estimation algorithm may be followed by a separate calculation of the reported likelihood. Check what the software actually used to produce the objective value rather than inferring it from the estimation algorithm’s name.

Worked Example: AIC and BIC From a Shared Approximation

Suppose two hypothetical nested models are fitted to the same concentration data from 120 subjects, using the same Laplace likelihood approximation and the same fitting settings. The smaller model has $Q_0=512.4$ and $k_0=6$; the larger has $Q_1=506.9$ and $k_1=8$. Assume both fits are adequately optimized. For this comparison, take $n=120$ subjects in the BIC formula.

AIC. Because $Q=-2\hat{\ell}$, the smaller model has $\mathrm{AIC}_0=512.4+2(6)=524.4$. The larger has $\mathrm{AIC}_1=506.9+2(8)=522.9$. Its AIC is lower by $524.4-522.9=1.5$, so AIC modestly favors the larger model among these two candidates.

BIC. Since $\log(120)\approx4.787$, $\mathrm{BIC}_0=512.4+6(4.787)=541.1$, and $\mathrm{BIC}_1=506.9+8(4.787)=545.2$. BIC favors the smaller model by about $545.2-541.1=4.1$. This difference in preference reflects the criteria’s different complexity penalties, not a difference in the fitted data.

Likelihood-ratio statistic. The objective-value difference is $Q_0-Q_1=512.4-506.9=5.5$. With two added parameters, the usual regular reference is chi-square with $df=2$. Its upper-tail p-value is approximately $P(\chi^2_2\ge5.5)=\exp(-5.5/2)=0.0639$. At $\alpha=0.05$, this does not meet the rejection threshold. That test conclusion is conditional on the applicable regularity assumptions; it is not a claim that the smaller model is true.

Why approximation errors can change a comparison

Let the approximation error for model $m$ at its fitted value be $\epsilon_m=\tilde{\ell}_m-\hat{\ell}_m$, where $\hat{\ell}_m$ denotes the corresponding exact maximized marginal log likelihood. For a fixed parameter count and sample-size convention, replacing the exact log likelihood with the approximation changes each criterion directly. In particular:

\widetilde{\mathrm{AIC}}_m-\mathrm{AIC}_m=-2\epsilon_m,\qquad \widetilde{\mathrm{BIC}}_m-\mathrm{BIC}_m=-2\epsilon_m.

For two models, the error in the approximate likelihood-ratio statistic is:

\widetilde{\Lambda}-\Lambda =2\left[(\tilde{\ell}_1-\tilde{\ell}_0)-(\hat{\ell}_1-\hat{\ell}_0)\right] =2(\epsilon_1-\epsilon_0).

Thus, an approximation error that is similar for both models may partly cancel in a difference, but cancellation is not guaranteed. If approximation error differs across models, it can alter the ranking or the likelihood-ratio statistic. A shared approximation method makes a comparison more coherent; it does not prove that approximation error is negligible or identical across models.

Worked Example: Why Cross-Approximation Rankings Are Not a Fair Contest

Consider two hypothetical models, A with $k_A=7$ and B with $k_B=8$, fitted to the same data. Under a FOCE calculation, suppose their reported objectives are $Q_A=486.0$ and $Q_B=488.0$. Under a Laplace calculation, suppose the reported objectives are $Q_A=480.2$ and $Q_B=477.0$. These values are illustrative, not results from a real study.

Within the FOCE calculation, $\mathrm{AIC}_A=486.0+14=500.0$ and $\mathrm{AIC}_B=488.0+16=504.0$, so that calculation favors A. Within the Laplace calculation, $\mathrm{AIC}_A=480.2+14=494.2$, while $\mathrm{AIC}_B=477.0+16=493.0$, so that calculation favors B.

The reversal is a warning that conclusions may depend on the likelihood calculation. It does not establish that either approximation is correct, nor that one model is scientifically better. In particular, it would be invalid to take A’s FOCE objective and B’s Laplace objective, compute one AIC comparison from that pair, and call it a common-scale comparison. First verify that the candidate models have been evaluated using the same likelihood construction and settings; if different valid approximations produce different within-method rankings, investigate the sensitivity rather than selecting whichever result is preferred.

Optimization errors are a separate failure mode

For a fixed approximation and model, maximization should give the best attainable approximate log likelihood; equivalently, it should minimize the corresponding objective $Q$. A fit that stops at a suboptimal point can report an objective that is too high. This can make a model look worse in AIC or BIC and can shrink a likelihood-ratio difference if the larger model is the one that failed to optimize. Convergence status alone is not enough: a numerical success message does not establish that the fitted solution is stable or globally satisfactory.

Worked Example: A Poorly Optimized Larger Model

Suppose, hypothetically, that a smaller model has $Q_0=620.0$, $k_0=6$, and a well-optimized fit under a fixed Laplace calculation. The larger nested model has $k_1=8$. One run reports $Q_1=617.0$, but diagnostic checks and a restart from different initial estimates find a better solution with $Q_1=613.2$. Assume the improved fit is adequately optimized and all regular likelihood-ratio conditions hold.

Using the first run, the statistic is $620.0-617.0=3.0$. With $df=2$, the upper-tail p-value is $\exp(-3.0/2)=0.2231$. The AIC values are $620.0+12=632.0$ for the smaller model and $617.0+16=633.0$ for the larger, favoring the smaller model.

Using the improved run, the statistic is $620.0-613.2=6.8$, with p-value $\exp(-6.8/2)=0.0334$. The larger model’s AIC is $613.2+16=629.2$, now 2.8 lower than the smaller model’s AIC. The change in conclusion arose from correcting the optimization, not from changing the model or approximation. The first run should not be retained simply because it converged first.

Diagnostics, alternatives, and limitations

When a comparison looks surprising or unstable, investigate both the objective calculation and the optimization. Useful checks include confirming the approximation and likelihood settings for every fit; verifying the same records, response scale, and likelihood constants; checking that the stated parameter counts match the fitted models; and repeating fits from dispersed starting values. Inspect convergence messages, gradients or other optimality diagnostics when available, parameter estimates, standard errors, and whether estimates sit at constraints. Compare objective values across successful restarts. Materially different solutions or objective values are evidence that the reported comparison may not be reliable.

A more accurate integration method, such as sufficiently fine adaptive quadrature when computationally feasible, can be used as a sensitivity check against a faster approximation. Agreement across methods is reassuring but not proof of exactness; disagreement calls for examination of integration accuracy, model behavior, and optimization. The main alternative to relying on an approximate marginal likelihood for selection is not to mix incompatible objective values: it is to obtain a more reliable likelihood calculation, or to use a different assessment strategy suited to the scientific question. AIC, BIC, and likelihood-ratio tests answer different questions, and none repairs a poorly evaluated likelihood.

Approximation-based criteria remain limited by the approximation, the model assumptions, and the candidate set. A small AIC difference does not certify a meaningful improvement, and a likelihood-ratio p-value computed from an approximate likelihood may not have its nominal calibration if approximation or regularity conditions are poor. These criteria do not replace checks of parameter plausibility, residual behavior, or predictive adequacy. As discussed in the earlier material on why AIC and BIC may choose different NLME models, their penalties can also favor different candidates even when based on the same likelihood.

Establish a shared basis.
Confirm the data, likelihood target, approximation, estimation settings, constants, and BIC sample-size convention are consistent across the models being compared.

Establish adequate fits.
Review convergence and optimality diagnostics, rerun from alternative starting values, and investigate materially different solutions or boundary estimates.

Calculate and qualify.
Compute criteria or test statistics from the comparable objective values, then describe conclusions as conditional on the likelihood approximation and the method’s assumptions.

Common pitfall: “Both outputs are called -2 log likelihood, so their difference is a valid likelihood-ratio statistic.” The label alone does not establish comparability. If the models use different approximations, different records, or different likelihood definitions, their numerical difference need not represent a valid likelihood-ratio comparison. Match the likelihood construction first, and then verify optimization and test assumptions.

Key takeaway: Information criteria and likelihood-ratio statistics inherit the quality and comparability of the likelihood values used to calculate them. Use a common likelihood approximation and consistent fitting conditions, verify that each fit is adequately optimized, and report sensitivity when reasonable calculations disagree.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Two models use the same data and have the same parameter count, but one objective is from FOCE and the other from Laplace. Is their AIC difference automatically meaningful? Explain.
If an approximate log likelihood for one model differs from its exact maximized value by $\epsilon$, what is the resulting AIC error, with parameter count held fixed?
Two nested models have comparable objective values $Q_0=400$ and $Q_1=394$, with a two-parameter difference. What is the likelihood-ratio statistic? Under the usual regular chi-square reference, what is its p-value?
A restarted fit of the larger model gives a lower objective than the first run under identical settings. What should be checked before reporting the model comparison?
Why can a likelihood-ratio statistic change when the same approximation method is used for both models?

Solutions. (1) No. AIC values should be based on compatible likelihood calculations; the common label does not make unlike approximations comparable. (2) The AIC error is $-2\epsilon$, because AIC contains $-2\hat{\ell}$ and the parameter penalty is unchanged. (3) The statistic is $Q_0-Q_1=400-394=6$. For $df=2$, the upper-tail p-value is $\exp(-6/2)=0.0498$, subject to the regularity and likelihood assumptions. (4) Check convergence and optimality diagnostics, initial-value sensitivity, parameter estimates, and whether the lower objective represents a stable, adequately optimized solution; use the reliable fit for comparison. (5) The approximation error need not be equal for the two models. The error in the statistic is $2(\epsilon_1-\epsilon_0)$, which is nonzero whenever the model-specific errors differ.