Comparing Non-Nested Candidate Models

Why Non-Nested Comparisons Need a Different Approach

Pharmacometric model development can produce plausible alternatives that are not nested: for example, one model may represent absorption with a transit-compartment structure while another uses a different absorption function. Neither candidate is a special case of the other simply by fixing one or more parameters. A likelihood ratio test is therefore generally not the right way to compare them. Instead, criteria such as AIC and BIC combine model fit with a penalty for the number of estimated parameters.

These criteria do not identify a model as scientifically true. They summarize evidence within a defined set of candidates and under particular assumptions. Their usefulness depends first on a fair comparison: the models must be fitted to comparable data with likelihoods that represent the same observations and response information.

Definition: Two models are non-nested when neither can be obtained from the other by imposing restrictions on its parameters. A likelihood-based comparison of such candidates uses their maximized likelihoods and a stated criterion, such as AIC or BIC; it does not treat the difference in their objective functions as a likelihood ratio test statistic.

What Must Be Comparable

For candidate $j$, let $\ell_j(\hat{\theta}_j)$ denote the maximized log likelihood, and let $k_j$ be the number of estimated parameters counted by the chosen criterion. A lower value of $-2\ell_j(\hat{\theta}_j)$ indicates a higher maximized likelihood. Before comparing criteria, check that the candidates use the same analysis records, response variable, likelihood target, and data handling, including treatment of missing observations and censoring. The model definitions and parameter counts should also be documented.

AIC and BIC are based on the likelihood, not just a software-reported number called an objective function. In many settings the objective function is $-2\ell$, possibly with a constant or other convention. Confirm what the reported quantity represents. If the candidates' objective values are not on the same likelihood scale, they cannot be inserted into a criterion comparison as though they were directly comparable.

Definition: For a candidate with maximized log likelihood $\ell(\hat{\theta})$, $k$ estimated parameters, and sample size $n$, the Akaike information criterion and Bayesian information criterion are $$ \mathrm{AIC}=-2\ell(\hat{\theta})+2k,\qquad \mathrm{BIC}=-2\ell(\hat{\theta})+k\log(n). $$ When the objective function is exactly $-2\ell(\hat{\theta})$, it can replace that term in these formulas. Lower values are preferred within the candidate set being compared.

AIC is motivated by estimating relative expected information loss when a model is used to represent the data-generating process. BIC is a large-sample criterion with a stronger penalty as $n$ grows; under its assumptions, it is associated with selecting among a set of candidate models. Neither criterion is a test with a conventional $p$-value. In pharmacometric analyses, the meaning of $n$ can require care: repeated records within an individual are not necessarily independent observations. State the sample-size convention used, especially for BIC, and apply it consistently.

Why the Penalty Matters

The maximized likelihood cannot decrease when a model is enlarged by adding freely estimated parameters, so a more complex model can improve fit even when the improvement is small. AIC and BIC offset that improvement with a complexity penalty. For two candidates, the criterion difference reflects both terms:

\mathrm{AIC}_2-\mathrm{AIC}_1 = \bigl[-2\ell_2(\hat{\theta}_2)+2\ell_1(\hat{\theta}_1)\bigr] +2(k_2-k_1).

This is a useful justification for not ranking models by objective function alone: the criterion difference is not merely the fit difference. For BIC the corresponding penalty difference is $(k_2-k_1)\log(n)$. In either case, a preferred value is a relative ranking under the chosen criterion, not proof that the candidate's structure or assumptions are correct.

Worked Example: Two Non-Nested Absorption Candidates

Suppose two hypothetical absorption candidates are fitted to the same 240 individuals and the same concentration observations, using comparable likelihoods. Candidate A has objective function $-2\ell=1204.6$ and $k=8$; candidate B has $-2\ell=1200.8$ and $k=10$. Assume these reported objective values are exactly $-2\ell$.

For A, $\mathrm{AIC}=1204.6+2(8)=1220.6$. For B, $\mathrm{AIC}=1200.8+2(10)=1220.8$. A has the lower AIC by $1220.8-1220.6=0.2$, a very small difference; this ranking alone gives little basis to claim a meaningful separation between the candidates.

For BIC, $\log(240)=5.48064$. Thus A has $\mathrm{BIC}=1204.6+8(5.48064)=1248.445$, and B has $\mathrm{BIC}=1200.8+10(5.48064)=1255.606$. BIC also ranks A lower, by about $7.161$. The criteria differ in the strength of their complexity penalties, so the size of their difference need not match.

Candidate B has the better maximized likelihood because its objective value is lower by $1204.6-1200.8=3.8$. That fact does not make 3.8 a likelihood ratio test statistic: the candidates are non-nested, and the criterion comparisons include penalties. The defensible summary is that A is slightly preferred by AIC and more clearly preferred by BIC under the stated comparison, while neither result establishes that A is the true absorption model.

Differences and Akaike Weights

For a candidate set of $R$ models, define $\Delta_j=\mathrm{AIC}_j-\min_r(\mathrm{AIC}_r)$. The model with the minimum AIC has $\Delta=0$. Akaike weights are a normalized summary of relative AIC support:

w_j=\frac{\exp(-\Delta_j/2)} {\sum_{r=1}^{R}\exp(-\Delta_r/2)}.

The weights sum to 1 and depend on which candidates are included. They are not probabilities that the respective models are true, nor are they posterior probabilities. They describe relative support from AIC within the specified candidate set; omitted candidates, poor model assumptions, or a different candidate set can change the summary.

Worked Example: Calculating Relative AIC Support

Consider three hypothetical, comparable candidate models with AIC values 1450, 1452, and 1457. Their differences from the minimum are $0$, $2$, and $7$. The unnormalized values are $1$, $\exp(-2/2)=\exp(-1)=0.367879$, and $\exp(-7/2)=\exp(-3.5)=0.030197$. Their sum is $1.398076$.

Dividing each value by that sum gives weights $1/1.398076=0.7153$, $0.367879/1.398076=0.2631$, and $0.030197/1.398076=0.0216$. The rounded weights sum to $1.0000$. Within this three-model set, the first candidate has the largest relative AIC support. It is incorrect to say there is a 71.53% probability that it is the true model.

A transparent report would give the criterion, each candidate's AIC and $\Delta$AIC, and—if useful—the weights, alongside the candidate definitions. Reporting only the winning model hides how close the alternatives were.

Small-Sample Correction and a Second Criterion

When the sample is not large relative to the number of fitted parameters, the small-sample corrected AIC, AICc, is often considered:

\mathrm{AICc}=\mathrm{AIC}+\frac{2k(k+1)}{n-k-1},\qquad n>k+1.

This correction grows as $k$ approaches $n-1$. As with BIC, the sample-size definition and parameter count must suit the likelihood and modeling setting. In complex hierarchical models, determining a defensible $k$ or effective $n$ may be difficult; do not present AICc or BIC as mechanically decisive when these inputs are ambiguous.

Worked Example: Checking the Small-Sample Penalty

For a hypothetical comparison with $n=150$, candidate A has $-2\ell=900$ and $k=6$; candidate B has $-2\ell=898$ and $k=8$. Their AIC values are $900+12=912$ and $898+16=914$, respectively, so AIC favors A by 2 points.

Both candidates satisfy $n>k+1$. For A, the AICc correction is $2(6)(7)/(150-6-1)=84/143=0.5874$, giving $\mathrm{AICc}=912.5874$. For B it is $2(8)(9)/(150-8-1)=144/141=1.0213$, giving $\mathrm{AICc}=915.0213$. A remains lower, now by $2.434$. The correction does not automatically change the ranking, but it can matter; it should be reported with the sample-size and parameter-count conventions used.

Diagnostics, Alternatives, and Limitations

A criterion calculation is only as credible as the fits and comparison inputs. Check that each run converged under the intended estimation method, that estimates are finite and plausible, and that the likelihood and parameter counts correspond to the saved candidate definition. A failed or materially different estimation outcome should not be treated as a genuine model ranking. Preserve the run identity, input data version, model version, estimation settings, objective value, parameter count, and criterion calculations so another analyst can reproduce the comparison.

The main alternatives answer different questions. A likelihood ratio test is appropriate for suitable nested models under its assumptions; it is not justified simply because two objective values are available. AIC and BIC compare candidate models through fit-penalty criteria without requiring nesting. Predictive assessment using held-out data can instead focus directly on prediction for new observations, but requires a defensible validation design and comparable predictions. These approaches need not give the same ranking because their targets differ.

Common failure signals include different records being included across fits, incompatible likelihood definitions, undocumented constants or objective-function conventions, nonconvergence, and sensitivity to the chosen criterion. Investigate these before interpreting a numerical difference. Even when the comparison is technically valid, a criterion cannot diagnose every model misspecification, validate the scientific plausibility of a structural assumption, or replace inspection of parameter estimates and model adequacy.

Common pitfall: “The objective function fell by 3.8, so the likelihood ratio test is significant.” This reasoning is wrong for non-nested candidates: the required nested-model reference distribution does not follow from the objective difference. Instead, verify likelihood comparability, calculate a stated criterion such as AIC or BIC, and report the relative ranking without calling it a likelihood ratio test.

Verify comparability.
Confirm common observations, response information, likelihood target, and documented candidate definitions.

Verify each fit.
Check estimation status, objective-function meaning, and parameter count before computing criteria.

Calculate and summarize.
Report AIC or BIC, criterion differences, and any weights with their candidate-set interpretation.

Qualify the conclusion.
State what the criterion ranks, note limitations or disagreement, and avoid claims of truth or a non-nested likelihood ratio test.

Check Your Understanding

Answer each question, then compare with the solutions below.

Two non-nested candidates have the same $n$ and likelihood target. Why is their objective-function difference not automatically a likelihood ratio test?
Candidate A has $-2\ell=500$ and $k=5$; candidate B has $-2\ell=498$ and $k=7$. Calculate their AIC values and state the AIC ranking.
What does an Akaike weight of 0.80 mean, and what does it not mean?
Name two conditions that should be checked before comparing candidate criteria.

Solutions. (1) A likelihood ratio test requires the appropriate nested-model relationship and its associated assumptions; non-nested candidates do not generally meet that condition. (2) A has AIC $500+2(5)=510$; B has AIC $498+2(7)=512$. A is lower by 2 and is preferred by AIC, but this does not establish that A is true. (3) It summarizes 80% of the normalized relative AIC support among the stated candidate set. It is not an 80% probability that the model is true. (4) Examples include checking that the same observations and response information were used, that the likelihood definitions are comparable, that objective values represent $-2\ell$ on a common scale, and that fits converged with documented parameter counts.