From Candidate Models to a Defensible Decision
A model-selection statement is useful only if a reader can reconstruct what was compared, see what evidence supported the decision, and understand why that evidence mattered for the pharmacometric question. A statement such as “the final model had the best fit” is not enough: it leaves the candidate set, comparison method, and scientific basis unclear. A strong report distinguishes the numerical comparison from the rationale for choosing a model.
The purpose is not to declare one model universally correct. It is to document why one candidate was preferred for the stated analysis, given the available data, prespecified decision rules, diagnostics, and scientific objectives. The earlier material on prespecifying model-development decisions and planning candidate alternatives provides the basis for describing which models were eligible for comparison.
What a Comparison Must Make Auditable
Before presenting a ranking, identify the candidate models in terms that distinguish them: for example, one- versus two-compartment disposition, alternative residual-error structures, or inclusion versus exclusion of a specified covariate effect. Give the relevant parameterization or refer to the reported equations. State the analysis population and data set used, and whether every candidate was fitted to the same observations with the same likelihood target. Comparisons of likelihood-based criteria are not interpretable as a simple ranking if the likelihoods refer to different data or different targets.
For each model, report the objective value used, the number of estimated parameters under the stated counting convention, and the comparison statistic. If using information criteria, name the criterion and its convention. For the usual maximum-likelihood forms, with \(k\) estimated parameters and \(n\) the stated number of independent sampling units, the Akaike information criterion (AIC) and Bayesian information criterion (BIC) are:
Here \(L\) is the maximized likelihood. Smaller values indicate a preferred trade-off under that criterion among the candidates compared; they do not measure the probability that a model is true. Report the definition of \(n\) used for BIC, particularly when the data have repeated observations or a hierarchical structure, and apply it consistently. AIC and BIC embody different penalties for model complexity, so disagreement is possible and should be described rather than concealed.
A likelihood-ratio test (LRT) can be used for a particular comparison when one model is nested in another: the simpler model is obtained by imposing restrictions on the more complex model. Its assumptions and scope are narrower than “compare any two fitted models.” State the degrees of freedom, the test statistic, and the p-value when an LRT is used, and make clear which term or restriction is being tested.
Worked Comparisons
Worked Example: Reporting AIC and BIC That Disagree
Suppose a hypothetical analysis uses the same likelihood and the same 240 independent subjects for two candidate models. Model A has \(-2\log L=1004.2\) and \(k=10\); Model B has \(-2\log L=999.8\) and \(k=12\). The prespecified convention defines \(n=240\) subjects for BIC. Calculate both criteria and describe the result.
For AIC, Model A has \(1004.2+2(10)=1024.2\), and Model B has \(999.8+2(12)=1023.8\). The AIC difference, A minus B, is \(1024.2-1023.8=0.400\), so AIC favors Model B by a small amount.
For BIC, \(\log(240)\approx5.480639\). Thus Model A has \(1004.2+10(5.480639)=1059.006\), or approximately 1059.01. Model B has \(999.8+12(5.480639)=1065.568\), or approximately 1065.57. The difference, B minus A, is \(1065.568-1059.006=6.561\), or approximately 6.56, so BIC favors Model A.
A defensible report would state that AIC favored B slightly, whereas BIC favored A by 6.56 units under the stated subject-count convention. It would then explain which criterion had priority under the analysis plan and assess whether Model B’s additional parameters improved scientifically relevant behavior. Neither criterion alone establishes which model generated the data.
Worked Example: A Nested Likelihood-Ratio Comparison
In a hypothetical population PK analysis, Model 0 omits a prespecified weight effect on clearance and Model 1 adds one estimated weight coefficient. Both models use the same subjects, observations, and likelihood. Their objective values are \(-2\log L_0=1018.6\) and \(-2\log L_1=1012.8\). Assume the added coefficient is an unconstrained parameter whose null value is not on a parameter boundary, and that the regularity conditions for the usual large-sample LRT are reasonable.
The likelihood-ratio statistic is
The models differ by one estimated coefficient, so \(df=1\). Under \(H_0\), the reference distribution is approximately chi-squared with one degree of freedom. The upper-tail probability is \(P(\chi^2_1\geq5.8)\approx0.0160\). At a prespecified \(\alpha=0.05\), this is evidence against the restriction that the weight coefficient is zero, within the assumptions of the test.
The result does not by itself show that weight causes a change in clearance, nor that the effect is large enough to matter clinically. The report should give the estimated effect and its uncertainty on an interpretable scale, and consider whether diagnostics and scientific plausibility support retaining it. The LRT addresses this nested restriction; it does not replace evaluation of the overall model.
Worked Example: Reconciling Criteria With Scientific and Diagnostic Evidence
Consider a hypothetical analysis of 180 subjects comparing Model P, a one-compartment model with first-order absorption, and Model Q, a two-compartment model. Both use the same observations and likelihood. Model P has \(-2\log L=820.0\) and \(k=8\); Model Q has \(-2\log L=811.5\) and \(k=11\). The analysis plan names BIC as the primary information criterion and AIC as supportive.
AIC for P is \(820.0+2(8)=836.0\); for Q it is \(811.5+2(11)=833.5\). AIC favors Q by \(836.0-833.5=2.5\) units. For BIC, \(\log(180)\approx5.192957\). BIC for P is \(820.0+8(5.192957)=861.544\), or 861.54. BIC for Q is \(811.5+11(5.192957)=868.623\), or 868.62. BIC therefore favors P by \(868.623-861.544=7.079\), or 7.08 units.
Suppose the hypothetical diagnostics show no important systematic pattern for P, while Q’s additional distribution parameters have large uncertainty and do not materially improve the predictive displays considered in the analysis. A concise rationale could read: “The prespecified BIC favored the one-compartment model by 7.08 units; AIC favored the two-compartment candidate by 2.5 units. Because the two-compartment model’s additional parameters were imprecise and its predictive displays did not show a material improvement, the one-compartment model was selected under the prespecified BIC-prioritized rule. This choice is conditional on the candidate set and available data.”
This wording gives the evidence on both sides, explains the decision rule and scientific considerations, and limits the claim. It does not describe P as proven true or say that Q is invalid.
Justifying the Likelihood-Ratio Statistic
The LRT statistic follows directly from the likelihood ratio. If Model 0 is nested in Model 1, fitting Model 1 cannot give a lower maximized likelihood, so \(L_1\geq L_0\). The likelihood ratio \(L_0/L_1\) is therefore no greater than 1, and its logarithm is no greater than 0. Multiplying by \(-2\) yields a nonnegative statistic:
Under the null restrictions and regular large-sample conditions, this statistic is approximately chi-squared with degrees of freedom equal to the number of independent restrictions. This reference distribution can fail when parameters are on boundaries, models are not nested, or regularity conditions are poor. For example, a variance component tested against zero is on a boundary of its allowable parameter space; the ordinary chi-squared reference distribution may then be inappropriate. In such cases, report the limitation and use a justified alternative rather than presenting a routine LRT p-value as definitive.
Building a Concise, Auditable Narrative
A short comparison table can carry the detailed evidence, leaving the text to explain the decision. Include model labels and distinguishing features, \(-2\log L\) or the relevant objective value, parameter count, criteria or test result, and a brief diagnostic or scientific note. In the narrative, identify the selected model and connect the decision to the prespecified rule and analysis objective.
State the model features that differ and confirm the comparison used the same analysis data and compatible likelihood target.
Give the criterion or test, its values, the relevant difference or statistic, and the direction of support. Include parameter counts and the BIC sample-size convention where applicable.
Apply the prespecified decision rule, then state how scientific relevance, parameter precision, and diagnostics informed the choice.
Note material disagreement or limitations, and describe the selected model as preferred among the candidates for the stated purpose, not as uniquely true.
Diagnostics should be described specifically enough to support the comparison: name the displays or checks examined and summarize the pattern relevant to the decision. Do not convert a visual assessment into an unsupported numerical test. Parameter uncertainty and shrinkage evidence can affect how confidently a difference is interpreted, but neither should be used as a substitute for reporting the comparison itself. Also distinguish “the fit improved” from “the model is preferable”: a more complex model will often fit the estimation data at least as well, while its added complexity may not be justified.
Alternatives, Limitations, and Common Errors
Different comparison approaches answer different questions. AIC and BIC compare candidates using fit and complexity penalties; their rankings can disagree, and differences are relative to the candidate set and conventions used. An LRT evaluates a specified nested restriction under its distributional assumptions. Predictive checks and other diagnostics assess whether model behavior is adequate for relevant features of the data, but they do not automatically supply a single ranking. Scientific plausibility and the intended use determine whether a numerical improvement is useful. A report should not imply these approaches are interchangeable or that one universally overrides the others.
A selection is limited by the candidate models, data quality and design, model assumptions, and the comparison rule. A model that ranks first among a narrow set may still be inadequate. Likewise, an apparently small criterion difference is not proof of practical equivalence, and a large difference does not guarantee useful prediction in a new setting. Keep claims tied to the analysis and state when the evidence is mixed.
- Reporting only “best fit.” This hides the criterion and the cost of added complexity. Name the criterion and give the compared values or their difference.
- Treating AIC or BIC as a probability. A difference of 6.56 does not mean a model is 6.56 times as likely, nor does a lower value give the probability that a model is true.
- Using an LRT for non-nested candidates. The usual nested-model statistic and reference distribution do not apply to arbitrary alternatives. Choose a comparison suited to the candidate relationship.
- Ignoring conflicting evidence. Report disagreements between criteria and explain which rule governed the decision; do not selectively report only the favorable result.
- Comparing incompatible fits. Different data subsets, likelihood targets, or parameter-counting conventions can invalidate a direct numerical ranking. State and verify the comparison basis.
- Overstating the selected model. Say it was preferred under stated criteria for the analysis purpose, not that the data proved it to be the true biological system.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions below.
- Two candidates use the same data and likelihood. Their AIC values are 912.4 and 914.1. What does the difference support, and what does it not establish?
- Why should a report state how \(n\) was defined when calculating BIC for repeated-measures data?
- A model adds a variance parameter constrained to be nonnegative, and the null value is zero. Why might the usual LRT reference distribution need reconsideration?
- Name three kinds of information, beyond a criterion ranking, that can support a model-selection rationale.
Solutions. (1) AIC favors the model with 912.4 by \(914.1-912.4=1.7\) units, but this is not the probability that model is true and does not by itself establish practical or scientific superiority. (2) Repeated observations within subjects are not necessarily independent sampling units; stating the convention makes the BIC calculation interpretable and reproducible. (3) Zero is a boundary of the allowable parameter space, so the regularity conditions for the standard chi-squared LRT may fail. (4) Examples include diagnostics relevant to the analysis objective, parameter precision or stability, scientific plausibility, and whether added complexity yields a meaningful improvement.