Why Information Criteria Need a Run Audit
A lower objective value does not by itself establish that one pharmacometric run is preferable: a more flexible model can often fit the data better simply because it has more parameters. The Akaike information criterion (AIC) and Bayesian information criterion (BIC) combine fit and a complexity penalty. Their values are useful for comparing candidate runs only when the likelihoods and the data being compared are compatible.
In a run comparison, the arithmetic is usually straightforward. The important work is verifying what each reported likelihood represents, which observations contributed to it, and which estimated quantities are counted as parameters. A numerically correct criterion calculated from mismatched runs does not provide a valid model comparison.
Definitions and Conditions for Comparison
These formulas use the maximized log likelihood, not the likelihood itself and not automatically any software field labeled “objective,” “deviance,” or “OFV.” If a reported objective is exactly \(-2\ell\), it can replace that term in the formulas. Confirm that identity from the software's definition and settings. An objective may instead omit constants, use a different likelihood approximation, or represent a different estimation target.
A meaningful comparison ordinarily requires the candidate runs to use the same response data, the same likelihood definition, and the same treatment of observations. For BIC, the value of \(n\) must also be defined consistently and correspond to the observations represented by the likelihood. With repeated measurements or hierarchical data, the appropriate count can depend on the likelihood formulation and the intended BIC convention; do not switch between counting records and counting subjects without justification. State the convention used.
The parameter count \(k\) is the number of free parameters estimated in the model likelihood. Depending on the model, these can include structural parameters, covariate effects, residual-error parameters, and estimated random-effects variance or covariance parameters. Do not count parameters fixed in advance as estimated, and do not assume that every reported individual estimate is a separate parameter in a marginal likelihood. The counting rule must match the likelihood and estimation procedure. If a run uses a penalty, constraint, approximation, or parameterization that makes \(k\) unclear, resolve that issue before treating the criterion as an ordinary AIC or BIC.
The formulas can be computed whenever \(\ell\), \(k\), and (for BIC) \(n\) are specified. Their usual model-selection rationale relies on regularity conditions, such as an appropriately defined likelihood and estimable parameters. In particular, BIC's penalty uses sample size to grow with the data, while AIC uses the fixed penalty \(2k\). In irregular or weakly identified models, the usual criteria may need careful qualification; a computed value does not remove those limitations.
How the Penalties Affect a Pairwise Comparison
The algebra makes the fit-complexity trade-off explicit. For candidates A and B, define \(\Delta\mathrm{AIC}=\mathrm{AIC}_B-\mathrm{AIC}_A\). Substituting the definition for each run gives:
Thus, a better-fitting B (larger \(\ell_B\)) lowers the first term, while additional parameters increase the second. B has a lower AIC exactly when its log-likelihood gain is greater than its parameter penalty: \(\ell_B-\ell_A>k_B-k_A\). For the same runs and the same \(n\), the corresponding BIC difference is:
When \(n>e^2\), \(\ln n>2\), so BIC penalizes each additional parameter more than AIC does. This helps explain why the criteria can rank candidates differently. It does not mean that one criterion is universally correct: they serve different selection rationales, and neither ranking is a hypothesis test or a guarantee of predictive performance in a new setting.
Worked Example: Two Runs With a Small Difference in Fit
Suppose two hypothetical runs use the same 60 observations and the same likelihood convention. Run A has \(k_A=4\) and \(\ell_A=-424.0943\); Run B has \(k_B=6\) and \(\ell_B=-422.0943\). The log likelihood in B is higher by 2.000 log-likelihood units, but B estimates two additional parameters.
AIC for A: Substitute \(\ell_A\) and \(k_A\): $$ \mathrm{AIC}_A=-2(-424.0943)+2(4)=848.1886+8=856.1886\approx856.19. $$
AIC for B: $$ \mathrm{AIC}_B=-2(-422.0943)+2(6)=844.1886+12=856.1886\approx856.19. $$ The improvement in log likelihood exactly balances AIC's penalty, so the AIC values tie at the displayed precision.
BIC for A: Using \(\ln(60)\approx4.094345\): $$ \mathrm{BIC}_A=848.1886+4\ln(60) \approx848.1886+16.3774=864.5660\approx864.57. $$
BIC for B: $$ \mathrm{BIC}_B=844.1886+6\ln(60) \approx844.1886+24.5661=868.7547\approx868.75. $$ Using the unrounded quantities, the difference is $$ \mathrm{BIC}_B-\mathrm{BIC}_A=2\ln(60)-4\approx4.1887\approx4.19. $$ Subtracting the displayed rounded values instead gives \(868.75-864.57=4.18\). That one-hundredth discrepancy is due to rounding; the difference based on unrounded values is the appropriate calculation. Under BIC, A has the lower score; under AIC, the scores tie. Neither result proves that A is true or that B's extra parameters are useless.
Comparing AIC and BIC in Context
| Criterion | Penalty | What a lower value means | Key caution |
|---|---|---|---|
| AIC | \(2k\) | Better relative balance of fit and complexity under AIC | Not a test of significance or a probability of model truth |
| BIC | \(k\ln(n)\) | Better relative balance of fit and complexity under the stated BIC convention | Check how \(n\) is defined, especially for dependent or hierarchical data |
AIC and BIC are not the only ways to assess candidate models. Predictive validation can directly assess performance on held-out data, but it answers a different question and depends on a defensible validation design. Likelihood-ratio comparisons are another approach in suitable settings; their assumptions and procedure are distinct from information-criterion ranking. Do not infer a likelihood-ratio test result from an AIC or BIC difference.
Information criteria also have limitations. They rank only the candidates supplied, so all candidates can be inadequate. Their values do not diagnose parameter identifiability, convergence, implausible estimates, or model misspecification. Use the run records and diagnostics to establish that a candidate is estimable and scientifically coherent before interpreting its criterion. A small difference is not a sharp boundary between “good” and “bad,” and a numerical ranking should be reported with the data and conventions that make it interpretable.
Worked Example: When AIC and BIC Rank Runs Differently
Two hypothetical models are fitted to the same 1,200 observations using the same likelihood convention. Model A has \(\ell_A=-512.4\) and \(k_A=5\); Model B has \(\ell_B=-507.9\) and \(k_B=8\). B improves the maximized log likelihood by 4.5 units and adds three parameters.
AIC: $$ \mathrm{AIC}_A=-2(-512.4)+2(5)=1024.8+10=1034.8, $$ $$ \mathrm{AIC}_B=-2(-507.9)+2(8)=1015.8+16=1031.8. $$ The AIC difference is \(1031.8-1034.8=-3.0\), so B has the lower AIC by 3.0 units among these candidates.
BIC: Since \(\ln(1200)\approx7.090077\), $$ \mathrm{BIC}_A=1024.8+5\ln(1200) \approx1024.8+35.4504=1060.2504\approx1060.25, $$ $$ \mathrm{BIC}_B=1015.8+8\ln(1200) \approx1015.8+56.7206=1072.5206\approx1072.52. $$ The BIC difference is approximately \(12.27\) in favor of A, which has the lower BIC. The criteria differ because BIC imposes a larger per-parameter penalty at this \(n\). These scores express relative preferences under different criteria; they do not establish that either model is correct.
Audit the Inputs Before Trusting the Ranking
Check that candidate runs use the same response observations and compatible likelihood definitions. Record any exclusions or different analysis populations.
Verify that the reported value is the maximized log likelihood \(\ell\), or that an objective is exactly \(-2\ell\). Record any constants or approximations that affect the reported value.
Count free estimated parameters under a consistent rule. For BIC, document the observation-count convention and apply it consistently to all candidates.
Compute both criteria from unrounded inputs, retain adequate precision for differences, and report the criterion, likelihood convention, \(k\), and BIC \(n\) with the result.
A practical failure mode is comparing a complete-case run to a run that excluded additional records. Even if both software outputs contain AIC and BIC, they score fits to different data and cannot be interpreted as a direct ranking of models on the same observations. Likewise, if one reported likelihood omits a constant and another includes it, absolute criterion values may shift. A constant omitted identically for all candidates on the same data leaves their within-set ranking unchanged, because it shifts each criterion by the same amount; mixing conventions makes the displayed values misleading and can prevent a reliable comparison.
Worked Example: A Lower Score From a Different Observation Set
In a hypothetical analysis, Model A uses all 100 records, with \(\ell_A=-100.0\) and \(k_A=3\). Model B excludes eight records and uses the remaining 92, with \(\ell_B=-94.0\) and \(k_B=4\). Suppose the stated BIC convention uses the number of records contributing to each run's likelihood.
Model A criteria: $$ \mathrm{AIC}_A=-2(-100.0)+2(3)=200+6=206.0, $$ $$ \mathrm{BIC}_A=200+3\ln(100) \approx200+13.8155=213.8155\approx213.82. $$
Model B criteria: $$ \mathrm{AIC}_B=-2(-94.0)+2(4)=188+8=196.0, $$ $$ \mathrm{BIC}_B=188+4\ln(92) \approx188+18.0873=206.0873\approx206.09. $$ B has lower computed values, but this is not a valid direct comparison: the runs used different records. The better scores cannot be attributed solely to the model change. Refit both candidates on the same 92-record set (or another common, justified set), use compatible likelihood conventions, and then compare their criteria. The arithmetic is valid for each run individually; the cross-run conclusion is not.
Common Mistakes and Interpretation
- Using an objective without checking its definition. A value labeled “objective” is not automatically \(-2\ell\). Verify the likelihood convention before substituting it into AIC or BIC.
- Counting every output estimate as a parameter. Count free parameters estimated in the likelihood, not every row in a parameter table or every individual-level estimate by default.
- Using inconsistent \(n\) values. BIC requires a stated observation-count convention. Use the same defensible convention for every candidate.
- Comparing different data sets. A criterion difference is not a fair model comparison if the runs use different response observations or likelihood targets.
- Rounding before subtraction. Calculate differences from unrounded criteria where possible. Rounded display values can produce a slightly different subtraction, as in the first worked example.
- Treating a lower criterion as proof. The lower score is preferred only among the compared, compatible candidates under that criterion. It is not a p-value, a probability of model truth, or proof of predictive superiority in every setting.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Two runs use the same data and likelihood convention. Their log likelihoods are \(-250\) and \(-247\), with \(k=4\) and \(k=6\), respectively. Which has the lower AIC, and by how much?
- Why can a lower BIC from a run using fewer observations not automatically be interpreted as evidence that its model is preferable?
- For two candidates with the same \(n=200\), one has three more estimated parameters. What is the additional BIC penalty for those parameters, before accounting for any fit improvement?
- A software output reports an “objective function value.” What should be checked before using it to calculate AIC?
Solutions. (1) The AIC difference for the second run minus the first is \(-2[(-247)-(-250)]+2(6-4)=-6+4=-2\). The second run has an AIC lower by 2 units. (2) Different observations mean the likelihoods summarize fits to different data; the lower score cannot isolate the effect of changing the model. Compare candidates on a common, justified observation set and compatible likelihoods. (3) The added BIC penalty is \(3\ln(200)\approx3(5.2983)=15.89\) units. The fit improvement must be assessed separately in the full criterion difference. (4) Verify that the objective is exactly \(-2\ell\) for the maximized likelihood being compared, including its constants, data, and likelihood convention. If not, obtain the compatible log likelihood or document the transformation before calculating AIC.