AIC, BIC, and Penalized Covariate Model Selection

Why Penalize Covariate Models?

Adding covariates to a population pharmacokinetic model can improve its fit, but the improvement alone is not a sufficient reason to retain every effect. A larger model has more flexibility and can fit observed data better even when some added parameters contribute little useful information. The Akaike information criterion (AIC) and Bayesian information criterion (BIC) balance fit against model complexity using explicit penalties.

Both criteria are calculated from a maximized likelihood and a count of estimated parameters. They can compare candidate models fitted to the same data, including models that are not nested. Their rankings are not p-values, and neither criterion establishes that a covariate has a causal effect or is biologically important. They provide a rule for comparing models under a stated likelihood and parameter-counting convention.

Definition: For a fitted model with maximized log likelihood $\hat{\ell}$, $k$ freely estimated parameters, and sample size $n$, the criteria are $$ \mathrm{AIC}=-2\hat{\ell}+2k,\qquad \mathrm{BIC}=-2\hat{\ell}+k\log(n). $$ Smaller values are preferred when comparing models fitted to the same data using comparable likelihoods. Here, $k$ includes all freely estimated model parameters, not just covariate coefficients.

In a population PK model, $k$ can include fixed-effect parameters for typical values and covariate effects, random-effects variance and covariance parameters, residual-error parameters, and other estimated parameters in the likelihood. It does not count each subject's latent random effect as a separate freely estimated parameter when those effects are integrated out in the marginal likelihood. The precise count depends on the fitted model and likelihood implementation.

The sample size in BIC must match the likelihood's sampling basis. For a marginal likelihood that factors over independent subjects, a common convention is to use the number of independent subjects. Other likelihood formulations can imply a different convention. It is not automatically correct to use the total number of concentration records: repeated observations within a subject are generally not independent sampling units. The convention should be stated and applied consistently to every candidate model.

How the Penalties Affect Nested Comparisons

Suppose a base model is nested within an expanded model, so the expanded model can reproduce the base model by setting its additional parameters to their null values. Let the expanded model add $d$ parameters, and let $\Delta\hat{\ell}=\hat{\ell}_{\mathrm{expanded}}-\hat{\ell}_{\mathrm{base}}$. Under maximum likelihood, the expanded model cannot have a lower maximized likelihood than the base model, apart from numerical failure. The likelihood-ratio statistic is $2\Delta\hat{\ell}$.

\begin{aligned} \mathrm{AIC}_{\mathrm{expanded}}-\mathrm{AIC}_{\mathrm{base}} &=-2\Delta\hat{\ell}+2d,\\ \mathrm{BIC}_{\mathrm{expanded}}-\mathrm{BIC}_{\mathrm{base}} &=-2\Delta\hat{\ell}+d\log(n). \end{aligned}

These expressions give a short derivation of the model-choice thresholds. AIC prefers the expanded model exactly when $2\Delta\hat{\ell}>2d$, or equivalently when the likelihood-ratio statistic exceeds $2d$. BIC prefers it when $2\Delta\hat{\ell}>d\log(n)$. Thus, AIC uses a fixed penalty per added parameter, whereas BIC's penalty per parameter grows with $n$.

For a regular nested-model likelihood-ratio test, the statistic is compared with a chi-square distribution with $df=d$ under the null hypothesis. Its significance cutoff is determined by the chosen $\alpha$, not by AIC or BIC. In particular, AIC's threshold $2d$ is not a universal significance threshold; BIC's threshold is not a testing rule either. These criteria and a likelihood-ratio test answer related but different questions.

The likelihood-ratio reference distribution requires regularity conditions, including identifiable parameters and null values that are not on problematic boundaries. Variance components equal to zero, for example, can violate the usual chi-square approximation. AIC and BIC can still be calculated in such settings, but their usual interpretations rely on an appropriate likelihood, sensible parameter count, and adequate estimation. Neither criterion repairs a misspecified model or an unreliable fit.

Worked Example: A Two-Parameter Covariate Addition

Consider hypothetical base and expanded population PK models fitted to the same 120 subjects with the same marginal-likelihood method. The base model has $k=12$ parameters and maximized log likelihood $-812.4$. The expanded model adds two covariate parameters, so $k=14$, and has maximized log likelihood $-808.9$. Assume both fits converged to comparable maxima and the base model is nested within the expanded model.

The log-likelihood improvement is $\Delta\hat{\ell}=-808.9-(-812.4)=3.5$, so the likelihood-ratio statistic is $2(3.5)=7.0$ on $df=2$. The AIC difference, expanded minus base, is $$ -2(3.5)+2(2)=-7+4=-3.0. $$ The expanded model has AIC 3.0 units lower and is preferred by AIC.

For BIC, $\log(120)\approx4.7875$, giving $$ -2(3.5)+2\log(120)=-7+9.575=2.575. $$ The expanded model's BIC is about 2.58 units higher, so BIC prefers the base model. These opposing rankings follow directly from the thresholds: the likelihood-ratio statistic 7.0 exceeds AIC's threshold of 4 but not BIC's threshold of about 9.575.

For context, under the regular chi-square approximation, $P(\chi^2_2\geq7.0)=e^{-7/2}\approx0.0302$. This is evidence against the nested-model null at a conventional 0.05 level, subject to the regularity assumptions; it does not make the AIC or BIC ranking a test result. For BIC, the specified $n=120$ is the subject count because the example uses a marginal likelihood factored over independent subjects.

Why Rankings Can Differ

AIC is motivated by estimating expected information loss when using a fitted model to represent the process generating new data. Its $2k$ term compensates, in expectation under regular large-sample conditions, for the optimism that arises because parameters were estimated using the data being evaluated. BIC has a different large-sample rationale: under regularity conditions, its penalty arises in an approximation to a model's marginal likelihood. These motivations do not imply that either criterion is always best for every scientific or predictive goal.

As sample size increases, $\log(n)$ eventually exceeds 2, so BIC penalizes each added parameter more heavily than AIC. For smaller $n$, it is possible for $\log(n)$ to be below 2; then BIC's per-parameter penalty is smaller than AIC's. Even when BIC penalizes more, a sufficiently large fit improvement can make both criteria favor the same larger model. A disagreement is therefore possible, not inevitable.

Worked Example: Competing Covariate Models

Suppose three hypothetical models are fitted to the same 150 subjects using the same likelihood. Model $M_0$ has $\hat{\ell}=-500.0$ and $k=9$; model $M_1$, with one covariate specification, has $\hat{\ell}=-496.8$ and $k=11$; and model $M_2$, with another specification, has $\hat{\ell}=-497.5$ and $k=10$. Treat these as candidate models without assuming that either covariate model is nested in the other.

Their AIC values are $$ \begin{aligned} \mathrm{AIC}_{M_0}&=-2(-500.0)+2(9)=1018.0,\\ \mathrm{AIC}_{M_1}&=-2(-496.8)+2(11)=1015.6,\\ \mathrm{AIC}_{M_2}&=-2(-497.5)+2(10)=1015.0. \end{aligned} $$ AIC ranks $M_2$ first, followed by $M_1$, then $M_0$.

Using $\log(150)\approx5.0106$, the BIC values are approximately $$ \begin{aligned} \mathrm{BIC}_{M_0}&=1000+9(5.0106)=1045.096,\\ \mathrm{BIC}_{M_1}&=993.6+11(5.0106)=1048.717,\\ \mathrm{BIC}_{M_2}&=995+10(5.0106)=1045.106. \end{aligned} $$ BIC ranks $M_0$ first, narrowly ahead of $M_2$, and ranks $M_1$ last. The BIC difference between $M_2$ and $M_0$ is only about 0.011, so the computed ordering is effectively a near tie at the displayed precision, not compelling evidence for one of those two models. Because the candidates need not be nested, a likelihood-ratio test between them is not justified merely from these likelihood values.

Parameter Counting, Comparability, and Diagnostics

A model comparison is only interpretable if the likelihoods and penalties are defined consistently. Before ranking models, check the following:

Same data and likelihood basis. Models should use the same observations and comparable likelihood definitions. A full marginal likelihood and a conditional or otherwise differently scaled objective should not be casually mixed.
Correct parameter count. Count all freely estimated parameters in each model, including variance and residual parameters. A categorical covariate represented by three levels usually needs two contrast coefficients relative to a reference level; the number of added parameters is not automatically one just because there is one named covariate.
Correct likelihood sign and scale. The formulas use the maximized log likelihood $\hat{\ell}$. If software reports an objective equal to $-2\hat{\ell}$, that objective is already the fit component of these criteria; do not multiply it by $-2$ again.
Reliable optimization. Check convergence, gradients, parameter boundaries, and whether different starting values lead to materially different maxima. A small apparent likelihood improvement can be numerical noise rather than evidence for an added effect.
Consistent BIC sample size. State the sampling unit used for $n$, and use the same convention across models. Dependence among repeated concentration measurements matters when choosing that unit.

When the sample is small relative to the number of parameters, the uncorrected AIC can favor overly complex models. A commonly used alternative is the corrected AIC, AICc, which adds a finite-sample penalty in settings where its assumptions and formulation are appropriate. In nonlinear mixed-effects models, the effective sample size and likelihood structure require care, so AICc should not be applied mechanically.

Worked Example: A Two-Degree-of-Freedom Effect

A hypothetical model has $k=15$ parameters and maximized log likelihood $-420.0$. Adding a categorical covariate requires two coefficients, giving $k=17$, and improves the maximized log likelihood to $-414.6$. The models are fitted to the same 60 independent subjects with a common marginal likelihood. Assume the optimization is reliable and the expanded model is nested within the base model.

The likelihood-ratio statistic is $2[-414.6-(-420.0)]=10.8$, with $df=2$ under the regular null approximation. The AIC difference is $$ -10.8+2(2)=-6.8, $$ so AIC favors the expanded model by 6.8 units. For BIC, $\log(60)\approx4.0943$, and $$ -10.8+2(4.0943)=-2.611. $$ BIC also favors the expanded model, by about 2.61 units. The chi-square approximation gives $P(\chi^2_2\geq10.8)=e^{-5.4}\approx0.00452$. This calculation is valid as a usual likelihood-ratio test only if the nested comparison satisfies its regularity conditions; it is not needed to calculate either information criterion.

How to Use the Rankings

For nested covariate additions, the earlier material on likelihood-ratio tests provides a direct inferential comparison with a prespecified significance level, while AIC and BIC rank models using their respective penalties. Earlier material on adjusted thresholds and false discovery control addresses error rates in repeated testing; an information criterion does not automatically control those error rates. If many candidate models are searched and only the winning model is reported, selection itself can make ordinary uncertainty summaries too optimistic.

Information criteria are relative: a difference has meaning only within the candidate set and comparable fitting setup. They do not report the probability that a model is true, and a lowest value does not establish that its covariate effects are clinically meaningful, stable across subgroups, or adequately supported by the data. Examine effect estimates and their uncertainty, model diagnostics, plausibility, and sensitivity to reasonable modeling choices alongside the ranking.

Common pitfall: “The expanded model has a higher likelihood, so it must be the better model.” The correction is that maximized likelihood cannot decrease when parameters are added to a nested model, so it rewards flexibility without penalty. AIC and BIC subtract that improvement against a complexity penalty; the preferred model depends on whether the fit gain exceeds the criterion-specific threshold.

Check Your Understanding

Use the definitions and comparison rules above. Assume reliable fits and a common likelihood unless a question says otherwise.

A nested model adds one parameter and increases the maximized log likelihood by 1.4. Which model does AIC prefer, and which does BIC prefer when $n=100$?
A covariate has three categories, with one category used as the reference. How many covariate coefficients are ordinarily added, and what else must be included in $k$?
Two models have AIC values of 820.0 and 823.5. What does the difference say, and what does it not say?
Why should the total number of concentration records not automatically be used as $n$ in BIC for a marginal likelihood based on independent subjects?

Solutions. (1) The likelihood-ratio statistic is $2(1.4)=2.8$, and $d=1$. AIC's threshold is $2$, so it prefers the expanded model. BIC's threshold is $\log(100)\approx4.605$, so it prefers the base model. (2) Two coefficients are ordinarily added, one for each non-reference category. The total $k$ must also count all other freely estimated parameters, including applicable fixed effects, random-effects covariance parameters, and residual-error parameters. (3) The model with AIC 820.0 ranks better within this comparison, by 3.5 AIC units. The difference is not a p-value, a probability that the model is true, or proof of a meaningful covariate effect. (4) Repeated records from one subject are dependent observations, not additional independent subjects; the BIC sample-size convention should reflect the likelihood's sampling basis and be applied consistently.