Why Covariance Structure Selection Requires a Fit–Complexity Trade-Off
In a population pharmacokinetic model, the omega matrix, \(\Omega\), describes the covariance of the subject-level random effects. A diagonal \(\Omega\) allows the random effects to vary across individuals but assumes their components are uncorrelated. A less constrained structure can estimate covariance terms as well. Those added terms may improve the likelihood, but they also increase the number of estimated parameters and can be difficult to estimate from limited information.
Information criteria summarize this trade-off. They can help rank candidate omega structures when the models are fitted to the same data under a common modeling framework. They do not establish that a selected structure is biologically true, and their ranking should be considered alongside model diagnostics and the scientific purpose of the analysis.
For a population PK analysis, \(k\) counts the estimated model parameters: for example, fixed-effect parameters, residual-error parameters, and the distinct free elements of \(\Omega\). It does not count each subject’s empirical Bayes estimate as a separate model parameter. When comparing omega structures with all other components held fixed, the difference in \(k\) is the difference in the number of free omega elements.
Conditions for a Meaningful Comparison
The likelihoods and parameter counts must correspond to comparable fits. A focused comparison of omega structures should use the same subjects and concentration records, the same response scale, the same structural model, and the same residual-error model. The candidate models should differ only in the covariance structure, unless the comparison is deliberately evaluating several modeling changes at once. For population models, the likelihood used for this purpose is ordinarily the marginal likelihood, which accounts for the distribution of individual random effects rather than treating their estimated values as known.
The fits must also be numerically trustworthy. Each candidate should converge to a credible optimum, and its estimated \(\Omega\) must be positive semidefinite. A covariance estimate at or near a boundary, a nearly singular covariance matrix, or substantial sensitivity to starting values can make the apparent likelihood improvement unstable. A failed or poorly converged fit is not fairly compared by simply inserting its reported likelihood into an information criterion.
For BIC in a hierarchical dataset, the independent sampling units matter. A common population-PK convention uses the number of independent subjects for \(N\), rather than the total number of concentration records, because repeated records within a subject are not independent sampling units. The convention should be stated and applied consistently; counting every correlated record as an independent unit can make the BIC complexity penalty misleadingly large.
These criteria are most useful as relative rankings within a set of defensible candidate models. If the data, likelihood definition, or BIC sample-size convention changes between fits, a difference in AIC or BIC may not isolate the effect of the omega structure.
Why the Criteria Penalize Added Parameters
A maximized likelihood cannot decrease when a model is expanded to include a nested model, because the larger model can reproduce the smaller model by setting its extra parameters to restricted values. That improved in-sample fit alone therefore tends to favor complexity. AIC and BIC add penalties to discourage selecting a larger model merely because it can adapt more closely to the data.
AIC’s \(2k\) penalty is motivated as an approximate correction for the optimism of evaluating a fitted model on the same data used to estimate it. Under regularity conditions, a model with \(k\) freely estimated parameters gains about \(k\) units of log-likelihood from fitting sampling noise; because deviance is \(-2\ell\), the corresponding optimism correction is about \(2k\). Thus AIC uses the maximized deviance plus this estimated optimism penalty.
BIC has a different motivation. If the model has a regular, identifiable interior optimum, the observations are independent at the sampling-unit level, and the prior density is positive and smooth near the optimum, a large-sample approximation to the integrated likelihood gives
Multiplying by \(-2\) yields the BIC expression up to terms that do not grow with \(N\). BIC therefore imposes a penalty that grows with sample size, while AIC uses a fixed penalty of two per parameter. The regularity assumptions behind these approximations can fail for variance and covariance parameters on boundaries, such as a variance estimated as zero. Information criteria remain useful summaries in such situations, but their theoretical approximations should not be mistaken for exact guarantees.
Worked Example: One Covariance Added to a Diagonal Omega
Suppose two hypothetical models use the same 120 subjects, the same data records, and the same structural and residual-error models. Model D has diagonal \(\Omega\); Model U adds one covariance between two random effects. Assume both fits converge properly, and the maximized log-likelihood increases by 3.7 units in Model U: \(\ell_U-\ell_D=3.7\). The parameter-count difference is \(k_U-k_D=1\).
The AIC contrast, defined as Model U minus Model D, is
Model U has an AIC 5.4 points lower than Model D, so AIC favors adding the covariance in this comparison. For BIC, using \(N=120\) independent subjects,
BIC also favors Model U: its BIC is about 2.613 points lower. The sign depends on the order of subtraction. Equivalently, \(\mathrm{BIC}_D-\mathrm{BIC}_U=2(3.7)-\ln(120)=2.613\), meaning Model D’s BIC is 2.613 points higher.
When a Larger Structure Fits Better but Is Penalized
Adding several covariances can produce a visible likelihood gain without producing a preferred information-criterion value. This is not a contradiction: the likelihood term rewards fit, while the penalty asks whether that improvement is large enough relative to the number of extra parameters.
Worked Example: Three Additional Covariances With Limited Subjects
Consider a hypothetical comparison based on 45 independent subjects. Model D is diagonal and Model U adds three covariance parameters. Suppose the log-likelihood improves by 4.2 units in Model U, so \(\ell_U-\ell_D=4.2\), and assume the fits are otherwise comparable and reliable.
For AIC, the U-minus-D contrast is
AIC is 2.4 points lower for Model U. For BIC, it is useful to state the contrast direction explicitly. The D-minus-U contrast is
Because this contrast is negative, Model D’s BIC is about 3.020 points lower than Model U’s, so BIC favors the diagonal model. Equivalently, \(\mathrm{BIC}_U-\mathrm{BIC}_D=3.020\). The three covariance terms improve the fit enough for AIC’s penalty, but not enough for BIC’s larger penalty at this sample size.
Comparing Several Plausible Structures
A realistic selection exercise may contain more than two candidates. The following hypothetical results compare a diagonal structure, a structure with one covariance, and a fully unstructured covariance for three random effects. All use the same dataset and the same non-omega model components. The total parameter count \(k\) includes all estimated parameters; parameters shared across these candidates are included in every count.
Worked Example: AIC and BIC Rank Three Candidate Structures
Suppose there are 80 independent subjects and the fitted results are as follows. The reported values of \(-2\ell(\hat{\theta})\) are maximized deviances.
| Omega structure | \(-2\ell(\hat{\theta})\) | \(k\) | AIC | BIC |
|---|---|---|---|---|
| Diagonal | 820.0 | 8 | 836.0 | 855.1 |
| One covariance | 816.8 | 9 | 834.8 | 856.2 |
| Unstructured | 815.6 | 11 | 837.6 | 863.8 |
For the one-covariance model, \(\mathrm{AIC}=816.8+2(9)=834.8\). For the diagonal model, \(\mathrm{BIC}=820.0+8\ln(80)=820.0+35.056=855.1\). Applying the same formulas gives the remaining entries. AIC selects the one-covariance structure, while BIC selects the diagonal structure. The unstructured model has the best likelihood, but its three additional covariance parameters relative to the diagonal model do not earn a lower AIC or BIC in this example.
This disagreement does not make either calculation erroneous. AIC and BIC target different approximations and can rank models differently. The results make the trade-off explicit: a more flexible structure improves fit, but the evidence for that flexibility depends on the chosen criterion and the available independent subjects.
Diagnostics, Alternatives, and Common Pitfalls
An information-criterion ranking is a starting point, not a substitute for checking whether the candidate models behave sensibly. Inspect convergence, parameter uncertainty where available, and whether covariance estimates are stable and compatible with a positive-semidefinite \(\Omega\). A covariance estimate that changes dramatically with starting values or is effectively at a boundary may indicate that the data do not identify the added term reliably. Also examine whether the fitted model reproduces relevant features of the data; a low relative AIC does not establish absolute adequacy.
The principal alternative is a likelihood-ratio comparison for nested models. Such tests can be useful when their assumptions and reference distribution are appropriate. For variance and covariance terms at boundaries, the ordinary chi-square reference may not apply, as considered in the earlier material on boundary likelihood-ratio tests. Information criteria instead give a penalized ranking and do not require interpreting a p-value, but they do not answer a null-hypothesis question. Scientific constraints, prespecified structures, stability, and predictive performance may also inform the choice; none should be replaced by treating a small numerical difference as decisive.
- Wrong reasoning: “The unstructured model has the largest likelihood, so it is automatically the best model.” The maximized likelihood generally improves as a model gains free parameters.
- Correction: Compare likelihood gain with the parameter penalty, and evaluate model stability and adequacy. In the three-covariance example, the unstructured model fit best by likelihood but was not favored by either information criterion.
- Wrong reasoning: “A lower BIC proves that the selected omega structure is true.” An information criterion ranks the candidates considered under a particular data and modeling setup; it does not prove truth or rule out omitted structures.
- Correction: Describe the result as relative support among the stated candidates and report the comparison convention, including the BIC sample size.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A candidate omega structure adds one covariance parameter and improves the maximized log-likelihood by 1.2. What is the AIC contrast, defined as the larger model minus the smaller model?
- Two models differ by two parameters. The larger model improves the log-likelihood by 5.0, with \(N=50\) independent subjects. Calculate the BIC contrast, larger minus smaller, and state which model has lower BIC.
- Why should a population-PK BIC calculation generally not count every repeated concentration record as an independent subject?
- A more complex omega model has the lowest AIC, but its covariance estimates vary greatly across starting values. Is the information-criterion result sufficient to select it? Explain.
Solutions. (1) \(\Delta\mathrm{AIC}=-2(1.2)+2(1)=-0.4\), so the larger model’s AIC is 0.4 lower. (2) \(\Delta\mathrm{BIC}=-2(5.0)+2\ln(50)=-10+7.824=-2.176\). The larger model has BIC about 2.176 lower. (3) Repeated records within a subject are correlated through that subject’s random effects, so they are not independent sampling units; using their count as \(N\) can overstate the BIC penalty. (4) No. The unstable estimates raise concern about convergence or identifiability. Check fit reliability and model diagnostics before relying on its relative AIC advantage.