Bayesian Information Criterion for NLME Models

Why BIC penalizes model complexity

A more flexible model can often achieve a higher maximized likelihood because it has more parameters available to fit the data. Comparing models therefore requires more than asking which has the largest likelihood. The Bayesian Information Criterion (BIC) balances fit against model dimension, with a penalty that increases with sample size. Its standard large-sample form can be understood as an approximation to how well a model’s likelihood fits after averaging over its parameter space, rather than evaluating fit only at the best-fitting parameter values.

For an NLME model, the likelihood used in this comparison is ordinarily the likelihood for the observed data after integrating over the subject-level random effects. The parameter count \(K\) refers to the estimated model parameters, including relevant fixed-effect, random-effect distribution, and residual-error parameters, as appropriate to the fitted model. It does not ordinarily count each subject’s latent random effect as a separate population parameter. The likelihoods being compared must be defined on the same data and be comparable; parameter-counting issues are covered in the earlier material on counting parameters for NLME information criteria.

The sample-size term makes BIC’s complexity penalty grow with \(n\): each additional parameter adds \(\log(n)\) to BIC. The likelihood contribution is on the deviance scale, \(-2\ell(\hat{\theta})\). Thus a larger model is favored only when its improved fit is enough to offset the added penalty. In NLME applications, the choice of \(n\) requires care because observations may be clustered within subjects; the formula does not by itself resolve which sampling unit should determine \(n\).

Deriving the penalty from model evidence

For a model with parameter vector \(\theta\), its marginal likelihood, also called model evidence, averages the likelihood over a prior distribution \(\pi(\theta)\):

This integral rewards parameter regions that fit the data, not merely a single high-likelihood point. To approximate it, suppose \(\hat{\theta}\) is an interior maximum-likelihood estimate and the log likelihood is smooth and approximately quadratic near that estimate. Let \(J(\hat{\theta})\) be the observed information matrix, the negative Hessian of the log likelihood at \(\hat{\theta}\). A second-order Taylor expansion gives

If the prior density changes little across the likelihood’s concentrated region, it can be approximated there by \(\pi(\hat{\theta})\). Substituting the quadratic approximation into the evidence integral and integrating the resulting \(K\)-dimensional Gaussian kernel gives the Laplace approximation:

The large-sample step follows when the data provide information at a rate proportional to \(n\): \(J(\hat{\theta})\) is approximately \(nI(\hat{\theta})\), where \(I\) is a per-unit information matrix. Therefore \(|J(\hat{\theta})|\) is approximately \(n^K|I(\hat{\theta})|\). Taking logarithms yields

For fixed \(K\), the prior and information terms do not grow like \(\log(n)\) under the regularity conditions below. Dropping these terms gives \(-2\log m(y)\approx -2\ell(\hat{\theta})+K\log(n)\), which is the BIC expression. This derivation explains both the form of the penalty and its asymptotic nature: BIC retains the term that grows with sample size and omits terms that remain bounded as the sample grows.

For NLME models, the subject effects are integrated out in the marginal likelihood before this approximation is applied. The regularity conditions can be strained when a random-effect variance is estimated at zero, when a covariance structure is singular, when parameters are weakly identified, or when the likelihood has multiple separated modes. These cases can make a single, interior, quadratic approximation a poor description of the evidence. BIC remains calculable, but its evidence-based interpretation should then be cautious.

How the penalty changes with sample size

The fit improvement required to justify additional parameters follows directly by subtracting the two BIC values. Let the expanded model have \(q\) more parameters than the simpler model. Write \(\Delta\ell=\ell_{\mathrm{expanded}}-\ell_{\mathrm{simple}}\). If the models use the same \(n\), then

The expanded model has lower BIC precisely when \(\Delta\ell>(q/2)\log(n)\). For a fixed number of added parameters, that required log-likelihood gain rises as \(n\) increases. This is the specific sense in which BIC imposes a sample-size-dependent penalty; it does not mean that BIC must favor the simpler model in every large data set, since the observed fit gain can also change.

Worked evidence approximation

The Laplace derivation also helps explain why BIC is not the exact model evidence. The omitted terms can matter for particular data sets, priors, and parameterizations, even though they are lower order under the stated asymptotic conditions.

Comparisons, diagnostics, and limitations

BIC is not interchangeable with every other model-comparison procedure. The earlier material on AIC and AICc motivates their penalties through a different objective: estimating relative predictive information loss, rather than approximating model evidence under the regularity conditions used here. A likelihood-ratio test instead uses a test statistic and a reference distribution for a specified nested-model hypothesis. BIC can compare non-nested models if their likelihoods and data basis are comparable, but it does not supply a hypothesis-test p-value.

Before interpreting BIC, check that models were fitted to the same observations and response definition, that their likelihood values are on a comparable basis, and that the parameter counts are defensible. Review optimization and identifiability diagnostics rather than treating a convergence message as sufficient. In particular, estimates near boundaries, very large parameter standard errors, near-singular covariance estimates, or materially different results from plausible starting values warn that the regular approximation may be fragile. BIC also depends on the chosen \(n\); applying a formula with an unjustified sample-size definition can alter the comparison.

A common mistake is to say, “The larger model has a higher likelihood, so it must be preferred.” Likelihood cannot decrease when parameters are added to a nested model, but BIC asks whether that gain offsets \(q\log(n)\). Another mistake is to interpret a BIC difference as the probability that one model is correct. BIC is a score; translating it into an evidence ratio requires additional asymptotic interpretation and assumptions, and it still does not establish that either candidate is true.