The Linear-Gaussian Benchmark for Marginalization

Why a Linear-Gaussian Case Is a Useful Benchmark

Marginal likelihoods in nonlinear mixed-effects models generally require integrating over the subject-specific random effects. In a special but important case, that integration is exact: the individual prediction is linear in Gaussian random effects, and the residual errors are Gaussian. The resulting marginal distribution gives a reference calculation against which approximate marginal likelihoods can be checked. It is a benchmark, not a claim that nonlinear pharmacokinetic or pharmacodynamic models are generally Gaussian after marginalization.

The key point is that marginalization changes the covariance as well as removing the random effects from the conditional mean. A random intercept, for example, makes observations from the same individual correlated, even when their residual errors are independent. The covariance expression makes that dependence explicit.

Here \(n_i\) is the number of observations for individual \(i\), and \(q\) is the number of random effects: \(y_i\) and \(\epsilon_i\) are \(n_i\)-vectors, \(X_i\) is \(n_i\) by the number of fixed effects, \(Z_i\) is \(n_i\) by \(q\), and \(b_i\) is a \(q\)-vector. The notation \(N(m,S)\) denotes a multivariate normal distribution with mean \(m\) and covariance \(S\). Independence across individuals is also assumed when the total likelihood is written as a product of subject likelihoods.

Marginal Distribution and Exact Likelihood

Conditionally on \(b_i\), the response is Gaussian with mean \(X_i\beta+Z_i b_i\) and covariance \(R_i\). Since \(Z_i b_i\) is a linear transformation of a Gaussian vector, it is Gaussian with mean zero and covariance \(Z_i\Omega Z_i^\mathsf{T}\). It is independent of \(\epsilon_i\). Adding the two independent Gaussian terms therefore gives the marginal distribution:

Thus the exact marginal density is a multivariate normal density. Defining \(r_i=y_i-X_i\beta\), its log-likelihood contribution is

The three terms have distinct roles: the normalizing constant depends on the number of observations; the log determinant accounts for the scale and shape of the marginal covariance; and the quadratic form measures the residual’s distance from the predicted mean in units of that covariance. The total log-likelihood is \(\ell=\sum_i\ell_i\) when individuals are independent. In estimation, \(V_i\) may depend on unknown variance parameters, so both the log determinant and the quadratic form must be recomputed as those parameters change.

Justifying the Marginalization by Integration

The Gaussian-sum argument establishes the distribution directly. The same result can be checked by evaluating the conditional-likelihood integral, which connects this benchmark to the general definition of a marginal likelihood. For clarity, omit the individual subscript, let \(r=y-X\beta\), and write \(A=\Omega^{-1}+Z^\mathsf{T}R^{-1}Z\) and \(h=Z^\mathsf{T}R^{-1}r\). The terms in the joint exponent that involve \(b\) satisfy

This is completion of the square: expanding the first term on the right produces \(b^\mathsf{T}Ab-2b^\mathsf{T}h+h^\mathsf{T}A^{-1}h\), and the final term cancels the added \(h^\mathsf{T}A^{-1}h\). Integrating the centered Gaussian kernel over \(b\) contributes the factor \((2\pi)^{q/2}|A|^{-1/2}\). Combining it with the conditional and random-effects normalizing constants gives the marginal density. The determinant identity \(|R||\Omega||A|=|R+Z\Omega Z^\mathsf{T}|\) and the inverse identity \(R^{-1}-R^{-1}ZA^{-1}Z^\mathsf{T}R^{-1}=(R+Z\Omega Z^\mathsf{T})^{-1}\) then yield the same determinant and quadratic form as the marginal Gaussian formula above. Positive definiteness ensures these inverses and Gaussian integrals exist.

Worked Calculations

Assumptions, Diagnostics, and Scope

The exact result relies on linearity in the random effects, Gaussian random effects, Gaussian residuals, and independence between those two sources of variation. The covariance matrices must be valid and the likelihood must use the correct residual covariance structure. Residuals need not be independent across occasions: a specified positive-definite \(R_i\) may include serial correlation. However, the formula only gives the intended likelihood if that structure is appropriate for the data.

Useful checks include confirming that \(V_i\) is symmetric positive definite, verifying dimensions before matrix multiplication, and checking that the marginal covariance contains the intended shared-parameter correlations. A non-positive determinant or an attempted inverse of a singular matrix signals a parameterization, data-design, or numerical problem; it is not a valid log-likelihood calculation. Also check whether observations have been centered around \(X_i\beta\), not around a conditional prediction that still includes a particular random-effect value.

The benchmark does not diagnose model adequacy by itself. A mathematically correct Gaussian marginal likelihood can still be a poor description of observations if the structural mean, residual distribution, covariance structure, or random-effects distribution is wrong. In nonlinear models, the prediction is generally not affine in the random effects, so the marginal distribution need not be Gaussian. FO, FOCE, and Laplace methods address marginalization in that broader setting by approximation; their performance can be assessed against this exact case when the model is linear-Gaussian. This tutorial establishes the reference likelihood, not the construction or behavior of those approximation procedures.

A common mistake is to use only \(R_i\) in the marginal likelihood because residual errors are the only errors visible in the observation equation once a subject’s prediction is specified. That is conditional reasoning: marginalization must include uncertainty in \(b_i\), so the correct covariance is \(Z_i\Omega Z_i^\mathsf{T}+R_i\). Another mistake is to add variances but leave off-diagonal covariances at zero; shared random effects induce off-diagonal terms whenever the corresponding rows of \(Z_i\) load on the same effects.