Why Integrate Over Random Effects?
In a nonlinear mixed-effects model, a subject’s observations depend on population parameters and on that subject’s random effects. The random effects are not observed. A subject’s data therefore contribute to population inference through all random-effect values allowed by the model, weighted by their probability under the population distribution. The mathematical operation that combines those possibilities is integration.
This matters because a likelihood evaluated at one chosen random-effect value answers a different question. It describes the data conditional on that value; it does not account for the full distribution of possible subject effects. The marginal likelihood does account for that distribution, and is the subject-level contribution used when the random effects are integrated out.
The usual density is $$ \phi_q(\eta_i;0,\Omega)=\frac{1}{(2\pi)^{q/2}|\Omega|^{1/2}} \exp\!\left(-\frac{1}{2}\eta_i^\mathsf{T}\Omega^{-1}\eta_i\right), $$ where \(\Omega\) is a positive-definite covariance matrix. The integral is over the entire random-effect space, not just over values that appear likely after seeing subject \(i\)’s data. Values with greater prior density receive greater weight, while the conditional data density also affects each integrand value.
From a Conditional Density to a Marginal Density
The conditional density \(f_i(y_i\mid\eta_i,\theta)\) treats \(\eta_i\) as fixed. The marginal likelihood \(L_i(\theta)\) instead averages that density with respect to the random-effect distribution. In general, this integral has no closed-form solution when the prediction or observation model is nonlinear in \(\eta_i\).
The derivation is the continuous version of the law of total probability. For any possible data value \(y_i\), the random effect can take values throughout \(\mathbb{R}^q\). The joint density of data and random effect is \(f_i(y_i\mid\eta_i,\theta)\phi_q(\eta_i;0,\Omega)\). Integrating this joint density over \(\eta_i\) leaves the density of the data alone:
This expression is a proper marginal density when the conditional data model is a proper density and the random-effect distribution is a proper density. For example, integrating over all possible data values gives $$ \int L_i(\theta)\,dy_i =\int \phi_q(\eta_i;0,\Omega) \left[\int f_i(y_i\mid\eta_i,\theta)\,dy_i\right]d\eta_i =\int \phi_q(\eta_i;0,\Omega)\,d\eta_i =1. $$ The interchange of integrals is valid for these nonnegative densities.
If subjects are independent under the model, each with an independent random-effect vector from the same population distribution, the likelihood for all subjects is the product of their marginal likelihoods:
Here \(N\) is the number of subjects. The product form relies on independence across subjects. Repeated observations within a subject are generally not independent after the random effect is integrated out; they share that subject’s \(\eta_i\), as the second example shows.
Worked Example: A Marginal Density for One Nonlinear-Scale Observation
Suppose a hypothetical concentration \(Y\), measured in mg/L, follows a lognormal model. Conditional on a subject’s random effect, \(\log Y=\eta+\epsilon\), where \(\eta\sim N(0,0.50^2)\) and \(\epsilon\sim N(0,0.20^2)\), independently. For an observed concentration \(y=1.00\) mg/L, find the marginal density and compare it with the conditional density evaluated at \(\eta=0\).
The conditional density on the concentration scale is $$ f(y\mid\eta)=\frac{1}{y(0.20)\sqrt{2\pi}} \exp\!\left[-\frac{(\log y-\eta)^2}{2(0.20)^2}\right]. $$ The random effect has density \(\phi(\eta;0,0.25)\). Thus the marginal likelihood contribution is the integral of their product over all real \(\eta\). Since the sum of the independent normal variables \(\eta+\epsilon\) is \(N(0,0.25+0.04)=N(0,0.29)\), this particular integral can be evaluated directly: $$ L(\theta)=f_Y(1.00)=\frac{1}{1.00\sqrt{2\pi(0.29)}} \exp\!\left[-\frac{(\log 1.00)^2}{2(0.29)}\right] \approx 0.7408\;(\mathrm{mg/L})^{-1}. $$
For comparison, setting \(\eta=0\) in the conditional density gives $$ f(1.00\mid 0)=\frac{1}{1.00(0.20)\sqrt{2\pi}}\exp(0) \approx 1.995\;(\mathrm{mg/L})^{-1}. $$ These are different densities: \(0.7408\) is the marginal contribution after integrating over the random-effect distribution, whereas \(1.995\) is conditional on the particular value \(\eta=0\). The conditional value is not a substitute for the integral.
Repeated Measurements Share One Random Effect
For subject \(i\), \(y_i\) is the full vector of that subject’s observations. If measurements are conditionally independent given \(\eta_i\), their conditional densities multiply inside \(f_i(y_i\mid\eta_i,\theta)\). The subject-level marginal likelihood integrates that product over the single shared random-effect vector. In general, this is not the same as multiplying marginal densities that each use a separate random effect.
Worked Example: Two Observations With a Shared Random Intercept
Consider a hypothetical subject with two positive measurements \(Y_1,Y_2\) in mg/L. Let \(\log Y_j=\eta+\epsilon_j\), with the same \(\eta\sim N(0,0.50^2)\) for both observations and independent errors \(\epsilon_j\sim N(0,0.20^2)\). Suppose both observed values are \(1.00\) mg/L. Calculate the joint marginal density and contrast it with the incorrect calculation that gives each observation an independent random effect.
Write \(Z_j=\log Y_j\). Each \(Z_j\) has variance \(0.25+0.04=0.29\), and the two log measurements have covariance \(0.25\), because they share \(\eta\). Their covariance matrix is therefore $$ \Sigma= \begin{pmatrix} 0.29&0.25\\ 0.25&0.29 \end{pmatrix}, \qquad |\Sigma|=(0.29)^2-(0.25)^2=0.0216. $$ At the observed values, \(Z_1=Z_2=0\). The bivariate normal density at zero is \(1/(2\pi\sqrt{|\Sigma|})\). The change from log measurements to \(Y_1,Y_2\) contributes the density Jacobian factor \(1/(y_1y_2)\), which equals \(1\) here. Thus $$ f_{Y_1,Y_2}(1.00,1.00) =\frac{1}{(1.00)(1.00)\,2\pi\sqrt{0.0216}} \approx 1.083\;(\mathrm{mg/L})^{-2}. $$
The incorrect independent-random-effect calculation would multiply the two one-observation marginal densities from the preceding example: $$ 0.7408\times0.7408\approx0.5488\;(\mathrm{mg/L})^{-2}. $$ That product describes a model in which each measurement has its own independent random effect. It does not describe the stated repeated-measurement model. Under the stated model, the correct operation is to multiply the two conditional densities given the shared \(\eta\), then integrate their product against the one random-effect density.
Combining Independent Subjects
Once each subject’s observations have been integrated over that subject’s random effects, independent subjects contribute multiplicatively. The following example keeps the same hypothetical lognormal model but uses two independent subjects, each with one measurement. This shows how the subject-level marginal densities form the population likelihood.
Worked Example: Likelihood for Two Independent Subjects
Suppose the two hypothetical observations are \(y_1=1.00\) mg/L and \(y_2=e^{0.50}\approx1.649\) mg/L. Each subject has an independent random effect with variance \(0.25\), and the residual variance on the log scale is \(0.04\). The marginal variance of each log measurement is \(0.29\), so the density for an observation \(y\) is $$ f_Y(y)=\frac{1}{y\sqrt{2\pi(0.29)}} \exp\!\left[-\frac{(\log y)^2}{2(0.29)}\right]. $$
For subject 1, \(\log y_1=0\), giving \(f_Y(y_1)\approx0.7408\;(\mathrm{mg/L})^{-1}\). For subject 2, \(\log y_2=0.50\), so $$ f_Y(y_2)=\frac{1}{e^{0.50}\sqrt{2\pi(0.29)}} \exp\!\left[-\frac{(0.50)^2}{2(0.29)}\right] =0.7408\,e^{-0.50}e^{-0.25/0.58} \approx0.2920\;(\mathrm{mg/L})^{-1}. $$
Because the subjects and their random effects are independent, the likelihood for these observations is $$ L(\theta)=0.7408\times0.2920\approx0.2163\;(\mathrm{mg/L})^{-2}, \qquad \ell(\theta)=\log(0.7408)+\log(0.2920)\approx-1.531. $$ The density product carries units of \((\mathrm{mg/L})^{-2}\), as expected for two continuous observations. This calculation first obtains each subject’s marginal density and then multiplies across subjects.
Assumptions, Computation, and Failure Modes
The definition is simple, but its use depends on specifying the model correctly. The conditional data density must describe the observations given the random effects, including the chosen residual error structure and any dependence that remains conditional on those effects. The random effects must have the stated distribution, here a mean-zero multivariate normal with covariance \(\Omega\). The random-effect distribution and conditional density together define the integrand; changing either changes the marginal likelihood.
For independent subjects, the likelihood product also assumes that subjects’ data and random effects are independent conditional on the population parameters. If the design or data introduce dependence across subjects, that dependence must be represented in the joint model rather than hidden in a product of subject-level terms.
In nonlinear models, the marginal integral often cannot be expressed in elementary functions. It may be evaluated numerically, but numerical accuracy must be checked. Useful checks include increasing integration accuracy, verifying stability of the resulting log likelihood, and examining whether important integrand mass lies in regions poorly covered by the computation. High-dimensional random-effect vectors can make integration particularly challenging. A positive-definite \(\Omega\) is needed for the displayed ordinary multivariate normal density; a variance estimated at or near a boundary can also create computational and inferential difficulties.
| Quantity | What is held fixed or integrated? | Role in the model |
|---|---|---|
| Conditional data likelihood \(f_i(y_i\mid\eta_i,\theta)\) | \(\eta_i\) is held fixed | Describes the data at a specified individual effect |
| Joint density \(f_i(y_i\mid\eta_i,\theta)\phi_q(\eta_i;0,\Omega)\) | Includes both data and random-effect density | Weights conditional fit by the population distribution of effects |
| Marginal likelihood \(L_i(\theta)\) | Integrates over all \(\eta_i\) | Subject-level contribution after random effects are unobserved |
A common mistake is to reason, “The subject’s best-fitting random effect is \(\eta_i^\ast\), so the subject likelihood is \(f_i(y_i\mid\eta_i^\ast,\theta)\).” The correction is that this is a conditional data likelihood at one selected value. The marginal likelihood is the integral of the conditional density times the random-effect density over all possible values. Selecting a value and integrating over a distribution are different operations.
The joint density is also not itself the marginal likelihood: it still contains \(\eta_i\). In contrast, integration removes \(\eta_i\) from the subject-level contribution. These distinctions are useful when interpreting model objectives and diagnosing calculations. If repeated observations have inadvertently been assigned separate random effects, the resulting marginal likelihood will represent a different dependence structure, as in the second example.
The integral is a definition of the likelihood contribution, not a guarantee that a chosen numerical method will compute it accurately or that the model is scientifically adequate. A poor conditional model, an unsuitable random-effect distribution, or an inadequate residual structure can yield a precisely computed likelihood for the wrong model. Marginalization handles latent random effects under the assumed model; it does not repair misspecification.
Check Your Understanding
Answer each question using the distinction between conditioning on a random effect and integrating over its distribution.
- In the definition of \(L_i(\theta)\), what is integrated over, and what does the normal density in the integrand represent?
- Why should two repeated observations with one shared random intercept be multiplied conditionally before integration?
- Under what condition can the full likelihood be written as a product of subject-level marginal likelihoods?
- In the first worked example, why is the conditional density at \(\eta=0\) not the marginal likelihood?
Solutions. (1) The integral is over every component of the subject’s random-effect vector \(\eta_i\); the normal density weights each value according to the specified population distribution. (2) Both observations depend on the same \(\eta_i\), so their conditional densities form one joint conditional density; integrating that product preserves their shared-effect dependence. (3) Subjects and their random effects must be independent under the model, conditional on the population parameters. (4) The conditional density fixes \(\eta\) at zero, while the marginal density averages the conditional density over the full \(N(0,0.25)\) distribution.