Marginal Likelihood by Integrating Random Effects

Why Integrate Over Random Effects?

In a shared random-effects model, a subject’s longitudinal measurements and event outcome are linked through latent subject-specific quantities. Conditional on those quantities, the subject’s data may have a relatively simple likelihood. But the observed data do not include the random effects themselves. To obtain a likelihood for the observed data, we must account for every random-effect value that could have generated them, weighting each value by its probability under the random-effects distribution.

The key operation is integration within each subject’s contribution. After that integration, independent subjects’ marginal contributions are multiplied. This is not the same as substituting an estimated random effect into the conditional likelihood, nor is it generally the same as integrating all subjects over one common random effect.

Definition: Let $O_i$ denote the observed data for subject $i$, and let $b_i$ be that subject’s random-effects vector. Write $f_i(O_i\mid b_i;\theta)$ for the conditional likelihood contribution given $b_i$, and $g(b_i;\psi)$ for its random-effects density. The subject’s marginal likelihood contribution is $$ L_i(\theta,\psi)=\int f_i(O_i\mid b_i;\theta)\,g(b_i;\psi)\,db_i. $$ Here $\theta$ denotes parameters in the conditional data model and $\psi$ denotes parameters of the random-effects distribution. The integral is over the full support of $b_i$.

This formula applies whether $O_i$ contains longitudinal observations, an event-time observation, censoring information, or several of these together. The conditional contribution $f_i$ must be constructed for the observed data and the model at hand; the integral then removes the unobserved random effects. The formula does not prescribe the detailed longitudinal or event-time contribution.

From Subject Contributions to the Sample Likelihood

Suppose there are $n$ subjects, each with a random effect $b_i$. Assume that subjects are independent conditional on the modeled covariates, and that their random effects are independently drawn from $g(\cdot;\psi)$. Given the random effects, assume the observations from different subjects are independent and have conditional contributions $f_i$. The joint density of the observed data and all random effects is then

\prod_{i=1}^{n} f_i(O_i\mid b_i;\theta)\,g(b_i;\psi).

The observed-data likelihood is obtained by integrating this joint density over all random effects. Since each factor involves only the random effect for its own subject, the multiple integral separates:

\begin{aligned} L(\theta,\psi) &=\int\cdots\int \prod_{i=1}^{n} f_i(O_i\mid b_i;\theta)g(b_i;\psi)\, db_1\cdots db_n\\ &=\prod_{i=1}^{n}\int f_i(O_i\mid b_i;\theta)g(b_i;\psi)\,db_i\\ &=\prod_{i=1}^{n}L_i(\theta,\psi). \end{aligned}

The separation follows from the assumed product form and integration over distinct $b_i$ variables. It is not a universal property of all latent-effect models: dependence between subjects’ random effects, or another shared latent quantity, can prevent this factorization. The log likelihood for independent subjects is correspondingly

\ell(\theta,\psi)=\log L(\theta,\psi) =\sum_{i=1}^{n}\log\left\{\int f_i(O_i\mid b_i;\theta)g(b_i;\psi)\,db_i\right\}.

In general, the logarithm is taken after integrating each subject’s conditional likelihood. The expression $\int \log f_i(O_i\mid b_i;\theta)\,g(b_i;\psi)\,db_i$ is an expectation of a conditional log likelihood, not the log of the marginal likelihood contribution.

Worked Example: A Gaussian Marginal Contribution

Consider one hypothetical observation $Y$ with $Y\mid b\sim N(b,2.25)$ and $b\sim N(6,4)$, where the second parameter of $N$ is the variance. Find the marginal density of $Y$ at $y=8$. This simple model illustrates the integration; it is not intended as a full joint-model specification.

The conditional density is $f(y\mid b)=\{2\pi(2.25)\}^{-1/2}\exp\{-(y-b)^2/[2(2.25)]\}$, and the random-effect density is $g(b)=\{2\pi(4)\}^{-1/2}\exp\{-(b-6)^2/(2(4))\}$. Thus the marginal density is the integral of their product over all real $b$. Equivalently, $Y=b+\varepsilon$, where $b$ and $\varepsilon$ are independent normal variables, with $\varepsilon\sim N(0,2.25)$. The sum is normal with mean $6$ and variance $4+2.25=6.25$, so $Y\sim N(6,6.25)$.

Therefore,

\begin{aligned} L(8) &=\frac{1}{\sqrt{2\pi(6.25)}} \exp\left\{-\frac{(8-6)^2}{2(6.25)}\right\}\\ &=\frac{\exp(-0.32)}{\sqrt{12.5\pi}} =\frac{0.7261}{\sqrt{39.270}} \approx 0.1159. \end{aligned}

The result is a density value, not the probability that $Y$ equals exactly 8. The integration accounts for uncertainty in $b$; it increases the variance of the marginal distribution from the conditional variance $2.25$ to $6.25$.

Marginalization Is Not Plug-In Prediction

A common temptation is to replace $b_i$ by its population mean or by an estimated subject-specific value and then evaluate $f_i$. That produces a conditional likelihood at a selected random-effect value, not the observed-data likelihood. In general,

\int f_i(O_i\mid b_i;\theta)g(b_i;\psi)\,db_i \ne f_i(O_i\mid E[b_i];\theta).

The difference matters especially when the conditional likelihood is nonlinear in $b_i$, as it often is in generalized linear models and event-hazard models. Integrating averages the likelihood contributions over the random-effects distribution; plugging in the mean evaluates the likelihood at only one point.

Worked Example: A Discrete Random-Effects Distribution

For a hypothetical binary outcome $Y$, suppose $P(Y=1\mid b)=\operatorname{logit}^{-1}(b)=1/(1+\exp(-b))$. Let $b$ equal $-1$ or $1$, each with probability $1/2$. Calculate the marginal likelihood contribution for observing $Y=1$, and compare it with using the mean random effect.

At $b=1$, the conditional probability is $1/(1+\exp(-1))\approx0.7311$. At $b=-1$, it is $1/(1+\exp(1))\approx0.2689$. Since the random-effect probabilities are both $1/2$, the marginal contribution is

L_i=\tfrac12(0.7311)+\tfrac12(0.2689)=0.5000.

The mean random effect is $E[b]=\tfrac12(-1)+\tfrac12(1)=0$. Plugging in this mean gives $P(Y=1\mid b=0)=0.5000$, which happens to match the integral in this symmetric example. That agreement is not general. To see the distinction, change the two values of $b$ to $-0.5$ and $1.5$, still with equal probabilities. The integrated contribution becomes

\tfrac12\operatorname{logit}^{-1}(-0.5) +\tfrac12\operatorname{logit}^{-1}(1.5) =\tfrac12(0.3775)+\tfrac12(0.8176) \approx0.5976.

The mean random effect is now $0.5$, and the plug-in probability is $\operatorname{logit}^{-1}(0.5)\approx0.6225$, not $0.5976$. The likelihood contribution is a weighted average of conditional probabilities, not the conditional probability at the average random effect.

Why the Subject-Specific Integration Matters

The product of marginal contributions relies on having a separate random effect for each subject, with the stipulated independence across subjects. The following calculation shows what changes if two observations instead share one random effect. It also demonstrates why an integral over all subjects using one common latent value would describe a different model.

Worked Example: Two Subjects With Independent Random Effects

Suppose two hypothetical observations satisfy $Y_i\mid b_i\sim N(b_i,1)$, with independent $b_i\sim N(0,1)$. The observed values are $y_1=0$ and $y_2=1$. For each subject the marginal distribution is $Y_i\sim N(0,2)$, so the individual contributions are

L_1=\frac{\exp(0)}{\sqrt{4\pi}}\approx0.2821,\qquad L_2=\frac{\exp(-1/4)}{\sqrt{4\pi}}\approx0.2197.

The sample likelihood contribution is their product:

L_1L_2\approx(0.2821)(0.2197)=0.0620.

For contrast, if both observations shared a single $b\sim N(0,1)$, integrating would produce a bivariate normal distribution with covariance matrix $\begin{pmatrix}2&1\\1&2\end{pmatrix}$, not the product of two independent $N(0,2)$ densities. The shared effect induces covariance $1$, because both observations contain the same random quantity. That is a different dependence structure and a different likelihood. The independent-subject formula cannot be used for it.

Assumptions, Checks, and Alternatives

Before using a marginal likelihood, verify the ingredients that make it a valid observed-data likelihood:

Proper random-effects distribution. $g(b_i;\psi)$ must be a density that integrates to one over the stated support.
Correct conditional contribution. $f_i(O_i\mid b_i;\theta)$ must describe the observed data conditional on the random effect, including the observation and censoring structure required by the model.
Correct independence structure. The product across subjects requires independent subject contributions after integrating independent subject-specific effects, conditional on modeled covariates.
Integration over the full support. Omitting part of the support or using an incorrectly normalized density changes the likelihood.

A basic check is that integrating a subject’s marginal data density over all possible observed data gives one, provided the conditional density is normalized for each $b_i$ and the random-effects density is proper. For a fixed observed dataset, however, $L_i$ is a likelihood contribution as a function of model parameters, not a probability distribution over those parameters.

When an integral has no convenient analytic form, it may be evaluated numerically or approximated. The target remains the same marginal likelihood integral; the computational approach does not change its definition. Numerical calculations should be checked for stability as integration accuracy is increased. Implausibly small or large contributions, sensitivity to computational settings, and disagreement between equivalent formulations can indicate an implementation or scaling problem.

The marginal likelihood is also distinct from the complete-data likelihood, which treats the random effects as if they were observed, and from a conditional likelihood evaluated at estimated random effects. The complete-data expression can be useful in estimation algorithms, but it is not the observed-data likelihood unless the random effects are actually observed. A plug-in conditional contribution can be useful for prediction or other purposes, but it does not perform the required averaging over latent effects.

Common pitfall: “The random effect is centered at zero, so I can set it to zero in every subject’s likelihood.” The correction is that centering describes the distribution, not the unobserved value for each subject. The marginal contribution averages the conditional likelihood over the entire random-effects distribution; nonlinear conditional models generally make that average different from the likelihood at zero.

Marginalization is central because model parameters are estimated from the observed data, not from latent effects that were never recorded. It preserves the uncertainty about those effects and the dependence they induce among a subject’s repeated measurements and outcomes. A correctly specified marginal likelihood therefore provides the appropriate basis for likelihood-based estimation and comparison.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Write the marginal likelihood contribution for one subject with random-effect vector $b_i$, conditional contribution $f_i$, and random-effects density $g$.
A binary outcome has $P(Y=1\mid b)=\operatorname{logit}^{-1}(b)$. Suppose $b=-1,0,2$ with probabilities $0.25,0.50,0.25$, respectively. Find the marginal contribution for observing $Y=1$.
Two independent subjects have already-integrated likelihood contributions $0.20$ and $0.30$. What is their sample likelihood contribution and log likelihood?
Why is integrating over one common random effect for all subjects not generally equivalent to integrating each subject’s independent random effect and multiplying the results?

Solutions. 1. $L_i=\int f_i(O_i\mid b_i;\theta)g(b_i;\psi)\,db_i$. 2. The contribution is $0.25(0.2689)+0.50(0.5000)+0.25(0.8808)\approx0.5374$. 3. The product is $(0.20)(0.30)=0.0600$, and its log is $\log(0.0600)\approx-2.813$. 4. A common effect induces dependence across subjects and defines a different joint distribution; independent subject-specific effects lead to a product of separately integrated contributions.