First-Order Information Versus Exact Marginal Information

Why Compare the Two Information Calculations?

Population-design calculations often replace integration over subject-specific random effects with a Gaussian approximation based on a local linearization. This can make the population Fisher matrix much easier to calculate. The approximation is exact when the prediction is linear in Gaussian random effects and the residual model is Gaussian with fixed covariance. When predictions are nonlinear, however, the distribution obtained by integrating over random effects need not be Gaussian, and its mean and covariance need not match the first-order approximation. The resulting information differences can affect comparisons between candidate designs.

The comparison is between two likelihoods for the same population model and design. It is not a choice between “using” and “ignoring” random effects: both calculations account for them, but in different ways. The exact marginal calculation integrates their contribution; the first-order calculation approximates that integration locally.

Exact Marginal Information and the Gaussian Approximation

Definition: Let $Y_i$ denote subject $i$'s observations, $\theta$ the population parameters of interest, and $\eta_i\sim N(0,\Omega)$ the subject's random effects. If the conditional density is $p(Y_i\mid\eta_i,\theta)$, the marginal likelihood is $$ p(Y_i\mid\theta)=\int p(Y_i\mid\eta_i,\theta)\,\phi(\eta_i;0,\Omega)\,d\eta_i. $$ The exact marginal score is the derivative of the log marginal likelihood with respect to $\theta$. The exact marginal Fisher information is the expected outer product of that score under the marginal model.

For a vector of observations, write the conditional model as $Y_i=f_i(\theta,\eta_i)+\epsilon_i$, with $\epsilon_i\sim N(0,R_i)$ independent of $\eta_i$. A first-order Taylor expansion around the random-effects mean, $\eta_i=0$, is

f_i(\theta,\eta_i)\approx f_i(\theta,0)+J_i(\theta)\eta_i,\qquad J_i(\theta)=\left.\frac{\partial f_i(\theta,\eta_i)}{\partial\eta_i}\right|_{\eta_i=0}.

Because a linear transformation of a Gaussian random vector is Gaussian, this approximation gives a Gaussian observation model with mean $\mu_i^{(1)}=f_i(\theta,0)$ and covariance $V_i^{(1)}=J_i\Omega J_i^\mathsf{T}+R_i$. If these quantities depend on $\theta$, the Gaussian Fisher information includes contributions from changes in both the mean and covariance:

I^{(1)}_{ab} = \left(\frac{\partial\mu_i^{(1)}}{\partial\theta_a}\right)^\mathsf{T} (V_i^{(1)})^{-1} \left(\frac{\partial\mu_i^{(1)}}{\partial\theta_b}\right) +\frac{1}{2}\operatorname{tr}\left[ (V_i^{(1)})^{-1}\frac{\partial V_i^{(1)}}{\partial\theta_a} (V_i^{(1)})^{-1}\frac{\partial V_i^{(1)}}{\partial\theta_b} \right].

This is the Fisher information of the approximating Gaussian model. It is not generally the information of the exact marginal likelihood. If the covariance is fixed with respect to the parameters, the covariance-derivative term is zero. For independent subjects, subject-level information matrices add; within-subject observations must still be treated jointly when their covariance is not diagonal.

What Nonlinearity Changes

A local linearization uses the prediction and its random-effects derivative at $\eta_i=0$. But the exact marginal mean is $E_{\eta_i}[f_i(\theta,\eta_i)]$, and the exact marginal covariance also depends on the full distribution of $f_i(\theta,\eta_i)$. Curvature can therefore shift the marginal mean, add variance not represented by the local derivative, and produce skewness or other non-Gaussian features. The first-order Gaussian model may miss any of these.

A useful identity explains why marginal scores differ from conditional scores. If $u_c(Y_i,\eta_i;\theta)$ is the conditional score, differentiation under the integral gives

u_m(Y_i;\theta) = \frac{\partial\log p(Y_i\mid\theta)}{\partial\theta} = E\left[u_c(Y_i,\eta_i;\theta) +\frac{\partial}{\partial\theta}\log\phi(\eta_i;0,\Omega(\theta)) \mid Y_i,\theta\right].

Thus the marginal score averages the complete-data score—comprising the conditional score and any score of the random-effects density—using the distribution of random effects conditional on the observed data, not the original random-effects distribution. If the random-effects distribution is fixed with respect to $\theta$, its score contribution is zero. In general, this posterior weighting is data-dependent. A first-order Gaussian calculation does not reproduce it exactly when the prediction is nonlinear.

Worked Example: Curvature Shifts the Marginal Mean and Variance

Suppose one observation has conditional mean $f(\theta,\eta)=\theta+\beta\eta+\gamma\eta^2$, with $\eta\sim N(0,\omega^2)$ and residual error $\epsilon\sim N(0,\sigma^2)$. Take hypothetical values $\theta=2$, $\beta=1$, $\gamma=0.5$, $\omega=0.8$, and $\sigma=0.6$. All values of $Y$, $\eta$, and $\epsilon$ are in the same measurement units.

Since $E(\eta)=0$ and $E(\eta^2)=\omega^2$, the exact marginal mean is

E(Y)=\theta+\gamma\omega^2 =2+0.5(0.8^2) =2+0.32 =2.32.

For a zero-mean normal random effect, $\operatorname{Var}(\eta^2)=2\omega^4$ and $\operatorname{Cov}(\eta,\eta^2)=0$. Therefore the exact marginal variance is

\operatorname{Var}(Y) =\beta^2\omega^2+2\gamma^2\omega^4+\sigma^2 =1^2(0.64)+2(0.5^2)(0.8^4)+0.36 =0.64+0.2048+0.36 =1.2048.

At $\eta=0$, the first-order prediction is $f(\theta,0)=2$, and its random-effects derivative is $\beta+2\gamma\eta=1$. Thus the first-order Gaussian model has mean $2$ and variance $1^2(0.64)+0.36=1.00$. In this example it understates the exact marginal mean by $0.32$ measurement units and the variance by $0.2048$ squared measurement units. It also misses the non-Gaussian shape induced by the squared random effect. These moment differences do not, by themselves, quantify the difference in Fisher information; information depends on how the full likelihood changes with the parameters.

Worked Marginal Likelihood and Score

Worked Example: Quadrature Reveals a Different Likelihood Score

Consider $Y\mid\eta\sim N(\theta+\eta^2,1)$, with $\eta\sim N(0,1)$, and evaluate the likelihood at the hypothetical observation $y=0$ and parameter value $\theta=0$. The exact marginal density is the integral of the conditional normal density over the standard normal random effect. To show the calculation, approximate that integral using three-point Gauss-Hermite quadrature for a standard normal expectation: nodes $0,\sqrt{3},-\sqrt{3}$, with weights $2/3,1/6,1/6$.

At node $\eta=0$, the conditional mean is $0$, so the density at $y=0$ is $\phi(0)=0.398942$. At either nonzero node, $\eta^2=3$, so that density is $\phi(3)=0.00443185$. The quadrature approximation to the marginal density is

p(y=0\mid\theta=0) \approx \frac{2}{3}\phi(0)+\frac{1}{3}\phi(3) =\frac{2}{3}(0.398942)+\frac{1}{3}(0.00443185) =0.267439.

For a normal conditional model with unit variance, the conditional score for $\theta$ is $y-\theta-\eta^2$. The marginal score is the conditional score averaged over random effects weighted by their contribution to the likelihood. The quadrature approximation gives

u_m(0;0) \approx \frac{\frac{2}{3}(0)(0.398942)+\frac{1}{3}(-3)(0.00443185)} {0.267439} = \frac{-0.00443185}{0.267439} \approx -0.01657.

The first-order expansion at $\eta=0$ has derivative $\partial f/\partial\eta=2\eta=0$. It therefore gives the approximating model $Y\sim N(\theta,1)$: at $y=\theta=0$, its density is $0.398942$ and its score for $\theta$ is $0$. The likelihood and score differ because the local approximation omits the quadratic random-effects contribution. The quadrature values are numerical approximations to the marginal integral, not exact closed-form values; more quadrature nodes can be used to check numerical convergence.

A score at one observation is not itself Fisher information. The exact information for $\theta$ is $E_\theta[u_m(Y;\theta)^2]$, where the expectation is over the exact marginal distribution of $Y$. The example shows how to obtain the marginal score at a particular observation; calculating the expected information requires evaluating or approximating that score across the marginal distribution.

A Linear Benchmark and an Information Identity

Worked Example: When First-Order and Exact Information Agree

Let $Y=\theta+\beta\eta+\epsilon$, with $\eta\sim N(0,\omega^2)$ and independent $\epsilon\sim N(0,\sigma^2)$. This prediction is linear in the random effect, so integrating it out gives exactly $Y\sim N(\theta,\beta^2\omega^2+\sigma^2)$; the first-order approximation makes no error. Set $\beta=1$, $\omega=0.8$, and $\sigma=0.6$. The marginal variance is

$$ V=1^2(0.8^2)+0.6^2=0.64+0.36=1.00. $$

For one observation, the score for the location parameter is $(Y-\theta)/V$. Its Fisher information is $E[(Y-\theta)^2]/V^2=V/V^2=1/V$, which equals $1.00$ inverse squared measurement units for these values. Since the first-order and exact marginal distributions coincide, both calculations give the same information.

A related identity helps interpret the role of latent random effects. Under regularity conditions that allow differentiation under the integral, the information in the marginal data is the information in the complete data minus the expected conditional variance of the complete-data score given the observations. Here $u_{\mathrm{complete}}(Y,\eta;\theta)=\partial_\theta\log[p(Y\mid\eta,\theta)\phi(\eta;0,\Omega(\theta))]$ is the score of the joint density, and $I_{\mathrm{complete}}=E[u_{\mathrm{complete}}u_{\mathrm{complete}}^\mathsf{T}]$. This is the missing-information identity. It says that uncertainty about the random effects can reduce information relative to observing them directly. It does not say the first-order approximation must always overstate or always understate the exact marginal information: the two calculations use different likelihoods, and either direction of discrepancy is possible.

I_{\mathrm{marg}} = I_{\mathrm{complete}} - E\left[\operatorname{Var}\{u_{\mathrm{complete}}(Y,\eta;\theta)\mid Y,\theta\}\right].

Diagnostics, Alternatives, and Limitations

A first-order information calculation is most credible when the prediction is close to linear over the plausible spread of the random effects and when conclusions are stable to reasonable changes in the calculation. Useful checks include comparing the first-order mean and covariance with marginal moments computed by numerical integration or simulation, and comparing information matrices or design rankings across methods. Pay particular attention to parameter directions associated with weak information: a modest change in a small eigenvalue or in a design ranking may matter even when overall matrix summaries look similar.

Numerical integration and Monte Carlo integration target the exact marginal likelihood more directly, but their accuracy and computational cost must be assessed. Quadrature can work well at low random-effects dimension; its cost can rise quickly as that dimension increases. Simulation can handle more complex integrals, but Monte Carlo error can obscure small differences in information unless enough samples are used. The first-order Gaussian approximation is faster and interpretable, but it is local to the expansion point and can miss curvature, skewness, and data-dependent posterior weighting. None of these calculations removes the need to specify the random-effects and residual models appropriately.

Common warning signs include large discrepancies between exact or simulated marginal moments and their first-order counterparts, unstable results as quadrature settings change, and substantial changes in design rankings across methods. These signal a need to investigate the approximation or the integration, not automatic proof that one calculation is wrong. In particular, do not infer from a mismatch in marginal variance alone that one design has more information: information depends on parameter sensitivities as well as the distribution of observations.

Key takeaway: First-order information is exact for a Gaussian observation model whose prediction is linear in Gaussian random effects, under the stated covariance assumptions. With nonlinear predictions, it is an approximation to the marginal likelihood calculation. Compare the likelihood-based information, not only the means or variances, whenever curvature could affect a design decision.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions.

For $f(\theta,\eta)=\theta+\gamma\eta^2$ with a zero-mean Gaussian random effect, what marginal-mean term does expansion at $\eta=0$ omit?
In the quadrature example, why is the marginal score not the unweighted average of the conditional scores at the quadrature nodes?
Under what conditions do exact marginal and Gaussian first-order likelihoods agree?
Does the missing-information identity establish that first-order information always overstates exact marginal information? Explain.
Name two checks that can reveal a consequential first-order approximation error in a design comparison.

Solutions. (1) It omits $\gamma E(\eta^2)=\gamma\omega^2$, so the exact marginal mean is $\theta+\gamma\omega^2$. (2) The conditional scores must be weighted by each node's conditional likelihood contribution, then normalized by the marginal likelihood; the data update the random-effects weights. (3) They agree when the conditional prediction is linear in Gaussian random effects and the residual model is Gaussian with the specified covariance, so integration gives the same Gaussian distribution as the approximation. (4) No. The identity compares marginal data with complete data, not the exact marginal model with a first-order approximation; either approximation discrepancy is possible. (5) Compare marginal moments by integration or simulation, and compare information matrices or design rankings across methods while checking numerical convergence.