Louis’ Identity for the Observed Information

Why Observed Information Requires More Than Averaging

In a joint model, each subject contributes longitudinal measurements and time-to-event information, while also having unobserved random effects. The likelihood for the observed data integrates over those effects. Consequently, the curvature of the observed-data log likelihood is not generally the average curvature of the complete-data log likelihood. Variation in the complete-data score across plausible random effects matters too.

Louis’ identity expresses this relationship exactly. It provides a way to calculate the observed score and observed information from posterior expectations and covariances of complete-data quantities. These calculations are useful when posterior expectations are available from numerical integration. The earlier material on marginal likelihood scores gives the corresponding first-derivative result; here the key additional feature is the score-covariance correction.

Definition: Let $y$ denote a subject’s observed longitudinal and event data, and let $b$ denote the subject’s latent random effects. Write $f_c(y,b;\theta)$ for the complete-data density, including the random-effects density, and define the observed-data density by $$ f(y;\theta)=\int f_c(y,b;\theta)\,db. $$ The complete-data log likelihood is $\ell_c(\theta;y,b)=\log f_c(y,b;\theta)$, its score is $S_c(\theta;y,b)=\nabla_\theta \ell_c(\theta;y,b)$, and its information matrix is $I_c(\theta;y,b)=-\nabla_\theta^2\ell_c(\theta;y,b)$. The observed log likelihood is $\ell(\theta;y)=\log f(y;\theta)$, and its observed information is $I_o(\theta;y)=-\nabla_\theta^2\ell(\theta;y)$.

All posterior expectations below are conditional on the observed data and evaluated at the same parameter value $\theta$. In particular, the relevant distribution of the random effects is $q_\theta(b\mid y)=f_c(y,b;\theta)/f(y;\theta)$. For vector-valued parameters, $S_c$ is a column vector, and $\operatorname{Cov}(S_c\mid y;\theta)$ is its covariance matrix.

Assumptions and Louis’ Identity

The identity applies when the integrated density is finite and positive, the complete-data density is sufficiently differentiable in $\theta$, and differentiation can be interchanged with integration. A standard sufficient condition is that, in a neighbourhood of the parameter value being considered, the first and second derivatives of the integrand are dominated by integrable functions. The support of $b$ is assumed not to depend on $\theta$; if it does, boundary terms or an appropriate transformation must be handled explicitly. These conditions let us differentiate the integral without omitting contributions.

Theorem: Under the stated regularity conditions, $$ \nabla_\theta \ell(\theta;y)=E\{S_c(\theta;y,b)\mid y;\theta\}, $$ and $$ I_o(\theta;y) =E\{I_c(\theta;y,b)\mid y;\theta\} -\operatorname{Cov}\{S_c(\theta;y,b)\mid y;\theta\}. $$ Equivalently, the observed Hessian is $$ \nabla_\theta^2\ell(\theta;y) =E\{\nabla_\theta^2\ell_c(\theta;y,b)\mid y;\theta\} +\operatorname{Cov}\{S_c(\theta;y,b)\mid y;\theta\}. $$

The first identity says that the observed-data score is the posterior mean of the complete-data score. The second says that observed information equals posterior expected complete-data information minus the posterior covariance of the complete-data score. The covariance is positive semidefinite, so in any direction it reduces information relative to the posterior expected complete-data information. The missing random effects leave uncertainty about the complete-data score, and that uncertainty reduces observed curvature.

Derivation

For clarity, first take derivatives with respect to a vector parameter. Since $\nabla_\theta f_c=f_c S_c$, differentiation under the integral gives $\nabla_\theta f(y;\theta)=\int f_c S_c\,db$. Dividing by $f(y;\theta)$ yields the score identity. To obtain the second derivative, differentiate $f_c S_c$: its derivative is $f_c\{S_cS_c^{\mathsf T}+\nabla_\theta^2\ell_c\}$. Applying the log-derivative formula then gives the observed Hessian.

\begin{aligned} \nabla_\theta\ell(\theta;y) &=\frac{\nabla_\theta f(y;\theta)}{f(y;\theta)} =\int q_\theta(b\mid y)S_c(\theta;y,b)\,db,\\ \nabla_\theta^2\ell(\theta;y) &=\frac{\nabla_\theta^2 f(y;\theta)}{f(y;\theta)} -\{\nabla_\theta\ell(\theta;y)\}\{\nabla_\theta\ell(\theta;y)\}^{\mathsf T}\\ &=E\{S_cS_c^{\mathsf T}+\nabla_\theta^2\ell_c\mid y;\theta\} -E(S_c\mid y;\theta)E(S_c\mid y;\theta)^{\mathsf T}\\ &=E\{\nabla_\theta^2\ell_c\mid y;\theta\} +\operatorname{Cov}(S_c\mid y;\theta). \end{aligned}

Negating the last line gives the observed-information formula because $I_c=-\nabla_\theta^2\ell_c$ and $I_o=-\nabla_\theta^2\ell$. This derivation also identifies an important implementation detail: the covariance is computed under the posterior distribution of $b$ given $y$, not under the prior random-effects distribution.

Worked Example: A Scalar Posterior Calculation

Suppose numerical integration at a fixed scalar parameter value gives three posterior support points for a subject’s random effect. The posterior probabilities are $0.2, 0.5, 0.3$; the corresponding complete-data scores are $-0.8, 0.2, 1.2$; and the complete-data information is $2.4$ at each point. These are hypothetical posterior summaries.

The observed score is the posterior-weighted mean: $$ 0.2(-0.8)+0.5(0.2)+0.3(1.2) =-0.16+0.10+0.36=0.30. $$

The posterior score variance is $$ 0.2(-0.8-0.3)^2+0.5(0.2-0.3)^2+0.3(1.2-0.3)^2 =0.2(1.21)+0.5(0.01)+0.3(0.81) =0.242+0.005+0.243=0.490. $$ The posterior expected complete-data information is $2.4$, since it is constant across the three points. Therefore, $$ I_o=2.4-0.490=1.91. $$

At this parameter value, the subject’s observed-data log likelihood has positive local information of $1.91$ in the scalar parameter’s units of inverse squared parameter units. The complete-data curvature alone would overstate that information by $0.490$, because it would ignore posterior uncertainty in the complete-data score.

Worked Example: A Two-Parameter Information Matrix

Consider two parameters, such as a longitudinal fixed effect and a survival association parameter. Suppose the posterior distribution has two possible score vectors, $(1,2)^{\mathsf T}$ with probability $0.25$, and $(-1,0)^{\mathsf T}$ with probability $0.75$. Suppose the complete-data information matrix is the same at both points: $$ I_c=\begin{pmatrix}3&0.4\\0.4&2\end{pmatrix}. $$

The posterior mean score is $$ E(S_c\mid y)=0.25\begin{pmatrix}1\\2\end{pmatrix} +0.75\begin{pmatrix}-1\\0\end{pmatrix} =\begin{pmatrix}-0.50\\0.50\end{pmatrix}. $$ The second-moment matrix is $$ E(S_cS_c^{\mathsf T}\mid y) =0.25\begin{pmatrix}1&2\\2&4\end{pmatrix} +0.75\begin{pmatrix}1&0\\0&0\end{pmatrix} =\begin{pmatrix}1&0.50\\0.50&1\end{pmatrix}. $$

Subtracting the outer product of the mean score gives $$ \operatorname{Cov}(S_c\mid y) =\begin{pmatrix}1&0.50\\0.50&1\end{pmatrix} -\begin{pmatrix}0.25&-0.25\\-0.25&0.25\end{pmatrix} =\begin{pmatrix}0.75&0.75\\0.75&0.75\end{pmatrix}. $$ Thus the observed information is $$ I_o=\begin{pmatrix}3&0.4\\0.4&2\end{pmatrix} -\begin{pmatrix}0.75&0.75\\0.75&0.75\end{pmatrix} =\begin{pmatrix}2.25&-0.35\\-0.35&1.25\end{pmatrix}. $$

The score covariance changes both diagonal entries and the cross-parameter entry. It is therefore not enough to adjust only separate variances or to calculate each parameter’s score variation while ignoring cross-products.

Using the Identity With Numerical Integration

In a joint model, posterior expectations over random effects are often approximated by quadrature. If integration produces nodes $b_k$ and normalized posterior weights $w_k$, then $\sum_k w_k=1$, and the posterior mean score is approximated by $\sum_k w_k S_c(b_k)$. The score covariance is approximated by the weighted sum of centered outer products. The same weights are used for the complete-data information. The resulting calculation is an approximation to Louis’ identity, so its accuracy depends on the accuracy of the posterior integration.

Worked Example: Posterior-Weighted Quadrature Summaries

At a candidate scalar parameter value, suppose quadrature gives two posterior nodes with normalized weights $0.6$ and $0.4$. Their complete-data scores are $-0.5$ and $1.0$, and their complete-data information values are $1.8$ and $2.3$, respectively. Treat these as hypothetical numerical integration output.

The score mean is $$ 0.6(-0.5)+0.4(1.0)=-0.30+0.40=0.10. $$ The score variance is $$ 0.6(-0.5-0.1)^2+0.4(1.0-0.1)^2 =0.6(0.36)+0.4(0.81)=0.216+0.324=0.540. $$ The expected complete-data information is $$ 0.6(1.8)+0.4(2.3)=1.08+0.92=2.00, $$ so the observed information approximation is $2.00-0.540=1.46$.

This calculation separates the two numerical tasks: average the complete-data information, then subtract the posterior score variance. If the posterior weights are not normalized, they must first be divided by their sum; otherwise these quantities are not posterior expectations.

Diagnostics, Alternatives, and Limitations

Several checks help detect an incorrect or unstable calculation. Posterior weights should be nonnegative and sum to one, up to numerical rounding. A scalar score variance cannot be negative, and a vector score-covariance matrix should be symmetric and positive semidefinite, up to small rounding error. The complete-data information and the score covariance should also be computed at the same parameter value and using the same posterior weights. Finally, increasing quadrature accuracy should stabilize the score, covariance, and resulting observed information; checking only the weighted mean score can miss error in the second moments.

A common failure is to use $E(I_c\mid y)$ as the observed information and omit the covariance term. That is correct only when the complete-data score has zero posterior variance, not merely when its posterior mean is small. For example, the first worked example has a positive observed score of $0.30$, but the correction is determined by the score variance $0.490$, not by the magnitude or sign of that mean.

An alternative is to differentiate the integrated log likelihood numerically, for example by finite differences of the marginal score. This directly targets the observed-data curvature and can serve as a useful check. However, numerical differentiation can amplify integration error, and the result can depend on step size. Louis’ identity instead uses complete-data derivatives and posterior moments; it makes the correction term explicit, but requires accurate posterior integration and correctly implemented derivatives. Agreement between the two approaches under tighter integration and sensible finite-difference step sizes is reassuring, though neither method removes model misspecification.

Louis’ identity is an exact identity under its regularity conditions, not a guarantee that a computed matrix is positive definite at every parameter value. Away from a local maximum, observed curvature may be indefinite. Even near a maximum, poor quadrature, unstable second derivatives, weak identification, or parameters on boundaries can make the numerical matrix unreliable. The identity also does not itself resolve whether the assumed joint model for longitudinal outcomes, event times, and random effects is appropriate.

Key takeaway: The observed information is posterior expected complete-data information minus posterior covariance of the complete-data score. The score covariance is the correction for uncertainty about the latent random effects, and it must be estimated with the same posterior distribution used for the other expectations.

Check Your Understanding

Try each question, then compare your reasoning with the solutions below.

In one sentence, state the scalar form of Louis’ identity for observed information.
Suppose the posterior expected complete-data information is $4.0$ and the posterior score variance is $1.3$. What is the observed information?
Why must the covariance in Louis’ identity be conditional on the observed data?
For the two-parameter example above, what is the posterior mean score?

Solutions. (1) $I_o=E(I_c\mid y)-\operatorname{Var}(S_c\mid y)$. (2) $I_o=4.0-1.3=2.7$, in inverse squared parameter units. (3) The observed likelihood integrates over latent random effects using their posterior distribution given the data; a prior covariance would use the wrong weights. (4) The posterior mean score is $(-0.50,\,0.50)^{\mathsf T}$, obtained by weighting the two score vectors by $0.25$ and $0.75$.