Observed Information From the Integrated Likelihood

Why the Marginal Likelihood Curvature Matters

In a shared-random-effects joint model, the random effects are latent: the likelihood for inference about population parameters must account for them rather than treating their unobserved values as known. Integrating them out gives the marginal likelihood. Its curvature describes how quickly that likelihood changes near a parameter value. The negative curvature, organized as a matrix, is the observed information.

This tutorial focuses on the Hessian of that integrated likelihood. The result is useful for describing local likelihood shape and is an important ingredient in likelihood-based inference. It is not generally the same as the curvature of a complete-data likelihood conditional on the random effects. In particular, integrating over latent effects can reduce information about a parameter.

Marginal Likelihood and Observed Information

Let $i=1,\ldots,n$ index independent subjects. For subject $i$, let $Y_i$ denote the observed longitudinal measurements and let $T_i$ and $\delta_i$ denote the observed event or censoring time and event indicator. Let $b_i$ be the subject’s latent random-effects vector, and let $\theta$ collect all unknown population parameters. Write $f_i(Y_i,T_i,\delta_i\mid b_i;\theta)$ for the subject’s conditional data likelihood and $p(b_i;\theta)$ for the random-effects density. The integrated subject likelihood is

L_i(\theta)=\int f_i(Y_i,T_i,\delta_i\mid b_i;\theta)\,p(b_i;\theta)\,db_i.

Under conditional independence across subjects, the marginal log likelihood is $\ell(\theta)=\sum_{i=1}^n \log L_i(\theta)$. The integration is over the random-effects vector, not over the observed data. For an $r$-dimensional parameter vector, the score is the vector of first derivatives and the Hessian is the $r\times r$ matrix of second derivatives.

Definition: The observed information at $\theta$ is the negative Hessian of the marginal log likelihood, $$ \mathcal I_{\mathrm{obs}}(\theta)=-\nabla^2\ell(\theta). $$ Its $(j,k)$ entry is $-\partial^2\ell(\theta)/(\partial\theta_j\,\partial\theta_k)$. At an interior maximum-likelihood estimate $\hat\theta$, the matrix is evaluated at $\hat\theta$. Its units depend on the units used for the parameters.

Observed information is distinct from expected Fisher information, which is an expectation of a likelihood curvature (or, under regularity conditions, of score products) over repeated data sets. It is also distinct from the complete-data information obtained by pretending the random effects had been observed. The target here is specifically the curvature of the likelihood after integration.

How Differentiation Passes Through the Integral

For clarity, write $g_i(b;\theta)=f_i(Y_i,T_i,\delta_i\mid b;\theta)p(b;\theta)$, suppressing the subject index temporarily, and let $U_c(b;\theta)=\nabla_\theta\log g_i(b;\theta)$ be the complete-data score. If the integral is finite, the support does not vary with $\theta$, and differentiation can be interchanged with integration, then

\nabla_\theta L_i(\theta)=\int g_i(b;\theta)U_c(b;\theta)\,db, \qquad \nabla_\theta\log L_i(\theta) =E\{U_c(b;\theta)\mid Y_i,T_i,\delta_i;\theta\}.

The conditional expectation is with respect to the random-effects distribution given that subject’s observed data and the current parameter value. This identity shows why a marginal score averages complete-data scores over the posterior distribution of the latent effects. It does not say that the marginal Hessian is just the average complete-data Hessian.

To obtain the Hessian, differentiate the score once more. Let $H_c=\nabla_\theta^2\log g_i(b;\theta)$. Differentiating the posterior weights as well as the score gives

\nabla_\theta^2\log L_i(\theta) =E(H_c\mid Y_i,T_i,\delta_i;\theta) +\operatorname{Cov}(U_c\mid Y_i,T_i,\delta_i;\theta).

Consequently, the subject’s observed information can also be written as

\mathcal I_{\mathrm{obs},i}(\theta) =E(-H_c\mid Y_i,T_i,\delta_i;\theta) -\operatorname{Cov}(U_c\mid Y_i,T_i,\delta_i;\theta).

The covariance term accounts for uncertainty about the latent effects. Summing these subject-level contributions gives the full observed information. This is Louis’ identity; it provides an alternative to differentiating the marginal log likelihood directly and a useful independent check on a calculation.

Worked Calculations

Worked Example: Scalar Curvature From Integrated Likelihood Values

Suppose $\theta$ is a scalar association parameter measured per one-unit increase in a longitudinal marker. At a candidate estimate $\hat\theta=0.40$, numerical integration gives the following marginal log likelihood values: $\ell(0.40)=-120.0000$, $\ell(0.41)=-120.0041$, and $\ell(0.39)=-120.0039$. Use a central second difference with step $h=0.01$.

For a twice-differentiable function, the central second-difference estimate of the second derivative is $\{\ell(\hat\theta+h)-2\ell(\hat\theta)+\ell(\hat\theta-h)\}/h^2$. Thus the observed information estimate is its negative:

\widehat{\mathcal I}_{\mathrm{obs}}(\hat\theta) =-\frac{-120.0041-2(-120.0000)-120.0039}{(0.01)^2} =-\frac{-0.0080}{0.0001}=80.0.

The estimated log-likelihood curvature is $-80.0$ log-likelihood units per squared unit of $\theta$, so the observed information is $80.0$ per squared unit of $\theta$. The negative curvature is consistent with a local maximum in this one-dimensional direction. This calculation alone does not establish that $0.40$ is a maximum in all directions or that the numerical integration is sufficiently accurate.

Worked Example: A Two-Parameter Information Matrix

Suppose two parameters are $\theta_1$ and $\theta_2$, and the log likelihood near a candidate estimate has value $-50.000$ at the center. Evaluations at offsets $+0.1$ or $-0.1$ in each coordinate give $\ell_{++}=-50.055$, $\ell_{+-}=-50.095$, $\ell_{-+}=-50.095$, and $\ell_{--}=-50.055$, where the signs refer to the offsets for $(\theta_1,\theta_2)$. Suppose the estimated diagonal second derivatives are $-10$ and $-5$.

The mixed derivative is estimated by the four-corner difference. Substitution gives

\frac{\ell_{++}-\ell_{+-}-\ell_{-+}+\ell_{--}} {4(0.1)(0.1)} = \frac{-50.055+50.095+50.095-50.055}{0.04} =\frac{0.080}{0.04}=2.

The estimated Hessian and observed information are therefore

\widehat{\nabla^2\ell}= \begin{pmatrix}-10&2\\2&-5\end{pmatrix}, \qquad \widehat{\mathcal I}_{\mathrm{obs}}= \begin{pmatrix}10&-2\\-2&5\end{pmatrix}.

The information matrix is symmetric, as a well-behaved Hessian should be when mixed derivatives are continuous. Its first leading diagonal entry is $10>0$, and its determinant is $10(5)-(-2)^2=46>0$; hence it is positive definite. Equivalently, the Hessian is negative definite, which is consistent with a strict local maximum for these two parameters. Positive definiteness is a local curvature check, not proof that the fitted model or the numerical integration is correct.

Worked Example: Interpreting the Latent-Effect Correction

At a particular parameter value for one subject, suppose a quadrature approximation to the conditional distribution of a latent effect assigns probability $0.25$ to a node with complete-data score $U_c=-2$ and negative complete-data second derivative $-H_c=8$. It assigns probability $0.75$ to a node with score $U_c=2$ and $-H_c=4$. Apply the scalar form of Louis’ identity.

The posterior mean score is $0.25(-2)+0.75(2)=1$. The expected negative complete-data curvature is $0.25(8)+0.75(4)=5$. The score variance is

\operatorname{Var}(U_c\mid\text{data}) =0.25(-2-1)^2+0.75(2-1)^2 =0.25(9)+0.75(1)=3.

Therefore the observed information contribution is $5-3=2$, not $5$. The subtraction reflects the uncertainty in the latent effect. These probabilities and scores are hypothetical quadrature summaries; they illustrate the identity, not data from an empirical study.

Numerical Choices, Diagnostics, and Alternatives

In practice, the marginal likelihood may itself be approximated by quadrature or another integration method. There are two broad routes to its Hessian: differentiate the integrated likelihood (analytically or with automatic differentiation), or use a finite-difference approximation to the marginal log likelihood. Louis’ identity offers a third route when posterior expectations of complete-data derivatives can be computed. These methods target the same observed information when their calculations are exact and the regularity conditions hold, but their numerical errors can differ.

Check the likelihood calculation.
Confirm that the marginal likelihood integrates the conditional data likelihood against the random-effects density and that the same parameterization and data contributions are used at every evaluation.

Check derivative stability.
For finite differences, compare more than one reasonable step size. A step that is too large gives truncation error; one that is too small can magnify rounding or integration error. If using quadrature, check that derivative estimates are stable as integration accuracy is increased.

Check the matrix structure.
Inspect symmetry, eigenvalues or positive definiteness at an interior fitted maximum, and the scale of entries. Meaningful asymmetry can indicate inconsistent mixed derivatives or numerical error. A nearly singular matrix can indicate weak identification or parameters on very different scales.

Cross-check when feasible.
Compare direct Hessian calculations with Louis’ identity or with a second differentiation method. Agreement across methods and integration resolutions is more persuasive than a single plausible-looking matrix.

For a smooth likelihood at an interior local maximum, the Hessian should be negative semidefinite; if the maximum is strict and locally well identified, it should be negative definite. An indefinite matrix at a claimed maximum may signal that optimization has not converged, that the point is a saddle, or that numerical error is substantial. Near a boundary, however, curvature in unconstrained coordinates need not satisfy the same simple interpretation, and ordinary interior-point likelihood approximations may be unreliable.

Parameterization matters. Under a linear change of coordinates, curvature transforms according to the corresponding change of scale. Under a nonlinear transformation, the Hessian also includes terms involving the score; at a stationary point those terms vanish, but away from one they need not. Thus information matrices from different parameterizations should not be compared entry by entry without accounting for the transformation.

Common Pitfalls and Scope

Using complete-data curvature as if it were observed information. The correction is the negative conditional covariance of the complete-data score, as shown by Louis’ identity.
Taking the Hessian without its negative sign. Observed information is $-\nabla^2\ell$; at a strict local maximum it is positive definite, not negative definite.
Differentiating an inaccurate integral. Small errors in integrated likelihood values can become large errors in finite-difference second derivatives. Check integration accuracy and step-size stability together.
Treating any computed matrix as reliable uncertainty information. A matrix can be symmetric and positive definite while still being based on a misspecified model, a poorly identified fit, or inaccurate integration.

Observed information summarizes local curvature of the fitted marginal likelihood. It supports likelihood-based inference, but its use for standard errors requires additional calculation and regularity considerations. A boundary estimate, a nearly singular matrix, or failure of the curvature and numerical checks calls for caution rather than an automatic interpretation.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

For a scalar parameter, a central difference gives $\ell(\hat\theta+h)-2\ell(\hat\theta)+\ell(\hat\theta-h)=-0.0036$ with $h=0.03$. What is the estimated observed information?
Why is the observed information not generally just the posterior expectation of negative complete-data curvature?
A computed two-parameter information matrix has a negative eigenvalue at a claimed strict interior maximum. Give two possible explanations.
What is one advantage and one limitation of finite differences compared with differentiating the integrated likelihood directly?

Solutions. (1) $-(-0.0036)/(0.03)^2=0.0036/0.0009=4.00$, in inverse squared units of the parameter. (2) The posterior distribution of the latent effects depends on the parameters; differentiating its weights contributes the score covariance. Louis’ identity subtracts that covariance from the expected negative complete-data Hessian. (3) The optimizer may not have reached a strict local maximum, or numerical error in the integration or derivatives may have distorted the matrix; weak identification or a boundary issue should also be investigated. (4) Finite differences are straightforward to apply to a marginal log likelihood that can be evaluated but is difficult to differentiate. They can be unstable because second derivatives amplify integration and rounding errors and depend on step size.