Observed Information From the Marginal Likelihood

Why Curvature Matters

An NLME estimate is obtained by fitting a model to the observed data, but the location of the maximum alone does not show how sharply the likelihood distinguishes that estimate from nearby parameter values. The local shape of the likelihood provides one measure of how much the data constrain the parameters. The matrix that records this local curvature is the observed-information matrix.

For an NLME model, the likelihood relevant to population parameters is generally marginal: it accounts for subject-level random effects by integrating them out. The observed information is therefore calculated from the marginal log-likelihood, not by treating fitted random effects as if they were known. This tutorial derives the matrix and explains what its curvature can—and cannot—tell us. It does not turn the matrix into a parameter covariance matrix.

The Marginal Log-Likelihood and Its Curvature

Let $y_i$ denote all observations for subject $i$, and let $b_i$ be that subject’s random effects. Let $\theta$ collect the model parameters being estimated, including population parameters and any estimated random-effects or residual-error parameters. Given the model, the contribution of subject $i$ to the marginal likelihood integrates over the possible random effects:

L_i(\theta)=\int f(y_i\mid b_i,\theta)\,g(b_i\mid\theta)\,db_i.

Here $f(y_i\mid b_i,\theta)$ is the conditional data density and $g(b_i\mid\theta)$ is the random-effects density. If subjects are independent under the model, the marginal likelihood for $N$ subjects is the product of these contributions, and its log is

L(\theta)=\prod_{i=1}^{N}L_i(\theta), \qquad \ell(\theta)=\log L(\theta)=\sum_{i=1}^{N}\log L_i(\theta).

The marginal log-likelihood $\ell(\theta)$ is the function whose curvature is used below. The observed data are fixed in this calculation; the word “observed” distinguishes this curvature, evaluated for the realized data, from information averaged over possible data sets.

Definition: If $\ell(\theta)$ is twice differentiable, its observed-information matrix at an estimate $\hat{\theta}$ is $$ J(\hat{\theta})=-\nabla^2\ell(\hat{\theta}), $$ where $\nabla^2\ell(\hat{\theta})$ is the matrix of second partial derivatives of the marginal log-likelihood. For parameter components $\theta_j,\theta_k$, its $j,k$ entry is $-\partial^2\ell(\theta)/(\partial\theta_j\,\partial\theta_k)$, evaluated at $\hat{\theta}$.

This definition requires a specified parameterization and a differentiable marginal log-likelihood near the estimate. A familiar regular setting has independent subjects, a correctly specified model, a well-defined marginal density, and a finite, identifiable parameter vector. To interpret curvature at an interior maximum, the log-likelihood must be locally smooth there. If the maximum is on a parameter boundary, the usual unconstrained stationary-point argument does not apply without qualification.

Why the Negative Hessian Appears

The score is the gradient $U(\theta)=\nabla\ell(\theta)$. At an interior differentiable maximum $\hat{\theta}$, the first-order condition is $U(\hat{\theta})=0$. A second-order Taylor expansion around that point gives

\ell(\hat{\theta}+\delta) = \ell(\hat{\theta}) + U(\hat{\theta})^{\mathsf T}\delta + \frac{1}{2}\delta^{\mathsf T}\nabla^2\ell(\hat{\theta})\delta + o(\|\delta\|^2).

The score term is zero at the interior maximum. Substituting $J(\hat{\theta})=-\nabla^2\ell(\hat{\theta})$ therefore yields

\ell(\hat{\theta}+\delta)-\ell(\hat{\theta}) \approx -\frac{1}{2}\delta^{\mathsf T}J(\hat{\theta})\delta.

This approximation is the central justification: the observed information is the negative curvature of the log-likelihood, so a direction with greater positive curvature in $J$ has a more rapid local decrease in log-likelihood when moving away from the maximum. At a local maximum, the Hessian must be negative semidefinite, making $J$ positive semidefinite. A strict, locally curved maximum has positive curvature in every nonzero direction, so $J$ is positive definite. Zero or nearly zero curvature in some direction signals a locally flat likelihood in that direction.

The quadratic form also shows why off-diagonal entries matter. A parameter change involving several components is assessed by $\delta^{\mathsf T}J\delta$, not by looking only at the diagonal entries. Eigenvectors of $J$ identify local directions of curvature, and the corresponding eigenvalues quantify the curvature along them. This is a local description: it does not guarantee that the likelihood remains quadratic far from the maximum.

Worked Example: Curvature for One Transformed Parameter

Suppose a hypothetical marginal log-likelihood for $\theta=\log(CL/(1\ \mathrm{L/h}))$ has the local form $\ell(\theta)=-40-4(\theta-0.20)^2$. The parameter $\theta$ is dimensionless because clearance $CL$ is divided by its reference unit. Find the maximum and observed information, then interpret a nearby point.

The squared term is nonnegative and is zero at $\hat{\theta}=0.20$, so this is the maximum. Differentiating, $$ \frac{d\ell}{d\theta}=-8(\theta-0.20), \qquad \frac{d^2\ell}{d\theta^2}=-8. $$ Thus $J(\hat{\theta})=-(-8)=8$ in inverse squared units of $\theta$ (here, dimensionless).

At $\theta=0.30$, the displacement is $\delta=0.10$. The exact log-likelihood difference is $$ \ell(0.30)-\ell(0.20)=-4(0.10)^2=-0.0400. $$ The curvature approximation gives $-\tfrac12(8)(0.10)^2=-0.0400$, which agrees exactly because the specified log-likelihood is quadratic. The value 8 describes local curvature; by itself it is not a confidence interval or a probability statement about $\theta$.

Reading a Matrix, Not Just Its Diagonal

For multiple parameters, each diagonal entry describes curvature when that parameter changes alone, holding the others fixed. Cross-derivatives describe how the joint shape tilts: a parameter combination can be much flatter or steeper than either coordinate direction suggests. The following calculation uses transformed, dimensionless parameters to avoid mixing the units of clearance and volume.

Worked Example: A Two-Parameter Observed-Information Matrix

Consider hypothetical estimates $\hat{\theta}_1=1.61$ and $\hat{\theta}_2=3.00$, where $\theta_1=\log(CL/(1\ \mathrm{L/h}))$ and $\theta_2=\log(V/(1\ \mathrm{L}))$. Suppose the marginal log-likelihood near the maximum is $$ \ell(\theta)=C-\frac12\{4u^2+2uv+9v^2\}, \qquad u=\theta_1-1.61,\quad v=\theta_2-3.00, $$ where $C$ is a constant. Find $J(\hat{\theta})$ and check whether the quadratic form indicates positive curvature in every direction.

The quadratic expression has matrix representation $$ 4u^2+2uv+9v^2 = \begin{pmatrix}u&v\end{pmatrix} \begin{pmatrix}4&1\\1&9\end{pmatrix} \begin{pmatrix}u\\v\end{pmatrix}. $$ Since the log-likelihood contains minus one-half of this expression, its Hessian is $$ \nabla^2\ell(\hat{\theta})= -\begin{pmatrix}4&1\\1&9\end{pmatrix}, \qquad J(\hat{\theta})= \begin{pmatrix}4&1\\1&9\end{pmatrix}. $$

For a symmetric two-by-two matrix, positive definiteness follows if its first diagonal entry and determinant are positive. Here $4>0$ and $\det(J)=4(9)-1(1)=35>0$, so the matrix is positive definite. The cross-entry 1 contributes to curvature for joint changes; for example, with $u=v=0.10$, the quadratic form is $4(0.01)+2(0.01)+9(0.01)=0.15$, and the approximate log-likelihood change is $-0.5(0.15)=-0.0750$. Reporting only the two diagonal entries would omit this joint contribution.

Numerical Calculation and Diagnostics

In practice, software may compute second derivatives analytically, by automatic differentiation, or numerically. For a scalar parameter, a centered finite-difference approximation to the second derivative is

\ell''(\hat{\theta}) \approx \frac{\ell(\hat{\theta}+h)-2\ell(\hat{\theta})+\ell(\hat{\theta}-h)}{h^2},

where $h$ is a small step in the parameter’s coordinate system. For a multivariate likelihood, corresponding second differences estimate diagonal entries, and mixed differences estimate cross-derivatives. The negative of the resulting Hessian estimates observed information. These approximations can be sensitive to step size, numerical integration error, and optimization error.

Worked Example: Checking Scalar Curvature by Finite Differences

Suppose a hypothetical fitted marginal log-likelihood has values $\ell(\hat{\theta})=-20.000$, $\ell(\hat{\theta}+0.10)=-20.040$, and $\ell(\hat{\theta}-0.10)=-20.040$. Estimate the second derivative and observed information using $h=0.10$.

The centered second difference is $$ \ell''(\hat{\theta}) \approx \frac{-20.040-2(-20.000)-20.040}{(0.10)^2} = \frac{-0.0800}{0.0100} =-8.00. $$ Therefore $J(\hat{\theta})\approx 8.00$. The equal likelihood values on either side are consistent with a locally symmetric maximum, and the negative estimated second derivative is consistent with local downward curvature of $\ell$. This check does not establish that the model is correct or that the curvature estimate is stable; it only checks the local numerical shape at the chosen step size.

Useful diagnostics examine whether the computed curvature behaves as the fitted maximum should. Check that the score is small at the reported estimate, that the Hessian is approximately symmetric, and that its eigenvalues have the expected signs. At an interior maximum, materially positive eigenvalues of the log-likelihood Hessian (equivalently, materially negative eigenvalues of $J$) contradict local maximality. A nearly zero eigenvalue can indicate weak identification, a redundant parameter combination, or inadequate numerical precision. Recomputing with different finite-difference steps, tighter integration tolerances, or a more accurate derivative method helps distinguish numerical instability from genuine flatness.

Curvature also depends on parameterization. If a model is fitted in log parameters and the Hessian is calculated in those coordinates, it describes changes in log parameters, not equal-sized changes in the original parameters. A change of coordinates alters the numerical Hessian; at a stationary point, its curvature transforms according to the local coordinate mapping. Thus, record the parameterization and units used when interpreting or comparing observed-information matrices.

How This Curvature Differs From Other Summaries

The observed information is the negative Hessian for the realized marginal log-likelihood. Expected information instead averages curvature over data sets generated under the model. Under suitable regularity conditions, these quantities can have the same expectation-based relationships to score variability, but a particular observed data set need not have the average curvature. Profile likelihood examines likelihood changes after optimizing nuisance parameters at each fixed value of a parameter of interest; it can reveal non-quadratic or asymmetric shape that a local Hessian does not capture. Bootstrap methods assess variation across resampled or simulated data sets rather than reading uncertainty from one local curvature calculation.

These approaches answer related but distinct questions. The Hessian is compact and local, and it can be useful for diagnosing flat or sharply curved directions near a fitted maximum. It can be misleading when the likelihood is strongly asymmetric, multimodal, close to a boundary, or poorly approximated by a quadratic over relevant parameter changes. A Hessian from a conditional likelihood that holds estimated random effects fixed is not a substitute for the Hessian of the marginal likelihood when the target is marginal population-parameter uncertainty.

Common Mistakes and Limitations

Dropping the minus sign. At a maximum, the log-likelihood Hessian is negative semidefinite; observed information is its negative, so it is positive semidefinite.
Using conditional curvature by accident. The target here is the marginal log-likelihood, which integrates over random effects. Conditional curvature answers a different calculation.
Interpreting a large entry as certainty on its own. Curvature depends on the parameter scale and the other parameters. Joint directions and scaling matter.
Treating local curvature as a complete uncertainty analysis. A local Hessian does not reveal distant modes, severe asymmetry, or all boundary effects, and it does not verify model adequacy.
Ignoring optimization and numerical error. If the score is not close to zero, or curvature changes substantially with numerical settings, the computed matrix may not describe the fitted maximum reliably.

At a boundary, the likelihood may have a constrained maximum even when the unconstrained score is not zero; the Taylor argument based on an interior stationary point then needs modification. Likewise, a singular or indefinite observed-information matrix should not be treated as an ordinary measure of well-behaved curvature. It calls for investigation of the optimization, parameterization, identifiability, and likelihood shape.

Key takeaway: Observed information is the negative Hessian of the marginal log-likelihood at the fitted estimate. Its quadratic form describes local likelihood loss in parameter directions, provided the maximum is sufficiently smooth and the numerical calculation is reliable. Curvature is informative, but it is not by itself a complete account of parameter uncertainty.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

At a smooth interior maximum, why is observed information defined with a minus sign in front of the log-likelihood Hessian?
A two-parameter observed-information matrix has a nearly zero eigenvalue. What does that suggest about the local likelihood, and what is one possible diagnostic follow-up?
Why must an NLME marginal likelihood integrate over subject-level random effects before calculating the observed information described here?
What does a cross-derivative contribute that cannot be learned by inspecting only the diagonal entries?

Solutions. (1) The log-likelihood curves downward at a smooth maximum, so its Hessian is negative semidefinite; negating it gives nonnegative curvature in the corresponding directions. (2) It suggests a nearly flat parameter combination and potentially weak identification or numerical instability; for example, check the score and recompute curvature with alternative derivative settings. (3) The marginal likelihood accounts for the random effects as latent quantities rather than treating their fitted values as known; its Hessian is the curvature of the likelihood relevant to the marginal parameters. (4) The cross-derivative captures how simultaneous changes in two parameters affect curvature, so joint directions may be steeper or flatter than either coordinate direction alone.