Observed Information and Likelihood Curvature

Information as Local Likelihood Curvature

A likelihood can be locally sharp or locally flat around a parameter value. Its curvature describes how quickly the log-likelihood changes when the parameter is perturbed, and information matrices quantify that curvature. Two related quantities are important: observed information, computed from the realized data, and expected information, averaged over data generated at a specified parameter.

For a regular exponential family, the distinction depends in part on the coordinates used. In natural coordinates, the log-likelihood’s Hessian does not involve the observed sufficient statistic, so observed and expected information coincide at every parameter value. After a nonlinear parameter transformation, an additional data-dependent term can appear. We derive both facts and examine why a stationary point in a lower-dimensional model does not, by itself, eliminate that term.

Definition: Let $\ell(\theta;x)$ be a twice differentiable log-likelihood for observed data $x$, with parameter $\theta\in\mathbb{R}^p$. The observed information is $J(\theta;x)=-\nabla_\theta^2\ell(\theta;x)$. The expected information is $I(\theta)=E_\theta[J(\theta;X)]$, provided this expectation exists. Expected information at a fixed parameter is not data-dependent. Observed information may depend on the sample, but in some models or parameterizations it is sample-independent.

Exponential-Family Setup

Consider $n$ independent observations $X_1,\ldots,X_n$ from a $k$-parameter canonical exponential family with density or mass function

f_\eta(x)=h(x)\exp\{\eta^T T(x)-A(\eta)\},\qquad \eta\in\mathcal{H}\subset\mathbb{R}^k,

where $\mathcal{H}$ is open, $T(x)\in\mathbb{R}^k$, and $A$ is the log-partition function. Assume the family is regular enough that $A$ is twice continuously differentiable and differentiation of its defining integral or sum is valid in a neighborhood of each $\eta\in\mathcal{H}$. For example, it suffices to have an integrable dominating function for the relevant first and second derivatives locally. Write $S=\sum_{i=1}^n T(X_i)$. Up to terms not depending on $\eta$, the log-likelihood is

\ell(\eta)=\eta^T S-nA(\eta)+\text{constant}.

Differentiation of the normalizing integral gives $\nabla A(\eta)=E_\eta[T(X)]$ and $\nabla^2 A(\eta)=\operatorname{Cov}_\eta(T(X))$. Thus the score and Hessian in natural coordinates are

U_\eta=\nabla_\eta\ell(\eta)=S-n\nabla A(\eta), \qquad \nabla_\eta^2\ell(\eta)=-n\nabla^2 A(\eta).

Theorem: Under the stated regularity conditions, the observed information in natural coordinates is $J_\eta(\eta;X)=n\nabla^2 A(\eta)$. The expected information is $I_\eta(\eta)=n\nabla^2 A(\eta)$ as well. In particular, these matrices are independent of the realized sample in natural coordinates.

Proof. The observed-information formula follows by taking the negative of the Hessian just derived. For the expected information, the same Hessian is a deterministic function of $\eta$, so taking its expectation over $X_1,\ldots,X_n$ leaves it unchanged:

I_\eta(\eta)=E_\eta[J_\eta(\eta;X)] =E_\eta[n\nabla^2 A(\eta)] =n\nabla^2 A(\eta).

This is a special feature of the canonical natural coordinates and this likelihood form; it is not a reason to regard observed information as sample-independent in every parameterization. $\square$

Worked Examples in Natural and Nonlinear Coordinates

Worked Example: Curvature in a Finite Natural Exponential Family

Let one observation take values $0,1,3$, with probability at $t$ proportional to $e^{\eta t}$. The log-partition function is $A(\eta)=\log(1+e^\eta+e^{3\eta})$. For $n=2$, suppose the observed values are $0$ and $3$, so $S=3$. Find observed and expected information at $\eta=0$.

At $\eta=0$, each of the three values has probability $1/3$. The mean of $T(X)$ is $(0+1+3)/3=4/3$. Its second moment is $(0^2+1^2+3^2)/3=10/3$, so

A''(0)=\operatorname{Var}_0(T(X)) =\frac{10}{3}-\left(\frac{4}{3}\right)^2 =\frac{14}{9}.

Consequently, observed information is $J_\eta(0;X)=2A''(0)=28/9$, and expected information is $I_\eta(0)=2A''(0)=28/9$. The equality holds for this sample, but the natural-coordinate result says it holds for every sample in this model.

Now write the natural parameter as a smooth function of another parameter, $\eta=\eta(\theta)$, where $\theta\in\mathbb{R}^p$ ranges over an open set and the map takes values in $\mathcal{H}$. Assume $\eta$ is twice continuously differentiable. Let $D\eta(\theta)$ be its $k\times p$ Jacobian, and let $\nabla^2\eta_j(\theta)$ denote the $p\times p$ Hessian of its $j$th component. The chain rule gives the score and Hessian in $\theta$-coordinates:

\nabla_\theta\ell =D\eta^T U_\eta, \qquad \nabla_\theta^2\ell =D\eta^T(\nabla_\eta^2\ell)D\eta +\sum_{j=1}^k (U_\eta)_j\nabla^2\eta_j.

Theorem: For the smooth parameterization above, the observed and expected information are $$ J_\theta =nD\eta^T\nabla^2 A(\eta)D\eta -\sum_{j=1}^k\{S_j-n(\nabla A(\eta))_j\}\nabla^2\eta_j, \qquad I_\theta =nD\eta^T\nabla^2 A(\eta)D\eta. $$ The expectation defining $I_\theta$ is at the specified $\theta$.

Proof. Substitute $\nabla_\eta^2\ell=-n\nabla^2 A(\eta)$ and $(U_\eta)_j=S_j-n(\nabla A(\eta))_j$ into the chain-rule Hessian. Negating that Hessian yields the displayed expression for $J_\theta$. Under the model at $\theta$, $E_\theta[S]=n\nabla A(\eta(\theta))$, because the observations are independent and $E_\eta[T(X_i)]=\nabla A(\eta)$. Therefore each residual in the sum has expectation zero. Taking expectations in $J_\theta$ leaves exactly $nD\eta^T\nabla^2 A(\eta)D\eta$, proving the formula for $I_\theta$. $\square$

Worked Example: A Nonlinear Coordinate Adds a Data-Dependent Term

Use the finite family from the preceding example, but set $\eta(\theta)=\theta+\theta^2/2$ near $\theta=0$. For the observations $0,3$, calculate both information quantities at zero.

Here $\eta(0)=0$, $\eta'(0)=1$, and $\eta''(0)=1$. The natural-coordinate score at zero is $U_\eta=S-nA'(0)=3-2(4/3)=1/3$. The scalar version of the transformation formula gives

J_\theta(0;X) =2A''(0)\{\eta'(0)\}^2-U_\eta\eta''(0) =\frac{28}{9}-\frac{1}{3} =\frac{25}{9}.

Expected information is $I_\theta(0)=2A''(0)\{\eta'(0)\}^2=28/9$, since the score residual has mean zero. Thus the observed and expected information differ for this sample in the nonlinear coordinate, even though they agreed in natural coordinates. The map is a valid local coordinate change at zero because $\eta'(0)\ne0$.

Curvature, Stationarity, and Parameterization

When the Hessian is continuous, a second-order Taylor expansion around a fixed $\theta$ gives

\ell(\theta+h) =\ell(\theta)+\nabla_\theta\ell(\theta)^T h -\frac{1}{2}h^T J_\theta(\theta;X)h +o(\lVert h\rVert^2).

At a stationary point, the linear term is zero, so the quadratic term gives the second-order contribution to the local change in log-likelihood. If $h^T J_\theta h>0$ for every nonzero direction $h$, the quadratic contribution is negative in every direction. It describes the leading nonzero change only when the relevant second-order curvature is nonzero; a vanishing quadratic term can leave higher-order terms to determine the local behavior.

A full-dimensional local coordinate change is a map with $p=k$ and nonsingular Jacobian. In that case, $\nabla_\theta\ell=D\eta^T U_\eta=0$ implies $U_\eta=0$. The score-weighted Hessian term in the information transformation therefore vanishes at a stationary point. This implication need not hold for a lower-dimensional parameterization: stationarity only requires $D\eta^T U_\eta=0$, which allows $U_\eta\ne0$.

Worked Example: A Stationary Point in a Lower-Dimensional Model

Consider a family with three possible statistic values $T=(0,0),(1,0),(0,1)$, each with unit base weight, and log-partition function $A(\eta)=\log(1+e^{\eta_1}+e^{\eta_2})$. Use three observations: one with statistic $(1,0)$ and two with statistic $(0,0)$, giving $S=(1,0)$. Restrict the natural parameter to the curve $\eta(\theta)=(\theta,-\theta^2/10)$. Compare observed and expected information at $\theta=0$.

At $\eta=(0,0)$, each outcome has probability $1/3$, so $\nabla A=(1/3,1/3)$. Hence $U_\eta=S-3\nabla A=(0,-1)$. The curve has $\eta'(0)=(1,0)$ and $\eta''(0)=(0,-1/5)$. Its scalar score is $\eta'(0)^T U_\eta=0$, so zero is a stationary point for this sample.

The variance of the first statistic component is $2/9$, and therefore the first diagonal entry of $\nabla^2 A(0,0)$ is $2/9$. The observed information formula gives

J_\theta(0;X) =3\left(\frac{2}{9}\right)-U_\eta^T\eta''(0) =\frac{2}{3}-\frac{1}{5} =\frac{7}{15}.

The expected information is $I_\theta(0)=3(2/9)=2/3$. These differ despite stationarity. Indeed, $U_\eta=(0,-1)$ is nonzero but orthogonal to the curve’s tangent $(1,0)$; its inner product with the curve’s second derivative is not zero. This verifies why stationarity alone does not remove the score-weighted term in a lower-dimensional model.

Common Pitfall and Practical Interpretation

A frequent error is to assert that observed information always depends on the sample, or, conversely, that it always equals expected information. Neither statement is generally correct. In natural coordinates for the canonical family above, the observed Hessian is sample-independent and equals expected information. In a nonlinear parameterization, the observed information can include a sample-dependent score residual multiplied by the curvature of the parameter map.

A second error is to infer $U_\eta=0$ from a zero score in a curved, lower-dimensional model. The score in the restricted model is $D\eta^T U_\eta$; a vector can have zero projection onto the model’s tangent directions without being zero. Finally, a zero score and a stationary point do not guarantee nonzero second-order curvature. The Hessian describes the second-order contribution; if it vanishes in a direction, higher-order behavior may matter.

Key takeaway: In natural coordinates, observed and expected information both equal $n\nabla^2 A(\eta)$. Under a smooth parameter map, expected information transforms by the Jacobian congruence, while observed information has an additional score-weighted Hessian term. Its disappearance at stationarity requires an appropriate condition, such as a full-dimensional local coordinate change.

Check Your Understanding

Use the definitions and derivations above to answer the following.

Why is observed information sample-independent in the canonical natural coordinates considered here?
What condition on the parameter map makes a stationary point imply $U_\eta=0$?
In a lower-dimensional model, what does stationarity say about $U_\eta$, and what does it not say?
Which term in the transformed observed information can depend on the realized sufficient statistic?
Why does a zero score at a stationary point not guarantee that the quadratic term is the leading nonzero local change?