Observed and Expected Information as Optimization Curvature

Why Two Curvature Measures?

A likelihood update depends on both the score, which indicates the direction of local increase, and curvature, which determines how far that direction is followed. In a generalized linear model (GLM), the likelihood curvature at the observed data is not always the same as its average over possible data sets at the current parameter value. These two quantities are called observed information and expected information.

The distinction is consequential for computation. If the curvature matrices differ, multiplying the same score by their respective inverses can give updates with different lengths and, in a model with multiple coefficients, different directions. Expected information is not simply a more accurate version of observed information: it averages over the response distribution, while observed information describes the particular data in hand.

Definitions and GLM Formulas

Consider independent responses \(Y_1,\ldots,Y_n\) from an exponential-dispersion family, with fixed design vectors \(x_i\) and linear predictors \(\eta_i=x_i^\mathsf{T}\beta\). For this derivation, treat the dispersion parameter as known and fixed. Write the log-likelihood as \(\ell(\beta)\), the score as \(U(\beta)=\partial\ell(\beta)/\partial\beta\), and the Hessian as \(\nabla^2\ell(\beta)\).

Definition: The observed information at \(\beta\) is \(J(\beta)=-\nabla^2\ell(\beta)\), the negative Hessian evaluated at the observed responses. The expected information is \(I(\beta)=E_\beta[J(\beta)]\), the expectation over the response distribution at \(\beta\). When the expectations and derivatives satisfy the regularity conditions stated below, \(I(\beta)=E_\beta[U(\beta)U(\beta)^\mathsf{T}]\) as well.

To make the distinction concrete, use the exponential-dispersion representation \(\ell_i=(Y_i\theta_i-b(\theta_i))/a(\phi)+c(Y_i,\phi)\), with \(\mu_i=b'(\theta_i)\) and \(V_i=\operatorname{Var}(Y_i)=a(\phi)b''(\theta_i)\). Let \(\mu_i'\) denote \(d\mu_i/d\eta_i\), and define \(A_i=\mu_i'/V_i\) and \(q_i=(\mu_i')^2/V_i\). Assume the link and mean functions are differentiable as needed and the variances are positive at the parameter values under consideration.

U(\beta)=\sum_{i=1}^n x_i A_i(Y_i-\mu_i),\qquad J(\beta)=\sum_{i=1}^n x_ix_i^\mathsf{T} \left[q_i-\frac{dA_i}{d\eta_i}(Y_i-\mu_i)\right],\qquad I(\beta)=\sum_{i=1}^n q_i x_ix_i^\mathsf{T}.

The score follows by differentiating each observation’s log-likelihood with respect to \(\eta_i\), then using \(d\eta_i/d\beta=x_i\). Differentiating its score contribution \(A_i(Y_i-\mu_i)\) with respect to \(\eta_i\) gives \((dA_i/d\eta_i)(Y_i-\mu_i)-A_i\mu_i'\). Its negative is the observed-curvature term in \(J\). Taking expectation replaces \(Y_i-\mu_i\) by zero, giving \(I\). Thus the observed and expected matrices differ through a data-dependent residual term.

Two General Results

The first result explains why expected information can also be described as score variability. Its hypotheses matter: the identity is not automatic if the support depends on the parameter or differentiation cannot be interchanged with integration.

Theorem: Suppose the response distribution has parameter-independent support, its density or mass function \(f(y;\beta)\) is twice differentiable in a neighbourhood of \(\beta\), and differentiation can be interchanged with integration (or summation) for the first and second derivatives. Suppose also that the score has finite second moments and the relevant derivatives and expectations exist. Then \(E_\beta[U(\beta)]=0\) and \(E_\beta[J(\beta)]=E_\beta[U(\beta)U(\beta)^\mathsf{T}]\).

Differentiate total probability.
Because the support is fixed and differentiation may pass under the integral, differentiating \(\int f(y;\beta)\,dy=1\) yields \(\int \nabla f(y;\beta)\,dy=0\). Since \(\nabla f=fU\), this gives \(E_\beta[U]=0\).

Differentiate the density a second time.
The identity \(\nabla f=fU\), differentiated again, gives \(\nabla^2 f=f\{\nabla^2\ell+UU^\mathsf{T}\}\). Integrating each entry and differentiating \(\int f(y;\beta)\,dy=1\) twice gives \(0=E_\beta[\nabla^2\ell+UU^\mathsf{T}]\).

Change the sign.
Since \(J=-\nabla^2\ell\), the preceding identity is \(E_\beta[J]=E_\beta[UU^\mathsf{T}]\). Together with the first step, the latter matrix is the covariance matrix of the score. \(\square\)

For the GLM formulas above, the expectation identity can also be checked directly: each residual has mean zero, so the residual term in \(J\) vanishes in expectation. Independence and \(\operatorname{Var}(Y_i)=V_i\) give the stated matrix \(I\).

The next result explains how a curvature matrix enters a local update. It is a statement about a quadratic approximation, not a guarantee that a full likelihood step will increase the likelihood.

Result: If \(J(\beta)\) is positive definite, the quadratic approximation to \(\ell(\beta+d)\) at \(d=0\) has its unique maximum at \(d=J(\beta)^{-1}U(\beta)\). If \(I(\beta)\) is positive definite, replacing observed by expected curvature gives the corresponding quadratic-approximation step \(d=I(\beta)^{-1}U(\beta)\).

To verify this, a twice-differentiable log-likelihood has the local expansion \(\ell(\beta+d)=\ell(\beta)+U^\mathsf{T}d-\tfrac12d^\mathsf{T}Jd+o(\lVert d\rVert^2)\). The quadratic part has gradient \(U-Jd\). Setting that gradient to zero yields \(d=J^{-1}U\), and positive definiteness makes the quadratic strictly concave, so this stationary point is its unique maximum. Replacing \(J\) by a positive-definite \(I\) gives \(d=I^{-1}U\) by the same calculation. \(\square\)

Worked Comparisons

Worked Example: One Bernoulli Observation

Let \(Y\) be Bernoulli with probability \(p=\beta\), using the identity link, and evaluate at \(\beta=0.4\) with observed value \(y=1\). The calculation is local: the identity-link mean must remain in the probability range. Here \(V=p(1-p)=0.24=6/25\), \(\mu'=1\), \(A=1/V=25/6\), and \(q=1/V=25/6\).

The score is \(U=A(y-p)=(25/6)(0.6)=5/2\). Also \(dA/d\eta=-(1-2p)/[p(1-p)]^2=-(1/5)/(36/625)=-125/36\). Therefore the observed information is

J=q-\frac{dA}{d\eta}(y-p) =\frac{25}{6}-\left(-\frac{125}{36}\right)\frac{3}{5} =\frac{25}{4}=6.25, \qquad I=q=\frac{25}{6}\approx4.167.

The observed-curvature step is \(U/J=(5/2)/(25/4)=0.4\). The expected-curvature step is \(U/I=(5/2)/(25/6)=0.6\). The score is the same in both calculations, but the smaller observed step results from the larger observed curvature at this data point.

Worked Example: Curvature Changes the Update Direction

Use a two-coefficient Bernoulli identity-link model with rows \(x_1=(1,0)^\mathsf{T}\), \(x_2=(1,1)^\mathsf{T}\), and current value \(\beta=(0.4,0)^\mathsf{T}\). Both fitted probabilities are \(0.4\). Let the observed responses be \(y_1=1\) and \(y_2=0\). At each row, \(A=25/6\), \(q=25/6\), and \(dA/d\eta=-125/36\).

The residuals are \(r_1=0.6=3/5\) and \(r_2=-0.4=-2/5\). The score contributions are \(x_i A r_i\), so

U=\frac{25}{6}\left(\frac{3}{5}x_1-\frac{2}{5}x_2\right) =\begin{pmatrix}5/6\\-5/3\end{pmatrix}.

The two observed scalar weights are \(q-(dA/d\eta)r_1=25/4\) and \(q-(dA/d\eta)r_2=25/9\). Thus

J=\frac{25}{4}x_1x_1^\mathsf{T}+\frac{25}{9}x_2x_2^\mathsf{T} =\frac{25}{36}\begin{pmatrix}13&4\\4&4\end{pmatrix}, \qquad I=\frac{25}{6}\begin{pmatrix}2&1\\1&1\end{pmatrix}.

For observed curvature, \(J^{-1}=(1/25)\begin{pmatrix}4&-4\\-4&13\end{pmatrix}\), giving \(J^{-1}U=(2/5,-1)^\mathsf{T}\). For expected curvature, \(I^{-1}=(6/25)\begin{pmatrix}1&-1\\-1&2\end{pmatrix}\), giving \(I^{-1}U=(3/5,-1)^\mathsf{T}\). The first coefficient changes by a different amount, so the two updates are not merely different-length multiples of one another: their directions differ.

Worked Example: A Poisson Log-Link Case With Matching Curvature

For one Poisson response with log link, take \(x=1\), current coefficient \(\beta=\log 2\), and observed count \(y=3\). Then \(\mu=e^\beta=2\), \(\operatorname{Var}(Y)=2\), and \(d\mu/d\eta=2\). Consequently \(A=1\) and \(q=2\).

The score is \(U=A(y-\mu)=3-2=1\). Since \(A\) is constant as a function of \(\eta\), \(dA/d\eta=0\), and the observed information is \(J=q=2\). Expected information is also \(I=q=2\). Both quadratic updates are \(1/2\). This example shows that the two curvature measures can agree at a particular model and parameter value; it does not imply they agree for every GLM.

Interpreting the Difference

The formula for \(J-I\) makes the role of the data explicit:

J-I=-\sum_{i=1}^n x_ix_i^\mathsf{T}\frac{dA_i}{d\eta_i}(Y_i-\mu_i).

Each residual term has expectation zero under the model, but its realized value can be positive or negative. The resulting change in curvature can affect different coefficient combinations unequally, as in the two-coefficient example. If both matrices are invertible, the steps \(J^{-1}U\) and \(I^{-1}U\) therefore need not coincide. Their difference also does not have a fixed sign or a universal effect on step size.

The formulas also identify a sufficient condition for equality at a parameter value: if \(dA_i/d\eta_i=0\) for every observation, then \(J=I\) for any observed responses. The Poisson log-link example has this property. Equality can also occur when the residual terms cancel after their matrix contributions are summed. It is important to check the actual model and data rather than assume that the matrices match.

Common mistake: “Expected information is the average of observed information, so it must give the same update on the observed data.” The first clause is correct when the expectation exists; the conclusion is not. Averaging a matrix and then inverting it is generally different from using the realized matrix, and both updates are multiplied by the same score only after those different curvature matrices are chosen. In general, \(I^{-1}U\ne J^{-1}U\).

A second useful caution is that information is a matrix measure of local curvature, not a direct measure of how far the likelihood will rise after a proposed step. The quadratic calculation describes local geometry; the likelihood itself can depart from that approximation away from the current coefficient value.

Check Your Understanding

Use the definitions and calculations above to answer the following questions.

In a GLM, which part of the observed-information formula disappears when the expectation is taken under the model?
What regularity conditions are needed for expected information to equal the score’s covariance matrix?
In the one-observation Bernoulli example, why is the observed-curvature update smaller than the expected-curvature update?
What feature of the two-coefficient example shows that the update directions differ?
Give a sufficient condition, stated in terms of \(A_i\), for observed and expected information to agree for any observed responses.