Observed and Expected Information in Gaussian Models

Why distinguish two kinds of information?

A likelihood is a function of the parameter for the data actually observed. Its curvature near a parameter value therefore depends on the data: one sample can make the likelihood sharply curved, while another sample from the same model can make it flatter. Observed information records this realized curvature. Expected information averages curvature over samples generated at a specified parameter value. They are closely related, but they are not interchangeable in every calculation.

This distinction is useful when interpreting likelihood-based uncertainty calculations. The inverse of an information matrix describes a local scale for parameter variation under a quadratic approximation to the log-likelihood. Whether that matrix is observed or expected changes the approximation used. The difference is especially visible when a model has more than one parameter, because observed information may also have nonzero off-diagonal entries.

Definitions and the curvature interpretation

Let \(X\) have density or mass function \(f(x;\theta)\), where \(\theta\) lies in an open subset of \(\mathbb{R}^q\). For observed data \(x\), write \(\ell(\theta;x)=\log f(x;\theta)\), and suppose the first and second derivatives exist. The score is the column vector of first derivatives. The information matrices are defined as follows.

Definition: The observed information at \(\theta\) for data \(x\) is the negative Hessian of the log-likelihood, \(J(\theta;x)=-\nabla^2_\theta \ell(\theta;x)\). When the expectation exists, the expected information (or Fisher information) is \(I(\theta)=E_\theta[J(\theta;X)]\). Under regularity conditions stated below, it also equals the covariance matrix of the score.

The minus sign makes positive curvature correspond to positive information: at a local maximum of the log-likelihood, a positive-definite observed information matrix means the log-likelihood bends downward in every parameter direction. More precisely, for a small displacement \(h\in\mathbb{R}^q\), twice differentiability gives

\ell(\theta+h;x) = \ell(\theta;x) + U(\theta;x)^{\mathsf T}h - \frac{1}{2}h^{\mathsf T}J(\theta;x)h + o(\|h\|^2),

where \(U(\theta;x)=\nabla_\theta\ell(\theta;x)\). This expansion explains the local-curvature interpretation. Expected information is the expectation of the quadratic curvature term, not the curvature from a particular realized sample.

A general identity and its conditions

The equality between expected negative curvature and score covariance is not automatic for every model. A standard sufficient set of conditions is that the support of the distribution does not depend on \(\theta\), the density is twice differentiable in \(\theta\), and differentiation can be passed under the integral sign for the normalization identity and its first derivatives. Also assume the score has finite second moments and the derivatives in the identities below are integrable. These conditions hold for the ordinary nonsingular Gaussian models considered here, at interior parameter values.

Theorem: Under those regularity conditions, \(E_\theta[U(\theta;X)]=0\) and \(I(\theta)=E_\theta[U(\theta;X)U(\theta;X)^{\mathsf T}] =-E_\theta[\nabla^2_\theta\ell(\theta;X)]\).

Proof. Since the density integrates to one and differentiation may be interchanged with integration,

E_\theta[U(\theta;X)] = \int \nabla_\theta\log f(x;\theta)\,f(x;\theta)\,dx = \int \nabla_\theta f(x;\theta)\,dx = \nabla_\theta 1 = 0.

For coordinates \(a,b\) of \(\theta\), differentiate the identity \(\int \partial_a f(x;\theta)\,dx=0\) with respect to \(\theta_b\), again using the stated interchange condition. The product rule and \(\partial_a f=f\,\partial_a\ell\) give

0 = \int \partial_{ab} f(x;\theta)\,dx = E_\theta\!\left[ \partial_{ab}\ell(\theta;X) + \partial_a\ell(\theta;X)\partial_b\ell(\theta;X) \right].

Thus the expected negative Hessian has entries equal to the expected products of score components. Since the score has mean zero, these products are its covariance entries. This proves both identities. \(\square\)

Worked Gaussian calculations

Worked Example: A Gaussian Mean With Known Covariance

Let \(X_1,\ldots,X_n\) be independent \(p\)-dimensional observations from \(N_p(\mu,\Sigma)\), where \(\Sigma\) is known and positive definite, and the parameter is \(\mu\). Up to a term not depending on \(\mu\), the log-likelihood is

\ell(\mu) = -\frac{1}{2}\sum_{i=1}^n (X_i-\mu)^{\mathsf T}\Sigma^{-1}(X_i-\mu).

Differentiating with respect to the \(p\)-vector \(\mu\) gives \(U(\mu)=\sum_i\Sigma^{-1}(X_i-\mu)\). Differentiating once more gives \(\nabla^2_\mu\ell(\mu)=-n\Sigma^{-1}\). Therefore

J(\mu)=n\Sigma^{-1}, \qquad I(\mu)=E_\mu[J(\mu)]=n\Sigma^{-1}.

Here the observed and expected information agree for every data set: the log-likelihood curvature in the mean does not depend on the observations. For example, if \(n=4\) and \(\Sigma=\begin{pmatrix}4&1\\1&2\end{pmatrix}\), then \(|\Sigma|=7\) and \(\Sigma^{-1}=\frac{1}{7}\begin{pmatrix}2&-1\\-1&4\end{pmatrix}\). Hence both information matrices equal \(\frac{4}{7}\begin{pmatrix}2&-1\\-1&4\end{pmatrix}\). The inverse is \(\Sigma/n\), the exact covariance matrix of the sample mean in this model. This is a particularly direct instance in which local curvature and sampling uncertainty have a simple correspondence.

Worked Example: Observed and Expected Information for Mean and Variance

Let \(X_1,\ldots,X_n\) be independent \(N(\mu,v)\) observations, with \(v>0\) the variance, and use parameter vector \((\mu,v)\). Define \(S(\mu)=\sum_i(X_i-\mu)^2\). Ignoring constants independent of the parameters,

\ell(\mu,v) = -\frac{n}{2}\log v-\frac{S(\mu)}{2v}.

The first derivatives are \(\ell_\mu=\sum_i(X_i-\mu)/v\) and \(\ell_v=-n/(2v)+S(\mu)/(2v^2)\). The second derivatives are \(\ell_{\mu\mu}=-n/v\), \(\ell_{\mu v}=-\sum_i(X_i-\mu)/v^2\), and \(\ell_{vv}=n/(2v^2)-S(\mu)/v^3\). Thus

J(\mu,v) = \begin{pmatrix} n/v & \displaystyle\sum_i(X_i-\mu)/v^2\\[4pt] \displaystyle\sum_i(X_i-\mu)/v^2 & -n/(2v^2)+S(\mu)/v^3 \end{pmatrix}.

At the true parameter, \(E[\sum_i(X_i-\mu)]=0\) and \(E[S(\mu)]=nv\). Taking expectations therefore yields

I(\mu,v) = \begin{pmatrix} n/v & 0\\ 0 & n/(2v^2) \end{pmatrix}.

For a numerical check, take data \(1,2,3,4,5\) and evaluate at the candidate \((\mu,v)=(2.5,1.5)\). The residuals are \(-1.5,-0.5,0.5,1.5,2.5\), so their sum is \(2.5\) and their squared sum is \(2.25+0.25+0.25+2.25+6.25=11\). The observed information is consequently

J(2.5,1.5) = \begin{pmatrix} 10/3 & 10/9\\ 10/9 & -10/9+88/27 \end{pmatrix} = \begin{pmatrix} 10/3 & 10/9\\ 10/9 & 58/27 \end{pmatrix}.

Its determinant is \((10/3)(58/27)-(10/9)^2=580/81-100/81=160/27>0\), and its first diagonal entry is positive, so this symmetric matrix is positive definite. At the same candidate parameter, expected information is \(\begin{pmatrix}10/3&0\\0&10/9\end{pmatrix}\); it is different because it averages over data sets rather than using these particular residuals. The inverse observed matrix is \(\begin{pmatrix}29/80&-3/16\\-3/16&9/16\end{pmatrix}\), whereas the inverse expected matrix is \(\begin{pmatrix}3/10&0\\0&9/10\end{pmatrix}\). These inverses show how the two curvature choices lead to different local uncertainty summaries at this candidate.

Worked Example: A Multivariate Gaussian Scale Parameter

Suppose \(X_1,\ldots,X_n\) are independent \(p\)-vectors with distribution \(N_p(0,\tau I_p)\), where \(\tau>0\) is a scalar variance parameter and \(I_p\) is the \(p\)-dimensional identity matrix. Put \(A=\sum_i X_i^{\mathsf T}X_i\). The log-likelihood, up to a parameter-independent constant, is \(\ell(\tau)=-np\log(\tau)/2-A/(2\tau)\). Hence

\ell'(\tau)=-\frac{np}{2\tau}+\frac{A}{2\tau^2}, \qquad J(\tau)=-\ell''(\tau) =-\frac{np}{2\tau^2}+\frac{A}{\tau^3}.

Since \(E_\tau[A]=np\tau\), expected information is \(I(\tau)=np/(2\tau^2)\). Now use \(p=2\), \(n=3\), and observations \((1,0)^{\mathsf T},(0,2)^{\mathsf T},(1,1)^{\mathsf T}\). Their squared lengths are \(1,4,2\), so \(A=7\). At \(\tau=2\),

J(2)=-\frac{6}{8}+\frac{7}{8}=\frac{1}{8}, \qquad I(2)=\frac{6}{8}=\frac{3}{4}.

The observed curvature is positive but much smaller than the expected curvature at this parameter value. Their reciprocals, \(8\) and \(4/3\), are the corresponding one-dimensional local variance scales from inverse information. They are not identical estimates of uncertainty: one uses the realized squared lengths, while the other averages over the model.

Using information for local uncertainty

Near an interior maximum-likelihood estimate \(\hat{\theta}\), the score is zero and the Taylor expansion is approximately \(\ell(\hat{\theta}+h)\approx \ell(\hat{\theta})-\frac12h^{\mathsf T}J(\hat{\theta})h\). If \(J(\hat{\theta})\) is positive definite, its inverse describes the scale and dependence of this local quadratic approximation. Expected information instead describes average curvature under the model at a specified parameter. Both matrices can therefore support uncertainty calculations, but the chosen matrix must match the intended calculation and assumptions.

The off-diagonal entries matter. In the mean-and-variance example, the observed cross-entry reflects the residual sum at the evaluated parameter, while the expected cross-entry is zero. Discarding the observed cross-entry before inverting the observed matrix would generally produce the wrong local uncertainty calculation. More broadly, information is parameterization-dependent as a matrix: when parameters are transformed, the local curvature and its inverse must be interpreted in the new coordinates.

Common pitfall

Confusing realized and averaged curvature. It is incorrect to compute the Hessian from the observed data and call it expected information. The correction is to take the model expectation of the negative Hessian, or use the score covariance under the regularity conditions that justify the identity.
Assuming the matrices always match. They match in some Gaussian models, such as the mean with known covariance, but not generally. The mean-and-variance example has a data-dependent observed cross-entry.
Reading inverse information as an exact finite-sample covariance in every model. Inverse curvature is a local uncertainty scale. It equals an exact sampling covariance in special cases, such as the known-covariance Gaussian sample mean, but that conclusion requires a model-specific justification.

Key takeaway: Observed information is the negative Hessian for the data in hand; expected information is its model average. Both describe local curvature, and their inverses can inform uncertainty calculations, but their definitions and interpretations must remain distinct.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

What regularity conditions allow expected negative Hessian and score covariance to be equated?
Why are observed and expected information identical for a Gaussian mean with known covariance?
In the Gaussian mean-and-variance model, which observed-information entry depends on the residual sum, and how?
For the multivariate scale example, explain why the observed information is \(1/8\) while expected information is \(3/4\).
When does inverse information give an exact covariance, and why should that interpretation not be assumed automatically?