Deriving the Information Matrix Equality

Why Two Measures of Information Can Agree

Likelihood contains two related descriptions of how informative data are about a parameter. The score records the local slope of the log likelihood; its outer product describes score variability. The Hessian records local curvature. Under suitable conditions, the expected score outer product equals the expected negative Hessian. This is the information matrix equality.

The equality is useful because it gives two ways to calculate Fisher information. Its assumptions matter, however: a formal calculation with derivatives is not enough if the density's support changes with the parameter or differentiation cannot be passed through an integral. We derive the result from the density normalization, then check it in Gaussian and Bernoulli models.

Definitions and Conditions

Let \(X\) be one observation with density or probability mass function \(f(x;\theta)\), with respect to a dominating measure that does not depend on \(\theta\). The parameter \(\theta\) is a \(d\)-dimensional vector in an open parameter region. At the parameter value under consideration, suppose \(f(x;\theta)>0\) on its support, and define the log density \(\ell(x;\theta)=\log f(x;\theta)\). Derivatives below are with respect to \(\theta\).

For the derivation, use the following sufficient conditions in a neighborhood of the parameter value. The support of \(f(x;\theta)\) is the same for all nearby \(\theta\); the density is twice differentiable in \(\theta\) on that support; and the derivatives can be integrated by differentiating under the integral sign. In particular, the first and second derivatives must be integrable, and an appropriate integrable dominating function is a standard way to justify these interchanges. These conditions are not the only possible technical formulation, but they make clear what the proof requires.

The density is normalized at each parameter value:

For discrete data, replace the integral with a sum. The argument applies component by component to vector parameters. In particular, it does not assume in advance that the score has mean zero; that fact follows from differentiating the normalization once under the same conditions.

Deriving the Information Matrix Equality

First differentiate the normalization with respect to each component \(\theta_j\). With differentiation under the integral justified, \(\int \partial f(x;\theta)/\partial\theta_j\,dx=0\). Since \(\partial f/\partial\theta_j=f\,\partial\ell/\partial\theta_j\), this also gives \(\mathbb{E}_\theta[s(X;\theta)]=0\). Differentiate a second time to obtain the matrix identity needed for the result.

Writing the componentwise result in matrix form gives the derivation in one line:

The mean-zero score result also clarifies the second expression: since \(\mathbb{E}_\theta[s]=0\), \(I(\theta)\) is the covariance matrix of the score when its second moments exist. The information matrix equality is an equality of expectations, not a claim that the negative Hessian and score outer product are the same for each realized observation.

Why Support Conditions Matter

The fixed-support condition prevents an important boundary contribution from being lost when differentiating the integral of a density. A useful contrast is the uniform model \(X\sim\mathrm{Uniform}(0,\theta)\), where \(\theta>0\). Its support depends on \(\theta\), so the derivation above does not apply. For an observed \(x\) in the support, the log density is \(\ell(x;\theta)=-\log\theta\). Its score is \(-1/\theta\) and its Hessian is \(1/\theta^2\), giving

These are not equal. In this model, the score also has nonzero expectation, \(\mathbb{E}_\theta[s(X;\theta)]=-1/\theta\). The missing contribution arises because the moving endpoint \(x=\theta\) affects the integral when it is differentiated; differentiating only the density within the support omits that boundary effect. The score outer product still defines \(I(\theta)\) when it exists, but existence of both expectations alone does not establish the information matrix equality.

Expected Information and Observed Curvature

For a sample of independent observations with a correctly specified common density, the log likelihood is the sum of the individual log densities. Its score and Hessian are sums of the corresponding individual quantities. Linearity of expectation therefore makes the sample expected negative Hessian and sample score outer product both equal to \(nI(\theta)\), provided the single-observation conditions hold. Independence ensures that cross terms in the expected score outer product vanish because each score has mean zero.

In an actual data set, the observed negative Hessian \(-H(x;\theta)\) can differ from \(s(x;\theta)s(x;\theta)'\). The information matrix equality says their expectations agree under its conditions; it does not say either realized quantity equals the other, or that the equality holds in an irregular model. Expected information and observed curvature are related but distinct calculations.

Common Mistake

A common error is to say, “Both expectations exist, so either one is Fisher information and they must be equal.” The uniform example disproves that reasoning: the score outer product has expectation \(1/\theta^2\), while the expected negative Hessian is \(-1/\theta^2\). Define Fisher information in the usual score-based way as \(\mathbb{E}_\theta[ss']\) when it exists. Call the expected negative Hessian an equal expression for information only after checking the conditions that support the information matrix equality. In particular, verify that the support is parameter-independent locally and that the necessary derivatives may be moved inside the integral.