Information Equality Under Correct Specification

Why Two Information Calculations Can Agree

For a likelihood model, information about a parameter can be calculated from the variability of the score or from the curvature of the log likelihood. Under correct specification and suitable regularity conditions, these two calculations have the same expectation. That equality is useful because it connects the behavior of likelihood derivatives to the probability model that generates the data.

The conditions matter. The usual derivation differentiates an integral that equals one. If the set of possible observations changes with the parameter, differentiating only the density while treating its support as fixed can omit boundary terms. The result may then fail, even though the formal derivatives of the log density look well behaved.

Score, Hessian, and Information

Let $Y$ have density or probability mass function $f(y;\theta)$, where $\theta$ is a $k$-dimensional parameter vector. Suppose the data-generating distribution is represented by the model at the true parameter $\theta_0$. The score is the gradient of the log density, and the Hessian is its matrix of second derivatives.

Definition: The score and Hessian for one observation are $$ s(y;\theta)=\frac{\partial \log f(y;\theta)}{\partial\theta}, \qquad H(y;\theta)=\frac{\partial^2 \log f(y;\theta)} {\partial\theta\,\partial\theta'}. $$ The score is a $k$-vector and the Hessian is a $k\times k$ matrix. The score outer product is $s(y;\theta)s(y;\theta)'$, while the observed information contribution is $-H(y;\theta)$.

Expectations below are taken under the true model $f(y;\theta_0)$, and derivatives are evaluated at $\theta_0$. The expected score outer product and the expected negative Hessian are, respectively,

\mathcal{I}_{\mathrm{OP}}(\theta_0) =E_{\theta_0}\!\left[s(Y;\theta_0)s(Y;\theta_0)'\right], \qquad \mathcal{I}_{\mathrm{H}}(\theta_0) =E_{\theta_0}\!\left[-H(Y;\theta_0)\right].

The subscript “OP” emphasizes that the first matrix is formed from an outer product. The second is the expectation of the observed information. In general, one should not identify either expectation with the information calculated from a particular realized sample: the equality here is an equality of expectations under the model.

Theorem: If the model is correctly specified and the regularity conditions for differentiating the density normalization twice hold, then $$ E_{\theta_0}\!\left[s(Y;\theta_0)s(Y;\theta_0)'\right] = E_{\theta_0}\!\left[-H(Y;\theta_0)\right]. $$

Conditions and Derivation

A standard sufficient set of conditions is that $\theta_0$ is an interior point of an open parameter space; $f(y;\theta)$ is twice differentiable in a neighborhood of $\theta_0$; the support of $Y$ does not depend on $\theta$; and differentiation can be passed under the integral sign. The score products and Hessian entries must also have finite expectations. The density is normalized for every parameter in that neighborhood:

\int_{\mathcal{Y}} f(y;\theta)\,d\mu(y)=1.

Here $\mu$ is a measure on the observation space, such as counting measure for discrete data or Lebesgue measure for continuous data. The common-support and differentiation conditions ensure that the derivatives of this integral are the integrals of the corresponding derivatives of the density. Correct specification means that the expectation is taken under the same density $f(\,\cdot\,;\theta_0)$ whose derivatives define the score and Hessian.

Differentiate the density.
Since $\partial \log f/\partial\theta=s$, the density derivative is $\partial f/\partial\theta=f s$. Differentiating its integral, which is constantly one, gives $E_{\theta_0}[s(Y;\theta_0)]=0$.

Differentiate again.
For each pair of parameter components $j,\ell$, the product rule gives $\partial^2 f/\partial\theta_j\partial\theta_\ell=f\{H_{j\ell}+s_js_\ell\}$.

Use the second derivative of normalization.
The second derivative of the integral of $f$ is zero. Integrating the expression in Step 2 therefore yields $E_{\theta_0}[H_{j\ell}+s_js_\ell]=0$ for every $j,\ell$.

Rearrange the matrix entries.
Thus $E_{\theta_0}[s_js_\ell]=-E_{\theta_0}[H_{j\ell}]$ for all entries, which proves the information equality.

The key identity used in the second step follows directly from $f=\exp(\log f)$. Differentiating once gives $\partial f/\partial\theta_j=f s_j$; differentiating this with respect to $\theta_\ell$ gives $f s_\ell s_j+fH_{j\ell}$. This also explains why both terms appear in the information equality.

Worked Examples

Worked Example: A Normal Model With Known Variance

Suppose one hypothetical observation follows $Y\sim N(\mu,4)$, with unknown mean $\mu$ and known variance $4$. At the true value $\mu_0=10$, verify the information equality for one observation.

The log density, ignoring constants that do not depend on $\mu$, is $\ell(\mu;y)=-(y-\mu)^2/8$. Its score and Hessian are

s(y;\mu)=\frac{y-\mu}{4}, \qquad H(y;\mu)=-\frac14.

Under the specified model, $E_{\mu_0}[Y-\mu_0]=0$ and $E_{\mu_0}[(Y-\mu_0)^2]=4$. Therefore the expected score outer product is

E_{\mu_0}[s(Y;\mu_0)^2] =\frac{E_{\mu_0}[(Y-\mu_0)^2]}{16} =\frac{4}{16}=0.25.

The expected negative Hessian is $-E_{\mu_0}[H(Y;\mu_0)]= -(-1/4)=0.25$. Both calculations give $0.25$ units of information per observation about the mean parameter, in inverse squared units of $Y$. The density has support on the whole real line for every $\mu$, so the support is fixed; its derivatives are integrable under the normal model.

Worked Example: Two Logistic Likelihood Contributions

Consider two hypothetical independent Bernoulli outcomes $Y_i$ with logistic probabilities $p_i(\beta)=\{1+\exp(-x_i\beta)\}^{-1}$, where $x_1=1$, $x_2=2$, and the true parameter is $\beta_0=0.5$. For a single observation with covariate $x_i$, the score and Hessian are

s_i(\beta)=x_i\{Y_i-p_i(\beta)\}, \qquad H_i(\beta)=-x_i^2p_i(\beta)\{1-p_i(\beta)\}.

Because $Y_i$ is Bernoulli with probability $p_i(\beta_0)$, its variance is $p_i(\beta_0)\{1-p_i(\beta_0)\}$. Thus the expected squared score for observation $i$ is $x_i^2p_i(1-p_i)$, exactly the negative Hessian contribution, which does not depend on the realized outcome.

For observation 1, $x_1\beta_0=0.5$, so $p_1=1/(1+\exp(-0.5))\approx0.622459$. Its information contribution is

1^2(0.622459)(1-0.622459)\approx0.2350037.

For observation 2, $x_2\beta_0=1$, so $p_2=1/(1+\exp(-1))\approx0.731059$. Its contribution is

2^2(0.731059)(1-0.731059)\approx0.7864477.

Independence makes the sample log likelihood derivatives additive. Consequently, the expected sample score outer product and the expected negative sample Hessian both equal $0.2350037+0.7864477=1.0214514$, or approximately $1.0215$ to four decimal places. The Bernoulli outcome space $\{0,1\}$ is fixed as $\beta$ changes, and the probabilities are smooth and strictly between zero and one at this finite parameter value, so the regularity conditions hold.

When Support Depends on the Parameter

A parameter-dependent support can produce terms from differentiating the integration limits. Those terms are absent from the usual derivation, which differentiates under an integral over a fixed set. The next example shows that the score-mean identity and information equality can fail when those boundary terms are ignored.

Worked Example: A Uniform Distribution With a Moving Endpoint

Let $X\sim\operatorname{Uniform}(0,\theta)$, with $\theta>0$. Its density is $f(x;\theta)=1/\theta$ for $0<x<\theta$ and zero otherwise. For an observation in the interior of its support, the log density is $\ell(\theta;x)=-\log\theta$. Hence

s(x;\theta)=-\frac1\theta, \qquad H(x;\theta)=\frac1{\theta^2}.

These derivatives give $E_\theta[s(X;\theta)]=-1/\theta$, $E_\theta[s(X;\theta)^2]=1/\theta^2$, and $E_\theta[-H(X;\theta)]=-1/\theta^2$. The score does not have mean zero, and the expected score outer product differs from the expected negative Hessian.

The missing contribution is visible by differentiating the normalization integral correctly:

\frac{d}{d\theta}\int_0^\theta \frac1\theta\,dx = \underbrace{\frac1\theta}_{\text{upper-boundary term}} + \int_0^\theta -\frac1{\theta^2}\,dx = \frac1\theta-\frac1\theta=0.

If the upper limit were incorrectly held fixed, only the integral term would remain, producing $-1/\theta$ rather than zero. Here the endpoint itself changes with $\theta$, so the fixed-support derivation does not apply. This example does not mean that every model with moving support must violate the equality; it shows that the usual proof cannot be used without accounting for boundary terms, and the equality can indeed fail.

Independent Observations and Practical Use

For independent observations $Y_1,\ldots,Y_n$ with correctly specified contributions $f_i(y_i;\theta)$, define $s_i(\theta)$ and $H_i(\theta)$ as the score and Hessian contributions. If each contribution satisfies the regularity conditions, the sample score and Hessian are sums:

s_n(\theta)=\sum_{i=1}^n s_i(\theta), \qquad H_n(\theta)=\sum_{i=1}^n H_i(\theta).

Each individual score has mean zero. Independence therefore makes the cross-products between distinct observations have expectation zero, and the expected sample score outer product is the sum of the individual expected outer products. The expected negative sample Hessian is also the sum of the individual expected negative Hessians. Applying the one-observation result contribution by contribution gives equality for the sample. In the identically distributed case, both matrices are $n$ times their one-observation value.

This result gives two ways to calculate model-based expected information and a useful theoretical check on likelihood derivatives. It does not say that the score outer product and negative Hessian are equal for every realized data set. Nor does it ensure equality when the probability model is not the true data-generating model: the expectations in the derivation must be taken under the density being differentiated. The distinction is important when assessing likelihood calculations and their assumptions.

Common Mistake

Wrong reasoning: “The density integrates to one, so differentiating under the integral always gives information equality.”
Correction: The differentiation step requires conditions that justify moving derivatives inside the integral. In particular, changing support can create boundary terms. Check the support and the interchange conditions before applying the result.
Another common error: Treating equality of expected information as equality of the two matrices for each sample. The theorem concerns expectations under the correctly specified model, not necessarily realized sample values.

Key takeaway: With a correctly specified, sufficiently smooth model and valid differentiation under a fixed-support integral, the expected score outer product equals the expected negative Hessian. Parameter-dependent boundaries can add terms that invalidate the standard argument.

Check Your Understanding

Use the definitions and conditions in the tutorial to answer these questions.

What are the dimensions of the score and Hessian when $\theta$ has $k$ components?
Why does differentiating the normalization condition twice produce both a Hessian term and a score outer-product term?
For a correctly specified Bernoulli model with probability $p$, what is the expected squared score for an intercept parameter, and what is the negative Hessian?
In the uniform example, identify the boundary term produced by differentiating the upper integration limit.
Does information equality assert that the score outer product equals negative Hessian for each realized observation? Explain.