The Expected Information Identity

Why Two Forms of Information Can Agree

The score describes the local change in log-likelihood as a parameter changes; the second derivative describes its local curvature. These are different features of a likelihood, so it may seem surprising that their expectations can be equal in magnitude. Under suitable regularity conditions, they are: the expected negative curvature equals the variance of the score. This equality is useful because it gives two ways to calculate Fisher information, and because its assumptions explain why some apparently simple likelihoods behave irregularly.

Consider a scalar parameter $\theta$ and a random variable $X$ with density or mass function $f(x;\theta)$, relative to a measure $\mu$. Write $\ell(\theta;x)=\log f(x;\theta)$ for the log-likelihood from one observation $x$. The score is its first derivative, $U(\theta;X)=\partial\ell(\theta;X)/\partial\theta$, and the log-likelihood curvature is its second derivative. The expectations below are taken under the model at the parameter value $\theta$.

Definition: For a scalar parameter, the expected information identity is $$ E_\theta[U(\theta;X)^2] = -E_\theta\!\left[\frac{\partial^2}{\partial\theta^2}\ell(\theta;X)\right]. $$ When the score has mean zero and finite variance, its second moment on the left is $\operatorname{Var}_\theta(U(\theta;X))$. The right side is the expected negative log-likelihood curvature.

Conditions for the Identity

The identity is not a formal consequence of differentiating a logarithm alone. Its proof differentiates the fact that a density integrates to one. For this step to be valid, differentiation must pass through the integral (or sum), and the support must not introduce parameter-dependent boundary terms.

Regularity conditions: Let $\theta$ lie in an open interval $\Theta$. At a fixed $\theta_0\in\Theta$, suppose there is a neighborhood $N\subset\Theta$ and a support set $S$, not depending on $\theta\in N$, such that $f(x;\theta)=0$ off $S$ and $f(x;\theta)>0$ for $\mu$-almost every $x\in S$. Suppose that for almost every $x\in S$, $f(x;\theta)$ is twice differentiable in $\theta$ on $N$. Assume there are $\mu$-integrable functions $M_1$ and $M_2$ such that, for all $\theta\in N$, $$ \left|\frac{\partial f(x;\theta)}{\partial\theta}\right|\leq M_1(x), \qquad \left|\frac{\partial^2 f(x;\theta)}{\partial\theta^2}\right|\leq M_2(x). $$ These bounds justify differentiating the normalization integral twice. Also assume the score squared and the log-likelihood second derivative are integrable at $\theta_0$, so the expectations in the identity are finite.

The envelope conditions are a convenient sufficient condition, not the only possible way to justify differentiation under the integral. The essential requirements are that the normalization can validly be differentiated twice and that the displayed expectations exist. The fixed-support assumption is likewise a standard regularity condition: when the support moves with $\theta$, differentiating a density formula on its interior can omit a boundary contribution.

Deriving the Expected Information Identity

The key pointwise calculus relations follow from $f=\exp(\ell)$ on the support: $$ \frac{\partial f}{\partial\theta} = f\frac{\partial\ell}{\partial\theta}, \qquad \frac{\partial^2 f}{\partial\theta^2} = f\left[ \left(\frac{\partial\ell}{\partial\theta}\right)^2 + \frac{\partial^2\ell}{\partial\theta^2} \right]. $$ The second relation is the one that connects score variability to curvature.

Theorem (Expected information identity): Under the regularity conditions above, at $\theta_0$, $$ E_{\theta_0}[U(\theta_0;X)]=0, \qquad E_{\theta_0}[U(\theta_0;X)^2] = -E_{\theta_0}\!\left[ \frac{\partial^2}{\partial\theta^2}\ell(\theta_0;X) \right]. $$ In particular, the score variance equals the expected negative curvature.

Proof. The density is normalized for each $\theta\in N$: $$ \int_S f(x;\theta)\,d\mu(x)=1. $$ By the assumed differentiation-under-the-integral conditions, differentiating once at $\theta_0$ gives $$ 0=\int_S \frac{\partial f(x;\theta_0)}{\partial\theta}\,d\mu(x) =\int_S f(x;\theta_0)U(\theta_0;x)\,d\mu(x) =E_{\theta_0}[U(\theta_0;X)]. $$ Differentiating the normalization a second time gives $\int_S \partial^2 f(x;\theta_0)/\partial\theta^2\,d\mu(x)=0$. Substituting the second pointwise relation yields $$ 0 = \int_S f(x;\theta_0) \left[ U(\theta_0;x)^2+ \frac{\partial^2}{\partial\theta^2}\ell(\theta_0;x) \right]d\mu(x) = E_{\theta_0}[U(\theta_0;X)^2] + E_{\theta_0}\!\left[ \frac{\partial^2}{\partial\theta^2}\ell(\theta_0;X) \right]. $$ Rearranging proves the claimed equality. Since the score has mean zero and a finite second moment, $\operatorname{Var}_{\theta_0}(U)=E_{\theta_0}[U^2]$. $\square$

For a vector parameter $\theta\in\mathbb{R}^p$, the same argument applies to each pair of coordinates, provided the corresponding partial derivatives can be passed under the integral and the expectations exist. If $U=\nabla_\theta\ell$ and $H_\ell$ is the Hessian matrix of $\ell$, then $$ E_\theta[U]=0, \qquad E_\theta[UU^\mathsf{T}]=-E_\theta[H_\ell]. $$ Indeed, differentiating the normalization once in coordinate $j$ gives $E[U_j]=0$. Differentiating twice in coordinates $j,k$ and using $\partial_{jk}f=f(U_jU_k+\partial_{jk}\ell)$ gives $E[U_jU_k]=-E[\partial_{jk}\ell]$. These coordinate equalities form the matrix identity.

Worked Examples

Worked Example: A Three-Outcome Model

Let $X$ take values $1,2,3$, with probabilities $\theta^2$, $2\theta(1-\theta)$, and $(1-\theta)^2$, respectively, where $0<\theta<1$. Verify the identity and evaluate both sides at $\theta=0.4$.

All three probabilities are positive on the parameter space, their sum is $\theta^2+2\theta(1-\theta)+(1-\theta)^2=1$, and the support is fixed. The log-likelihood scores for outcomes $1,2,3$ are, respectively, $$ \frac{2}{\theta},\qquad \frac{1}{\theta}-\frac{1}{1-\theta},\qquad -\frac{2}{1-\theta}. $$ Thus the score second moment is $$ E_\theta[U^2] =\theta^2\frac{4}{\theta^2} +2\theta(1-\theta)\left(\frac{1-2\theta}{\theta(1-\theta)}\right)^2 +(1-\theta)^2\frac{4}{(1-\theta)^2} =8+\frac{2(1-2\theta)^2}{\theta(1-\theta)} =\frac{2}{\theta(1-\theta)}. $$ For the curvature calculation, the three log-likelihood second derivatives are $-2/\theta^2$, $-1/\theta^2-1/(1-\theta)^2$, and $-2/(1-\theta)^2$. Therefore $$ -E_\theta[\ell''(\theta;X)] =2+2\theta(1-\theta)\left(\frac{1}{\theta^2}+\frac{1}{(1-\theta)^2}\right)+2 =\frac{2}{\theta(1-\theta)}. $$ At $\theta=0.4$, both sides equal $2/(0.4\cdot0.6)=25/3$. The two calculations agree, as required.

Worked Example: A Continuous Density on a Fixed Interval

On $0<x<1$, consider $f(x;\theta)=1+\theta(2x-1)$, with $-1<\theta<1$. At $\theta=0$, calculate the score variance and expected negative curvature.

Writing $z=2x-1$, the density is positive on the interval and integrates to one because $\int_0^1 z\,dx=0$. The support is fixed. The log-likelihood derivatives are $$ \ell'(\theta;x)=\frac{z}{1+\theta z}, \qquad \ell''(\theta;x)=-\frac{z^2}{(1+\theta z)^2}. $$ At $\theta=0$, $f(x;0)=1$, $U(0;x)=z$, and $-\ell''(0;x)=z^2$. Hence both quantities equal $$ \int_0^1 (2x-1)^2\,dx =\left[\frac{(2x-1)^3}{6}\right]_{0}^{1} =\frac{1}{3}. $$ In a neighborhood such as $|\theta|\leq 1/2$, the denominator $1+\theta z$ is at least $1/2$, so the first and second derivatives of the density are bounded on the interval. This example satisfies the regularity conditions, and the identity gives $1/3=1/3$.

Worked Example: A Moving Support Where the Identity Fails

Let $X$ have a uniform distribution on $(0,\theta)$, with $\theta>0$. On the interior of its support, $f(x;\theta)=1/\theta$, so the log-likelihood derivatives for a fixed interior observation are $$ \ell'(\theta;x)=-\frac{1}{\theta}, \qquad \ell''(\theta;x)=\frac{1}{\theta^2}. $$ Consequently, $E_\theta[U^2]=1/\theta^2$, but $-E_\theta[\ell'']= -1/\theta^2$. They are not equal. Also $E_\theta[U]=-1/\theta$, rather than zero.

There is no contradiction with the theorem: the support $(0,\theta)$ depends on $\theta$. Although $\int_0^\theta (1/\theta)\,dx=1$, differentiating this integral requires accounting for its moving upper limit. Treating the support as fixed and differentiating only the density in the interior loses that boundary contribution. This is a direct warning against applying the identity solely from the algebra of log-likelihood derivatives.

Why the Identity Matters—and a Common Mistake

When the regularity conditions hold, the identity equates two population quantities: score variance and expected negative curvature. The observed curvature, $-\ell''(\theta;x)$ for a particular data value, need not equal the squared score for that value, nor must the two calculations agree observation by observation. The equality is after taking expectations under the model. In a sample, the distinction between observed curvature and its expectation remains important.

A common mistake is to write $E[\ell'']= -E[(\ell')^2]$ for any likelihood because $f''=f\{\ell''+(\ell')^2\}$. That pointwise algebra is correct, but the conclusion additionally requires that the second derivative of the normalized density integrates to zero, which depends on valid differentiation and support conditions. The uniform example shows that a moving boundary can invalidate the step. Another related error is to identify $E[U^2]$ with the score variance without first checking $E[U]=0$; under the regularity conditions this mean-zero result follows from differentiating normalization once, but it need not hold in nonregular models.

Key takeaway: The expected information identity follows by differentiating $\int f(x;\theta)\,d\mu(x)=1$ twice and using $f''=f\{\ell''+(\ell')^2\}$. Its valid use requires the differentiation and support conditions that make those steps legitimate.

Check Your Understanding

Use the conditions and derivation above to answer these questions.

What normalization equation is differentiated to obtain the expected information identity?
Why does the first derivative of normalization imply that the score has mean zero?
In the three-outcome example, what is the common value of the two information calculations at $\theta=0.5$?
Why is equality of score variance and expected negative curvature an expectation-level statement rather than a pointwise one?
Which regularity condition fails for the uniform model on $(0,\theta)$, and what extra feature of differentiation is then needed?