Score Identities and Information Equality

Why Score Identities Matter

The score records how a log-likelihood changes as the parameter changes. Its expected value and variability are central to likelihood inference: under suitable regularity conditions, the score is centered at the true parameter, and its covariance equals the expected curvature of the log-likelihood with the sign reversed. These are not merely algebraic coincidences. They follow from differentiating the fact that a probability density integrates to one.

The phrase “under suitable regularity conditions” is essential. Differentiating an integral requires justification, and the argument can fail when the support of the distribution depends on the parameter. We first state sufficient conditions explicitly, then prove the identities and check them in familiar models.

Definitions and Regularity Conditions

Definition: Let $X$ have density or mass function $f_\theta(x)$, with respect to a measure $\nu$, where $\theta=(\theta_1,\ldots,\theta_k)^T$ belongs to an open set $\Theta\subseteq\mathbb{R}^k$. At points where $f_\theta(x)>0$, define the score vector and log-density Hessian by $$ U_\theta(x)=\nabla_\theta\log f_\theta(x), \qquad H_\theta(x)=\nabla_\theta^2\log f_\theta(x). $$ The score covariance matrix is $\operatorname{Cov}_\theta(U_\theta(X))$. The expected Fisher information is $I(\theta)=E_\theta[U_\theta(X)U_\theta(X)^T]$ when the score has mean zero; the expected negative Hessian is $J(\theta)=E_\theta[-H_\theta(X)]$.

For the identities below, fix an interior parameter value $\theta$. Assume the model has common support $S$: the density is zero outside $S$ for every parameter in a neighborhood of $\theta$, and $f_\vartheta(x)>0$ for $\nu$-almost every $x\in S$ throughout that neighborhood. Assume $f_\vartheta(x)$ is twice continuously differentiable in $\vartheta$ on the neighborhood for almost every $x\in S$.

Here is one explicit sufficient condition for the required differentiation under the integral. For some neighborhood $V$ of $\theta$, suppose that for each coordinate $i$, $|\partial_i f_\vartheta(x)|\leq G_i(x)$ for all $\vartheta\in V$, and for each pair $i,j$, $|\partial_{ij} f_\vartheta(x)|\leq G_{ij}(x)$ for all $\vartheta\in V$, where each $G_i$ and $G_{ij}$ is integrable with respect to $\nu$. Also assume the score components have finite second moments and the Hessian entries have finite expectations at $\theta$. The dominating functions allow the dominated convergence theorem to justify differentiating the integral of the density once and twice. These are sufficient conditions, not a claim that every model satisfying the identities must meet this particular set of bounds.

The Two Score Identities

Theorem: Under the conditions just stated, $$ E_\theta[U_\theta(X)]=0 \qquad\text{and}\qquad \operatorname{Cov}_\theta(U_\theta(X)) =E_\theta[U_\theta(X)U_\theta(X)^T] =E_\theta[-H_\theta(X)]. $$

The first equality is the zero-mean score identity. The second is the information equality. Both can be derived from the normalization $\int_S f_\theta(x)\,d\nu(x)=1$, but the second also uses the relation between derivatives of a density and derivatives of its logarithm.

Proof of the Identities

For coordinate $i$, differentiate the normalization with respect to $\theta_i$. The common support and the stated domination condition justify bringing the derivative inside the integral: $$ 0=\partial_i\int_S f_\theta(x)\,d\nu(x) =\int_S \partial_i f_\theta(x)\,d\nu(x). $$ On $S$, positivity of the density and the chain rule give $\partial_i f_\theta=f_\theta\,\partial_i\log f_\theta$. Therefore $$ 0=\int_S f_\theta(x)\,\partial_i\log f_\theta(x)\,d\nu(x) =E_\theta[U_{\theta,i}(X)]. $$ This holds for every coordinate $i$, so $E_\theta[U_\theta(X)]=0$.

Now differentiate the same normalization twice, once with respect to $\theta_i$ and once with respect to $\theta_j$. The second-derivative domination justifies this interchange as well: $$ 0=\partial_{ij}\int_S f_\theta(x)\,d\nu(x) =\int_S \partial_{ij}f_\theta(x)\,d\nu(x). $$ To express this derivative using the log density, differentiate $\partial_i f_\theta=f_\theta\,\partial_i\log f_\theta$ with respect to $\theta_j$: $$ \partial_{ij}f_\theta =f_\theta\left[ \partial_{ij}\log f_\theta+ (\partial_i\log f_\theta)(\partial_j\log f_\theta) \right]. $$ Integrating and dividing the integrand into expectations yields $$ 0=E_\theta[H_{\theta,ij}(X)] +E_\theta[U_{\theta,i}(X)U_{\theta,j}(X)]. $$ This holds for every pair $i,j$, so $E_\theta[UU^T]=E_\theta[-H_\theta]$. Finally, by the definition of covariance and the first identity, $$ \operatorname{Cov}_\theta(U_\theta) =E_\theta[(U_\theta-E_\theta U_\theta)(U_\theta-E_\theta U_\theta)^T] =E_\theta[U_\theta U_\theta^T]. $$ Combining the equalities proves the theorem. $\square$

Independent Observations and Likelihood Information

For independent observations $X_1,\ldots,X_n$, let the joint log-likelihood be $\ell(\theta)=\sum_{r=1}^n\log f_\theta(X_r)$. Its score is the sum of the individual scores, and its Hessian is the sum of the individual Hessians. If each observation satisfies the conditions above, linearity of expectation gives $$ E_\theta[\nabla_\theta\ell(\theta)]=0,\qquad E_\theta[-\nabla_\theta^2\ell(\theta)] =\operatorname{Cov}_\theta(\nabla_\theta\ell(\theta)) =\sum_{r=1}^n E_\theta[-H_{\theta,r}(X_r)]. $$ The covariance sum follows because independent observations have independent score vectors, hence their cross-covariance matrices vanish. For identically distributed observations, this expected information is $n$ times the one-observation information.

The observed information is the random matrix $-\nabla_\theta^2\ell(\theta)$; its expectation is the expected information in the display. The equality says that its expectation matches the covariance of the score, not that the observed information equals the score outer product for each realized data set.

Worked Example: Bernoulli Probability

Let $X\sim\operatorname{Bernoulli}(\theta)$, where $0<\theta<1$, so $f_\theta(x)=\theta^x(1-\theta)^{1-x}$ for $x\in\{0,1\}$. The log-density is $x\log\theta+(1-x)\log(1-\theta)$. Differentiating gives $$ U_\theta(x)=\frac{x}{\theta}-\frac{1-x}{1-\theta} =\frac{x-\theta}{\theta(1-\theta)},\qquad H_\theta(x)=-\frac{x}{\theta^2}-\frac{1-x}{(1-\theta)^2}. $$

Since $E[X]=\theta$, the score mean is $$ E[U_\theta(X)]=\frac{E[X]-\theta}{\theta(1-\theta)}=0. $$ Also, $\operatorname{Var}(X)=\theta(1-\theta)$, so $$ \operatorname{Var}(U_\theta(X)) =\frac{\operatorname{Var}(X)}{\theta^2(1-\theta)^2} =\frac{1}{\theta(1-\theta)}. $$ For the negative Hessian, $$ E[-H_\theta(X)] =\frac{E[X]}{\theta^2}+\frac{1-E[X]}{(1-\theta)^2} =\frac{1}{\theta}+\frac{1}{1-\theta} =\frac{1}{\theta(1-\theta)}. $$ Thus both sides of the information equality agree.

Worked Example: Poisson Rate

Let $X\sim\operatorname{Poisson}(\lambda)$, with $\lambda>0$. Its log-mass function is $x\log\lambda-\lambda-\log(x!)$, so $$ U_\lambda(x)=\frac{x}{\lambda}-1,\qquad H_\lambda(x)=-\frac{x}{\lambda^2}. $$ Using $E[X]=\lambda$ and $\operatorname{Var}(X)=\lambda$, $$ E[U_\lambda(X)]=\frac{E[X]}{\lambda}-1=0, \qquad \operatorname{Var}(U_\lambda(X)) =\frac{\operatorname{Var}(X)}{\lambda^2} =\frac{1}{\lambda}. $$ The expected negative Hessian is $$ E[-H_\lambda(X)]=\frac{E[X]}{\lambda^2}=\frac{1}{\lambda}. $$ This agrees with the score variance. For $n$ independent observations, the total score has variance $n/\lambda$, and the expected negative log-likelihood Hessian is also $n/\lambda$.

Worked Example: Normal Mean with Known Variance

Let $X\sim N(\mu,\sigma^2)$, where $\sigma^2>0$ is known and $\mu\in\mathbb{R}$. The log-density, up to its normalizing constant, is $-(x-\mu)^2/(2\sigma^2)$. Therefore $$ U_\mu(x)=\frac{x-\mu}{\sigma^2},\qquad H_\mu(x)=-\frac{1}{\sigma^2}. $$

The mean score is $E[U_\mu(X)]=(E[X]-\mu)/\sigma^2=0$. Its variance is $$ \operatorname{Var}(U_\mu(X)) =\frac{\operatorname{Var}(X)}{\sigma^4} =\frac{\sigma^2}{\sigma^4} =\frac{1}{\sigma^2}. $$ Since the Hessian is constant in $x$, $E[-H_\mu(X)]=1/\sigma^2$ as well. The two quantities match, with no approximation involved.

When the Argument Fails

A common mistake is to differentiate the log-density and then assume the score must have mean zero, without checking whether differentiation can pass through the integral. The missing issue is often the support. Consider $X$ uniformly distributed on $(0,\theta)$, with $\theta>0$. Inside its support, the density is $1/\theta$, and the pointwise log-density derivative is $-1/\theta$. Averaging this expression under the model gives $-1/\theta$, not zero.

There is no contradiction: the support $(0,\theta)$ changes with $\theta$, so the integral cannot be differentiated as though it were over a fixed support with no boundary contribution. Indeed, writing the normalization with its limits gives $$ \int_0^\theta \frac{1}{\theta}\,dx=1. $$ Differentiating this expression includes the contribution from the moving upper limit; differentiating only the integrand omits it. The common-support condition rules out this particular failure. It is therefore not enough for a formula for the score to exist pointwise; the regularity conditions must also justify the integral argument.

A second error is to identify the covariance with $E[UU^T]$ before establishing that the score has mean zero. In general, $$ \operatorname{Cov}(U)=E[UU^T]-E[U]E[U]^T. $$ The zero-mean identity is what removes the second term. In applications, check the assumptions and the centering step rather than treating the information equality as an unconditional identity.

Key takeaway: For a regular model with common support and justified differentiation under the integral, normalization forces the score to have mean zero. Differentiating normalization twice then gives $E[UU^T]=E[-H]$, and zero mean makes this matrix the score covariance.

Check Your Understanding

Use the definitions and conditions above to answer these questions.

Which step in the proof of the zero-mean score identity requires differentiation under the integral sign?
Why does $E[UU^T]$ equal $\operatorname{Cov}(U)$ only after the score mean has been established?
For a model with independent observations, why do the cross-covariance terms between distinct observation scores vanish?
In the uniform $(0,\theta)$ example, what regularity condition fails, and what term is missed by differentiating only the density inside its support?
For one observation from a Poisson model with rate $\lambda$, what are the score variance and expected negative Hessian?