Score Variance and Fisher Information

Why the Score Measures Information

A likelihood changes as its parameter changes. The score records the local rate of that change on the log scale: a large score means the observed outcome makes the log-likelihood sensitive to a small parameter change. But the score is random before the data are observed. Fisher information summarizes its variability under the model, which is why the conditions behind its definition matter.

This tutorial focuses on a scalar parameter $\theta$. In particular, it establishes when Fisher information is the variance of the score. The key step is showing that the score has mean zero, and that step requires conditions allowing differentiation to pass through an integral or sum.

Definition: Let $X$ have density or mass function $f(x;\theta)$ with respect to a measure that does not depend on $\theta$. At a parameter value where $f(x;\theta)>0$ and the derivative exists, the score is $$ U(\theta;X)=\frac{\partial}{\partial\theta}\log f(X;\theta). $$ When $E[U(\theta;X)^2]$ is finite, the Fisher information in one observation is $$ I(\theta)=E[U(\theta;X)^2]. $$

The score is defined almost surely; values at points where the density is zero do not affect its expectation under that parameter value. Where the derivative and log are defined, $U(\theta;X)=\partial_\theta f(X;\theta)/f(X;\theta)$. Consequently, the information can also be written as an integral or sum involving the squared derivative of the density.

When the Score Has Mean Zero

A standard sufficient set of regularity conditions makes the mean-zero claim precise. Let the parameter space $\Theta$ be an open interval, and fix $\theta\in\Theta$. Assume the model has densities $f(x;t)$ with respect to one fixed measure $\mu$, for all $t$ in a neighborhood of $\theta$. Assume that $f(x;t)$ is differentiable in $t$ for $\mu$-almost every $x$. Also assume there is an integrable function $g$ such that, throughout a neighborhood of $\theta$, $|\partial_t f(x;t)|\leq g(x)$ for $\mu$-almost every $x$. These conditions justify differentiating the integral of the density.

Theorem: Under the conditions just stated, if $X$ has density $f(x;\theta)$, then $E[U(\theta;X)]=0$, provided the score is defined almost surely.

Proof. Since $f(\cdot;t)$ is a density with respect to the fixed measure, $\int f(x;t)\,d\mu(x)=1$ for every $t$ in the neighborhood. The derivative bound by the integrable function $g$ permits differentiation under the integral at $\theta$. Therefore,

0=\left.\frac{\partial}{\partial t}\int f(x;t)\,d\mu(x)\right|_{t=\theta} =\int \left.\frac{\partial}{\partial t}f(x;t)\right|_{t=\theta}\,d\mu(x).

Where $f(x;\theta)>0$, the derivative of the log density satisfies $\partial_\theta f(x;\theta)=f(x;\theta)U(\theta;x)$. The set where $f(x;\theta)=0$ has probability zero under $f(\cdot;\theta)$. It follows that

E[U(\theta;X)] =\int U(\theta;x)f(x;\theta)\,d\mu(x) =\int \partial_\theta f(x;\theta)\,d\mu(x) =0.

$\square$

The theorem yields the score-variance expression for information, as long as the score has a finite second moment. This finiteness requirement is separate: the differentiation argument establishes the mean, not the second moment.

Corollary: Under the theorem’s conditions, if $E[U(\theta;X)^2]<\infty$, then $$ I(\theta)=\operatorname{Var}(U(\theta;X)) =\int \frac{[\partial_\theta f(x;\theta)]^2}{f(x;\theta)}\,d\mu(x), $$ where the integrand is taken on the set where $f(x;\theta)>0$.

Derivation. By definition, $I(\theta)=E[U^2]$. The theorem gives $E[U]=0$, so $\operatorname{Var}(U)=E[U^2]-(E[U])^2=I(\theta)$. On the set where $f>0$, $U=(\partial_\theta f)/f$, and hence

E[U^2] =\int \left(\frac{\partial_\theta f(x;\theta)}{f(x;\theta)}\right)^2 f(x;\theta)\,d\mu(x) =\int \frac{[\partial_\theta f(x;\theta)]^2}{f(x;\theta)}\,d\mu(x).

The assumed finite second moment ensures this information is finite; if it is not finite, the same expression may have value $+\infty$. $\square$

Worked Calculations in Regular Models

For a continuous model, the score is often easiest to find by differentiating the log density. After that, calculate its expectation and variance under the stated distribution. The next examples show both the score-variance calculation and a discrete version of the same reasoning.

Worked Example: Information in One Exponential Observation

Let $X$ have an exponential distribution with rate $\theta>0$, so $f(x;\theta)=\theta e^{-\theta x}$ for $x\geq 0$, and zero otherwise. Find the score and Fisher information.

The support $[0,\infty)$ does not depend on $\theta$, and the density is differentiable in $\theta$. For any fixed $\theta>0$, choose a sufficiently small neighborhood with lower endpoint $a>0$ and upper endpoint $b<\infty$. In that neighborhood, $|\partial_t f(x;t)|=e^{-tx}|1-tx|\leq e^{-ax}(1+bx)$ for $x\geq0$. The right-hand side is integrable, so the differentiation condition holds.

For $x\geq0$, the log density is $\log f(x;\theta)=\log\theta-\theta x$. Thus

U(\theta;X)=\frac{1}{\theta}-X.

An exponential random variable with rate $\theta$ has $E[X]=1/\theta$ and $\operatorname{Var}(X)=1/\theta^2$. Therefore $E[U]=1/\theta-1/\theta=0$, and

I(\theta)=\operatorname{Var}(U) =\operatorname{Var}\left(\frac1\theta-X\right) =\operatorname{Var}(X) =\frac{1}{\theta^2}.

For example, at $\theta=2$, the score is $1/2-X$, its mean is zero, and the information is $1/4$.

Worked Example: Information in a Geometric Observation

Suppose $X$ counts the number of failures before the first success, with $P(X=k)=(1-p)^k p$ for $k=0,1,2,\ldots$ and $0<p<1$. Find the score and information for $p$.

The support is fixed. The log mass is $k\log(1-p)+\log p$, so

U(p;X)=\frac{1}{p}-\frac{X}{1-p}.

For this geometric distribution, $E[X]=(1-p)/p$ and $\operatorname{Var}(X)=(1-p)/p^2$. Substitution checks that the score is centered:

E[U]=\frac{1}{p}-\frac{E[X]}{1-p} =\frac{1}{p}-\frac{(1-p)/p}{1-p}=0.

The only random part of the score is $-X/(1-p)$. Therefore,

I(p)=\operatorname{Var}(U) =\frac{\operatorname{Var}(X)}{(1-p)^2} =\frac{(1-p)/p^2}{(1-p)^2} =\frac{1}{p^2(1-p)}.

At $p=1/2$, this gives $I(p)=1/[(1/2)^2(1/2)]=8$. The score-variance calculation applies because the support is fixed and, locally around any $p\in(0,1)$, the derivatives of the geometric probabilities are bounded by a summable sequence. For example, restrict $t$ to $a\leq t\leq b$ with $0<a<b<1$; then $|\partial_t[(1-t)^k t]|\leq (1-a)^k + kb(1-a)^{k-1}$, whose sum over $k\geq0$ is finite.

Parameter-Dependent Support: A Necessary Warning

A density can be differentiable at each point where it is positive and still fail the regularity conditions. In particular, if the support changes with the parameter, differentiating the density at fixed data values may not account for how probability enters or leaves through the moving boundary. The next example shows why the mean-zero conclusion cannot be applied automatically.

Worked Example: Uniform Support and a Nonzero Mean Score

Let $X\sim\operatorname{Uniform}(0,\theta)$, where $\theta>0$. Its density is $f(x;\theta)=1/\theta$ for $0<x<\theta$ and zero otherwise. For $0<x<\theta$, the log density is $-\log\theta$, giving

U(\theta;X)=-\frac{1}{\theta}.

Since $X$ lies in $(0,\theta)$ with probability one, this score is constant under the model. Thus $E[U]=-1/\theta$, $\operatorname{Var}(U)=0$, and $E[U^2]=1/\theta^2$. Under the definition of information as the expected squared score, $I(\theta)=1/\theta^2$; it is not the variance of the score.

The regularity condition fails because the support $(0,\theta)$ depends on $\theta$. In particular, the integral of the pointwise derivative of the density over the fixed real line is $\int_0^\theta -1/\theta^2\,dx=-1/\theta$, not zero. Differentiating the full integral requires accounting for the moving endpoint: $\partial_\theta\int_0^\theta (1/\theta)\,dx=0$, because the boundary contribution is $1/\theta$ and cancels the integral of the pointwise derivative. This is exactly the step the fixed-support argument cannot make.

Common Mistake and Practical Use

A frequent error is to define information as the variance of the score without checking that the score has mean zero. The correct general definition used here is $I(\theta)=E[U^2]$; it equals $\operatorname{Var}(U)$ when the score is centered and has finite second moment. The uniform example shows that these quantities can differ.

In a regular model, the score-variance form is useful because it turns information into a measure of the score’s spread across possible observations. A calculation can then use familiar expectations and variances rather than directly integrating a squared derivative. But that shortcut is justified only after checking the model’s support and the conditions that permit differentiation under the integral or sum. For independent observations, scores can also be formed from the joint likelihood; the information contributed by multiple observations is treated separately.

Differentiate the log density or mass.
Obtain the score as a function of the observation and parameter.

Check the support and regularity.
Confirm that a fixed dominating measure applies and that differentiation under the integral or sum is justified.

Calculate the second moment.
Use $I(\theta)=E[U^2]$; use $\operatorname{Var}(U)$ only when the score is centered and has finite variance.

Key takeaway: Fisher information is the expected squared score. Under regularity conditions that imply $E[U]=0$, and when the second moment is finite, it is also the score variance. A parameter-dependent support is a warning that this identity may fail.

Check Your Understanding

Answer each question using the definitions and conditions above.

State the definition of the scalar score and Fisher information.
Which step in the proof of $E[U]=0$ requires a regularity condition, and what does that condition justify?
For the exponential-rate model, why does subtracting $X$ from the constant $1/\theta$ leave the variance unchanged?
In the geometric example, verify that the score has mean zero using $E[X]=(1-p)/p$.
For the uniform $(0,\theta)$ model, explain why $E[U^2]$ is not equal to $\operatorname{Var}(U)$, and identify the violated condition.