Deriving the Fisher Information Identity

Why Curvature and Score Variability Are Connected

A likelihood changes as its parameter changes. Its score records the local slope of the log-likelihood, while its curvature records how quickly that slope changes. Under suitable conditions, the expected negative curvature equals the variance of the score. This gives two ways to describe the same information in a regular model.

That equality is not automatic. In particular, differentiating an integral or sum involving a density requires conditions that allow differentiation to pass through the integral or sum. If the possible observations depend on the parameter, a boundary contribution can be missed. The distinction matters when interpreting information calculations for severity models and likelihoods based on incomplete observations.

Definition: For one observation \(X\) with density or probability mass function \(f_\theta(x)\), where the scalar parameter \(\theta\) is an interior point of its parameter space, the score is \(U_\theta(X)=\frac{\partial}{\partial\theta}\log f_\theta(X)\). The expected Fisher information is \(I(\theta)=E_\theta[U_\theta(X)^2]\) when the score has mean zero and a finite second moment. Under the regularity conditions below, this also equals \(E_\theta[-\frac{\partial^2}{\partial\theta^2}\log f_\theta(X)]\).

The identity is for expected quantities under the model at the stated parameter. It does not say that the square of the score for one observed data set equals that data set’s negative log-likelihood curvature. Nor does it assert that curvature is constant across possible data sets.

Deriving the Identity

Write \(\ell_\theta(x)=\log f_\theta(x)\). For a continuous model with support that does not depend on \(\theta\), suppose the density is twice differentiable in \(\theta\), differentiation may be interchanged with integration, and the required derivatives are integrable. Also suppose the score has finite variance. The same argument applies to a discrete model by replacing integrals with sums.

Because a density integrates to one, differentiating its normalization gives a first useful result. The density derivatives satisfy \(\partial f_\theta/\partial\theta=f_\theta U_\theta\) wherever \(f_\theta>0\).

1=\int f_\theta(x)\,dx \quad\Longrightarrow\quad 0=\int \frac{\partial f_\theta(x)}{\partial\theta}\,dx =\int f_\theta(x)U_\theta(x)\,dx =E_\theta[U_\theta(X)].

Thus the score has mean zero under these conditions. Differentiate the normalization a second time. The derivative of \(f_\theta U_\theta\) is \(f_\theta U_\theta^2+f_\theta\,\partial U_\theta/\partial\theta\), and \(\partial U_\theta/\partial\theta=\partial^2\ell_\theta/\partial\theta^2\).

0=\int \frac{\partial^2 f_\theta(x)}{\partial\theta^2}\,dx =\int f_\theta(x)\left[ U_\theta(x)^2+ \frac{\partial^2}{\partial\theta^2}\ell_\theta(x) \right]dx.

Rearranging proves the identity. Since the score has mean zero, its second moment is its variance. This is why the expected negative curvature equals the variance of the score, rather than merely being a convenient algebraic resemblance.

E_\theta\!\left[-\frac{\partial^2}{\partial\theta^2}\ell_\theta(X)\right] =E_\theta[U_\theta(X)^2] =\operatorname{Var}_\theta(U_\theta(X)) =I(\theta).

Regularity conditions: A standard sufficient set of conditions is that \(\theta\) lies in the interior of the parameter space; the support of \(X\) does not depend on \(\theta\); \(f_\theta\) is twice differentiable in \(\theta\); differentiation can be interchanged with integration (or summation) at both steps; and the score and required derivatives have finite expectations. These conditions are sufficient, not a claim that every possible model must satisfy them in exactly this form. If they fail, the identity must not be used without a separate justification.

Applying the Identity in Regular Models

For independent observations, the joint log-likelihood is the sum of the individual log-likelihoods. Its score is therefore a sum of individual scores. If the observations share the same regular model and have finite score variances, their scores are independent with mean zero, so the variance of their sum is the sum of their variances. Expected negative curvature is additive for the same reason. The examples below work directly with these expected quantities.

Worked Example: Normal Measurement Errors With Unknown Mean

A hypothetical calibration study records three independent measurement errors \(X_1,X_2,X_3\), each modeled as normal with unknown mean \(\mu\) and known standard deviation \(\sigma=2\). The observed values are \(8,10,12\). Calculate the Fisher information for \(\mu\) from the three observations and check the identity. The normal model has fixed support on the real line and is regular for every real \(\mu\).

Method. Write the log-density, differentiate to find the score and curvature, then take their expectations. This method avoids treating the score from the particular observed sample as if it were an expected information.

For one observation, the log-density, score, and second derivative are

\ell_\mu(x)=-\frac12\log(2\pi\cdot 4)-\frac{(x-\mu)^2}{8}, \qquad U_\mu(x)=\frac{x-\mu}{4}, \qquad \frac{\partial^2\ell_\mu(x)}{\partial\mu^2}=-\frac14.

Under the model, \(E_\mu[X-\mu]=0\) and \(\operatorname{Var}_\mu(X)=4\). Therefore the one-observation score has mean zero and variance \(4/16=1/4\). The expected negative second derivative is also \(1/4\).

For three independent observations, the total score is \(U_\mu=\sum_{j=1}^3(X_j-\mu)/4\). Its variance is \(3(1/4)=3/4\), and the total negative second derivative is \(3/4\) for every sample. Thus the Fisher information is \(I_3(\mu)=0.75\).

As a check using the stated observations, at \(\mu=10\) the individual scores are \(-0.5,0,0.5\), giving total score \(0\). The observed negative curvature is still \(0.75\). The realized score being zero does not make the information zero: information is the variance of the score over samples, not the square of this one realized score.

Worked Example: Bernoulli Claim Indicators

In a hypothetical portfolio, four independent policies each produce a claim indicator \(X_j\), where \(X_j=1\) denotes at least one claim and \(X_j=0\) denotes none. Assume \(P(X_j=1)=p\), with \(p=0.6\) for the information calculation. The observed indicators contain three ones and one zero. Find the expected information and compare it with the observed curvature.

Method. Use the Bernoulli likelihood with \(p\) as the parameter. Since \(0<p<1\) and the support \(\{0,1\}\) is fixed, the regular identity applies. This avoids confusing a realized score squared with its variance.

For \(k\) successes out of \(n=4\), the log-likelihood, score, and second derivative are

\ell_p=k\log p+(4-k)\log(1-p), \qquad U_p=\frac{k}{p}-\frac{4-k}{1-p}, \qquad \frac{\partial^2\ell_p}{\partial p^2} =-\frac{k}{p^2}-\frac{4-k}{(1-p)^2}.

The expected number of successes is \(E_p[k]=4p=2.4\). The expected negative curvature at \(p=0.6\) is

E_p[-\ell_p''(p)] =\frac{E_p[k]}{p^2}+\frac{4-E_p[k]}{(1-p)^2} =\frac{2.4}{0.36}+\frac{1.6}{0.16} =6.6667+10 =16.6667.

For one indicator, the score is \(1/p\) with probability \(p\), and \(-1/(1-p)\) with probability \(1-p\). Its mean is \(p(1/p)+(1-p)(-1/(1-p))=0\), and its variance is \(p(1/p^2)+(1-p)(1/(1-p)^2)=1/[p(1-p)]=1/0.24=4.1667\). Four independent indicators therefore have score variance \(4(4.1667)=16.6667\), agreeing with expected negative curvature.

For the observed \(k=3\), the score is \(3/0.6-1/0.4=2.5\), whose square is \(6.25\). The observed negative curvature is \(3/0.36+1/0.16=14.5833\). Neither observed quantity needs to equal the expected information \(16.6667\). This difference is not a contradiction; the identity equates expectations, not individual sample values.

When the Identity Can Fail

The key step in the derivation was differentiating the normalization condition under the integral. If the range of integration changes with the parameter, differentiating only the density inside the existing support can omit a boundary term. The score may then have nonzero mean, and its variance need not equal its second moment. The uniform model below demonstrates both problems.

Worked Example: A Uniform Severity With Parameter-Dependent Support

Suppose two hypothetical severities \(X_1,X_2\) are independent and uniformly distributed on \((0,\theta)\), with \(\theta=10\). The observed values are \(4\) and \(7\). Compare the variance of the score with expected negative log-likelihood curvature. Identify why the regular identity does not apply.

Method. Differentiate the likelihood for observations strictly inside the support, then check the support condition before invoking the identity. The common mistake is to use a formal score calculation while overlooking that the upper endpoint depends on \(\theta\).

For the observed sample, the likelihood is \(\theta^{-2}\) when \(\theta\geq 7\), and zero otherwise. In the interior region \(\theta>7\), the log-likelihood and its derivatives are

\ell(\theta)=-2\log\theta, \qquad U_\theta=-\frac{2}{\theta}, \qquad \ell''(\theta)=\frac{2}{\theta^2}.

At \(\theta=10\), the score is \(-2/10=-0.2\) for every possible sample whose observations lie in the support. It is therefore constant under the model, so its variance is \(0\). The expected negative curvature is also constant: \(-E_\theta[\ell''(\theta)]=-2/100=-0.02\). Thus the variance of the score is \(0\), whereas the expected negative curvature is \(-0.02\); they are not equal. Moreover, the score mean is \(-0.2\), not zero.

There is no contradiction with the derivation: its fixed-support condition fails because the upper endpoint \(\theta\) moves as the parameter changes. Differentiating the density only for \(0<x<\theta\) omits the contribution from that moving boundary. The negative expected curvature is even negative here, another warning that it should not be interpreted as regular Fisher information. The maximum likelihood estimate for this sample is also at the boundary, reinforcing that standard interior regularity arguments do not apply.

Common Traps and Practical Checks

Equating observed and expected quantities. The identity concerns expectations under the model. A realized score squared, a realized curvature, and expected information are generally different.
Forgetting the mean-zero condition. Variance equals the second moment only when \(E_\theta[U_\theta]=0\). Under regularity, the first normalization derivative gives that result. If the derivative cannot pass through the integral, do not assume it.
Ignoring the support. Check whether the possible values of the observation depend on the parameter. Truncation, censoring, or a parameter-dependent endpoint can require careful treatment of the likelihood and its conditions; do not import the regular identity mechanically.
Using the wrong sign. The identity compares the variance of the score with the negative expected second derivative of the log-likelihood. The expected second derivative itself has the opposite sign in regular models.

A reliable exam calculation is to write the log-likelihood, differentiate it, check the parameter space and support, and only then take expectations. If those checks establish the regularity conditions, either side of the identity can be used as a check on the other.

Key takeaway: Under suitable regularity conditions, \(E_\theta[-\ell_\theta''(X)]=\operatorname{Var}_\theta(U_\theta(X))\). The proof depends on differentiating the density’s normalization and obtaining a mean-zero score. A parameter-dependent support can invalidate those steps.

Check Your Understanding

For each problem, distinguish expected information from quantities computed for one observed sample, and check whether the stated model is regular.

A hypothetical study has five independent Bernoulli indicators with claim probability \(p=0.2\). Under the regular model, calculate the variance of the total score and the expected negative log-likelihood curvature. (Answer: both are \(5/[0.2(0.8)]=31.25\).)
Six independent observations follow a normal model with unknown mean and known standard deviation \(3\). Find the Fisher information for the mean. (Answer: \(6/3^2=2/3\).)
For five Bernoulli indicators with \(p=0.4\), an observed sample has four successes. Calculate the observed score, its square, the observed negative curvature, and the expected information. (Answer: score \(=4/0.4-1/0.6=8.3333\); score squared \(=69.4444\); observed negative curvature \(=4/0.16+1/0.36=27.7778\); expected information \(=5/[0.4(0.6)]=20.8333\).)
Three observations are independent uniform random variables on \((0,\theta)\), with \(\theta=12\). Using the interior likelihood derivatives, find the score variance and expected negative curvature, and state why the identity is unavailable. (Answer: score is constantly \(-3/12=-0.25\), so its variance is \(0\); expected negative curvature is \(-3/12^2=-1/48\approx-0.0208\); the support depends on \(\theta\).)
State the normalization result that makes the score’s second moment equal its variance in the regular derivation. (Answer: differentiating \(\int f_\theta(x)\,dx=1\) under the integral gives \(E_\theta[U_\theta(X)]=0\).)

What You'll Learn