Deriving Fisher Information for a Normal Mean

Why Derive Information From the Likelihood?

In a group sequential design, the amount of statistical information available at a look is a measure of how precisely the data can estimate the parameter of interest. For a normal mean with known variance, this information has a particularly simple form. Deriving it from the likelihood explains both the role of the variance and why independent observations add information. The result is also a useful benchmark when checking calculations under more complicated models.

Here the parameter is the population mean $\mu$, and the population variance $\sigma^2$ is treated as known and positive. The observations are assumed to be independent and normally distributed. The tutorial focuses on deriving information for this model, not on converting it to a monitoring schedule or an information fraction.

Definition: For a parameter $\mu$, the score is the derivative of the log-likelihood with respect to $\mu$. Under the usual regularity conditions, the Fisher information is the expected squared score, equivalently the expected negative second derivative of the log-likelihood: $$ I(\mu)=E_\mu\!\left[\left(\frac{\partial \ell(\mu)}{\partial\mu}\right)^2\right] =-E_\mu\!\left[\frac{\partial^2\ell(\mu)}{\partial\mu^2}\right]. $$ Here $\ell(\mu)$ is the log-likelihood and the expectation is taken under the model with parameter $\mu$.

One Normal Observation

Let $X\sim N(\mu,\sigma^2)$, with known $\sigma^2>0$. Its density, viewed as a function of $\mu$ for observed value $x$, is

f(x;\mu)=\frac{1}{\sqrt{2\pi\sigma^2}} \exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right).

Taking the natural logarithm and differentiating with respect to $\mu$ gives the log-likelihood and score:

\ell(\mu)=-\frac{1}{2}\log(2\pi\sigma^2)-\frac{(x-\mu)^2}{2\sigma^2}, \qquad U(\mu)=\frac{\partial\ell(\mu)}{\partial\mu} =\frac{x-\mu}{\sigma^2}.

The score measures the local direction and steepness of the log-likelihood: it is positive when $x>\mu$, negative when $x<\mu$, and zero at $x=\mu$. The second derivative is constant in $x$ and $\mu$:

\frac{\partial^2\ell(\mu)}{\partial\mu^2}=-\frac{1}{\sigma^2}.

Therefore, the expected negative second derivative is $1/\sigma^2$. The squared-score definition gives the same result because $E_\mu[(X-\mu)^2]=\sigma^2$:

I_1(\mu) =E_\mu\!\left[\frac{(X-\mu)^2}{\sigma^4}\right] =\frac{\sigma^2}{\sigma^4} =\frac{1}{\sigma^2}.

This equality also verifies the equivalence of the two definitions for this model. The information is greater when the variance is smaller: observations with less variability provide more information about their mean. Its units are the inverse of the measurement units squared. For example, if $X$ is measured in milligrams, information is in inverse milligrams squared.

Worked Example: Information From One Observation

Suppose a hypothetical measurement $X$ is normally distributed with mean $\mu$ in millimeters and known standard deviation $\sigma=2$ millimeters. Calculate its Fisher information for $\mu$.

The variance is $\sigma^2=2^2=4$ square millimeters. For one normal observation, $I_1(\mu)=1/\sigma^2$, so

I_1(\mu)=\frac{1}{4}=0.2500\ \text{mm}^{-2}.

The result does not depend on the observed measurement or the value of $\mu$. It depends on the assumed variance and the parameterization in millimeters. It describes the model's expected precision about the mean, not the direction of the observed estimate's error.

Independent Observations Add Information

Now suppose $X_1,\ldots,X_n$ are independent observations, each with distribution $N(\mu,\sigma^2)$, where the same known variance applies to every observation. Independence makes the joint likelihood the product of the individual densities. Consequently, the joint log-likelihood is a sum, and so is its score:

\ell_n(\mu)=\sum_{i=1}^{n}\ell_i(\mu), \qquad U_n(\mu)=\frac{\partial\ell_n(\mu)}{\partial\mu} =\sum_{i=1}^{n}\frac{X_i-\mu}{\sigma^2}.

For the expected negative second derivative, differentiating the sum gives $n$ identical terms. For the squared-score definition, the cross-products have expectation zero: each individual score has mean zero, and independence makes the expectation of a product of distinct scores equal the product of their means. The following derivation displays both arguments.

\begin{aligned} I_n(\mu) &=-E_\mu\!\left[\frac{\partial^2\ell_n(\mu)}{\partial\mu^2}\right] =-E_\mu\!\left[-\frac{n}{\sigma^2}\right] =\frac{n}{\sigma^2},\\ E_\mu[U_n(\mu)^2] &=\sum_{i=1}^{n}E_\mu[U_i(\mu)^2] +2\sum_{i<j}E_\mu[U_i(\mu)U_j(\mu)]\\ &=\frac{n}{\sigma^2} +2\sum_{i<j}E_\mu[U_i(\mu)]E_\mu[U_j(\mu)] =\frac{n}{\sigma^2}. \end{aligned}

Thus, the Fisher information in $n$ independent, equal-variance observations is $n/\sigma^2$. The additivity is not a special arithmetic shortcut: it follows from the factorization of the likelihood under independence. The sample mean has variance $\sigma^2/n$, so its standard error is $\sigma/\sqrt{n}$. Equivalently, the inverse of its variance is $n/\sigma^2$, matching the Fisher information in this model.

Worked Example: Combining Independent Measurements

A hypothetical experiment records 30 independent measurements of a mean level $\mu$, each in the same units and each with known standard deviation 3 units. Calculate the total Fisher information and the standard error of the sample mean under the model.

Each measurement has variance $3^2=9$ square units. Independence and the common known variance give

I_{30}(\mu)=\frac{30}{9}=3.333\ \text{units}^{-2}.

The model variance of the sample mean is $\sigma^2/n=9/30=0.3000$ square units, and its standard error is

SE(\bar{x})=\sqrt{\frac{9}{30}}=0.5477\ \text{units}.

As a check, $1/\sqrt{I_{30}(\mu)}=1/\sqrt{3.333}=0.5477$ units, with rounding. Adding independent measurements increases information in direct proportion to their number, while the standard error decreases in proportion to $1/\sqrt{n}$.

Independent Observations With Different Known Variances

The same likelihood argument allows independent observations to have different known variances, provided they share the same mean parameter. Suppose $X_i\sim N(\mu,\sigma_i^2)$, independently, where each $\sigma_i^2$ is known and positive. The $i$th log-likelihood contributes a second derivative of $-1/\sigma_i^2$. Summing these contributions gives

I_n(\mu)=\sum_{i=1}^{n}\frac{1}{\sigma_i^2}.

An observation with a smaller variance contributes more information. This formula relies on independence: if observations are correlated, their joint likelihood generally does not factor into separate terms, and one cannot simply add the marginal information values as if the observations were independent.

Worked Example: Adding Information With Unequal Variances

Consider three hypothetical independent measurements of the same mean $\mu$, all in centimeters, with known standard deviations 2, 3, and 4 centimeters. Their observed values are 10, 11, and 8 centimeters. Calculate the Fisher information for $\mu$. Also calculate the score at the candidate value $\mu=10$ centimeters to distinguish the score from the information.

The variances are 4, 9, and 16 square centimeters. The information adds as the sum of the inverse variances:

I(\mu)=\frac{1}{4}+\frac{1}{9}+\frac{1}{16} =0.2500+0.1111+0.0625 =0.4236\ \text{cm}^{-2}.

At $\mu=10$ centimeters, the observed score is the sum of the three residuals divided by their respective variances:

U(10)=\frac{10-10}{4}+\frac{11-10}{9}+\frac{8-10}{16} =0+\frac{1}{9}-\frac{2}{16} =-0.01389\ \text{cm}^{-1}.

The score is one data-dependent quantity indicating the local direction of the likelihood at the candidate mean. The information is a model-based measure of expected curvature; here it is positive and does not depend on those three observed values. They answer different questions.

Assumptions, Diagnostics, and Limits

The formulas above require a suitable model for the observations and the way they relate to one another. Before using $n/\sigma^2$, check that the observations are plausibly independent, share a common mean, are adequately modeled by normal distributions, and have the specified common variance. For the unequal-variance formula, the observations must still be independent and share the same mean, but their known variances may differ.

In practice, the variance is often estimated rather than known. Substituting an estimated variance may be useful, but it is no longer exactly the calculation under the known-variance model derived here. Likewise, dependence, changing means, outliers, or a poor normal approximation can make the nominal information a poor description of the actual data. Diagnostics should therefore address the assumptions, not just confirm that the arithmetic was performed correctly.

For example, a time plot can reveal drift or serial patterns inconsistent with independent, common-mean observations. A plot of residuals can reveal outliers or changing spread that warrant investigating the normal and constant-variance assumptions. If repeated measurements are correlated, counting each one as a separate independent contribution generally overstates the available information. The correct calculation must account for the joint model.

The main alternative to a likelihood-based information calculation is to describe precision through the variance or standard error of an estimator. In this normal model, the two views agree: the sample mean has variance $\sigma^2/n$, and its reciprocal is $I_n(\mu)$. The likelihood derivation is valuable because it states exactly which assumptions produce that relationship. In other models, the information calculation may require a different likelihood and need not reduce to this simple formula.

Common mistake: Treating the observed score as the Fisher information, or adding information values while ignoring dependence. The score changes with the observed data and may be positive, negative, or zero. Fisher information is an expected measure of likelihood curvature. Additivity follows from independent likelihood contributions; it should not be assumed for correlated observations.

Check Your Understanding

Answer each question using the known-variance normal-mean model. Full solutions follow the questions.

One observation has a known standard deviation of 5 seconds. What is its Fisher information for the mean, including units?
For 16 independent normal observations with known variance 4 square meters, calculate the Fisher information for their common mean.
Two independent observations share a mean, but have known variances 1 and 9 square centimeters. What is their total Fisher information?
Explain why an observed score of zero does not imply that the Fisher information is zero.

Solutions. (1) The variance is $5^2=25$ square seconds, so $I_1=1/25=0.04000\ \text{seconds}^{-2}$. (2) $I_{16}=n/\sigma^2=16/4=4.000\ \text{m}^{-2}$. (3) Independence gives $I=1/1+1/9=1.111\ \text{cm}^{-2}$. (4) The score is data-dependent and can equal zero when positive and negative residual contributions balance. Information is expected squared score, or expected negative log-likelihood curvature; in this model it is $n/\sigma^2>0$ for positive $n$ and $\sigma^2$, regardless of the observed score.