Information for Normal Mean and Variance

Why Calculate Information for Both Normal Parameters?

A normal model is often used when observations vary around a common mean, with the variance describing their spread. When both quantities are unknown, the likelihood has two parameters, and its Fisher information is a matrix: its diagonal entries describe information about each parameter, while the off-diagonal entry describes their interaction in the information calculation. For the normal mean and variance, that off-diagonal entry is zero. The derivation shows why, and also clarifies what this orthogonality does—and does not—say.

Model, Likelihood, and Score

Let \(X_1,\ldots,X_n\) be independent and identically distributed as \(N(\mu,\sigma^2)\), where \(\mu\in\mathbb{R}\) and \(\sigma^2>0\) are both unknown. Write \(v=\sigma^2\) for the variance while differentiating, so the parameter vector is \(\theta=(\mu,v)\). In particular, \(v^2=\sigma^4\).

Definition: For this model, the score is the vector of first derivatives of the log-likelihood, \(U(\theta)=\left(\partial\ell/\partial\mu,\partial\ell/\partial v\right)^\mathsf{T}\). The Fisher information matrix is \(I_n(\theta)=E_\theta[-\nabla^2\ell(\theta)]\), the expected negative Hessian of the log-likelihood. Here the normal density is smooth for \(v>0\), and its support does not depend on \(\theta\).

For observed values \(x_1,\ldots,x_n\), independence gives the log-likelihood

\ell(\mu,v)=-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log v-\frac{1}{2v}\sum_{i=1}^n(x_i-\mu)^2,\qquad v>0.

Differentiating with respect to \(\mu\) and \(v\) gives the score components

U_\mu(\mu,v)=\frac{1}{v}\sum_{i=1}^n(x_i-\mu),\qquad U_v(\mu,v)=-\frac{n}{2v}+\frac{1}{2v^2}\sum_{i=1}^n(x_i-\mu)^2.

The variance score is a derivative with respect to \(v=\sigma^2\), not with respect to \(\sigma\). That distinction determines the powers of the variance in the information matrix.

Worked Example: Scores and Observed Curvature for Four Observations

Suppose the observed values are \(1,2,4,5\), and evaluate the likelihood derivatives at \(\mu=2\) and \(v=9\). The residuals are \(-1,0,2,3\), so their sum is \(4\) and their squared sum is \(1+0+4+9=14\).

The score components at this parameter point are

U_\mu=\frac{4}{9},\qquad U_v=-\frac{4}{18}+\frac{14}{2(9^2)} =-\frac{2}{9}+\frac{7}{81} =-\frac{11}{81}.

These are sample-specific derivatives; a score need not equal zero at an arbitrary parameter value. The Hessian entries, obtained by differentiating the score again, are

\ell_{\mu\mu}=-\frac{n}{v},\qquad \ell_{\mu v}=-\frac{\sum_i(x_i-\mu)}{v^2},\qquad \ell_{vv}=\frac{n}{2v^2}-\frac{\sum_i(x_i-\mu)^2}{v^3}.

At the stated values, \(\ell_{\mu\mu}=-4/9\), \(\ell_{\mu v}=-4/81\), and

\ell_{vv}=\frac{4}{162}-\frac{14}{729} =\frac{18-14}{729} =\frac{4}{729}.

Thus the observed negative Hessian has off-diagonal entry \(4/81\), which is not zero. This does not contradict orthogonality: Fisher information uses the expected negative Hessian, not the negative Hessian for one particular data set.

Deriving the Fisher Information Matrix

The Hessian formulas above hold for any observed sample. To take their expectations under the model, use \(E_\theta[X_i-\mu]=0\) and \(E_\theta[(X_i-\mu)^2]=v\). The expected negative curvature can therefore be calculated entry by entry.

Theorem: If \(X_1,\ldots,X_n\) are independent \(N(\mu,v)\) observations with \(\mu\in\mathbb{R}\) and \(v>0\), then for the parameterization \((\mu,v)\) the Fisher information matrix is

I_n(\mu,v)= \begin{pmatrix} \dfrac{n}{v} & 0\\[6pt] 0 & \dfrac{n}{2v^2} \end{pmatrix} = \begin{pmatrix} \dfrac{n}{\sigma^2} & 0\\[6pt] 0 & \dfrac{n}{2\sigma^4} \end{pmatrix}.

Proof. Negating the Hessian entries gives

-\ell_{\mu\mu}=\frac{n}{v},\qquad -\ell_{\mu v}=\frac{\sum_i(X_i-\mu)}{v^2},\qquad -\ell_{vv}=-\frac{n}{2v^2}+\frac{\sum_i(X_i-\mu)^2}{v^3}.

The first expression is constant, so its expectation is \(n/v\). Since each centered observation has mean zero, the expected cross entry is

E_\theta[-\ell_{\mu v}] =\frac{1}{v^2}\sum_{i=1}^n E_\theta[X_i-\mu]=0.

For the variance entry, each centered observation has second moment \(v\), and hence

E_\theta[-\ell_{vv}] =-\frac{n}{2v^2}+\frac{1}{v^3}\sum_{i=1}^n E_\theta[(X_i-\mu)^2] =-\frac{n}{2v^2}+\frac{nv}{v^3} =\frac{n}{2v^2}.

Combining the three entries proves the stated matrix. \(\square\)

Worked Example: Information at a Specified Mean and Variance

For a sample of \(n=4\) independent observations from \(N(\mu,9)\), the information is evaluated at any chosen mean because this matrix does not depend on \(\mu\). Its mean entry is \(n/v=4/9\), and its variance entry is \(n/(2v^2)=4/(2\cdot81)=2/81\). Thus

I_4(\mu,9)= \begin{pmatrix} 4/9 & 0\\ 0 & 2/81 \end{pmatrix}.

The zero cross-entry holds at every \(\mu\) and every positive variance, not just for this numerical choice. The two diagonal entries have different units because they refer to derivatives with respect to different parameters.

Checking Information Through the Score

For this model, the score has mean zero, and its covariance matrix gives the same information. This provides a second route to the result and makes the source of the zero cross-entry especially transparent. Set \(Z_i=(X_i-\mu)/\sqrt{v}\). Each \(Z_i\) is standard normal, and the scores can be written as

U_\mu=\frac{1}{\sqrt v}\sum_{i=1}^n Z_i,\qquad U_v=\frac{1}{2v}\sum_{i=1}^n(Z_i^2-1).

Result: In this normal model, \(E_\theta[U]=0\) and \(\operatorname{Cov}_\theta(U)=I_n(\mu,v)\). In particular, the mean and variance score components have covariance zero.

Derivation. The standard normal moments needed here are \(E[Z_i]=0\), \(E[Z_i^2]=1\), \(E[Z_i^3]=0\), and \(E[Z_i^4]=3\). The odd moments vanish by symmetry; the even moments follow from the standard normal density. These give \(E[U_\mu]=E[U_v]=0\).

For the first score component, independence and zero means make all cross terms vanish when its square is expanded:

\operatorname{Var}(U_\mu) =\frac{1}{v}\operatorname{Var}\left(\sum_i Z_i\right) =\frac{1}{v}\sum_i\operatorname{Var}(Z_i) =\frac{n}{v}.

For the second, the variables \(Z_i^2-1\) are independent, have mean zero, and have variance \(E[Z_i^4]-(E[Z_i^2])^2=3-1=2\). Therefore

\operatorname{Var}(U_v) =\frac{1}{4v^2}\sum_i\operatorname{Var}(Z_i^2-1) =\frac{2n}{4v^2} =\frac{n}{2v^2}.

Finally, terms involving distinct observations have zero expectation by independence and the zero mean of each score contribution. For the terms from the same observation, the relevant moment is \(E[Z_i(Z_i^2-1)]=E[Z_i^3]-E[Z_i]=0\). Consequently,

\operatorname{Cov}(U_\mu,U_v) =\frac{1}{2v\sqrt v}\sum_{i=1}^n E[Z_i(Z_i^2-1)] =0.

These three covariance entries give the same matrix as the expected-Hessian calculation. \(\square\)

Worked Example: Verifying the Score Covariance for Two Observations

Take \(n=2\) and \(v=4\). Then \(U_\mu=(Z_1+Z_2)/2\) and \(U_v=(Z_1^2+Z_2^2-2)/8\). The score means are zero because \(E[Z_i]=0\) and \(E[Z_i^2-1]=0\).

The mean-score variance is \((1/4)(1+1)=1/2\). The variance-score variance is \((1/64)(2+2)=1/16\), since each \(Z_i^2-1\) has variance \(2\). The covariance is zero: the same-observation terms have expectation \(E[Z_i^3-Z_i]=0\), and cross-observation terms have expectation zero by independence. Thus

\operatorname{Cov}(U)= \begin{pmatrix} 1/2&0\\ 0&1/16 \end{pmatrix} = \begin{pmatrix} n/v&0\\ 0&n/(2v^2) \end{pmatrix}_{n=2,\,v=4}.

Orthogonality and Its Interpretation

Two parameters are called orthogonal in the Fisher-information sense when the off-diagonal entry of their Fisher information matrix is zero throughout the parameter space. For the parameter vector \((\mu,v)\), the calculation shows that the mean and variance are orthogonal: \(I_{\mu v}(\mu,v)=0\) for every \(\mu\in\mathbb{R}\) and \(v>0\).

This is a statement about expected curvature, equivalently here about the covariance of the score components. It does not say that the observed cross-derivative vanishes for every sample. As the first worked example showed, an individual sample can give a nonzero observed cross-entry even though its expectation is zero. Nor does orthogonality mean that the parameters are unrelated in every possible sense; it is a precise property of this likelihood and this parameterization.

For example, with \(n=10\) and \(v=2\), the information matrix is

I_{10}(\mu,2)= \begin{pmatrix} 10/2&0\\ 0&10/(2\cdot 2^2) \end{pmatrix} = \begin{pmatrix} 5&0\\ 0&5/4 \end{pmatrix}.

If the same sample size has \(v=8\), the entries become \(10/8=5/4\) and \(10/(2\cdot64)=5/64\), while the cross-entry remains zero. The information about the mean depends on the variance through \(1/v\); the information about the variance depends on it through \(1/v^2\). Orthogonality does not make the diagonal entries constant or equal.

Common Mistakes

Differentiating with respect to the wrong variance parameter. If the parameter is \(v=\sigma^2\), the variance-information entry is \(n/(2v^2)=n/(2\sigma^4)\), not \(n/(2\sigma^2)\). The derivative is with respect to the variance itself.
Confusing observed and expected information. A nonzero observed cross-derivative for one sample does not refute orthogonality. Fisher information is the expectation of negative curvature, and the expected cross-entry is zero.
Treating a zero cross-entry as a claim about all dependence. Orthogonality here means zero Fisher-information cross-entry. It is not a general claim that every statistic or feature of the data associated with the two parameters is independent.

Key takeaway: For an independent \(N(\mu,\sigma^2)\) sample with both parameters unknown, the Fisher information in the \((\mu,\sigma^2)\) parameterization is diagonal, with entries \(n/\sigma^2\) and \(n/(2\sigma^4)\). The zero cross-entry follows from the symmetry of the centered normal distribution.

Check Your Understanding

Use the parameterization \((\mu,v)\), where \(v=\sigma^2>0\).

Write the log-likelihood for \(n\) independent \(N(\mu,v)\) observations.
What are the score components for \(\mu\) and \(v\)?
For \(n=6\) and \(v=3\), calculate both diagonal entries of the Fisher information matrix.
Why can the observed cross-entry be nonzero even when the mean and variance are orthogonal?
Which standard normal moment makes the covariance of the two score components zero?