Fisher Information Matrices and Parameter Dependence

Why Fisher Information Is a Matrix

When a likelihood contains more than one unknown parameter, information about those parameters need not be separate. A change in one parameter can affect how strongly the likelihood responds to another. The Fisher information matrix records both the information associated with each parameter and this cross-parameter coupling.

The diagonal entries summarize information in each parameter direction when the other coordinate is held fixed. The off-diagonal entries describe how the score components move together, or equivalently how the expected curvature in one direction is connected to the other. Looking only at separate diagonal entries can therefore miss an important feature of a multiparameter model.

Definition: For a model with parameter vector \(\boldsymbol{\theta}=(\theta_1,\ldots,\theta_p)^\mathsf{T}\) and one observation \(X\), let \(\ell(\boldsymbol{\theta};X)=\log f(X;\boldsymbol{\theta})\) be the log-likelihood and let \(\mathbf U(\boldsymbol{\theta})=\nabla_{\boldsymbol{\theta}}\ell(\boldsymbol{\theta};X)\) be the score vector. Under regularity conditions, the Fisher information matrix is

\mathbf I(\boldsymbol{\theta}) =E_{\boldsymbol{\theta}}\!\left[\mathbf U\mathbf U^\mathsf{T}\right] =-E_{\boldsymbol{\theta}}\!\left[\nabla_{\boldsymbol{\theta}}^2\ell(\boldsymbol{\theta};X)\right].

Its \(j,k\) entry is \(I_{jk}=E[U_jU_k]\), or equivalently \(-E[\partial^2\ell/(\partial\theta_j\partial\theta_k)]\).

The two expressions for information agree when the model has a differentiable density, its support does not depend on the unknown parameter, differentiation can be interchanged with integration, and the relevant expectations exist. In particular, these conditions imply \(E[\mathbf U]=\mathbf 0\). The score-product expression makes the covariance interpretation transparent: \(I_{jk}=\operatorname{Cov}(U_j,U_k)\). The expected-Hessian expression describes average log-likelihood curvature.

A Two-Parameter Model

Consider a hypothetical measurement \(X\) with a normal distribution whose mean and variance are linked to two parameters:

X\sim N(\alpha+\beta,\beta),\qquad \alpha\in\mathbb R,\quad \beta>0.

Here the second argument of \(N(\mu,\sigma^2)\) is the variance. The parameters are \(\alpha\) and \(\beta\); \(\beta\) controls the variance and also contributes to the mean. For a realized observation \(x\), define the centered residual \(r=x-\alpha-\beta\). The log-likelihood is

\ell(\alpha,\beta;x) =-\frac12\log(2\pi\beta)-\frac{r^2}{2\beta}.

The model has fixed support on the real line and is regular for \(\beta>0\), so the information identities above apply. The next derivation constructs each entry rather than treating the matrix as a collection of unrelated scalar formulas.

Worked Example: Deriving the Full Information Matrix

Method: Differentiate the one-observation log-likelihood, then take score products using the centered normal moments. Find \(\mathbf I(\alpha,\beta)\) for the model \(X\sim N(\alpha+\beta,\beta)\). The common mistake avoided is setting the off-diagonal entry to zero merely because there are two named parameters.

Step 1: Find the score components. Since \(r=x-\alpha-\beta\), its derivative with respect to \(\alpha\) and \(\beta\) is \(-1\) in each case. Differentiating gives

U_\alpha=\frac{\partial\ell}{\partial\alpha}=\frac{r}{\beta}, \qquad U_\beta=\frac{\partial\ell}{\partial\beta} =-\frac{1}{2\beta}+\frac{r}{\beta}+\frac{r^2}{2\beta^2}.

Step 2: Compute \(I_{\alpha\alpha}\). Under the model, \(r=X-\alpha-\beta\sim N(0,\beta)\), so \(E[r^2]=\beta\). Therefore

I_{\alpha\alpha}=E[U_\alpha^2] =\frac{E[r^2]}{\beta^2} =\frac{1}{\beta}.

Step 3: Compute the cross entry. A centered normal variable has \(E[r]=E[r^3]=0\). Thus

\begin{aligned} I_{\alpha\beta} &=E[U_\alpha U_\beta]\\ &=E\!\left[ \frac{r}{\beta}\left(-\frac{1}{2\beta}+\frac{r}{\beta}+\frac{r^2}{2\beta^2}\right) \right]\\ &=-\frac{E[r]}{2\beta^2}+\frac{E[r^2]}{\beta^2} +\frac{E[r^3]}{2\beta^3} =\frac{1}{\beta}. \end{aligned}

Step 4: Compute \(I_{\beta\beta}\). It is convenient to use expected curvature. Differentiating \(U_\beta\) once more gives

\frac{\partial^2\ell}{\partial\beta^2} =\frac{1}{2\beta^2}-\frac{1}{\beta} -\frac{2r}{\beta^2}-\frac{r^2}{\beta^3}.

Taking its expectation uses \(E[r]=0\) and \(E[r^2]=\beta\):

I_{\beta\beta} =-E\!\left[\frac{\partial^2\ell}{\partial\beta^2}\right] =-\left(\frac{1}{2\beta^2}-\frac{1}{\beta}-\frac{1}{\beta^2}\right) =\frac{1}{\beta}+\frac{1}{2\beta^2}.

Result and check. Symmetry gives \(I_{\beta\alpha}=I_{\alpha\beta}\), so

\mathbf I(\alpha,\beta)= \begin{pmatrix} \frac{1}{\beta} & \frac{1}{\beta}\\[4pt] \frac{1}{\beta} & \frac{1}{\beta}+\frac{1}{2\beta^2} \end{pmatrix}.

The cross entry is positive and nonzero. As a check, calculating \(E[U_\beta^2]\) from the score gives the same \(1/\beta+1/(2\beta^2)\) diagonal entry. The nonzero cross term arises because \(\beta\) affects both the mean and the variance.

What the Entries Tell You

For a two-parameter vector \((\theta_1,\theta_2)\), the information matrix has the form

\mathbf I(\boldsymbol{\theta})= \begin{pmatrix} I_{11}&I_{12}\\ I_{12}&I_{22} \end{pmatrix}.

Diagonal entries: \(I_{11}=E[U_1^2]\) and \(I_{22}=E[U_2^2]\) are the variances of the respective score components under regularity. They are also the expected negative curvature in their own coordinate directions.
Off-diagonal entry: \(I_{12}=E[U_1U_2]\) is the covariance of the two score components. A positive value means they tend to move in the same direction; a negative value means they tend to move in opposite directions. Zero means the score components are uncorrelated at that parameter value.
Matrix rather than separate numbers: Information for a simultaneous change in parameters depends on all entries, not just the two diagonal entries. A nonzero cross term should not be discarded.

The off-diagonal entry is sometimes described as parameter coupling or lack of orthogonality in the chosen coordinates. This wording concerns the likelihood's behavior in parameter directions; it does not mean that the fixed parameters are random variables that are statistically dependent. Also, Fisher information is not itself the covariance matrix of parameter estimates.

Because \(\mathbf I=E[\mathbf U\mathbf U^\mathsf{T}]\), it is positive semidefinite: for any fixed vector \(\mathbf a\),

\mathbf a^\mathsf{T}\mathbf I\mathbf a =E[(\mathbf a^\mathsf{T}\mathbf U)^2]\geq 0.

For a symmetric two-by-two matrix, positive definiteness requires \(I_{11}>0\) and \(\det(\mathbf I)=I_{11}I_{22}-I_{12}^2>0\). Positive definiteness indicates that no nonzero linear combination of the parameter directions has zero information locally. A singular matrix can signal a locally unidentifiable direction or a model that does not distinguish parameter changes in that direction.

Worked Example: Evaluating and Checking the Matrix

Method: Substitute the parameter value into the derived matrix, then check its determinant. In a hypothetical measurement study, assume \(X\sim N(\alpha+\beta,\beta)\) with \(\alpha=2\) and \(\beta=4\). The variance is therefore \(4\), not \(4^2\). Find the information matrix and assess its definiteness. The common mistake avoided is confusing the variance with a standard deviation.

Substitute \(\beta=4\) into each entry:

I_{\alpha\alpha}=\frac14,\qquad I_{\alpha\beta}=\frac14,\qquad I_{\beta\beta}=\frac14+\frac{1}{2(4^2)} =\frac14+\frac{1}{32}=\frac{9}{32}.

Thus

\mathbf I(2,4)= \begin{pmatrix} \frac14&\frac14\\[2pt] \frac14&\frac{9}{32} \end{pmatrix}.

Its leading diagonal entry is \(1/4>0\), and its determinant is

\det(\mathbf I)=\frac14\cdot\frac{9}{32} -\frac14\cdot\frac14 =\frac{9}{128}-\frac{8}{128} =\frac{1}{128}>0.

The matrix is positive definite. The cross entry \(1/4\) is as large as the first diagonal entry, so ignoring it would not be a minor bookkeeping change. This is expected: increasing \(\beta\) changes the model's mean as well as its variance. The calculation is checked by the positive determinant and the positive first diagonal entry.

Parameterization Changes the Matrix

Information describes a model in specified parameter coordinates. The same probability distributions can have different-looking information matrices under different parameterizations. In particular, an off-diagonal entry can arise because the chosen coordinates combine effects that are separate in another coordinate system.

For the model above, let \(\mu=\alpha+\beta\). In coordinates \((\mu,\beta)\), the distribution is \(X\sim N(\mu,\beta)\): \(\mu\) is the mean and \(\beta\) is the variance. Its score components are

U_\mu=\frac{r}{\beta},\qquad U_\beta\big|_{\mu}=-\frac{1}{2\beta}+\frac{r^2}{2\beta^2}, \qquad r=x-\mu.

The cross score product has expectation zero because it is a combination of \(E[r]\) and \(E[r^3]\), both zero for a centered normal variable. The resulting matrix is diagonal:

\mathbf I_{(\mu,\beta)}= \begin{pmatrix} \frac{1}{\beta}&0\\[2pt] 0&\frac{1}{2\beta^2} \end{pmatrix}.

The coordinates \((\mu,\beta)\) separate the mean and variance effects; the coordinates \((\alpha,\beta)\) do not. The matrix transformation makes the relationship explicit. For a differentiable one-to-one change \(\boldsymbol{\phi}=g(\boldsymbol{\theta})\), the score chain rule gives

\mathbf I_{\boldsymbol{\theta}} =\mathbf J^\mathsf{T}\mathbf I_{\boldsymbol{\phi}}\mathbf J, \qquad \mathbf J=\frac{\partial\boldsymbol{\phi}}{\partial\boldsymbol{\theta}^\mathsf{T}},

provided the transformation is regular. Here \(\boldsymbol{\phi}=(\mu,\beta)^\mathsf{T}\), \(\boldsymbol{\theta}=(\alpha,\beta)^\mathsf{T}\), and

\mathbf J= \begin{pmatrix}1&1\\0&1\end{pmatrix}.

Multiplying \(\mathbf J^\mathsf{T}\mathbf I_{(\mu,\beta)}\mathbf J\) gives the earlier matrix with cross entry \(1/\beta\). This is why an off-diagonal entry must be interpreted in the stated parameterization, rather than treated as an immutable property of the probability distribution.

Worked Example: Comparing Two Parameterizations

Method: Compute information in mean-variance coordinates and transform it using the Jacobian. For a hypothetical observation with \(X\sim N(\alpha+\beta,\beta)\), take \(\alpha=3\) and \(\beta=1\), so the mean is \(\mu=4\) and the variance is \(1\). Compare the two matrices. The common mistake avoided is claiming that the cross term must be zero in every coordinate system because it is zero for the mean and variance coordinates.

In \((\mu,\beta)\) coordinates, the information is

\mathbf I_{(\mu,\beta)} = \begin{pmatrix} 1&0\\ 0&\frac12 \end{pmatrix}.

Transforming to \((\alpha,\beta)\), use \(\mathbf J=\begin{pmatrix}1&1\\0&1\end{pmatrix}\):

\begin{aligned} \mathbf I_{(\alpha,\beta)} &=\mathbf J^\mathsf{T}\mathbf I_{(\mu,\beta)}\mathbf J\\ &= \begin{pmatrix}1&0\\1&1\end{pmatrix} \begin{pmatrix}1&0\\0&\frac12\end{pmatrix} \begin{pmatrix}1&1\\0&1\end{pmatrix}\\ &= \begin{pmatrix}1&1\\1&\frac32\end{pmatrix}. \end{aligned}

As a check, substituting \(\beta=1\) in the directly derived formula gives the same matrix. Its score correlation is

\frac{I_{\alpha\beta}}{\sqrt{I_{\alpha\alpha}I_{\beta\beta}}} =\frac{1}{\sqrt{1\cdot(3/2)}} =\sqrt{\frac23}\approx 0.8165.

The positive correlation describes the score components in \((\alpha,\beta)\) coordinates. It is not a claim that the parameters themselves are random or dependent. The comparison is checked by both methods giving the same transformed information matrix.

Common Exam Traps

Dropping the cross entry: A matrix is not diagonal unless the model or parameterization makes the score components uncorrelated. Calculate \(I_{12}\) rather than assuming it is zero.
Mixing up expected and observed curvature: Fisher information uses the expectation of the negative log-likelihood Hessian. A Hessian evaluated at a particular data value is not automatically the Fisher information.
Using the wrong variance convention: In \(N(\mu,\sigma^2)\), the second argument is the variance. Check whether a problem provides variance or standard deviation before substitution.
Reading a nonzero cross entry as parameter dependence: Parameters in a fixed-parameter likelihood are not random. The entry describes the relationship between score components in the chosen coordinates.
Interpreting diagonal entries in isolation: The determinant depends on all three distinct entries, \(I_{11},I_{22},I_{12}\); two positive diagonal entries do not by themselves guarantee positive definiteness.

Key takeaway: Build the Fisher information matrix from the full score vector or the expected log-likelihood Hessian. The diagonal entries measure direction-specific information, while the off-diagonal entries record score coupling. Both are needed to describe the model locally in its chosen parameterization.

Check Your Understanding

Work each problem from the definition and check the displayed final answer.

For one observation from \(X\sim N(\alpha+\beta,\beta)\), calculate the information matrix at \(\beta=3\). What is its determinant? (Answer: \(\begin{pmatrix}1/3&1/3\\1/3&7/18\end{pmatrix}\), determinant \(1/54\).)
For the same model at \(\beta=2\), calculate the correlation between the two score components. Give four decimal places. (Answer: \(0.8944\).)
Use coordinates \((\mu,\beta)\), where \(\mu=\alpha+\beta\), to calculate the information matrix at \(\beta=4\). (Answer: \(\begin{pmatrix}1/4&0\\0&1/32\end{pmatrix}\).)
State the regularity-based identity connecting the score-product definition of Fisher information and the expected Hessian definition. (Answer: \(\mathbf I=E[\mathbf U\mathbf U^\mathsf{T}]=-E[\nabla^2\ell]\), when the regularity conditions permit differentiation under the integral and the expectations exist.)
A proposed symmetric information matrix is \(\begin{pmatrix}4&-1\\-1&2\end{pmatrix}\). Is it positive definite, and what does the negative off-diagonal entry signify? (Answer: Yes; its determinant is \(8-1=7>0\) and its first diagonal entry is positive. The score components have negative covariance.)

What You'll Learn