Information for Exponential-Family Dispersion

Why Parameterization Matters

In a model with both a mean and a dispersion parameter, Fisher information describes how sensitive the distribution is to changes in each parameter. It also records whether the information for one parameter is coupled to the other. The calculations are specific to the distribution: in particular, gamma dispersion does not have the same information formula as normal variance.

This tutorial calculates information for a single observation in two models. For a parameter vector \(\theta\), the expected information matrix is \(I(\theta)=E[(\nabla_\theta\ell)(\nabla_\theta\ell)^\mathsf{T}]\), when the score has mean zero and the expectation exists. Under the usual regularity conditions, it also equals the negative expected Hessian. We will display both diagonal contributions and the mean–dispersion cross-information.

The score-variance and expected-information identities used here require a differentiable density on a parameter-independent support, integrable first and second derivatives that permit differentiation under the integral, and finite score second moments. For the normal and gamma densities below, these conditions hold when the mean and dispersion parameters are strictly positive (and, for the normal mean, the mean may be any real number). Gamma expectations involving logarithms are finite for every positive shape. We compute each information matrix directly rather than relying on a general rule for transforming information.

Normal Mean and Dispersion

Let \(Y\sim N(\mu,\phi)\), where the second argument is the variance, \(\mu\in\mathbb R\), and \(\phi>0\). The log-density, up to the displayed normalizing terms, is

Differentiating with respect to the stated parameters gives

The centered normal moments are \(E[Y-\mu]=0\), \(E[(Y-\mu)^2]=\phi\), \(E[(Y-\mu)^3]=0\), and \(E[(Y-\mu)^4]=3\phi^2\). Thus both score components have mean zero. Their variances and covariance give the information matrix:

Gamma Mean and Dispersion

For the gamma model, use the convention \(E[Y]=\mu\) and \(\operatorname{Var}(Y)=\phi\mu^2\), with \(\mu>0\) and \(\phi>0\). Equivalently, \(Y\) has shape \(t=1/\phi\) and scale \(\phi\mu\). Its density is

Writing \(Z=Y/\mu\), the variable \(Z\) has a gamma distribution with shape \(t=1/\phi\) and scale \(\phi\); in particular, \(E[Z]=1\) and \(\operatorname{Var}(Z)=\phi\). Let \(\psi(t)=\frac{d}{dt}\log\Gamma(t)\) be the digamma function and \(\psi_1(t)=\frac{d}{dt}\psi(t)\) the trigamma function. Differentiating the log-density yields

To calculate their variances, the gamma integral gives \(E[\log Z]=\psi(t)+\log\phi\) and \(\operatorname{Var}(\log Z)=\psi_1(t)\). These identities follow by differentiating the gamma normalizing integral with respect to its shape; differentiation is valid here because the integrals of the density times \(|\log z|\) and \((\log z)^2\) are finite for \(t>0\). Also,

For the last equality, the gamma-function identity \(\psi(t+1)-\psi(t)=1/t\) applies for \(t>0\). The score \(U_\mu\) has variance \(\operatorname{Var}(Z)/(\phi^2\mu^2)=1/(\phi\mu^2)\). The bracket in \(U_\phi\) has mean zero, and its variance is

Finally, the cross-information is proportional to \(\operatorname{Cov}(Z-1,-\log Z+Z)\). This covariance is \(-\phi+\phi=0\), using the two moment identities above. These calculations prove the gamma result:

What Changes Under a Different Dispersion Parameter?

The gamma model is also commonly written using its shape parameter \(\kappa=1/\phi\). In this parameterization its density is

so that \(\operatorname{Var}(Y)=\mu^2/\kappa\). Direct differentiation gives

The variance of the first score is \(\kappa/\mu^2\). For the second, differentiating \(U_\kappa\) with respect to \(\kappa\) gives \(1/\kappa-\psi_1(\kappa)\); its expectation is already constant in \(Y\). Thus the negative expected second derivative is \(\psi_1(\kappa)-1/\kappa\). As in the dispersion parameterization, the cross-information is zero. The different formula for the shape entry is not a different distribution: it reflects a different parameter and is obtained directly here from the shape-parameter score.

Parameterization of Normal Dispersion

A related distinction occurs for a normal observation if the model is parameterized by standard deviation \(\sigma>0\), rather than variance \(\phi=\sigma^2\). From \(Y\sim N(\mu,\sigma^2)\), direct differentiation gives \(U_\sigma=-1/\sigma+(Y-\mu)^2/\sigma^3\). Since \(\operatorname{Var}((Y-\mu)^2)=2\sigma^4\), the information for \(\sigma\) is \(I_{\sigma\sigma}=2/\sigma^2\), not \(1/(2\sigma^4)\), which is the information for the variance parameter. The mean information is \(1/\sigma^2\) in either description of the spread, and the cross-entry is zero. Always identify the parameter before interpreting an information entry.

Common Mistake: Reusing the Normal Dispersion Formula

A tempting but incorrect shortcut is to assign \(1/(2\phi^2)\) as the dispersion information in every model. That formula was derived from the squared centered residual of a normal observation; it is not a general formula for a dispersion parameter. For the gamma model with \(\operatorname{Var}(Y)=\phi\mu^2\), the dispersion information involves the trigamma function and is \(\{\psi_1(1/\phi)-\phi\}/\phi^4\). For example, at \(\phi=1/2\), the normal formula would give \(2\), whereas the gamma calculation gives approximately \(2.318945\). The distributions and their score functions differ, so the expected squared scores differ too.

A second error is to label information for a reciprocal parameter as though it belonged to the original parameter. The gamma shape \(\kappa\) and dispersion \(\phi\) are reciprocals, and the entries \(I_{\kappa\kappa}\) and \(I_{\phi\phi}\) are not numerically interchangeable. State the parameterization alongside every information matrix and use the score for that parameter.