Fisher Information in Natural Coordinates

Why Information Is a Covariance Matrix

In an exponential family, the natural parameter multiplies the sufficient statistic in the log likelihood. This makes the score a centered version of that statistic: it records how the observed statistic differs from its model expectation. Fisher information, which averages the outer product of the score with itself, is therefore governed by the covariance of the sufficient statistic. The important qualification is that a covariance matrix need not be positive definite: redundant components of the statistic can make it singular.

We work with a regular exponential family whose density, relative to a fixed measure \(\nu\), is

f_\eta(x)=\exp\{\eta^{\mathsf T}T(x)-A(\eta)\}, \qquad A(\eta)=\log\int \exp\{\eta^{\mathsf T}T(x)\}\,d\nu(x).

Here \(T(x)\) is a \(d\)-dimensional statistic, \(\eta\in\mathbb R^d\) is the natural parameter, and \(A\) is the log-partition function. The natural parameter space is the set of \(\eta\) for which the integral defining \(A(\eta)\) is finite. In this tutorial, \(\eta\) is an interior point of that space, which we assume is open. This condition permits the local differentiation under the integral used below.

Definition: For one observation, the score in natural coordinates is \(U_\eta(X)=\nabla_\eta\log f_\eta(X)\). The Fisher information matrix is \(I(\eta)=E_\eta[U_\eta(X)U_\eta(X)^{\mathsf T}]\), provided the expectation exists. For \(n\) independent observations from this family, the Fisher information for the full sample is \(nI(\eta)\).

Deriving the Moment and Information Identities

The log-partition function normalizes the density: \(\int \exp\{\eta^{\mathsf T}T(x)-A(\eta)\}\,d\nu(x)=1\). Differentiating this normalizing integral reveals the moments of \(T(X)\). The differentiations are valid at interior points. To see why, take a small closed box around such a point that lies inside the natural parameter space. For parameters in a smaller box, each first- or second-derivative integrand is bounded by a constant times a finite sum of integrable exponential terms at points in the larger box. Polynomial factors in the components of \(T\) are absorbed by a slightly larger exponential bound. Dominated differentiation therefore applies.

Theorem: At every interior natural parameter \(\eta\) for which the family above is defined, \(\nabla A(\eta)=E_\eta[T(X)]\) and \(\nabla^2 A(\eta)=\operatorname{Cov}_\eta(T(X),T(X))\). Consequently, the score is \(U_\eta(X)=T(X)-E_\eta[T(X)]\), and the Fisher information is \(I(\eta)=\operatorname{Cov}_\eta(T(X),T(X))=\nabla^2 A(\eta)\).

Proof. Let \(Z(\eta)=\exp\{A(\eta)\}\), and write \(T_j\) for component \(j\) of \(T\). The differentiation justified above gives

\frac{\partial Z(\eta)}{\partial \eta_j} =\int T_j(x)\exp\{\eta^{\mathsf T}T(x)\}\,d\nu(x).

Dividing by \(Z(\eta)\) yields

\frac{\partial A(\eta)}{\partial \eta_j} =\frac{1}{Z(\eta)}\frac{\partial Z(\eta)}{\partial\eta_j} =\int T_j(x)f_\eta(x)\,d\nu(x) =E_\eta[T_j(X)].

For the second derivative, differentiate the ratio expression for \(\partial A/\partial\eta_j\). The numerator derivative contributes \(E_\eta[T_j(X)T_k(X)]\); differentiating the denominator contributes the subtraction of \(E_\eta[T_j(X)]E_\eta[T_k(X)]\). Thus

\frac{\partial^2 A(\eta)}{\partial\eta_j\partial\eta_k} =E_\eta[T_j(X)T_k(X)] -E_\eta[T_j(X)]E_\eta[T_k(X)] =\operatorname{Cov}_\eta(T_j(X),T_k(X)).

Also, \(\log f_\eta(X)=\eta^{\mathsf T}T(X)-A(\eta)\), so differentiating with respect to \(\eta\) gives \(U_\eta(X)=T(X)-\nabla A(\eta)=T(X)-E_\eta[T(X)]\). Taking the expected outer product of this centered statistic gives its covariance matrix, proving the information identity. For \(n\) independent observations, the sample score is the sum of the \(n\) individual scores. Each has mean zero, so cross-covariances between distinct observations vanish, and the information is \(nI(\eta)\). \(\square\)

When Is the Information Positive Definite?

A covariance matrix is always positive semidefinite: for any fixed vector \(a\in\mathbb R^d\), its quadratic form is the variance of a scalar random variable.

a^{\mathsf T}I(\eta)a =\operatorname{Var}_\eta\{a^{\mathsf T}T(X)\} \geq 0.

It is positive definite only if every nonzero linear combination of the components of \(T(X)\) has positive variance. A linear combination has zero variance exactly when it is constant with probability one.

Theorem: At an interior natural parameter \(\eta\), \(I(\eta)\) is positive definite if and only if there is no nonzero \(a\in\mathbb R^d\) for which \(a^{\mathsf T}T(X)\) is constant \(P_\eta\)-almost surely. If the family is minimal—meaning there is no nonzero \(a\) and constant \(c\) such that \(a^{\mathsf T}T(x)=c\) for \(\nu\)-almost every \(x\)—then \(I(\eta)\) is positive definite at every interior natural parameter.

Proof. For any \(a\), \(a^{\mathsf T}I(\eta)a=\operatorname{Var}_\eta\{a^{\mathsf T}T(X)\}\). This is zero exactly when \(a^{\mathsf T}T(X)\) is constant \(P_\eta\)-almost surely. Positive definiteness means that this quadratic form is strictly positive for every \(a\ne 0\), giving the first claim.

For the second claim, the density \(f_\eta(x)\) is strictly positive \(\nu\)-almost everywhere on the family’s support. Thus \(P_\eta\) and \(\nu\) have the same null sets there. A linear combination that is constant \(P_\eta\)-almost surely would consequently be constant \(\nu\)-almost everywhere, contrary to minimality. Hence no nonzero \(a\) gives zero variance, and \(I(\eta)\) is positive definite. \(\square\)

The equivalence of null sets matters: the criterion is not merely that the components of \(T\) look different in a formula. They must not satisfy an almost-sure affine relation under the model. Redundant statistics can represent a family correctly while making its information singular and its natural coordinates non-unique.

Worked Information Calculations

Worked Example: Bernoulli Information at a Specific Parameter

Let \(X\) be Bernoulli with success probability \(p\). In natural coordinates, \(\eta=\log\{p/(1-p)\}\), \(T(X)=X\), and \(A(\eta)=\log(1+e^\eta)\). Find the information at \(\eta=\log 3\).

At this parameter, \(e^\eta=3\), so \(p=3/(1+3)=3/4\). Differentiating the log-partition function gives

A'(\eta)=\frac{e^\eta}{1+e^\eta}=p, \qquad A''(\eta)=\frac{e^\eta}{(1+e^\eta)^2}=p(1-p).

Therefore \(I(\eta)=\operatorname{Var}(X)=p(1-p)=(3/4)(1/4)=3/16\). This is positive for every finite \(\eta\), since then \(0<p<1\). At the limiting parameter values where \(p\) approaches 0 or 1, the variance approaches zero; those boundary limits do not contradict positive definiteness at finite natural parameters.

Worked Example: Redundant Poisson Statistics Give Singular Information

Let \(X\sim\operatorname{Poisson}(\lambda)\), with \(\lambda>0\), but use the two-component statistic \(T(X)=(X,2X)^{\mathsf T}\) and natural parameter \(\eta=(\eta_1,\eta_2)^{\mathsf T}\). Write \(q=\eta_1+2\eta_2\). The density has the exponential-family form with \(A(\eta)=e^q\), which is finite for every \(\eta\), and \(\lambda=e^q\).

Since \(\operatorname{Var}(X)=\lambda\), the covariance of \(T(X)\) is

I(\eta) =\lambda \begin{pmatrix} 1&2\\ 2&4 \end{pmatrix}.

For example, at \(\eta=(0,0)^{\mathsf T}\), \(\lambda=1\), so the matrix is \(\begin{pmatrix}1&2\\2&4\end{pmatrix}\). Its determinant is \(4-4=0\). More directly, for \(a=(-2,1)^{\mathsf T}\), \(a^{\mathsf T}T(X)=-2X+2X=0\) for every \(X\), so \(a^{\mathsf T}I(\eta)a=0\). The representation is not minimal: the statistic’s second component is twice its first. The singularity reflects that redundancy, not a failure of the covariance identity.

Worked Example: A Two-Parameter Categorical Family with Positive Definite Information

Let \(Y\) take values \(0,1,2\), with natural statistic \(T(Y)=(\mathbf 1\{Y=1\},\mathbf 1\{Y=2\})^{\mathsf T}\). Its log-partition function is \(A(\eta)=\log(1+e^{\eta_1}+e^{\eta_2})\). At \(\eta=(0,0)^{\mathsf T}\), each of the three outcomes has probability \(1/3\).

Each statistic component is an indicator with mean \(1/3\) and variance \((1/3)(2/3)=2/9\). The two indicators cannot both equal 1, so their product is always 0 and their covariance is \(0-(1/3)(1/3)=-1/9\). Hence

I(0,0) =\begin{pmatrix}2/9&-1/9\\-1/9&2/9\end{pmatrix}, \qquad \det I(0,0)=\frac{4-1}{81}=\frac{1}{27}>0.

For any nonzero \(a=(a_1,a_2)^{\mathsf T}\), the quadratic form is \((a_1^2+a_2^2-a_1a_2)/9\), which is positive: its numerator equals \(\{(a_1-a_2)^2+a_1^2+a_2^2\}/2\). Thus the information is positive definite. The two statistic components have no nonzero linear combination that is constant across all three outcomes.

Sample Information and a Common Pitfall

For independent observations \(X_1,\ldots,X_n\), the log likelihood in natural coordinates is

\ell(\eta) =\eta^{\mathsf T}\sum_{i=1}^n T(X_i)-nA(\eta) +\sum_{i=1}^n\log h(X_i),

where the final term is absent if the reference measure already incorporates the carrier factor. The sample score is \(\sum_{i=1}^n T(X_i)-nE_\eta[T(X)]\), and its covariance is \(n\operatorname{Cov}_\eta(T(X))\). Thus the per-observation and full-sample information differ by the factor \(n\); neither is automatically nonsingular.

Common mistake: “The statistic is sufficient and has a covariance matrix, so Fisher information must be positive definite.” Sufficiency does not rule out redundant components. In the Poisson example, \(T(X)=(X,2X)\) is a sufficient statistic for the represented family, but its covariance matrix is singular. The correction is to check whether any nonzero linear combination of the statistic is constant almost surely. Positive definiteness requires that none is; sufficiency alone does not.

Key takeaway: In regular natural coordinates, Fisher information is the covariance matrix of the sufficient statistic. It is always positive semidefinite, and it is positive definite exactly when the statistic has no nontrivial almost-sure constant linear combination.

Check Your Understanding

Use the covariance identity and the positive-definiteness criterion to answer the following.

In a natural exponential family, what is the score in terms of \(T(X)\) and \(E_\eta[T(X)]\)?
Why is the Hessian of the log-partition function positive semidefinite?
For the statistic \(T(X)=(X,2X)^{\mathsf T}\), give a nonzero vector that demonstrates singular information.
What condition on \(a^{\mathsf T}T(X)\), for every nonzero \(a\), ensures positive definite information?
For \(n\) independent observations, how does the information compare with the information for one observation?