Deriving Fisher Information from the Log-Partition Function

Why the Log-Partition Function Determines Information

In a canonical exponential family, the parameter enters the log density through a linear term involving a statistic of the data and a normalizing term that ensures the density integrates to one. Differentiating the log density gives the score. The same normalization term then determines the score’s variability, which is the Fisher information. This connection turns a potentially difficult expectation into derivatives of one function.

The key result is specific to the canonical parameter: the score is the sufficient statistic centered at its expectation, and the Fisher information is its covariance matrix. We derive these identities under explicit regularity conditions, then calculate them for three familiar models.

Canonical Exponential-Family Setup

Definition: Let $X$ take values in a sample space with a fixed dominating measure $\mu$. A $p$-dimensional regular canonical exponential family has density $$ f_\eta(x)=h(x)\exp\!\left\{\eta^{\mathsf T}T(x)-A(\eta)\right\},\qquad \eta\in\Theta, $$ where $T(x)\in\mathbb{R}^p$ is the sufficient statistic, $h(x)\geq 0$ is the carrier density, $\Theta$ is an open subset of $\mathbb{R}^p$, and $$ A(\eta)=\log\int h(x)\exp\!\left\{\eta^{\mathsf T}T(x)\right\}\,d\mu(x) $$ is the log-partition function. The integral defining $A(\eta)$ is required to be finite for every $\eta\in\Theta$.

For the identities below, assume the support does not depend on $\eta$, and that differentiation under the integral defining the normalizing constant is valid through second order in a neighborhood of each $\eta\in\Theta$. In particular, the first and second derivatives of that integral are the integrals of the corresponding derivatives of its integrand. These conditions hold, for example, at interior points of a natural parameter space when the local exponential integrability needed for those derivatives is satisfied. They make the differentiation steps below legitimate rather than merely formal.

The score is the column vector of derivatives of the log density with respect to $\eta$. The Fisher information for one observation is the expected outer product of that score with itself, when the expectation exists. Under the conditions just stated, the score has finite second moments.

Deriving the Score and Information

Take logarithms of the density. Since $h(x)$ does not depend on $\eta$, differentiating gives

\nabla_\eta\log f_\eta(x)=T(x)-\nabla A(\eta).

Thus, once we know the gradient of $A$, the score follows. To find that gradient, write the normalizing constant as $Z(\eta)=\exp\{A(\eta)\}$. Differentiation under the integral gives

\frac{\partial Z(\eta)}{\partial\eta_a} =\int T_a(x)h(x)\exp\!\left\{\eta^{\mathsf T}T(x)\right\}\,d\mu(x).

Dividing by $Z(\eta)$ turns the right-hand side into an expectation under $f_\eta$. Since $\partial A/\partial\eta_a=(1/Z)\partial Z/\partial\eta_a$, this proves $\nabla A(\eta)=E_\eta[T(X)]$. Differentiating once more gives the covariance identity.

Theorem: Under the stated regularity conditions, for one observation from a canonical exponential family, $$ \nabla A(\eta)=E_\eta[T(X)],\qquad \nabla^2 A(\eta)=\operatorname{Cov}_\eta(T(X)). $$ The score and Fisher information are therefore $$ U_\eta(X)=T(X)-E_\eta[T(X)],\qquad \mathcal I(\eta)=E_\eta[U_\eta(X)U_\eta(X)^{\mathsf T}] =\nabla^2 A(\eta)=\operatorname{Cov}_\eta(T(X)). $$ Also, the observed Hessian of the log density is $-\nabla^2 A(\eta)$, so the negative expected Hessian equals $\mathcal I(\eta)$.

Proof. Set $Z(\eta)=\exp\{A(\eta)\}$. For components $a,b\in\{1,\ldots,p\}$, differentiation under the integral yields

\frac{\partial Z}{\partial\eta_a} =\int T_a(x)h(x)e^{\eta^{\mathsf T}T(x)}\,d\mu(x), \qquad \frac{\partial^2 Z}{\partial\eta_a\partial\eta_b} =\int T_a(x)T_b(x)h(x)e^{\eta^{\mathsf T}T(x)}\,d\mu(x).

Dividing the first equation by $Z(\eta)$ proves that $\partial A/\partial\eta_a=E_\eta[T_a(X)]$. For the second derivative, use the derivative of a logarithm:

\frac{\partial^2 A}{\partial\eta_a\partial\eta_b} =\frac{1}{Z}\frac{\partial^2 Z}{\partial\eta_a\partial\eta_b} -\frac{1}{Z^2}\frac{\partial Z}{\partial\eta_a}\frac{\partial Z}{\partial\eta_b} =E_\eta[T_a(X)T_b(X)]-E_\eta[T_a(X)]E_\eta[T_b(X)].

The last expression is $\operatorname{Cov}_\eta(T_a(X),T_b(X))$, proving the Hessian identity. The log-density derivative is $U_\eta(X)=T(X)-\nabla A(\eta)$; substituting $\nabla A(\eta)=E_\eta[T(X)]$ centers the statistic. Its expected outer product is its covariance matrix, hence is $\nabla^2 A(\eta)$. Finally, differentiating the log density a second time gives $\nabla_\eta^2\log f_\eta(x)=-\nabla^2 A(\eta)$, which does not depend on $x$. Taking its negative expectation gives the same matrix. $\square$

This theorem also shows that $A$ is convex: for any $v\in\mathbb{R}^p$,

v^{\mathsf T}\nabla^2 A(\eta)v =\operatorname{Var}_\eta\!\left(v^{\mathsf T}T(X)\right)\geq 0.

The information is positive definite exactly when no nonzero linear combination $v^{\mathsf T}T(X)$ is constant almost surely under $P_\eta$. If such a combination is constant, its variance is zero and the information has a zero direction. This is a property of the statistic’s variation under the model, not just of the algebraic appearance of the Hessian.

Worked Examples

Worked Example: Bernoulli Information in Logit Coordinates

Let $X\in\{0,1\}$ be Bernoulli with success probability $\pi$, where $0<\pi<1$. Set $\eta=\log\{\pi/(1-\pi)\}$, so $\pi=e^\eta/(1+e^\eta)$. The mass function can be written $$ P_\eta(X=x)=\exp\!\left\{\eta x-\log(1+e^\eta)\right\},\qquad x\in\{0,1\}. $$ Thus $T(x)=x$ and $A(\eta)=\log(1+e^\eta)$.

Differentiate the log-partition function: $$ A'(\eta)=\frac{e^\eta}{1+e^\eta}=\pi,\qquad A''(\eta)=\frac{e^\eta}{(1+e^\eta)^2}=\pi(1-\pi). $$ The score for an observation $x$ is $x-\pi$. Its expectation is $E_\eta[X]-\pi=\pi-\pi=0$, and its information is $$ E_\eta[(X-\pi)^2]=\operatorname{Var}_\eta(X)=\pi(1-\pi)=A''(\eta). $$ In particular, the information is $1/4$ when $\pi=1/2$, and is positive for every allowed $\pi$.

Worked Example: Multinomial Category Probabilities

Suppose one observation is a category $X\in\{1,\ldots,K\}$, with probabilities $p_1,\ldots,p_K>0$ summing to one. Use category $K$ as the reference and define $\eta_j=\log(p_j/p_K)$ for $j=1,\ldots,K-1$. Then $$ p_j=\frac{e^{\eta_j}}{1+\sum_{\ell=1}^{K-1}e^{\eta_\ell}}\quad(j<K), \qquad p_K=\frac{1}{1+\sum_{\ell=1}^{K-1}e^{\eta_\ell}}. $$ For $T_j(X)=\mathbf{1}\{X=j\}$, the canonical representation has $A(\eta)=\log(1+\sum_{\ell=1}^{K-1}e^{\eta_\ell})$.

For each $j<K$, differentiation gives $\partial A/\partial\eta_j=p_j$. Differentiating again gives $$ \frac{\partial^2 A}{\partial\eta_j\partial\eta_k} =p_j\mathbf{1}\{j=k\}-p_jp_k. $$ This also follows directly from the indicators: $E[T_j]=p_j$, $E[T_j^2]=p_j$, and for $j\ne k$, $T_jT_k=0$, so $\operatorname{Cov}(T_j,T_k)=-p_jp_k$. The score vector is $(T_1-p_1,\ldots,T_{K-1}-p_{K-1})^{\mathsf T}$, and its covariance is the displayed Hessian. The result includes the off-diagonal dependence among the indicator statistics; treating their information as diagonal would be incorrect.

Worked Example: Multivariate Normal Mean with Known Covariance

Let $X\in\mathbb{R}^d$ have distribution $N(\mu,\Sigma)$, where $\Sigma$ is fixed and positive definite. Set $\eta=\Sigma^{-1}\mu$, so $\mu=\Sigma\eta$. Expanding the normal density gives $$ f_\eta(x)=h(x)\exp\!\left\{\eta^{\mathsf T}x-\frac{1}{2}\eta^{\mathsf T}\Sigma\eta\right\}, $$ where $h(x)=(2\pi)^{-d/2}|\Sigma|^{-1/2}\exp\{-x^{\mathsf T}\Sigma^{-1}x/2\}$. Thus $T(x)=x$ and $A(\eta)=\tfrac12\eta^{\mathsf T}\Sigma\eta$.

Because $\Sigma$ is symmetric, $\nabla A(\eta)=\Sigma\eta=\mu$ and $\nabla^2 A(\eta)=\Sigma$. The score is $x-\mu$, and its information is $$ E_\eta[(X-\mu)(X-\mu)^{\mathsf T}]=\operatorname{Cov}(X)=\Sigma. $$ This verifies the identity for a vector-valued statistic: the information includes the covariances between coordinates, not only their individual variances.

Independent Observations and Practical Interpretation

For $n$ independent observations $X_1,\ldots,X_n$ from the same canonical family, the log-likelihood is the sum of the individual log densities. The total score is therefore the sum of centered sufficient statistics, and its expected outer product is $$ \mathcal I_n(\eta)=n\mathcal I(\eta)=n\nabla^2 A(\eta). $$ Indeed, when the outer product of the sum is expanded, each cross term has expectation zero: independence gives $E[U_\eta(X_i)U_\eta(X_j)^{\mathsf T}]=E[U_\eta(X_i)]E[U_\eta(X_j)]^{\mathsf T}=0$ for $i\ne j$. The $n$ diagonal terms each contribute $\mathcal I(\eta)$.

The identity matters because it links three views of the same local information: curvature of the log-partition function, variability of the sufficient statistic, and variability of the score. For a canonical model, any of these can make the calculation easiest. It also gives a useful check on algebra: a proposed information matrix must be symmetric and positive semidefinite, since it is a covariance matrix.

Common Mistakes

Dropping the minus sign in the log-likelihood Hessian. The score is $T-\nabla A$, but its derivative is $-\nabla^2 A$. The information is the negative expected Hessian, so it is $+\nabla^2 A$.
Using the uncentered second moment as information. The score is centered, so its outer-product expectation is a covariance: $E[TT^{\mathsf T}]-E[T]E[T]^{\mathsf T}$, not just $E[TT^{\mathsf T}]$.
Ignoring the canonical-coordinate qualification. The formula $\mathcal I(\eta)=\nabla^2 A(\eta)$ is derived for the canonical parameter used in the density above. It should not be applied unchanged after replacing that parameter with a different one.
Assuming positive definiteness automatically. A covariance matrix can be singular. If a nonzero linear combination of the sufficient-statistic components is constant almost surely, the information has a zero eigenvalue.

Key takeaway: In a regular canonical exponential family, $A$ is the cumulant-generating normalizer: its gradient gives the mean of $T(X)$, and its Hessian gives both $\operatorname{Cov}(T(X))$ and the Fisher information for one observation.

Check Your Understanding

Use the canonical representation and the theorem to answer the following.

For a canonical family, express the score in terms of $T(X)$ and $E_\eta[T(X)]$.
Why does the Hessian of the log-partition function have to be positive semidefinite?
In the multinomial example, why are the off-diagonal information entries negative?
For $n$ independent observations, what is the total Fisher information, and why do cross terms in the score outer product vanish?
What property of $T(X)$ makes the Fisher information singular in a nonzero direction?