Log-Partition Hessians Are Covariance Matrices

Why the Hessian Contains Covariance Information

In an exponential family, the log-partition function \(A(\eta)\) does more than make the density integrate to one. Its first derivatives give the means of the canonical statistics; its second derivatives describe how those statistics vary together. In particular, the mixed derivative in coordinates \(i\) and \(j\) is the covariance between \(T_i(X)\) and \(T_j(X)\).

This connection is useful both for calculations and for understanding the family’s local behavior. Once \(A\) is known, one can obtain an entire covariance matrix by differentiation, rather than computing each expectation and centered product separately. The essential condition is that the natural parameter lies in the interior of the set where the normalizing integral is finite.

Setup and Regularity

Definition: Consider a \(d\)-parameter exponential family with density, relative to a fixed carrier measure \(\nu\), \(p_\eta(x)=\exp\{\eta^\mathsf{T}T(x)-A(\eta)\}\), where \(T=(T_1,\ldots,T_d)^\mathsf{T}\) is measurable and \(A(\eta)=\log K(\eta)\), with \(K(\eta)=\int \exp\{\eta^\mathsf{T}T(x)\}\,\nu(dx)\). The natural parameter space is the set of \(\eta\in\mathbb{R}^d\) for which \(K(\eta)<\infty\). The covariance identity below applies at every point in the interior of this space.

The interior condition is what justifies differentiating under the integral. To see why, fix an interior point \(\eta\). There is an \(r>0\) such that the cube of radius \(2r\) around \(\eta\) lies in the natural parameter space. For parameters \(\eta'\) in the smaller cube of radius \(r\), the integrands needed for first and second derivatives are dominated by an integrable function. Indeed, polynomial factors such as \(|T_iT_j|\) can be bounded by a constant times \(\exp(r\sum_k|T_k|)\). The resulting exponential is bounded by a finite sum of exponentials with parameter vectors at the corners of the larger cube. Each such exponential has a finite integral by the choice of the cube. Dominated convergence therefore justifies the differentiations locally.

This argument also ensures that the relevant first and second moments exist at \(\eta\). No claim is being made here about boundary points of the natural parameter space: there the required derivatives or moments may fail to exist.

The Covariance-Hessian Identity

Theorem: At every interior natural parameter \(\eta\), \(A\) is twice differentiable and \(\nabla A(\eta)=E_\eta[T(X)]\), while \(\nabla^2 A(\eta)=\operatorname{Cov}_\eta(T(X))\). In coordinates, \(\frac{\partial^2 A}{\partial\eta_i\partial\eta_j}=\operatorname{Cov}_\eta(T_i(X),T_j(X))\).

Here the covariance matrix has \((i,j)\) entry \(E_\eta[(T_i-E_\eta[T_i])(T_j-E_\eta[T_j])]\). The identities follow directly by differentiating the normalizing integral and then accounting for the derivative of its logarithm.

Proof. By the regularity argument above, differentiation under the integral gives \(\partial K/\partial\eta_i=\int T_i(x)\exp\{\eta^\mathsf{T}T(x)\}\,\nu(dx)\), and \(\partial^2 K/\partial\eta_i\partial\eta_j=\int T_i(x)T_j(x)\exp\{\eta^\mathsf{T}T(x)\}\,\nu(dx)\). Dividing by \(K(\eta)\), these are respectively \(E_\eta[T_i]\) and \(E_\eta[T_iT_j]\). Since \(A=\log K\), the first derivative is \(\partial A/\partial\eta_i=(\partial K/\partial\eta_i)/K=E_\eta[T_i]\). Differentiating this log expression a second time yields

\frac{\partial^2 A}{\partial\eta_i\partial\eta_j} = \frac{1}{K}\frac{\partial^2 K}{\partial\eta_i\partial\eta_j} - \frac{1}{K^2}\frac{\partial K}{\partial\eta_i}\frac{\partial K}{\partial\eta_j} = E_\eta[T_iT_j]-E_\eta[T_i]E_\eta[T_j].

The final expression is the covariance by definition, proving the coordinate identity and hence the matrix identity. \(\square\)

A second consequence describes every direction in parameter space. For a fixed vector \(a\in\mathbb{R}^d\), let \(g(t)=A(\eta+ta)\) for \(t\) close enough to zero that the parameter remains in the interior. The chain rule and the theorem give

g''(0)=a^\mathsf{T}\nabla^2A(\eta)a =\operatorname{Var}_\eta\!\left(a^\mathsf{T}T(X)\right)\geq 0.

Thus the Hessian is positive semidefinite: its quadratic form cannot be negative. This conclusion does not imply that it is positive definite. If \(a^\mathsf{T}T(X)\) is constant with probability one, its variance is zero and the Hessian has zero curvature in direction \(a\).

Worked Covariance Calculations

Worked Example: A Three-Outcome Categorical Family

Let \(X\) take values \(0,1,2\), with counting measure as carrier measure, and choose \(T(0)=(0,0)^\mathsf{T}\), \(T(1)=(1,0)^\mathsf{T}\), and \(T(2)=(0,1)^\mathsf{T}\). The normalizing integral is the sum of the three unnormalized weights:

K(\eta)=1+e^{\eta_1}+e^{\eta_2}, \qquad A(\eta)=\log(1+e^{\eta_1}+e^{\eta_2}).

At \(\eta_1=\log 2\) and \(\eta_2=\log 3\), the weights for \(X=0,1,2\) are \(1,2,3\). Their sum is \(6\), so the probabilities are \(1/6,2/6,3/6\). Write \(p_1=P_\eta(X=1)=1/3\) and \(p_2=P_\eta(X=2)=1/2\). Differentiating \(A\), or applying the theorem, gives

\nabla^2 A(\eta)= \begin{pmatrix} p_1(1-p_1) & -p_1p_2\\ -p_1p_2 & p_2(1-p_2) \end{pmatrix} = \begin{pmatrix} 2/9 & -1/6\\ -1/6 & 1/4 \end{pmatrix}.

The statistics are \(T_1(X)=\mathbf{1}\{X=1\}\) and \(T_2(X)=\mathbf{1}\{X=2\}\). They cannot both equal one, so their product is always zero. Consequently \(E[T_1T_2]=0\) and \(\operatorname{Cov}(T_1,T_2)=0-(1/3)(1/2)=-1/6\), matching the off-diagonal entry. The diagonal entries are the Bernoulli variances \( (1/3)(2/3)=2/9\) and \( (1/2)(1/2)=1/4\).

Worked Example: Two Independent Poisson Variables

Let \(X=(X_1,X_2)\) have independent Poisson coordinates with means \(\lambda_1,\lambda_2>0\). Use counting measure on pairs of nonnegative integers and carrier factor \(1/(x_1!x_2!)\). With \(T(x)=(x_1,x_2)^\mathsf{T}\), set \(\eta_i=\log\lambda_i\). The normalizing sum factors:

K(\eta) =\sum_{x_1,x_2\geq 0} \frac{e^{\eta_1x_1+\eta_2x_2}}{x_1!x_2!} =\exp(e^{\eta_1})\exp(e^{\eta_2}), \qquad A(\eta)=e^{\eta_1}+e^{\eta_2}.

Differentiating twice gives

\nabla^2 A(\eta)= \begin{pmatrix} e^{\eta_1}&0\\ 0&e^{\eta_2} \end{pmatrix} = \begin{pmatrix} \lambda_1&0\\ 0&\lambda_2 \end{pmatrix}.

Therefore \(\operatorname{Var}(X_1)=\lambda_1\), \(\operatorname{Var}(X_2)=\lambda_2\), and \(\operatorname{Cov}(X_1,X_2)=0\). For instance, at \((\eta_1,\eta_2)=(\log 2,\log 5)\), the covariance matrix is \(\begin{pmatrix}2&0\\0&5\end{pmatrix}\). The zero covariance also agrees with the stipulated independence.

Worked Example: The First Two Powers of a Normal Observation

Consider the family on the real line with carrier measure equal to Lebesgue measure and \(T(x)=(x,x^2)^\mathsf{T}\). Its natural parameter region is \(\eta_2<0\). Completing the square in the normalizing integral gives

K(\eta) =\int_{\mathbb{R}}e^{\eta_1x+\eta_2x^2}\,dx = \sqrt{\frac{\pi}{-\eta_2}}\, \exp\!\left(-\frac{\eta_1^2}{4\eta_2}\right), \qquad A(\eta)=-\frac{\eta_1^2}{4\eta_2} +\frac12\log\!\left(\frac{\pi}{-\eta_2}\right).

The Hessian, obtained by differentiating this expression, is

\nabla^2 A(\eta)= \begin{pmatrix} -\dfrac{1}{2\eta_2} & \dfrac{\eta_1}{2\eta_2^2}\\[6pt] \dfrac{\eta_1}{2\eta_2^2} & \dfrac{1}{2\eta_2^2}-\dfrac{\eta_1^2}{2\eta_2^3} \end{pmatrix}.

At \((\eta_1,\eta_2)=(0,-1/2)\), the density is \(e^{-x^2/2}/\sqrt{2\pi}\), the \(N(0,1)\) density. Substitution gives \(\nabla^2 A=\begin{pmatrix}1&0\\0&2\end{pmatrix}\). Thus \(\operatorname{Var}(X)=1\), \(\operatorname{Cov}(X,X^2)=0\), and \(\operatorname{Var}(X^2)=2\). In particular, the last result is the variance of the canonical statistic \(X^2\), not the variance of \(X\).

Interpreting the Identity and Avoiding a Common Error

The examples show how the same derivative rule handles different supports and carrier measures. The carrier measure affects the normalizer \(A\), but once the family is in canonical form, the Hessian entries correspond to covariances of the components of \(T\). For any linear combination \(a^\mathsf{T}T\), its variance is the Hessian’s quadratic form in direction \(a\); this is also a practical check on a computed matrix.

A common mistake is to identify the second derivative with \(E_\eta[T_iT_j]\) and stop there. That is only the first term in the derivative of \(\log K\). The second term, \(E_\eta[T_i]E_\eta[T_j]\), comes from differentiating the denominator \(K(\eta)\). Omitting it gives an uncentered second moment, not a covariance. Another error is to infer that every covariance matrix here must be positive definite. The correct general conclusion is positive semidefiniteness; a nonzero linear combination of the statistics can still be constant with probability one.

Key takeaway: At an interior natural parameter, \(\nabla^2A(\eta)=\operatorname{Cov}_\eta(T(X))\). Its diagonal entries are variances, its off-diagonal entries are covariances, and its quadratic form in direction \(a\) is \(\operatorname{Var}_\eta(a^\mathsf{T}T(X))\).

Check Your Understanding

Use the covariance-Hessian identity and the conditions stated above to answer these questions.

Why is interiority of the natural parameter important for the differentiation argument?
For the three-outcome example, why must the covariance between the two indicator statistics be negative?
If a Hessian calculation gives a negative value for \(a^\mathsf{T}\nabla^2A(\eta)a\), what does that indicate about the calculation or assumptions?
In the Poisson example, identify which Hessian entries give the two variances and which gives the covariance.
Can a nonzero vector \(a\) satisfy \(a^\mathsf{T}\nabla^2A(\eta)a=0\)? State the probabilistic condition for this to happen.