Higher Log-Partition Derivatives Give Cumulants

Introduction

The gradient and Hessian of a log-partition function describe the means and covariances of canonical statistics. Higher derivatives continue this pattern: they encode the higher-order cumulants, which describe features such as asymmetry and departures from Gaussian fourth-moment behavior. This connection is useful only when the relevant derivatives exist, so the natural parameter must be in an appropriate part of the parameter space.

The key idea is local. Tilting a canonical distribution by a small vector changes its natural parameter by that vector. Consequently, the cumulant-generating function of the canonical statistics is a difference of log-partition values. Differentiating this identity gives all orders of joint cumulants at once.

Canonical Statistics and Cumulants

For a random vector \(T\) with a moment-generating function finite near zero, its cumulant-generating function is \(K(t)=\log E[\exp\{t^{\mathsf T}T\}]\). The joint cumulant of \(T_{i_1},\ldots,T_{i_r}\) is the mixed derivative \(\partial_{i_1}\cdots\partial_{i_r}K(t)\) evaluated at \(t=0\), where repeated indices mean repeated differentiation with respect to the same coordinate. In particular, the first cumulants are means, the second are covariances, and higher orders describe additional joint distributional features.

For \(\eta\in\Omega\), the moment-generating function of \(T\) under \(P_\eta\) is finite for all \(t\) sufficiently close to zero. Indeed, \(\eta+t\) then remains in \(\Omega\), and direct integration gives \(E_\eta[\exp\{t^{\mathsf T}T\}]=\exp\{A(\eta+t)-A(\eta)\}\). The next result makes the derivative-cumulant connection precise.

Proof. First, differentiation of the integral defining \(A\) is justified locally. Choose \(\delta>0\) sufficiently small that every point \(\eta+\delta s\), for \(s\in\{-1,1\}^d\), lies in \(\Omega\). For parameters \(u\) in a sufficiently small neighborhood of \(\eta\), any fixed derivative of \(\exp\{u^{\mathsf T}T(x)\}\) is bounded in absolute value by a constant times \(\exp\{\eta^{\mathsf T}T(x)+\delta\sum_j|T_j(x)|\}\). This last expression is bounded by the sum of the \(2^d\) integrable functions \(\exp\{(\eta+\delta s)^{\mathsf T}T(x)\}\). Dominated convergence therefore permits differentiation under the integral for each finite order. Since the integral is positive, its logarithm \(A\) has the same local differentiability.

Next, for \(t\) near zero, normalization yields \[ E_\eta[\exp\{t^{\mathsf T}T\}] =\exp\{-A(\eta)\}\int \exp\{(\eta+t)^{\mathsf T}T(x)\}\,d\nu(x) =\exp\{A(\eta+t)-A(\eta)\}. \] Taking logarithms gives \(K_\eta(t)=A(\eta+t)-A(\eta)\). Differentiating \(r\) times with respect to the indicated coordinates and evaluating at \(t=0\) gives the claimed identity, by the definition of a joint cumulant. \(\square\)

Third- and Fourth-Order Cumulants

For one statistic \(T\), write \(\mu=E[T]\), \(\sigma^2=\operatorname{Var}(T)\), and let \(A^{(r)}(\eta)\) denote the \(r\)th derivative. The theorem says that \(A^{(r)}(\eta)\) is the \(r\)th cumulant. The third cumulant is the third central moment, but the fourth cumulant is not the fourth central moment.

To see the third- and fourth-order formulas, center the statistic by writing \(Y=T-\mu\) and \(M(t)=E[\exp(tY)]\). Then \(M(0)=1\) and \(M'(0)=0\). The cumulant-generating function of \(Y\) is \(\log M(t)\); centering changes only its first derivative, not its derivatives of order two or higher. Differentiating twice gives \((\log M)''=M''/M-(M'/M)^2\), so at zero the second derivative is \(E[Y^2]\). Differentiating once more and using \(M'(0)=0\) gives \((\log M)'''(0)=M'''(0)=E[Y^3]\). A fourth differentiation, again using \(M(0)=1\) and \(M'(0)=0\), gives \((\log M)''''(0)=M''''(0)-3M''(0)^2=E[Y^4]-3E[Y^2]^2\). This proves the formulas.

The multivariate fourth-order formula follows by the same differentiation, now with four possibly repeated coordinate indices. If \(Y_j=T_j-E[T_j]\), then \[ \kappa_{ijkl} =E[Y_iY_jY_kY_l] -\operatorname{Cov}(T_i,T_j)\operatorname{Cov}(T_k,T_l) -\operatorname{Cov}(T_i,T_k)\operatorname{Cov}(T_j,T_l) -\operatorname{Cov}(T_i,T_l)\operatorname{Cov}(T_j,T_k). \] The three subtracted terms correspond to the three ways to pair four indices. For order three, all first centered moments vanish, so the joint cumulant is simply \(E[Y_iY_jY_k]\). These formulas also show why higher derivatives capture dependence patterns that means and covariances alone cannot describe.

Worked Calculations

Why the Conditions and Corrections Matter

The cumulant identity is a local statement about an interior natural parameter. At such a point, small perturbations in every coordinate remain in the natural parameter space, so the moment-generating function exists around zero and the derivatives are justified. At a boundary point, this argument may fail: a moment-generating function might not exist on a full neighborhood, or the log-partition function might not have the required finite derivatives. One should not apply the identity at a boundary without checking the relevant conditions separately.

A common mistake is to identify the fourth derivative with the fourth central moment. The correction is the subtraction of three variance products in one dimension, or three covariance pairings in the joint case. For a standard normal variable, for instance, the fourth central moment is \(3\sigma^4\), while the fourth cumulant is zero. A second common mistake is to infer all higher derivatives merely from the covariance interpretation of the Hessian. The general result requires the local cumulant-generating-function identity and the existence of the derivatives; it is not an automatic extension at parameter values where regularity fails.