Cumulant Functions, Moments, and Covariance

Why the Log-Partition Function Contains Moments

In an exponential family, a single normalizing function records how the distribution changes as its natural parameter changes. Its first derivatives describe the average sufficient statistic; its second derivatives describe variation and dependence among the components. Higher derivatives encode higher-order cumulants. These identities are useful both for understanding a model’s sufficient statistics and for calculating their distributional properties.

The parameter must lie where the normalizing integral is finite, and differentiation under that integral needs justification. We establish the identities under conditions that make this step valid, then work through discrete and continuous multivariate examples.

Setup and Regularity Conditions

Definition: A regular natural exponential family with statistic $T(x)\in\mathbb{R}^q$, natural parameter $\eta\in\mathbb{R}^q$, and base measure $\nu$ has density or mass function $$ f_\eta(x)=\exp\{\eta^\mathsf{T}T(x)-A(\eta)\} $$ with respect to $\nu$, where $$ Z(\eta)=\int \exp\{\eta^\mathsf{T}T(x)\}\,d\nu(x), \qquad A(\eta)=\log Z(\eta). $$ The natural parameter space $\mathcal{H}=\{\eta:Z(\eta)<\infty\}$ is assumed to have nonempty interior, and the identities below are stated for $\eta$ in that interior.

The interior condition supplies a neighborhood of natural parameters, not merely a finite integral at one isolated point. In particular, around any such $\eta$, choose a small closed coordinate box that stays inside $\mathcal{H}$. Exponential tilting within this box controls polynomial factors in $T(x)$: for any fixed derivative order, those factors can be bounded by a finite sum of exponentials with parameters in the box. Since the corresponding partition integrals are finite, differentiation under the integral is justified locally by dominated convergence. Thus $Z$ and $A$ have the derivatives used below. This condition is important; finiteness at a boundary point alone does not guarantee these identities by differentiation.

First and Second Derivatives

Theorem: At an interior natural parameter $\eta$, $$ \nabla A(\eta)=E_\eta[T(X)], \qquad \nabla^2 A(\eta)=\operatorname{Cov}_\eta(T(X)). $$ In coordinates, $\partial A/\partial\eta_i=E_\eta[T_i]$ and $\partial^2A/\partial\eta_i\partial\eta_j=\operatorname{Cov}_\eta(T_i,T_j)$.

Proof. Differentiating $Z$ with respect to coordinate $\eta_i$ gives

\frac{\partial Z(\eta)}{\partial\eta_i} =\int T_i(x)\exp\{\eta^\mathsf{T}T(x)\}\,d\nu(x).

Because $A=\log Z$, its first derivative is $Z^{-1}\partial Z/\partial\eta_i$. The density formula then identifies this ratio as $E_\eta[T_i(X)]$. Differentiate that ratio with respect to $\eta_j$. The quotient rule gives

\frac{\partial^2 A(\eta)}{\partial\eta_j\partial\eta_i} = \frac{1}{Z}\int T_iT_j e^{\eta^\mathsf{T}T}\,d\nu - \frac{1}{Z^2} \left(\int T_i e^{\eta^\mathsf{T}T}\,d\nu\right) \left(\int T_j e^{\eta^\mathsf{T}T}\,d\nu\right) = E_\eta[T_iT_j]-E_\eta[T_i]E_\eta[T_j].

The last expression is the covariance of $T_i(X)$ and $T_j(X)$, proving the result. $\square$

In particular, $\nabla^2 A(\eta)$ is positive semidefinite: for any vector $a$, $a^\mathsf{T}\nabla^2A(\eta)a=\operatorname{Var}_\eta(a^\mathsf{T}T(X))\geq 0$. This is a statement about the distribution of the sufficient statistic, not a claim that the covariance matrix must be invertible.

Higher Derivatives and Cumulants

The link to cumulants follows from the moment-generating function of $T(X)$. For $t$ such that $\eta+t\in\mathcal{H}$,

E_\eta[\exp\{t^\mathsf{T}T(X)\}] = \frac{Z(\eta+t)}{Z(\eta)}, \qquad K_\eta(t) = \log E_\eta[\exp\{t^\mathsf{T}T(X)\}] = A(\eta+t)-A(\eta).

Here $K_\eta(t)$ is the cumulant-generating function. Differentiating it at $t=0$ shows that the first derivatives are means, the second derivatives are covariances, and its higher derivatives are joint cumulants. Since $K_\eta(t)=A(\eta+t)-A(\eta)$, every derivative of positive order of $K_\eta$ at zero equals the corresponding derivative of $A$ at $\eta$.

Result: For indices $i_1,\ldots,i_r$, the derivative $$ \frac{\partial^r A(\eta)} {\partial\eta_{i_1}\cdots\partial\eta_{i_r}} $$ is the joint cumulant of $T_{i_1}(X),\ldots,T_{i_r}(X)$. For $r=3$, this is $$ E_\eta[(T_i-E_\eta T_i)(T_j-E_\eta T_j)(T_k-E_\eta T_k)]. $$

For the third-order formula, one can differentiate the covariance identity once more. The derivative of $E_\eta[T_iT_j]$ with respect to $\eta_k$ is $E_\eta[T_iT_jT_k]-E_\eta[T_iT_j]E_\eta[T_k]$. Differentiating the product $E_\eta[T_i]E_\eta[T_j]$ and subtracting gives

E_\eta[T_iT_jT_k] -E_\eta[T_iT_j]E_\eta[T_k] -E_\eta[T_iT_k]E_\eta[T_j] -E_\eta[T_jT_k]E_\eta[T_i] +2E_\eta[T_i]E_\eta[T_j]E_\eta[T_k].

Expanding the product of the three centered variables gives exactly this expression. This proves the stated third-order identity; the general cumulant identity follows from the defining derivatives of $K_\eta$. $\square$

Worked Examples

Worked Example: A Bernoulli Sufficient Statistic

Let $X_1,\ldots,X_n$ be independent Bernoulli variables with success probability $p$, where $0<p<1$, and write $S=\sum_{r=1}^n X_r$. Set $\eta=\log\{p/(1-p)\}$. The joint mass function is $$ \exp\{\eta S-n\log(1+e^\eta)\}, $$ so $T=S$ and $A(\eta)=n\log(1+e^\eta)$.

The success probability is $p=e^\eta/(1+e^\eta)$, and $dp/d\eta=p(1-p)$. Therefore $$ A'(\eta)=np,\qquad A''(\eta)=np(1-p),\qquad A'''(\eta)=np(1-p)(1-2p). $$

These are, respectively, $E[S]$, $\operatorname{Var}(S)$, and the third cumulant $E[(S-E[S])^3]$. For instance, at $p=1/4$ they equal $n/4$, $3n/16$, and $3n/32$. The last value follows from $n(1/4)(3/4)(1-1/2)=3n/32$.

Worked Example: Covariance and a Mixed Cumulant in a Multinomial Model

Suppose $n$ independent trials each fall into one of three categories, with probabilities $p_1,p_2,p_0$, where $p_0=1-p_1-p_2$. Let $T=(N_1,N_2)^\mathsf{T}$ be the counts in the first two categories. With category 0 as the baseline, the natural parameters are $\eta_i=\log(p_i/p_0)$, and $$ A(\eta)=n\log(1+e^{\eta_1}+e^{\eta_2}). $$

At $p_1=0.2,\ p_2=0.3,\ p_0=0.5$, the first derivatives give $E[N_1]=0.2n$ and $E[N_2]=0.3n$. The covariance matrix is $$ \nabla^2 A(\eta) =n \begin{pmatrix} p_1(1-p_1)&-p_1p_2\\ -p_1p_2&p_2(1-p_2) \end{pmatrix} =n \begin{pmatrix} 0.16&-0.06\\ -0.06&0.21 \end{pmatrix}. $$

The mixed third derivative with two indices 1 and one index 2 is the third joint cumulant of $N_1,N_1,N_2$. For one trial, let $I_1,I_2$ denote the two category indicators. Its centered product has expectation $$ E[(I_1-p_1)^2(I_2-p_2)] =0.2(0.8)^2(-0.3)+0.3(0.2)^2(0.7)+0.5(0.2)^2(-0.3) =-0.0384+0.0084-0.006=-0.036. $$

The $n$ trials are independent, so cumulants add; hence $\partial^3 A/\partial\eta_1^2\partial\eta_2=-0.036n$. This also illustrates why a mixed third derivative need not be positive.

Worked Example: A Multivariate Normal Statistic

Let $X\sim N_d(\mu,\Sigma)$, with known positive-definite covariance matrix $\Sigma$ and unrestricted mean vector $\mu$. Put $\eta=\Sigma^{-1}\mu$. Completing the quadratic form in the normal density gives the canonical representation $$ f_\eta(x)=\exp\{\eta^\mathsf{T}x-A(\eta)\}h(x), \qquad A(\eta)=\tfrac12\eta^\mathsf{T}\Sigma\eta, $$ where $h(x)=(2\pi)^{-d/2}|\Sigma|^{-1/2}\exp(-x^\mathsf{T}\Sigma^{-1}x/2)$.

Differentiating yields $\nabla A(\eta)=\Sigma\eta=\mu$ and $\nabla^2 A(\eta)=\Sigma$. Every derivative of order three or higher is zero because $A$ is quadratic. Thus $X$ has mean $\mu$, covariance $\Sigma$, and zero cumulants of order three and above. The conclusion is consistent with the log moment-generating function of a Gaussian being quadratic.

Interpreting and Using the Derivatives

These calculations give a direct route from a model’s canonical representation to the moments of its sufficient statistic. In a sample of $n$ independent observations from the same family, the joint log-partition function is $nA(\eta)$ for the combined statistic $\sum_{r=1}^nT(X_r)$. Consequently, its mean, covariance matrix, and cumulants are $n$ times the corresponding one-observation quantities.

The derivative identities also describe the geometry of the log-partition function: its Hessian is a covariance matrix and therefore cannot have a negative quadratic form. A zero variance in some direction means that the corresponding linear combination of the statistic is almost surely constant under that parameter. It does not, by itself, imply that the Hessian is positive definite.

Common Mistake

A frequent error is to identify $\nabla A(\eta)$ with the natural parameter $\eta$, or to identify the Hessian with a matrix of uncentered second moments. The derivative is instead $E_\eta[T]$, and the Hessian is $E_\eta[TT^\mathsf{T}]-E_\eta[T]E_\eta[T]^\mathsf{T}$. Another pitfall is to apply the formulas at a boundary point without checking that the required derivatives exist. The safe procedure is to verify that $\eta$ is an interior point of the finite-partition domain, then differentiate $A$ and interpret each derivative as the appropriate cumulant.

Key takeaway: For an interior natural parameter, the gradient of the log-partition function is the mean of the sufficient statistic, its Hessian is that statistic’s covariance matrix, and its higher derivatives are its higher joint cumulants.

Check Your Understanding

Use the derivative identities and the model assumptions stated in each question.

Why is finiteness of the partition integral at a single parameter value not, on its own, the regularity condition used for differentiation?
For a two-component sufficient statistic, what does the off-diagonal entry of the Hessian of $A$ represent?
In the multinomial example, why is the covariance between $N_1$ and $N_2$ negative?
What is the third cumulant of the Bernoulli sum when $p=1/2$, and what feature of the formula gives this value?
Why does a zero third derivative for the normal example not imply that the statistic has zero variance?