Moment-Generating Functions and Cumulants

Why Use Moment-Generating Functions?

A random vector can have many joint moments: moments of each coordinate, products of coordinates, and products with repeated powers. A multivariate moment-generating function collects these quantities into one function. Differentiating it at the origin recovers joint raw moments, while differentiating its logarithm gives cumulants, which often have simpler behavior under independence.

The important condition is local: the moment-generating function must be finite on an open neighborhood of the origin. Finiteness just at the origin is automatic, since the function there equals one, and finiteness at a few isolated points is not enough to justify the usual differentiation formulas.

Definition: Let $X=(X_1,\ldots,X_p)^\mathsf{T}$ be a random vector in $\mathbb{R}^p$. Its moment-generating function is $$ M_X(t)=E\!\left[\exp(t^\mathsf{T}X)\right], \qquad t\in\mathbb{R}^p, $$ at every $t$ for which this expectation is finite. We say the moment-generating function exists near the origin if it is finite for all $t$ in some open set containing $0$. On such a neighborhood, $M_X(t)>0$, and the cumulant-generating function is $K_X(t)=\log M_X(t)$.

For a multi-index $\alpha=(\alpha_1,\ldots,\alpha_p)$ of nonnegative integers, write $|\alpha|=\sum_{j=1}^p\alpha_j$, $X^\alpha=\prod_{j=1}^p X_j^{\alpha_j}$, and $\partial^\alpha=\partial^{|\alpha|}/(\partial t_1^{\alpha_1}\cdots\partial t_p^{\alpha_p})$. A derivative involving, for example, $t_1$ twice and $t_2$ once corresponds to the product $X_1^2X_2$.

Differentiation Recovers Joint Moments

The neighborhood condition provides enough exponential integrability to justify differentiating under the expectation. In particular, it gives finite absolute moments of every order, not merely the particular moments one hopes to calculate.

Theorem: If $M_X(t)$ is finite on an open neighborhood of $0$, then for every multi-index $\alpha$, $M_X$ has the corresponding derivative at $0$, and $$ \partial^\alpha M_X(0)=E[X^\alpha]. $$

Proof. Because the domain contains a neighborhood of $0$, choose $\delta>0$ small enough that every corner of the cube $[-\delta,\delta]^p$ is in the domain. For any $x\in\mathbb{R}^p$, $$ \exp\!\left(\delta\sum_{j=1}^p |x_j|\right) =\max_{s\in\{-1,1\}^p}\exp(\delta s^\mathsf{T}x) \leq \sum_{s\in\{-1,1\}^p}\exp(\delta s^\mathsf{T}x). $$ Taking expectations shows that $E[\exp(\delta\sum_j|X_j|)]<\infty$, since the sum on the right has finitely many terms, each with finite expectation.

Fix any derivative order $k$. For $t$ in a sufficiently small cube around $0$, a derivative of order $k$ of $\exp(t^\mathsf{T}X)$ is $X^\alpha\exp(t^\mathsf{T}X)$, with $|\alpha|=k$. The factor $|X^\alpha|$ is bounded by $(1+\sum_j|X_j|)^k$. By shrinking the cube for $t$, this polynomial times $\exp(t^\mathsf{T}X)$ is bounded by a constant times $\exp(\delta\sum_j|X_j|)$. The latter is integrable by the preceding argument. The dominated convergence theorem therefore permits differentiation under the expectation, one derivative at a time. Evaluating at $t=0$ gives $E[X^\alpha]$. Since $k$ was arbitrary, the result holds for every multi-index. $\square$

For example, in two dimensions, $M_{t_1t_2}(0)=E[X_1X_2]$, while $M_{t_1t_1}(0)=E[X_1^2]$. These are raw joint moments; they are not automatically centered.

Worked Example: Independent Bernoulli Coordinates

Let $X_1$ and $X_2$ be independent Bernoulli random variables with success probabilities $1/4$ and $1/2$, respectively. For a Bernoulli variable with success probability $q$, its moment-generating function is $1-q+qe^t$, since its values are zero and one. Independence therefore gives $$ M_X(t_1,t_2)=\left(\frac34+\frac14e^{t_1}\right)\left(\frac12+\frac12e^{t_2}\right). $$

The logarithm separates into two terms: $$ K_X(t_1,t_2)=\log\left(\frac34+\frac14e^{t_1}\right) +\log\left(\frac12+\frac12e^{t_2}\right). $$

For a Bernoulli coordinate with probability $q$, the first three derivatives at zero of $\log(1-q+qe^t)$ are $q$, $q(1-q)$, and $q(1-q)(1-2q)$. Thus the first three cumulants of $X_1$ are $1/4,\ 3/16,\ 3/32$, while those of $X_2$ are $1/2,\ 1/4,\ 0$. Every mixed cumulant is zero, because $K_X$ is a sum of a function of $t_1$ and a function of $t_2$; any mixed derivative differentiates each term with respect to a variable on which it does not depend.

The moment-generating function also gives a joint raw moment directly. Differentiating once in each variable and setting $t_1=t_2=0$ gives $$ E[X_1X_2]=M_{t_1t_2}(0,0) =\left(\frac14\right)\left(\frac12\right)=\frac18. $$

What Cumulants Measure

For a multi-index $\alpha$, the joint cumulant $\kappa_\alpha$ is defined as the derivative of the cumulant-generating function at the origin:

Definition: If $M_X$ exists near $0$, the joint cumulant with index $\alpha$ is $$ \kappa_\alpha=\partial^\alpha K_X(0). $$ The first-order cumulants are the coordinate means. The second-order cumulant involving coordinates $i$ and $j$ is $E[X_iX_j]-E[X_i]E[X_j]$, the covariance of those coordinates.

To see the second-order identity, differentiate $K_X=\log M_X$ twice: $$ \partial_{t_i}\partial_{t_j}K_X(t) =\frac{\partial_{t_i}\partial_{t_j}M_X(t)}{M_X(t)} -\frac{(\partial_{t_i}M_X(t))(\partial_{t_j}M_X(t))}{M_X(t)^2}. $$ At $t=0$, $M_X(0)=1$; the differentiation result above gives the joint second raw moment in the first numerator and the two means in the second term. Their difference is the stated second-order cumulant. This identity is useful for interpreting the derivatives, without changing the definition: cumulants of every order are defined by derivatives of $K_X$, not by treating all of them as ordinary moments.

A practical distinction is that the moment-generating function encodes raw moments, whereas the cumulant-generating function encodes cumulants. For instance, the second mixed cumulant is obtained from a second derivative of $K_X$, not from the mixed derivative of $M_X$ alone.

Worked Example: A Bivariate Gaussian Vector

Let $Z_1,Z_2$ be independent $N(0,1)$ random variables, and define $X_1=1+Z_1+Z_2$ and $X_2=-1+Z_1+2Z_2$. For $t=(t_1,t_2)^\mathsf{T}$, $$ t^\mathsf{T}X=t_1-t_2+(t_1+t_2)Z_1+(t_1+2t_2)Z_2. $$ For a standard normal $Z$, $E[e^{aZ}]=e^{a^2/2}$. Applying this to the two independent terms yields $$ M_X(t_1,t_2)=\exp\left(t_1-t_2+\frac12(t_1+t_2)^2+\frac12(t_1+2t_2)^2\right). $$

The exponent is the cumulant-generating function. Expanding its quadratic terms gives $$ K_X(t_1,t_2)=t_1-t_2+t_1^2+3t_1t_2+\frac52t_2^2. $$

The first derivatives at zero give $\kappa_{10}=1$ and $\kappa_{01}=-1$. The second derivatives give $\kappa_{20}=2$, $\kappa_{11}=3$, and $\kappa_{02}=5$. For example, the coefficient of $t_1t_2$ is $3$, and differentiating once in each variable gives $3$, not $3/2$. All cumulants of order three or higher are zero because $K_X$ is quadratic.

The raw mixed moment is not the mixed cumulant: the second-order identity gives $E[X_1X_2]=\kappa_{11}+E[X_1]E[X_2]=3+(1)(-1)=2$. Directly, $X_1X_2$ has expectation $E[(1+Z_1+Z_2)(-1+Z_1+2Z_2)]= -1+1+2=2$; terms linear in either centered normal have expectation zero, and $E[Z_1Z_2]=0$.

Cumulants of Independent Sums

Cumulants are especially convenient when independent random vectors are added: their cumulant-generating functions, and hence their cumulants, add. This follows directly from the definitions.

Theorem: Let $X$ and $Y$ be independent random vectors of the same dimension, each with a moment-generating function finite on a neighborhood of $0$. Then $X+Y$ has a moment-generating function finite on a neighborhood of $0$, and $$ K_{X+Y}(t)=K_X(t)+K_Y(t) $$ there. Consequently, every joint cumulant of $X+Y$ is the sum of the corresponding joint cumulants of $X$ and $Y$.

Proof. The intersection of the two neighborhoods on which the individual moment-generating functions are finite is itself a neighborhood of $0$. For $t$ in that intersection, independence and the definition of expectation give $$ M_{X+Y}(t)=E[e^{t^\mathsf{T}X}e^{t^\mathsf{T}Y}] =E[e^{t^\mathsf{T}X}]E[e^{t^\mathsf{T}Y}] =M_X(t)M_Y(t). $$ Both factors are finite and positive there, so taking logarithms gives $K_{X+Y}(t)=K_X(t)+K_Y(t)$. Differentiating this identity at $0$ gives additivity for every joint cumulant. $\square$

Worked Example: A Shared Poisson Component

Let $U\sim\operatorname{Poisson}(2)$ and $V\sim\operatorname{Poisson}(1)$ be independent, and set $X_1=U+V$, $X_2=V$. For a Poisson variable $W$ with rate $\lambda$, summing its probability mass function gives $E[e^{tW}]=\exp(\lambda(e^t-1))$. Therefore, $$ M_X(t_1,t_2)=\exp\left(2(e^{t_1}-1)+(e^{t_1+t_2}-1)\right), \qquad K_X(t_1,t_2)=2(e^{t_1}-1)+(e^{t_1+t_2}-1). $$

The derivatives of $2(e^{t_1}-1)$ contribute $2$ to any cumulant involving only $t_1$, and zero to derivatives involving $t_2$. The derivatives of $e^{t_1+t_2}-1$ contribute $1$ to every positive-order derivative involving either variable, because each differentiation leaves the same exponential factor, which equals $1$ at the origin.

It follows that $\kappa_{10}=3,\ \kappa_{01}=1,\ \kappa_{20}=3,\ \kappa_{11}=1,\ \kappa_{02}=1$. At third order, $\kappa_{30}=3$, while $\kappa_{21}=\kappa_{12}=\kappa_{03}=1$. The shared component $V$ is responsible for mixed cumulants: differentiating the term $e^{t_1+t_2}-1$ in both variables gives $1$.

As a check on the distinction between moments and cumulants, $E[X_1X_2]=\kappa_{11}+E[X_1]E[X_2]=1+(3)(1)=4$. Directly, $X_1X_2=UV+V^2$. Independence gives $E[UV]=E[U]E[V]=2$, and for a Poisson variable with rate $1$, $E[V^2]=\operatorname{Var}(V)+(E[V])^2=1+1=2$; hence the raw mixed moment is $2+2=4$.

Common Pitfall and Why the Condition Matters

A common mistake is to differentiate an expectation as though the operation always passes through it. The formula $\partial^\alpha M_X(0)=E[X^\alpha]$ requires justification; it is not guaranteed just because $M_X(0)=1$. Finiteness near the origin is the condition used above to control the integrands and justify the interchange. A random variable can have some finite moments but no moment-generating function on any open neighborhood of zero, so the two situations must not be conflated.

Another frequent error is to call $\partial_{t_i}\partial_{t_j}M_X(0)$ a covariance. It is the raw moment $E[X_iX_j]$; the second cumulant is the derivative of $\log M_X$, which subtracts the product of the means. Keeping track of whether a derivative is applied to $M_X$ or to $K_X$ prevents this confusion.

In applications, moment-generating functions provide a compact way to obtain many joint moments, and cumulants make independent components particularly easy to combine. Their neighborhood-of-zero condition also identifies when these derivative calculations are valid. These tools prepare the way for using joint moments to obtain means and covariance quantities explicitly.

Check Your Understanding

Use the definitions and derivative rules to answer the following questions.

For a vector $X$, what does it mean for its moment-generating function to exist near the origin?
Which derivative of $M_X$ gives $E[X_1^2X_2]$ in a three-dimensional vector?
Why is the mixed second derivative of $M_X$ generally not the mixed second cumulant?
If $X$ and $Y$ are independent and both have moment-generating functions near zero, how does the cumulant-generating function of $X+Y$ relate to theirs?
In the shared Poisson example, which component creates the nonzero mixed cumulants, and how can this be seen from $K_X$?