Factorial Moments From PGF Derivatives

Why Differentiate a Probability Generating Function?

A probability generating function (PGF) packages the probabilities of a count-valued random variable into a single function. Its derivatives do more than provide another way to write the same probabilities: at one, they yield expectations of products of consecutive integers. These are called factorial moments. The first factorial moment is the ordinary mean, and the second combines with the mean to give the variance.

Let $(\Omega,\mathcal{F},\mathbb{P})$ be a probability space, and let $X:\Omega\to\{0,1,2,\ldots\}$ be a random variable. Write $p_k=\mathbb{P}(X=k)$. Its probability generating function is $G(s)=\mathbb{E}[s^X]=\sum_{k=0}^{\infty}p_k s^k$ for $0\le s\le1$, with the convention $s^0=1$, including at $s=0$. In particular, $G(1)=\sum_k p_k=1$.

Definition: For an integer $r\ge1$, the falling factorial of order $r$ is $(x)_r=x(x-1)\cdots(x-r+1)$, and $(x)_0=1$. If $X$ is nonnegative integer-valued, its $r$th factorial moment is $\mathbb{E}[(X)_r]$, which is allowed to be infinite. For $k<r$, $(k)_r=0$.

The falling factorial appears naturally upon differentiating a power: the $r$th derivative of $s^k$ is $(k)_r s^{k-r}$ when $k\ge r$, and is zero when $k<r$. Evaluating the resulting expression at one removes the power of $s$, leaving precisely the factorial moment. The endpoint requires care: if a moment is infinite, the derivative at one is not a finite number. We therefore first take derivatives for $0<s<1$, then take their limit as $s$ increases to one.

The Derivative Formula

Theorem (PGF derivatives and factorial moments): For each integer $r\ge1$ and $0<s<1$, $$ G^{(r)}(s)=\sum_{k=r}^{\infty}(k)_r p_k s^{k-r} =\mathbb{E}\!\left[(X)_r s^{X-r}\mathbf{1}_{\{X\ge r\}}\right]. $$ As $s\uparrow1$, $G^{(r)}(s)$ increases to $\mathbb{E}[(X)_r]$, with the limit allowed to be infinite. If $\mathbb{E}[(X)_r]<\infty$, the derivative has a finite left-hand limit at one, denoted $G^{(r)}(1-)$, and $G^{(r)}(1-)=\mathbb{E}[(X)_r]$.

Proof. Fix $r\ge1$. The PGF is a power series with coefficients $p_k$, and it converges for $0\le s<1$ because $\sum_k p_k s^k\le\sum_k p_k=1$. The power-series differentiation rule therefore applies inside its interval of convergence. Differentiating $r$ times gives zero for terms with $k<r$ and gives $(k)_r p_k s^{k-r}$ for each term with $k\ge r$. Summing these terms proves the series formula. Since $X$ takes the value $k$ with probability $p_k$, the series is also the stated expectation. For every $k\ge r$, $s^{k-r}$ increases to one as $s\uparrow1$, and all summands are nonnegative. The monotone convergence theorem then gives $\lim_{s\uparrow1}G^{(r)}(s)=\sum_{k=r}^{\infty}(k)_r p_k=\mathbb{E}[(X)_r]$, whether that sum is finite or infinite. If it is finite, this limit is the finite left-hand limit $G^{(r)}(1-)$. $\square$

For the first two derivatives, the theorem gives the practical formulas $G'(1-)=\mathbb{E}[X]$ and $G''(1-)=\mathbb{E}[X(X-1)]$, provided the corresponding moments are finite. The second formula is not the ordinary second moment. The pointwise identity $X^2=X(X-1)+X$ supplies the conversion.

Corollary (Mean and variance from the first two derivatives): If $\mathbb{E}[X^2]<\infty$, then $$ \mathbb{E}[X]=G'(1-),\qquad \operatorname{Var}(X)=G''(1-)+G'(1-)-\bigl(G'(1-)\bigr)^2. $$

Proof. The finite second-moment assumption implies that both $\mathbb{E}[X]$ and $\mathbb{E}[X(X-1)]$ are finite, since $0\le X\le 1+X^2$ and $0\le X(X-1)\le X^2$ for nonnegative integer-valued $X$. By the theorem, these expectations equal $G'(1-)$ and $G''(1-)$, respectively. The identity $X^2=X(X-1)+X$ gives $\mathbb{E}[X^2]=G''(1-)+G'(1-)$. Substituting this and $\mathbb{E}[X]=G'(1-)$ into $\operatorname{Var}(X)=\mathbb{E}[X^2]-(\mathbb{E}[X])^2$ proves the formula. $\square$

If $\mathbb{E}[X(X-1)]=\infty$, then $\mathbb{E}[X^2]=\infty$ as well, by the same identity and nonnegativity. In that case the second derivative limit is infinite and there is no finite variance. The finite-variance formula should not be used as though all its terms were finite.

Worked Calculations

Worked Example: A Count with Three Possible Values

Suppose $X$ takes values $0,1,3$ with probabilities $\tfrac14,\tfrac12,\tfrac14$, respectively. Find its mean and variance using the PGF, then verify them directly from the distribution.

The probabilities sum to $\tfrac14+\tfrac12+\tfrac14=1$, so they define a distribution. The PGF is $G(s)=\tfrac14+\tfrac12s+\tfrac14s^3$. Differentiating gives $G'(s)=\tfrac12+\tfrac34s^2$, hence $G'(1-)=\tfrac12+\tfrac34=\tfrac54$. The second derivative is $G''(s)=\tfrac32s$, so $G''(1-)=\tfrac32$. Therefore $\operatorname{Var}(X)=\tfrac32+\tfrac54-(\tfrac54)^2 =\tfrac{24}{16}+\tfrac{20}{16}-\tfrac{25}{16} =\tfrac{19}{16}$.

Direct check. The mean from the probabilities is $0\cdot\tfrac14+1\cdot\tfrac12+3\cdot\tfrac14=\tfrac12+\tfrac34=\tfrac54$. The second ordinary moment is $0^2\cdot\tfrac14+1^2\cdot\tfrac12+3^2\cdot\tfrac14=\tfrac12+\tfrac94=\tfrac{11}{4}$. Thus $\operatorname{Var}(X)=\tfrac{11}{4}-(\tfrac54)^2=\tfrac{44}{16}-\tfrac{25}{16}=\tfrac{19}{16}$, agreeing with the PGF calculation.

Worked Example: A Binomial Count

Let $X$ have a binomial distribution with $n=4$ trials and success probability $p=\tfrac13$. Use its PGF to find the mean and variance, and check by summing its probabilities.

The PGF is $G(s)=(\tfrac23+\tfrac13s)^4$. Differentiating once, $G'(s)=\tfrac43(\tfrac23+\tfrac13s)^3$, so $G'(1-)=\tfrac43(1)^3=\tfrac43$. Differentiating again, $G''(s)=4(\tfrac23+\tfrac13s)^2\cdot\tfrac13=\tfrac43(\tfrac23+\tfrac13s)^2$, so $G''(1-)=\tfrac43$. The variance is $\tfrac43+\tfrac43-(\tfrac43)^2=\tfrac83-\tfrac{16}{9}=\tfrac89$.

Check from the probability mass function. For $k=0,1,2,3,4$, the probabilities are, in order, $\tfrac{16}{81},\tfrac{32}{81},\tfrac{24}{81},\tfrac{8}{81},\tfrac{1}{81}$; their numerators sum to $16+32+24+8+1=81$. The mean is $\mathbb{E}[X]=(0\cdot16+1\cdot32+2\cdot24+3\cdot8+4\cdot1)/81=108/81=\tfrac43$. The second moment is $\mathbb{E}[X^2]=(0^2\cdot16+1^2\cdot32+2^2\cdot24+3^2\cdot8+4^2\cdot1)/81=216/81=\tfrac83$. Consequently $\operatorname{Var}(X)=\tfrac83-(\tfrac43)^2=\tfrac{24}{9}-\tfrac{16}{9}=\tfrac89$, as before.

Worked Example: A Poisson Count

Suppose $X$ has a Poisson distribution with parameter $2$. Its PGF is $G(s)=e^{2(s-1)}$. Find the mean and variance from derivatives, and verify both by summing the Poisson probabilities.

Differentiating gives $G'(s)=2e^{2(s-1)}$ and $G''(s)=4e^{2(s-1)}$. At one these give $G'(1-)=2$ and $G''(1-)=4$. Hence $\mathbb{E}[X]=2$ and $\operatorname{Var}(X)=4+2-2^2=2$.

Check by summation. Here $\mathbb{P}(X=k)=e^{-2}2^k/k!$ for $k=0,1,\ldots$. For the mean, the $k=0$ term is zero, and for $k\ge1$, $k2^k/k!=2\cdot2^{k-1}/(k-1)!$. Thus $\mathbb{E}[X]=2e^{-2}\sum_{j=0}^{\infty}2^j/j!=2e^{-2}e^2=2$. For the second factorial moment, terms with $k<2$ are zero, and for $k\ge2$, $k(k-1)2^k/k!=4\cdot2^{k-2}/(k-2)!$. Therefore $\mathbb{E}[X(X-1)]=4e^{-2}\sum_{j=0}^{\infty}2^j/j!=4$. Using $X^2=X(X-1)+X$ gives $\mathbb{E}[X^2]=4+2=6$, and the direct variance calculation is $6-2^2=2$.

Interpreting the Formula Correctly

The derivative at one counts ordered selections of distinct items from a count. For a realized count $X=k$, there are $k$ ways to select one item, giving $(k)_1=k$. There are $k(k-1)$ ways to select an ordered pair of distinct items, giving $(k)_2$. Averaging these quantities over the distribution produces the first and second factorial moments. This interpretation is useful when the random variable counts occurrences, individuals, events, or other objects.

A frequent error is to treat $G''(1-)$ as $\mathbb{E}[X^2]$. The derivatives instead give $\mathbb{E}[X(X-1)]$, so the missing term is $\mathbb{E}[X]=G'(1-)$.

Common Mistake: Omitting the First Derivative

Let $X$ be Bernoulli with success probability $\tfrac12$, so $G(s)=\tfrac12+\tfrac12s$. A wrong calculation says $G''(s)=0$, and therefore $\mathbb{E}[X^2]=G''(1-)=0$. But $X^2=X$ for a Bernoulli variable, and its second moment is $\tfrac12$, not zero.

The correction is $G'(s)=\tfrac12$ and $G''(s)=0$. Thus $\mathbb{E}[X]=\tfrac12$, $\mathbb{E}[X(X-1)]=0$, and $\mathbb{E}[X^2]=G''(1-)+G'(1-)=0+\tfrac12=\tfrac12$. The resulting variance is $\tfrac12-(\tfrac12)^2=\tfrac14$. Directly, the two possible squared values are $0$ and $1$, each with probability $\tfrac12$, which also gives $\mathbb{E}[X^2]=\tfrac12$.

There is a second issue to watch: a PGF is guaranteed to be finite on $0\le s\le1$, but its derivatives at one need not be finite. The theorem uses a left-hand limit, which may equal infinity. For example, if a count has an infinite second factorial moment, $G''(s)$ grows without bound as $s\uparrow1$; assigning it a finite value at one would produce an invalid variance calculation. Check moment finiteness before using the formulas as finite numerical identities.

Check Your Understanding

Use the falling-factorial identities and verify any numerical answer by substituting into the relevant formula.

For $G(s)=\tfrac13+\tfrac23s^2$, find $G'(1-)$, $G''(1-)$, the mean, and the variance.
Explain why $G''(1-)$ is the second factorial moment rather than the ordinary second moment.
If $G'(1-)=3$ and $G''(1-)=6$, calculate $\mathbb{E}[X^2]$ and $\operatorname{Var}(X)$, assuming both derivatives are finite.
What does it mean if $G''(s)$ increases without bound as $s\uparrow1$? Which moment is then infinite?
For a Bernoulli random variable with success probability $p$, explain why the second factorial moment is zero while the second ordinary moment is $p$.