The Joint Generating Function of Multinomial Counts

Why Use a Joint Generating Function?

A multinomial count vector records how many of \(n\) independent trials fall in each of \(k\) categories. Its joint distribution contains several kinds of information: the probability of a particular count vector, the mean and variance of each count, and the dependence between counts. A joint probability-generating function packages that information into one polynomial. Differentiating the polynomial recovers moments, while its coefficients are the probabilities themselves.

Here \(n\) is a fixed nonnegative integer, and each trial falls in exactly one of \(k\) categories. The probability of category \(j\) on any trial is \(p_j\), where \(p_j\geq 0\) and \(\sum_{j=1}^k p_j=1\). The count in category \(j\) is \(X_j\), so the count vector is \((X_1,\ldots,X_k)\) and its entries sum to \(n\).

Definition and Derivation

To derive the multinomial joint generating function, let \(C_r\) be the category of trial \(r\), for \(r=1,\ldots,n\). On that trial, the factor contributed to \(s_1^{X_1}\cdots s_k^{X_k}\) is \(s_{C_r}\): it is \(s_j\) if the trial falls in category \(j\). Multiplying these factors across trials gives \(s_1^{X_1}\cdots s_k^{X_k}=\prod_{r=1}^n s_{C_r}\). The category outcomes of the trials are independent, so the expectation of this product is the product of the single-trial expectations. For one trial, that expectation is \(\sum_{j=1}^k p_j s_j\). Therefore:

This derivation also covers \(n=0\), when the count vector is identically zero and the generating function is the constant \(1\). Because \(G\) is a finite polynomial, its partial derivatives can be taken term by term; no interchange of an infinite sum and an expectation is needed.

Derivatives Give Moments

For a probability-generating function, differentiating with respect to \(s_i\) brings down a factor of \(X_i\). The other variables then track the remaining counts. Evaluating every variable at \(1\) removes those remaining powers, because \(1^{X_j}=1\). More generally, a second derivative with respect to the same variable brings down \(X_i(X_i-1)\), while a mixed derivative with respect to two distinct variables brings down \(X_iX_j\).

Proof. Write \(q(s_1,\ldots,s_k)=\sum_{j=1}^k p_js_j\), so that \(G=q^n\). For \(n\geq 1\), differentiating once gives

At \(s_1=\cdots=s_k=1\), \(q=\sum_jp_j=1\). By the definition of \(G\), its first partial derivative there is \(E[X_i]\), giving \(E[X_i]=np_i\). For \(n=0\), both sides are zero as well.

For \(n\geq 2\), differentiating twice with respect to \(s_i\) gives \(\frac{\partial^2G}{\partial s_i^2}=n(n-1)p_i^2q^{n-2}\). At all variables equal to one, the derivative is \(E[X_i(X_i-1)]\), so this factorial moment is \(n(n-1)p_i^2\). When \(n=0\) or \(n=1\), both the second derivative and the stated factorial moment are zero, so the formula holds for all \(n\geq0\).

For distinct \(i,j\), differentiating once with respect to each variable gives, when \(n\geq2\),

At all variables equal to one this is \(E[X_iX_j]=n(n-1)p_ip_j\). For \(n=0\) or \(n=1\), the mixed derivative and the product \(X_iX_j\) are both zero, so the identity again holds.

Finally, \(X_i^2=X_i(X_i-1)+X_i\), and therefore

For \(i\ne j\), the covariance is

These formulas also apply when some \(p_i=0\), since the corresponding counts are always zero. \(\square\)

What the Dependence Means

The negative covariance between distinct counts is a consequence of the fixed total \(X_1+\cdots+X_k=n\). If one category count increases while the total remains \(n\), there are fewer trials available for the other categories. More formally, for \(i\ne j\), the covariance is \(-np_ip_j\), which is nonpositive and is strictly negative when both category probabilities are positive.

The trials are independent of one another, but the category counts are generally not independent. Independence across trials does not imply independence among totals formed from those trials. In fact, when \(n>0\) and two category probabilities are positive, the negative covariance just derived rules out independence: independent random variables with finite variances have covariance zero.

Common Pitfall

A common mistake is to treat the counts as independent because the trials themselves are independent. That reasoning confuses two different collections of random variables: the outcomes on separate trials are independent, but the category totals are linked by their fixed sum. For example, with \(n=8\), \(p_1=0.2\), and \(p_2=0.5\), independence would require \(\operatorname{Cov}(X_1,X_2)=0\). The derivative calculation instead gives \(-8(0.2)(0.5)=-0.8\), so the counts are not independent.

A related computational error is to use \(E[X_i^2]=n(n-1)p_i^2\). That derivative gives \(E[X_i(X_i-1)]\), not \(E[X_i^2]\). The correction is to add \(E[X_i]\), because \(X_i^2=X_i(X_i-1)+X_i\). This extra term is essential when calculating the variance.