Multinomial Counts and Their Joint Normal Approximation

From Cell Counts to a Joint Normal Limit

In a multinomial experiment, each observation falls into exactly one of a fixed number of cells. The resulting counts therefore fluctuate together: if one cell receives an extra observation, fewer observations remain for the others. A separate normal approximation for each count can miss this dependence. The joint normal limit captures both the individual fluctuations and their covariance.

Suppose there are \(k\) cells, with fixed probabilities \(p_1,\ldots,p_k>0\) satisfying \(\sum_{i=1}^k p_i=1\). In \(n\) independent trials, let \(N_i\) be the count in cell \(i\), and write \(N=(N_1,\ldots,N_k)^\mathsf{T}\) and \(p=(p_1,\ldots,p_k)^\mathsf{T}\). Thus \(N\) has a multinomial distribution with sample size \(n\) and cell probabilities \(p\).

A multivariate normal distribution is allowed to have a singular covariance matrix. That detail matters here: the total count is always \(n\), so the centered counts always sum to zero and cannot vary freely in all \(k\) directions.

Exact Covariances and the Joint Limit

For trial \(r\), let \(Z_r\) be the \(k\)-vector whose \(i\)th entry is 1 if trial \(r\) lands in cell \(i\), and 0 otherwise. Then \(N=\sum_{r=1}^n Z_r\). For each \(i\), \(E[(Z_r)_i]=p_i\) and \(\operatorname{Var}((Z_r)_i)=p_i(1-p_i)\). For distinct cells \(i\) and \(j\), the indicators cannot both equal 1, so \(\operatorname{Cov}((Z_r)_i,(Z_r)_j)=0-p_i p_j=-p_i p_j\). Independence across trials now gives the exact covariance matrix of the counts.

The diagonal entries of \(\Sigma\) are \(p_i(1-p_i)\); its off-diagonal entries are \(-p_i p_j\). The negative covariances express competition among cells for a fixed total number of observations.

Proof. The vectors \(Z_1,\ldots,Z_n\) are independent and identically distributed, have mean \(p\), and have covariance matrix \(\Sigma\), as calculated above. Their coordinates are bounded, so they have finite second moments. The multivariate central limit theorem for independent, identically distributed vectors therefore gives \[ \frac{\sum_{r=1}^n(Z_r-p)}{\sqrt n}\xrightarrow{d}N_k(0,\Sigma). \] Since \(\sum_{r=1}^n Z_r=N\), the expression on the left is \((N-np)/\sqrt n\), proving the result. \(\square\)

The limiting covariance matrix is singular for a structural reason, not because the approximation has failed. For any vector \(x\in\mathbb R^k\), \[ x^\mathsf{T}\Sigma x=\sum_{i=1}^k p_i x_i^2-\left(\sum_{i=1}^k p_i x_i\right)^2 =\sum_{i=1}^k p_i(x_i-\bar x_p)^2, \qquad \bar x_p=\sum_{i=1}^k p_i x_i. \] This quantity is zero when all the \(x_i\) are equal. Since every \(p_i>0\), it is positive for any vector that is not constant across cells. Thus the all-ones vector is a null direction and, for \(k\) positive cell probabilities, \(\Sigma\) has rank \(k-1\). Consistently, \(\sum_i(N_i-np_i)=0\) exactly.

Linear Contrasts of the Counts

Many questions concern a single weighted combination of counts rather than the entire vector. For a fixed vector \(a=(a_1,\ldots,a_k)^\mathsf{T}\), define the contrast \(a^\mathsf{T}N=\sum_i a_iN_i\). A cell difference is one example: choosing \(a_i=1\), \(a_j=-1\), and all other entries zero gives \(N_i-N_j\). The contrast has an exact mean and variance, and its normal limit follows directly from the scalar central limit theorem.

Proof. In trial \(r\), define the scalar \(W_r=a^\mathsf{T}Z_r\). These variables are independent and identically distributed, bounded because \(k\) and \(a\) are fixed, and satisfy \(E[W_r]=\mu_a\) and \(\operatorname{Var}(W_r)=v_a\). Also, \(a^\mathsf{T}N=\sum_{r=1}^n W_r\). The scalar central limit theorem gives the stated standardized limit when \(v_a>0\). To justify the moving-threshold conclusion, write \(x_n=(c_n-n\mu_a)/\sqrt{nv_a}\), so \(x_n\to x\). For any \(\epsilon>0\), eventually \(x-\epsilon<x_n<x+\epsilon\), and therefore \[ P\left(\frac{a^\mathsf{T}N-n\mu_a}{\sqrt{nv_a}}\le x-\epsilon\right) \le P(a^\mathsf{T}N\le c_n) \le P\left(\frac{a^\mathsf{T}N-n\mu_a}{\sqrt{nv_a}}\le x+\epsilon\right). \] Taking limits and then letting \(\epsilon\) decrease to zero gives \(\Phi(x)\), since the standard normal distribution function is continuous. \(\square\)

The contrast variance can also be read directly from the covariance matrix: \[ \operatorname{Var}(a^\mathsf{T}N)=a^\mathsf{T}\operatorname{Cov}(N)a =n\,a^\mathsf{T}\Sigma a=nv_a. \] If all cell probabilities are positive, \(v_a=0\) precisely when all entries of \(a\) are equal. In that case the contrast is just a constant times the fixed total \(n\), so there is no random fluctuation to approximate.

Using the Approximation Carefully

The joint limit is useful when cell counts are large enough for their centered fluctuations to be reasonably represented by a Gaussian vector. It gives a coherent covariance structure for simultaneous fluctuations and, through linear contrasts, approximations to many questions of practical interest: differences between groups, weighted scores, or totals over selected cells. The scale is important: counts fluctuate on the order of \(\sqrt n\), while proportions fluctuate on the order of \(1/\sqrt n\).

A frequent mistake is to approximate the cell counts by independent normal variables with variances \(np_i(1-p_i)\). Those are the correct marginal variances, but independence is wrong. For \(i\ne j\), the exact covariance is \(-np_ip_j\), and the count total is fixed. Ignoring the negative covariances gives the wrong variance for contrasts and may assign probability to count vectors whose entries do not sum to \(n\). Instead, use \(n\Sigma\) for the joint count covariance or calculate \(a^\mathsf{T}\Sigma a\) for the particular contrast.

A second caution concerns a contrast with zero variance. In that case standardization is undefined, but the contrast itself is deterministic; there is no nondegenerate normal approximation to make. Finally, a limiting distribution describes behavior as \(n\) grows. At a particular finite sample size, especially for a rare cell or a far-tail event, it is an approximation rather than an exact probability.