Covariances Between Multinomial Cell Counts

Why Multinomial Counts Move Together

In a multinomial experiment, each of \(n\) independent trials produces exactly one category. If more trials fall in one category, fewer are available for the others. This built-in competition is reflected in the covariances between the cell counts: counts for distinct categories have negative covariance, except in degenerate cases where a category has probability zero.

The calculation is easiest at the trial level. An indicator records whether a particular trial produced a particular category. Indicators from different trials are independent, but indicators for different categories on the same trial cannot both equal one. Those two facts determine the covariance between the total counts.

For each trial, exactly one of \(I_{r1},\ldots,I_{rk}\) equals one. For two distinct categories \(j\) and \(\ell\), therefore, \(I_{rj}I_{r\ell}=0\) on every trial. By contrast, indicators belonging to different trials are independent. We use these facts to derive the count variances and covariances.

Variances and Pairwise Covariances

A single indicator \(I_{rj}\) is Bernoulli with success probability \(p_j\), so \(\operatorname{Var}(I_{rj})=p_j(1-p_j)\). Summing the indicators over trials gives the variance of one cell count. For two different cell counts, expand their covariance as a double sum of covariances between indicators.

Proof. Since \(N_j=\sum_{r=1}^n I_{rj}\), bilinearity of covariance gives \[ \operatorname{Var}(N_j) =\sum_{r=1}^n\sum_{s=1}^n\operatorname{Cov}(I_{rj},I_{sj}). \] When \(r\neq s\), the indicators come from independent trials and their covariance is zero. When \(r=s\), the covariance is the variance \(p_j(1-p_j)\). There are \(n\) diagonal terms, so \[ \operatorname{Var}(N_j)=np_j(1-p_j). \]

For \(j\neq\ell\), expand the covariance in the same way: \[ \operatorname{Cov}(N_j,N_\ell) =\sum_{r=1}^n\sum_{s=1}^n\operatorname{Cov}(I_{rj},I_{s\ell}). \] If \(r\neq s\), independence of the trials makes the covariance zero. If \(r=s\), the two indicators cannot both be one, so \(E[I_{rj}I_{r\ell}]=0\). Since \(E[I_{rj}]=p_j\) and \(E[I_{r\ell}]=p_\ell\), \[ \operatorname{Cov}(I_{rj},I_{r\ell}) =E[I_{rj}I_{r\ell}]-E[I_{rj}]E[I_{r\ell}] =0-p_jp_\ell. \] There are \(n\) such same-trial terms. Hence \(\operatorname{Cov}(N_j,N_\ell)=-np_jp_\ell\). \(\square\)

The diagonal entries of the covariance matrix are variances; its off-diagonal entries are covariances between distinct cells. In particular, if both category probabilities are positive, then \(-np_jp_\ell<0\). This is the precise sense in which the counts are pairwise negatively correlated.

The Covariance Matrix and the Fixed Total

Writing the covariances together makes the structure visible. The diagonal entries depend on the probability of each category and its complement; every off-diagonal entry is negative when both associated probabilities are positive.

More generally, \(\sum_{j=1}^k N_j=n\) for every outcome of the experiment. Thus the total count is constant, not random. The covariance formulas reflect this constraint: the covariances with other cells exactly offset the variance of each cell. Indeed, for any \(j\), \[ \sum_{\ell=1}^k \operatorname{Cov}(N_j,N_\ell) =np_j(1-p_j)+\sum_{\ell\neq j}(-np_jp_\ell) =np_j(1-p_j)-np_j(1-p_j)=0, \] because \(\sum_{\ell\neq j}p_\ell=1-p_j\).

Negative Covariance and Negative Association

A negative covariance says that two counts have a negative linear association, as measured by covariance. The stronger term negative association has a specific meaning: for any two disjoint groups of coordinates, increasing functions of the counts in those groups have nonpositive covariance. It is useful to distinguish the two statements; pairwise negative covariances alone do not prove negative association.

For multinomial counts, negative association follows from the one-category-per-trial structure, not just from the covariance formula. On one trial, take disjoint groups of categories \(A\) and \(B\). The vector of indicators in group \(A\) is either all zero or has a single one. If \(g\) is increasing, its value on this vector is \(g(0)+\sum_{j\in A}a_j I_j\), where \(a_j=g(e_j)-g(0)\geq0\). Similarly, \(h\) has the form \(h(0)+\sum_{\ell\in B}b_\ell I_\ell\), with \(b_\ell\geq0\). Since the groups are disjoint, \(I_j I_\ell=0\) for \(j\in A,\ell\in B\). Expanding the covariance therefore gives \[ \operatorname{Cov}(g(I_A),h(I_B)) =-\left(\sum_{j\in A}a_jp_j\right) \left(\sum_{\ell\in B}b_\ell p_\ell\right)\leq0. \] Thus the category-indicator vector from one trial is negatively associated.

Two standard closure facts finish the argument. Independent negatively associated vectors can be concatenated while preserving negative association: condition on the first vector, apply negative association to the disjoint coordinate groups in the second, and then apply it to the conditional expectations, which remain increasing functions of disjoint coordinate groups in the first. The law of total covariance shows that both resulting covariance terms are nonpositive. Also, applying coordinatewise increasing functions to disjoint blocks of a negatively associated vector preserves negative association, directly by composing the functions in its definition.

The trials are independent, so their indicator vectors concatenate to a negatively associated vector. Each count \(N_j\) is the sum of the indicators for category \(j\) across trials. Counts from disjoint category groups are increasing functions of disjoint blocks of these indicators. Therefore the multinomial count vector is negatively associated. This stronger conclusion explains why increasing a function of one set of cell counts tends, in the covariance sense, to oppose an increasing function of disjoint cell counts.

Common Mistakes

• Assuming all indicator pairs are independent. Indicators from different trials are independent, but indicators for two distinct categories on the same trial are mutually exclusive. Their covariance is \(-p_jp_\ell\), not zero.
• Using the wrong sign. For distinct cells, \(E[I_{rj}I_{r\ell}]=0\), so the covariance is \(0-p_jp_\ell\), which is negative, not positive.
• Calling covariance and negative association the same thing. Pairwise covariance checks only pairs of counts. Negative association concerns increasing functions of any two disjoint groups. For multinomial counts the stronger property holds, but it requires more than the pairwise covariance calculation.
• Ignoring a boundary case. If \(p_j=0\), then \(N_j=0\) with probability one, so its variance and its covariance with every other count are zero. A correlation involving this constant count is undefined because its variance is zero.