Multinomial Counts and Sample Proportions

From Categorical Observations to Counts

A categorical observation records which one of a fixed set of categories occurred. When $n$ independent observations share the same category probabilities, their category counts have an exact joint sampling law. This law describes not only how often each category appears, but also the dependence among the counts: an observation assigned to one category cannot simultaneously be assigned to another.

Let there be $K$ categories, labelled $1,\ldots,K$. For observation $r$, let $X_r$ denote its category, and suppose $X_1,\ldots,X_n$ are independent with $P(X_r=i)=p_i$ for every $r$ and $i$. The probabilities satisfy $p_i\geq 0$ and $\sum_{i=1}^K p_i=1$. The common-probability assumption matters: the sampling law below is for independent observations with the same categorical distribution.

Definition: The count $N_i$ for category $i$ is the number of observations in that category: $N_i=\sum_{r=1}^n I(X_r=i)$, where $I(A)$ is 1 when event $A$ occurs and 0 otherwise. The count vector is $N=(N_1,\ldots,N_K)^\mathsf{T}$. The sample proportion for category $i$ is $\hat p_i=N_i/n$, and the sample-proportion vector is $\hat p=(\hat p_1,\ldots,\hat p_K)^\mathsf{T}$.

Every observation is assigned to exactly one category, so $\sum_i N_i=n$, and consequently $\sum_i\hat p_i=1$. Thus the count vector can take only nonnegative integer values whose entries sum to $n$. Its coordinates are not independent, even though the observations themselves are independent.

The Joint Sampling Law

To obtain a particular count vector, first consider one ordered sequence of observations that produces it. Independence gives the probability of that sequence as a product of the category probabilities. The number of distinct ordered sequences producing the same counts supplies the remaining factor.

Theorem: For nonnegative integers $n_1,\ldots,n_K$ with $\sum_i n_i=n$, the joint probability of the count vector is $$ P(N_1=n_1,\ldots,N_K=n_K) =\frac{n!}{n_1!\cdots n_K!}\prod_{i=1}^K p_i^{n_i}. $$ If the counts do not sum to $n$, or any count is negative, the probability is zero. This joint law is the multinomial distribution with sample size $n$ and category probabilities $p_1,\ldots,p_K$.

For the derivation, a sequence with $n_i$ occurrences of category $i$ has probability $\prod_i p_i^{n_i}$, because the observations are independent. To count such sequences, choose the $n_1$ positions for category 1, then the $n_2$ positions for category 2 from those remaining, and continue. The product of these choices is $$ \binom{n}{n_1}\binom{n-n_1}{n_2}\cdots =\frac{n!}{n_1!\cdots n_K!}. $$ The events corresponding to these distinct ordered sequences are disjoint and together describe exactly the specified count vector. Multiplying their number by the probability of one sequence proves the formula.

Worked Example: A Specified Three-Category Count

Suppose $n=5$ independent observations fall into three categories with probabilities $(p_1,p_2,p_3)=(0.5,0.3,0.2)$. Find the probability of counts $(N_1,N_2,N_3)=(2,2,1)$.

The counts are nonnegative and sum to 5, so the multinomial formula applies. There are $5!/(2!2!1!)=30$ ordered sequences with these counts. Each such sequence has probability $(0.5)^2(0.3)^2(0.2)^1$. Therefore $$ P(N_1=2,N_2=2,N_3=1) =30(0.5)^2(0.3)^2(0.2) =30(0.25)(0.09)(0.2) =0.135. $$ The probability is $0.135$, or 13.5%. The factor 30 is essential: the product alone is the probability of one particular ordering, not of all sequences with these counts.

Marginal Moments and Covariances

Each count is a sum of indicators, one for each observation. For a fixed category $i$, those indicators are independent across observations, each with success probability $p_i$. This gives the marginal mean and variance directly. For distinct categories, indicators from the same observation are mutually exclusive, which produces negative covariance.

Theorem: For $i=1,\ldots,K$, $$ E[N_i]=np_i,\qquad \operatorname{Var}(N_i)=np_i(1-p_i). $$ For $i\neq j$, $$ \operatorname{Cov}(N_i,N_j)=-np_ip_j. $$ Consequently, $$ E[\hat p_i]=p_i,\qquad \operatorname{Var}(\hat p_i)=\frac{p_i(1-p_i)}{n},\qquad \operatorname{Cov}(\hat p_i,\hat p_j)=-\frac{p_ip_j}{n}\quad(i\neq j). $$

To prove these statements, write $I_{ri}=I(X_r=i)$, so $N_i=\sum_{r=1}^n I_{ri}$. Each $I_{ri}$ is Bernoulli with success probability $p_i$, hence $E[I_{ri}]=p_i$ and $\operatorname{Var}(I_{ri})=p_i(1-p_i)$. Independence across observations gives $$ E[N_i]=\sum_{r=1}^n E[I_{ri}]=np_i,\qquad \operatorname{Var}(N_i)=\sum_{r=1}^n\operatorname{Var}(I_{ri})=np_i(1-p_i). $$ For $i\neq j$, the two indicators from the same observation cannot both equal 1, so $E[I_{ri}I_{rj}]=0$. Thus $\operatorname{Cov}(I_{ri},I_{rj})=0-p_ip_j=-p_ip_j$. Indicators from different observations are independent and have covariance zero. Summing all the pairwise covariances therefore yields $\operatorname{Cov}(N_i,N_j)=n(-p_ip_j)$. Finally, $\hat p_i=N_i/n$, and scaling a covariance by $1/n$ in each argument divides it by $n^2$. This proves the proportion formulas as well. $\square$

The same indicator representation shows that $N_i$ has a binomial marginal distribution with parameters $n$ and $p_i$: it is the sum of $n$ independent indicators with that success probability. This marginal statement does not make the different category counts independent. Their joint law and their negative covariances record the competition for the same $n$ observations.

Worked Example: Count and Proportion Covariance Matrices

Let $n=10$ and $p=(0.2,0.5,0.3)^\mathsf{T}$. Find the mean count vector, the count covariance matrix, and the covariance matrix of the sample proportions.

The expected counts are $np=(2,5,3)^\mathsf{T}$. The diagonal entries of the count covariance matrix are $np_i(1-p_i)$, giving $1.6$, $2.5$, and $2.1$. The off-diagonal entries are $-np_ip_j$: they are $-1$, $-0.6$, and $-1.5$ for category pairs $(1,2)$, $(1,3)$, and $(2,3)$. Hence $$ \operatorname{Cov}(N) = \begin{pmatrix} 1.6&-1&-0.6\\ -1&2.5&-1.5\\ -0.6&-1.5&2.1 \end{pmatrix}. $$

Since $\hat p=N/10$, its covariance matrix is the count covariance matrix divided by $10^2=100$: $$ \operatorname{Cov}(\hat p) = \begin{pmatrix} 0.016&-0.010&-0.006\\ -0.010&0.025&-0.015\\ -0.006&-0.015&0.021 \end{pmatrix}. $$ For example, the standard deviation of the first sample proportion is $\sqrt{0.016}\approx0.1265$, while the covariance between the first and second proportions is $-0.010$. The expected sample-proportion vector is $p$, so each proportion is unbiased for its category probability.

Matrix Form and the Fixed-Sum Constraint

Let $D=\operatorname{diag}(p_1,\ldots,p_K)$, the diagonal matrix of category probabilities, and regard $p$ as a column vector. The componentwise covariance formulas combine into $$ \operatorname{Cov}(N)=n(D-pp^\mathsf{T}),\qquad \operatorname{Cov}(\hat p)=\frac{1}{n}(D-pp^\mathsf{T}). $$ The diagonal entries of $D-pp^\mathsf{T}$ are $p_i-p_i^2$, and its off-diagonal entries are $-p_ip_j$, so these matrix expressions reproduce the formulas above.

The matrix is singular because the entries of the count vector sum to $n$ exactly and the entries of the proportion vector sum to 1 exactly. Indeed, $(D-pp^\mathsf{T})\mathbf{1}=p-p(\sum_i p_i)=0$, where $\mathbf{1}$ is the all-ones vector. When every $p_i$ is positive, this is the only null direction: for any vector $a$, $$ a^\mathsf{T}(D-pp^\mathsf{T})a =\sum_i p_i a_i^2-\left(\sum_i p_i a_i\right)^2 =\sum_i p_i(a_i-\bar a_p)^2, \qquad \bar a_p=\sum_i p_i a_i. $$ This quantity is zero exactly when all $a_i$ are equal, so the matrix has rank $K-1$. The singularity is a consequence of the fixed-sum constraint, not a defect in the covariance calculation.

Worked Example: Checking the Zero-Sum Covariance Direction

Use the covariance matrix in the preceding example to check the fixed-sum constraint. Each row sum is zero: $1.6-1-0.6=0$, $-1+2.5-1.5=0$, and $-0.6-1.5+2.1=0$. Thus $\operatorname{Cov}(N)\mathbf{1}=0$.

This also follows directly from the sum of the counts. Since $N_1+N_2+N_3=10$ with probability 1, its variance is zero. Expanding that variance gives $$ 0=\operatorname{Var}(N_1+N_2+N_3) =\sum_i\operatorname{Var}(N_i)+2\sum_{i<j}\operatorname{Cov}(N_i,N_j) =1.6+2.5+2.1+2(-1-0.6-1.5)=0. $$ The same calculation for the proportions gives zero variance for their sum, which is always 1.

Common Mistake: Treating Category Counts as Independent

A tempting but incorrect argument is that independent observations must produce independent counts. Independence applies to the observations $X_1,\ldots,X_n$, not to the counts formed by sorting those observations into mutually exclusive categories. For two distinct categories, the correct covariance is $-np_ip_j$, not zero. If one count is higher while the total sample size stays fixed, less room remains for the other counts.

A useful check on any proposed covariance matrix is that every row must sum to zero: each row sum is the covariance of one count with the fixed total $\sum_j N_j=n$. Likewise, the row sums of the sample-proportion covariance matrix must be zero. A matrix with positive off-diagonal entries or nonzero row sums may be appropriate in another model, but it is not the covariance matrix for these multinomial counts under the stated assumptions.

Key takeaway: Independent, identically distributed categorical observations produce a multinomial count vector. Its expected counts are $np_i$, its diagonal covariances are $np_i(1-p_i)$, and its off-diagonal covariances are $-np_ip_j$. Sample proportions divide the count means by $n$ and the count covariance matrix by $n^2$.

Check Your Understanding

Use the definitions and formulas to answer the following questions.

For $n=6$ and category probabilities $(0.4,0.6)$, calculate the probability of counts $(2,4)$, showing the multinomial coefficient and probability factors.
For category $i$, explain why $N_i$ has mean $np_i$ and variance $np_i(1-p_i)$.
Why is the covariance between counts in distinct categories negative under multinomial sampling?
State the covariance matrix of the sample-proportion vector in terms of $D$, $p$, and $n$.
What fixed-sum property makes the covariance matrix singular, and what does that imply about the sum of each matrix row?