Tutorials › Statistical Theory I - Probability and Distribution Theory › Exact Laws of Categorical Sample Contrasts

Home  /  Tutorials  /  Exact Sampling Distributions

Exact Sampling Distributions · Tutorial 788 of 994

Exact Laws of Categorical Sample Contrasts

Learn to turn multinomial counts into exact distributions for categorical sample contrasts, including their support, probabilities, and common pitfalls.

Advanced 9 min read

What You'll Learn

  • Map multinomial count vectors to the attainable values of a weighted categorical total.
  • Calculate exact probabilities when different count vectors produce the same statistic.
  • Derive the law of a difference between two sample proportions.
  • Use probability-generating functions to obtain distributions for integer-valued scores.
  • Distinguish a weighted count total from a zero-sum contrast of category values.
  • Check exact probability calculations by verifying that the masses sum to one.

From Multinomial Counts to a Contrast

A categorical sample can be summarized by its count in each category, but a statistical question often concerns a single number formed from those counts. Examples include the difference between two category proportions and a total that assigns a numerical score to each category. The exact distribution of that number is obtained by mapping every possible count vector to its corresponding statistic value and adding the probabilities of count vectors that map to the same value.

This mapping matters because the statistic can have fewer possible values than the count vector. In particular, distinct count vectors may produce the same weighted total. An exact law must combine their probabilities rather than treat them as separate values of the statistic.

Definition: Let \(N=(N_1,\ldots,N_m)\) be the category-count vector from \(n\) independent observations, where each observation falls in category \(j\) with probability \(p_j\), \(\sum_{j=1}^m p_j=1\). Then \(N\) has the multinomial distribution \[ P(N_1=k_1,\ldots,N_m=k_m)=\frac{n!}{k_1!\cdots k_m!}\prod_{j=1}^m p_j^{k_j} \] for nonnegative integers \(k_j\) satisfying \(\sum_j k_j=n\), and probability zero otherwise. For fixed real scores \(c_1,\ldots,c_m\), the weighted total is \(T=\sum_{j=1}^m c_jN_j\). Its support is the set of values \(\sum_j c_jk_j\) attained by feasible count vectors having positive multinomial probability.

When \(\hat p_j=N_j/n\) is the sample proportion in category \(j\), a weighted sum of proportions is just a rescaled weighted total: \(\sum_j c_j\hat p_j=T/n\). The finite-sample law can therefore be found first for \(T\), then rescaled.

The Exact Probability Rule

For any real number \(t\), the event \(T=t\) consists of all feasible count vectors whose weighted scores add to \(t\). The multinomial probabilities of those vectors must be summed. This gives a general finite-sample formula, whether or not the scores are equally spaced or integer-valued.

Theorem: For \(T=\sum_{j=1}^m c_jN_j\), its probability mass at \(t\) is \[ P(T=t)= \sum_{\substack{k_1+\cdots+k_m=n\\ k_j\geq 0\\ \sum_j c_jk_j=t}} \frac{n!}{k_1!\cdots k_m!}\prod_{j=1}^m p_j^{k_j}. \] An empty sum is zero. Thus the support is precisely the set of weighted sums attained by feasible count vectors having positive multinomial probability.

To justify the formula, the sample space for the count vector is finite: it contains the nonnegative integer vectors summing to \(n\). For each such vector \(k\), the statistic has the single value \(\sum_j c_jk_j\). The event \(T=t\) is the disjoint union of the count-vector events for which this value equals \(t\). Additivity of probability and the multinomial mass formula give the result.

A useful equivalent view starts with one observation. Let \(Y_i=c_j\) when observation \(i\) falls in category \(j\). Then \(T=\sum_{i=1}^nY_i\), where the \(Y_i\) are independent and identically distributed with \(P(Y_i=c_j)\) receiving a contribution \(p_j\) from category \(j\). If scores repeat, their category probabilities combine. This sum-of-independent-scores view often makes the calculation shorter.

Worked Example: An Exact Difference of Two Sample Proportions

A sample of \(n=4\) independent observations has three categories with probabilities \(p_1=1/4\), \(p_2=1/2\), and \(p_3=1/4\). Find the exact distribution of \(\hat p_1-\hat p_2\).

Set \(D=N_1-N_2\), so that \(\hat p_1-\hat p_2=D/4\). For one observation, the contribution to \(D\) is \(+1\) with probability \(1/4\), \(-1\) with probability \(1/2\), and \(0\) with probability \(1/4\). Equivalently, the probability-generating expression for the sum is \[ \left(\frac14z+\frac12+\frac14z^{-1}\right)^4. \] The exponent of \(z\) records the value of \(D\). Expanding and collecting equal powers gives the following probabilities, with a common denominator of 256:

\(d\)\(P(D=d)\)\(P(\hat p_1-\hat p_2=d/4)\)
\(-4\)\(16/256\)\(16/256\)
\(-3\)\(32/256\)\(32/256\)
\(-2\)\(56/256\)\(56/256\)
\(-1\)\(56/256\)\(56/256\)
\(0\)\(49/256\)\(49/256\)
\(1\)\(28/256\)\(28/256\)
\(2\)\(14/256\)\(14/256\)
\(3\)\(4/256\)\(4/256\)
\(4\)\(1/256\)\(1/256\)

The correct generating expression is as stated. Its coefficients for \(D=4,3,2,1\) are \(1/256, 4/256, 14/256,\) and \(28/256\), respectively. The distribution is not symmetric because the probabilities of scores \(+1\) and \(-1\) differ; the masses for \(D=-4,-3,-2,-1\) are \(16/256, 32/256, 56/256,\) and \(56/256\). The remaining mass at zero is \(49/256\), so all the masses sum to \(256/256\).

Worked Laws for Other Categorical Scores

A generating function provides a compact way to organize the count-vector mapping when the scores are integers. For one observation, form \(g(z)=\sum_jp_jz^{c_j}\). Independence makes the generating function of \(T\) equal to \(g(z)^n\); the coefficient of \(z^t\) is \(P(T=t)\). Negative integer scores are allowed, producing a Laurent polynomial. With arbitrary real scores, the count-vector sum remains valid even when this polynomial method is inconvenient.

Derivation: If observation \(i\) contributes score \(Y_i\), then \(E[z^{Y_i}]=\sum_jp_jz^{c_j}\) for integer scores. Independence gives \(E[z^T]=E[z^{Y_1+\cdots+Y_n}]=\prod_iE[z^{Y_i}]=(\sum_jp_jz^{c_j})^n\). Multiplying out assigns one score choice to each observation; collecting terms with the same exponent adds the probabilities of all sample arrangements producing that total.

Worked Example: A Weighted Total With Unequally Spaced Scores

Suppose \(n=2\), with category probabilities \((0.2,0.5,0.3)\) and scores \((0,1,3)\). Find the distribution of \(T=0N_1+N_2+3N_3\).

Each observation contributes 0, 1, or 3 with probabilities 0.2, 0.5, and 0.3. The possible totals are 0, 1, 2, 3, 4, and 6. Pairing the two observations gives:

  • \(T=0\): both scores are 0, with probability \(0.2^2=0.04\).
  • \(T=1\): scores 0 and 1 in either order, with probability \(2(0.2)(0.5)=0.20\).
  • \(T=2\): both scores are 1, with probability \(0.5^2=0.25\).
  • \(T=3\): scores 0 and 3 in either order, with probability \(2(0.2)(0.3)=0.12\).
  • \(T=4\): scores 1 and 3 in either order, with probability \(2(0.5)(0.3)=0.30\).
  • \(T=6\): both scores are 3, with probability \(0.3^2=0.09\).

The masses add to \(0.04+0.20+0.25+0.12+0.30+0.09=1\). The value 5 is not in the support: no pair of available scores sums to 5. The gap illustrates why a support is the set of attainable values, not automatically every value between the minimum and maximum.

Worked Example: A Zero-Sum Score Vector

Let \(n=3\), with category probabilities \((1/2,1/4,1/4)\), and scores \(c=(1,-2,1)\). Find the law of \(T=N_1-2N_2+N_3\).

Let \(K=N_2\). Each of the three observations contributes \(-2\) with probability \(1/4\), and contributes \(+1\) with probability \(3/4\), since categories 1 and 3 have the same score. Thus \(K\) is binomial with size 3 and success probability \(1/4\), and \[ T=(3-K)-2K=3-3K. \] For \(K=0,1,2,3\), the corresponding values of \(T\) are \(3,0,-3,-6\). Their probabilities are respectively \[ (3/4)^3=\frac{27}{64},\quad 3(1/4)(3/4)^2=\frac{27}{64},\quad 3(1/4)^2(3/4)=\frac{9}{64},\quad (1/4)^3=\frac{1}{64}. \] These masses sum to \(64/64\). If the statistic is the weighted sum of sample proportions, \(T/3\), its support values are the support values of \(T\) divided by 3, and the corresponding probabilities remain unchanged.

Contrasts, Shifts, and a Common Pitfall

The word contrast is used for different but related linear combinations, so the object being shifted must be clear. In the weighted total \(T=\sum_jc_jN_j\), the \(c_j\) are category scores and the \(N_j\) are counts. Adding a constant \(a\) to every score changes the statistic to \[ \sum_j(c_j+a)N_j=T+a\sum_jN_j=T+an, \] because the counts always sum to \(n\). It does not leave this weighted total unchanged.

A separate, standard contrast of fixed category values \(x_1,\ldots,x_m\) has the form \(\sum_ja_jx_j\), where the contrast coefficients satisfy \(\sum_ja_j=0\). Shifting every category value by \(a\) then leaves that contrast unchanged: \[ \sum_ja_j(x_j+a)=\sum_ja_jx_j+a\sum_ja_j=\sum_ja_jx_j. \] These are different expressions with different roles for the scores, counts, and coefficients. The zero-sum condition alone does not make a weighted count total invariant to shifting its category scores.

A common mistake is to say that \(\sum_jc_jN_j\) is unchanged when a constant is added to every \(c_j\), because \(\sum_jc_j=0\). That reasoning substitutes the sum of the scores for the sum of the counts. The relevant identity is \(\sum_jN_j=n\), so the shift is \(an\). Always identify which coefficients are being shifted and which quantities they multiply.

Exact laws are useful when the sample size is modest, the score set is discrete, or an exact tail probability is needed. They also clarify the structure of a statistic before any approximation is considered: its support, probability mass, and symmetry all follow from the finite set of multinomial count vectors. A final arithmetic check is indispensable: probabilities assigned to all distinct support values must sum to one.

Check Your Understanding

Use the count-vector mapping or the one-observation score representation to answer each question.

  1. For \(T=2N_1-N_2\), what constraints define the count vectors contributing to \(P(T=1)\)?
  2. If two different feasible count vectors give the same weighted total, how should their probabilities be combined?
  3. With scores \(0\) and \(2\), sample size \(n=3\), and category probabilities \(1/2\) each, what is the support of the weighted total?
  4. For \(T=\sum_jc_jN_j\), what is the effect of replacing every score \(c_j\) by \(c_j+a\)?
  5. Why can a weighted total have gaps in its support even between its minimum and maximum values?

Previous

Next

Back to Tutorials