Tutorials › Introduction to Statistics › Exact Multinomial Goodness-of-Fit Tests

Home  /  Tutorials  /  Chi-Square Goodness of Fit

Chi-Square Goodness of Fit · Tutorial 815 of 996

Exact Multinomial Goodness-of-Fit Tests

Learn how to define extremeness with a goodness-of-fit statistic and compute an exact p-value by summing the probabilities of qualifying count vectors under the null model.

Intermediate 10 min read

What You'll Learn

  • State the fixed-probability multinomial model needed for an exact goodness-of-fit test.
  • Calculate a count vector’s probability under the null model.
  • Use the Pearson statistic to define which count vectors are at least as extreme as observed.
  • Enumerate the relevant count vectors without omitting ties.
  • Sum their probabilities and interpret the resulting exact p-value.
  • Distinguish an exact p-value from a chi-square approximation.

Why Calculate an Exact Goodness-of-Fit P-Value?

A Pearson goodness-of-fit test compares observed category counts with the counts expected under a null probability model. When the model’s expected counts are small, the chi-square approximation used to obtain a p-value may be unreliable. An exact multinomial test takes a different route: it considers the possible count vectors for the fixed sample size and adds their probabilities under the null model.

“Exact” means that the calculation uses the specified null probability model directly, rather than approximating the distribution of the test statistic with a chi-square distribution. It does not mean that every aspect of the analysis is automatically correct. The categories, sampling assumptions, null probabilities, and rule for deciding which outcomes count as at least as extreme must all be specified.

The Model and the Exact P-Value

Suppose \(n\) independent observations each fall into exactly one of \(k\) categories. Under the null hypothesis, the probability of category \(i\) is a specified value \(p_i\), where \(p_i>0\) and \(\sum_{i=1}^{k}p_i=1\). The total sample size \(n\) is fixed. Write the observed count vector as \((O_1,\ldots,O_k)\), with \(\sum_{i=1}^{k}O_i=n\).

Definition: Under the fixed-probability null model, the probability of a possible count vector \((x_1,\ldots,x_k)\) is $$ P(X_1=x_1,\ldots,X_k=x_k) =\frac{n!}{x_1!x_2!\cdots x_k!}\prod_{i=1}^{k}p_i^{x_i}, \qquad \sum_{i=1}^{k}x_i=n. $$ The factor \(n!/(x_1!\cdots x_k!)\) counts how many ordered sequences of observations produce that count vector.

For a Pearson goodness-of-fit test, the expected count in category \(i\) is \(E_i=np_i\), and a possible vector’s Pearson statistic is

$$ T(x_1,\ldots,x_k)=\sum_{i=1}^{k}\frac{(x_i-E_i)^2}{E_i}. $$

The exact p-value for the observed vector is the sum of the null probabilities of all possible vectors whose statistic is at least as large as the observed statistic. “At least as large” includes ties.

Definition: For the Pearson statistic, the exact upper-tail p-value is $$ p_{\mathrm{exact}} =\sum_{\substack{x_1+\cdots+x_k=n\\ T(x_1,\ldots,x_k)\ge T(O_1,\ldots,O_k)}} \frac{n!}{x_1!\cdots x_k!}\prod_{i=1}^{k}p_i^{x_i}. $$ The statistic specifies what “at least as extreme” means. Another statistic could lead to a different exact p-value.

This procedure assumes that the observations are independent and follow the same category probabilities under the null, that the category probabilities are fixed and correctly specified, and that the sample size is fixed. It also assumes each observation is counted once in exactly one category. The formula is exact for this model; it cannot repair dependence between observations or an unsuitable null model.

Worked Example: Enumerating a Three-Category Tail

Worked Example: Enumerating a Three-Category Tail

Suppose four independent observations fall into categories A, B, and C. Under the null hypothesis, their probabilities are \(0.50\), \(0.25\), and \(0.25\). The observed counts are \((4,0,0)\). Calculate the exact p-value using the Pearson statistic.

The model has fixed \(n=4\), with the same probabilities for each independent observation; all three probabilities are positive and sum to 1. The expected counts are

$$ (E_A,E_B,E_C)=(4(0.50),4(0.25),4(0.25))=(2,1,1). $$

For the observed vector, the statistic is

$$ T_{\mathrm{obs}} =\frac{(4-2)^2}{2}+\frac{(0-1)^2}{1}+\frac{(0-1)^2}{1} =2+1+1=4. $$

There are \(\binom{6}{2}=15\) possible nonnegative count vectors summing to 4. Enumerating them and retaining those with \(T\ge 4\) gives the following eight vectors. Their probabilities use the multinomial formula; for example, the probability of \((1,3,0)\) is \(4!/(1!3!0!)(0.50)(0.25)^3=1/32\).

Count vectorPearson statistic \(T\)Null probability
(4, 0, 0)41/16
(1, 3, 0)5.51/32
(1, 0, 3)5.51/32
(0, 0, 4)121/256
(0, 1, 3)64/256
(0, 2, 2)46/256
(0, 3, 1)64/256
(0, 4, 0)121/256

For instance, the tied vector \((0,2,2)\) has statistic \(2+(2-1)^2+(2-1)^2=4\). Its probability is \(4!/(0!2!2!)(0.25)^4=6/256\). It must be included because its statistic equals, rather than exceeds, the observed statistic.

Adding the probabilities of all eight tail vectors gives

$$ p_{\mathrm{exact}} =\frac{1}{16}+\frac{1}{32}+\frac{1}{32} +\frac{1}{256}+\frac{4}{256}+\frac{6}{256} +\frac{4}{256}+\frac{1}{256} =\frac{48}{256} =0.1875. $$

Thus, under the stated null model, 18.75% of samples of four observations have a Pearson statistic at least as large as the one observed. At significance level \(\alpha=0.05\), this p-value does not provide sufficient evidence to reject the null model.

Why the Sum Is a Valid Tail Probability

The possible count vectors form a finite set: every coordinate is a nonnegative integer and their sum is \(n\). Under the null model, the multinomial probabilities of all these vectors sum to 1. This follows from the multinomial theorem:

$$ \sum_{\substack{x_1+\cdots+x_k=n\\x_i\ge 0}} \frac{n!}{x_1!\cdots x_k!}\prod_{i=1}^{k}p_i^{x_i} =(p_1+\cdots+p_k)^n=1. $$

Consequently, summing the probabilities of the vectors meeting the stated statistic threshold gives their probability under the null model. The procedure also explains why an exact test can be conservative at a chosen significance level: because the possible outcomes have discrete probabilities, attainable p-values may jump over \(\alpha\). The probability of rejecting at level \(\alpha\) is therefore no greater than \(\alpha\) when the model holds, but it need not equal \(\alpha\).

To see this, group the possible vectors by their statistic values, starting with the largest. For a vector at a particular statistic level, its p-value is the total probability at that level and all larger levels. The rejection rule \(p_{\mathrm{exact}}\le\alpha\) includes only upper groups whose cumulative probability is at most \(\alpha\). Thus the total null probability of rejection cannot exceed \(\alpha\). Tied vectors belong to the same statistic level and must be treated together under this rule.

Worked Example: A Two-Category Exact Test

Worked Example: A Two-Category Exact Test

In a hypothetical setting, six independent observations are classified into categories A and B. The null probabilities are \(0.50\) and \(0.50\), and the observed counts are \((6,0)\). Use the Pearson statistic to find the exact p-value.

The fixed sample size is \(n=6\), and the expected counts are \(E_A=E_B=3\). The observed statistic is

$$ T_{\mathrm{obs}}=\frac{(6-3)^2}{3}+\frac{(0-3)^2}{3}=3+3=6. $$

Let \(x\) be the count in A, so the possible vectors are \((x,6-x)\) for \(x=0,1,\ldots,6\). Their statistics and probabilities are:

\(x\)Count vector\(T\)Null probability
0(0, 6)61/64
1(1, 5)8/36/64
2(2, 4)2/315/64
3(3, 3)020/64
4(4, 2)2/315/64
5(5, 1)8/36/64
6(6, 0)61/64

For example, the probability of \(x=1\) is \(\binom{6}{1}(0.5)^1(0.5)^5=6/64\). Only the two endpoint vectors have \(T\ge6\), so

$$ p_{\mathrm{exact}}=\frac{1}{64}+\frac{1}{64} =\frac{2}{64}=0.03125. $$

At \(\alpha=0.05\), the result is statistically significant under this specified test. The exact p-value is the probability, assuming the null model, of a Pearson statistic at least 6—not the probability that the null hypothesis is true.

Worked Example: An Unequal Null Probability

Worked Example: An Unequal Null Probability

Suppose five independent observations fall into A or B, with null probabilities \(P(A)=0.20\) and \(P(B)=0.80\). The observed counts are \((0,5)\). Calculate the exact Pearson p-value.

The expected counts are \(E_A=5(0.20)=1\) and \(E_B=5(0.80)=4\). The observed statistic is

$$ T_{\mathrm{obs}} =\frac{(0-1)^2}{1}+\frac{(5-4)^2}{4} =1+0.25=1.25. $$

Let \(x\) denote the count in A. The statistic for vector \((x,5-x)\) simplifies to

$$ T(x,5-x)=\frac{(x-1)^2}{1}+\frac{(1-x)^2}{4} =1.25(x-1)^2. $$

The following table lists every possible value of \(x\), so the tail can be identified directly. Each probability is \(\binom{5}{x}(0.20)^x(0.80)^{5-x}\).

\(x\)\(T\)Null probabilityIncluded in tail?
01.250.32768Yes
100.40960No
21.250.20480Yes
350.05120Yes
411.250.00640Yes
5200.00032Yes

The observed statistic is 1.25, so both \(x=0\) and \(x=2\) are included: they tie at the threshold. Summing all included probabilities gives

$$ p_{\mathrm{exact}} =0.32768+0.20480+0.05120+0.00640+0.00032 =0.59040. $$

This is not the probability of observing a count of zero or fewer in A. The p-value is defined by the Pearson statistic, and an outcome with two observations in A is just as extreme by that statistic as the observed outcome.

Enumeration, Interpretation, and Common Pitfalls

For a small sample and a modest number of categories, the possible count vectors can be listed by assigning nonnegative integer counts that sum to \(n\). For each vector, calculate the statistic and its null probability; then sum probabilities for all vectors at or beyond the observed statistic. A probability table, careful hand enumeration, or software can perform the arithmetic, but the definition of the tail should remain explicit.

In larger problems, enumerating every vector may require substantial computation. This tutorial’s method is the exact finite enumeration procedure; simulation-based approaches are a separate method. Whatever computational tool is used, verify that the listed or selected outcomes satisfy the fixed-total constraint and that the tail includes every tie.

  • Omitting a tie. A vector with a statistic equal to the observed statistic belongs in the tail. Leaving it out understates the p-value.
  • Summing only the probability of the observed vector. The p-value sums probabilities over every count vector meeting the extremeness rule, not just the observed vector.
  • Using a one-sided count tail automatically. In more than two categories, “larger count in one category” is not a general definition of extremeness. Use the stated statistic and its threshold.
  • Calling the p-value the probability that the null is true. It is a probability of data outcomes calculated assuming the null model, not a probability of the hypothesis.
  • Assuming exact means assumption-free. The calculation is exact only under the specified probability model and sampling assumptions.
Key takeaway: State the fixed-probability null model and the statistic first. Then identify every possible count vector at least as extreme as the observed vector, include ties, and sum their multinomial probabilities.

Check Your Understanding

Use the definition of the exact p-value and the stated statistic to answer these questions.

  1. Why must the counts in a possible vector sum to the fixed sample size \(n\)?
  2. For a Pearson exact test, should a vector with statistic equal to the observed statistic be included in the p-value sum? Explain.
  3. Under a null model with three category probabilities \(0.2\), \(0.3\), and \(0.5\), what are the expected counts when \(n=10\)?
  4. What assumptions make the multinomial probability formula appropriate for a goodness-of-fit calculation?
  5. Explain why an exact p-value need not be equal to the significance level when the null model is true.

Previous

Next

Back to Tutorials