Sparse Expected Counts and Approximation Failure

Why Expected Counts Matter

A Pearson chi-square goodness-of-fit test compares observed category counts with the counts expected under a null probability model. Its statistic is calculated from those differences, but its p-value usually comes from a chi-square reference distribution: an approximation to the statistic’s distribution when the null model is true. The approximation works better when the expected counts are not too small.

This distinction matters in practice. A test can be calculated mechanically even when its reference distribution is a poor fit. In that case, the resulting p-value may not accurately represent how unusual the data would be under the null model. Checking expected counts is therefore part of deciding whether the usual chi-square p-value is defensible, not just a calculation to complete after the test.

Definition: For category \(i\), the expected count under a null model is \(E_i=np_i\), where \(n\) is the total number of observations and \(p_i\) is the null probability for that category. It is a model-based average count, not the observed count \(O_i\). A count is called sparse in this context when one or more expected counts are small enough to make the chi-square approximation questionable.

For the usual one-way goodness-of-fit analysis, the model assumes independent observations, each assigned to exactly one of the specified categories, and null probabilities that describe those categories. The probabilities must sum to 1. Under a fixed-probability null model, the Pearson statistic is

X^2=\sum_{i=1}^{k}\frac{(O_i-E_i)^2}{E_i}.

The statistic measures how far the observed counts are from their expected counts relative to the expected-count scale. When the model probabilities are fixed and the conditions for chi-square calibration are adequate, it is compared with a chi-square distribution with \(k-1\) degrees of freedom. If probabilities are estimated, the degrees of freedom must account for that estimation, as covered in the earlier material on degrees of freedom when probabilities are estimated. Correcting the degrees of freedom does not, by itself, fix sparse expected counts.

Guidelines, Not Guarantees

A common guideline for using the chi-square approximation is that no expected count should be below 1 and no more than 20% of expected counts should be below 5. A stricter, easy-to-apply guideline is that every expected count should be at least 5. These are rules of thumb, not mathematical guarantees. They flag situations that deserve caution; they do not establish that every passing test has an accurate p-value or that every failing test must be unusable.

The relevant quantities are the expected counts calculated from the null model. A small observed count does not, by itself, establish that the approximation fails. Conversely, a sample can have a large total \(n\) and still have sparse expected counts if the null model assigns very small probabilities to some categories. Always examine the full set of expected counts, not only the total sample size or the largest discrepancy in the data.

Worked Example: A Rare Category With a Sparse Expected Count

Suppose 20 independent, hypothetical observations are classified into four exhaustive categories. The null probabilities are \(0.70\), \(0.20\), \(0.08\), and \(0.02\). The observed counts are \(15, 4, 1,\) and \(0\). Check the expected counts and assess whether the usual chi-square reference p-value is dependable.

Calculate the expected counts. Multiplying each null probability by \(n=20\) gives

$$ (E_1,E_2,E_3,E_4) =(20(0.70),20(0.20),20(0.08),20(0.02)) =(14,4,1.6,0.4). $$

One expected count is below 1, and three of the four expected counts are below 5. The guideline is plainly not met. The observed zero in the last category is not the diagnostic; the expected count of \(0.4\) is.

For completeness, the Pearson statistic is

X^2=\frac{(15-14)^2}{14} +\frac{(4-4)^2}{4} +\frac{(1-1.6)^2}{1.6} +\frac{(0-0.4)^2}{0.4} =0.0714+0+0.225+0.400 =0.696.

If the probabilities are fixed, the nominal degrees of freedom are \(4-1=3\). But computing \(X^2\) and identifying those degrees of freedom does not make the chi-square approximation reliable. A p-value obtained from that reference distribution should not be presented as if its calibration were secure. A finite-sample method based on the null model, or more data if feasible, would be more defensible.

Why Sparse Counts Cause Trouble

The approximation relies on the statistic behaving sufficiently like a chi-square random variable under the null model. A useful way to see the difficulty is to consider one category with null probability \(p\). Its observed count \(O\) has a binomial marginal distribution with \(n\) trials and probability \(p\), and its expected count is \(E=np\). In particular, the probability of seeing no observations in that category is

$$ P(O=0)=(1-p)^n. $$

When \(p\) is small, this probability can be substantial even if the null model is true. For example, with \(n=20\) and \(p=0.02\), \(E=0.4\), and

P(O=0)=(1-0.02)^{20}\approx 0.668.

So a zero count in that rare category would be quite ordinary under the null model. More generally, small expected counts produce discrete counts with limited possible values, rather than the smooth behavior that supports the chi-square approximation. This is one reason a statistic’s chi-square reference tail probability can be inaccurate. The direction and size of the error are not reliably fixed: the nominal p-value might be too small or too large.

This calculation is a justification for treating very small expectations cautiously, not a derivation of a universal cutoff. The quality of an approximation depends on the full null model and the statistic’s distribution. A guideline is a practical warning system, not a theorem about an exact boundary between valid and invalid tests.

Worked Example: Adequate Expected Counts in a Specified Model

In a hypothetical setting, 120 independent observations are assigned to three exhaustive categories. The null probabilities are \(0.50\), \(0.30\), and \(0.20\), and the observed counts are \(54, 42,\) and \(24\). Assess the expected-count condition and calculate the usual test statistic.

Check the model and expectations. The observations are assumed independent, the categories are mutually exclusive and exhaustive, and the specified probabilities sum to 1. The expected counts are

$$ (E_1,E_2,E_3) =(120(0.50),120(0.30),120(0.20)) =(60,36,24). $$

Every expected count is at least 5, so the common expected-count guideline is satisfied. This supports, but does not guarantee, using the chi-square reference distribution; the independence and category assumptions must also be reasonable.

The Pearson statistic is

X^2=\frac{(54-60)^2}{60} +\frac{(42-36)^2}{36} +\frac{(24-24)^2}{24} =\frac{36}{60}+\frac{36}{36}+0 =1.60.

With fixed null probabilities, \(df=3-1=2\). The chi-square reference distribution with 2 degrees of freedom gives an approximate p-value of \(P(\chi^2_2\ge 1.60)\approx 0.449\). The data do not provide strong evidence against this null model. The word “approximate” remains appropriate: the reference distribution is an approximation, even when the expected-count guideline is met.

What to Do When Counts Are Sparse

When expected counts are sparse, first check that the calculation reflects the intended null model: the categories, probabilities, total \(n\), and observed counts must all be correct. Then decide whether a chi-square approximation is adequate for the question at hand. If it is not, there are several defensible responses, depending on the design and scientific purpose.

Use exact inference. For a fully specified categorical null model, an exact test can evaluate the observed result against the finite-sample distribution implied by that model, rather than relying on the chi-square approximation. Exact inference still requires appropriate assumptions about the observations and null probabilities, and the test’s definition of outcomes as at least as extreme must be made clear.
Use simulation-based inference. Repeatedly generating data under the null model can approximate the statistic’s null distribution when a simple reference approximation is doubtful. The simulated p-value has Monte Carlo uncertainty, so the simulation must be large enough for the precision needed. Simulation does not repair an inappropriate null model or dependent observations.
Collect more data if practical. Increasing \(n\) can increase expected counts while leaving the null probabilities and categories unchanged. Whether this is feasible depends on the study, and collecting more data does not fix violations such as non-independent observations.
Report the limitation transparently. If no suitable alternative is available, report the observed counts and expected counts and explain why the usual approximate p-value may be unreliable. Do not present a questionable approximation as a definitive result.

These approaches answer a common practical question—what to do when the usual calibration is doubtful—without making a small-count rule into a substitute for judgment. The exact and simulation-based methods have their own assumptions and implementation details; the key decision here is recognizing when the ordinary chi-square reference distribution needs to be questioned.

Worked Example: Choosing a Defensible Route for Very Small Counts

A hypothetical pilot study records eight independent observations in four exhaustive categories. Under a specified null model, all four categories have probability \(0.25\), and the observed counts are \(5,2,1,\) and \(0\). What does the expected-count check imply, and how should the result be handled?

Calculate expected counts. Each expected count is \(8(0.25)=2\), so none reaches 5. The expected-count guideline is not met. The assumptions stated for this example—independent observations, one category per observation, and specified null probabilities—support the categorical model, but do not make the chi-square approximation adequate.

The statistic, calculated for comparison, is

X^2=\frac{(5-2)^2}{2} +\frac{(2-2)^2}{2} +\frac{(1-2)^2}{2} +\frac{(0-2)^2}{2} =4.5+0+0.5+2 =7.00.

For fixed probabilities, the nominal degrees of freedom are \(4-1=3\). A chi-square reference calculation would produce an approximate p-value of about \(0.072\), but the expected counts indicate that this approximation may be poorly calibrated. That number alone is not a secure basis for a conclusion.

A defensible next step is to use an exact method based on the specified null probabilities, or a simulation-based method that approximates the statistic’s null distribution under the same model. Either route should use the same categories and test statistic chosen for the scientific question. The calculation above remains a useful description of discrepancy, but its conventional chi-square p-value should be labelled unreliable rather than treated as decisive.

Common Mistakes

Checking observed counts instead of expected counts. The approximation diagnostic is based on counts implied by the null probabilities. A small observed count can occur even when its expected count is adequate; a large observed count does not make a sparse expected count disappear.
Assuming a large total sample guarantees success. If the model gives some categories tiny probabilities, their expected counts can remain small even when \(n\) is large. Calculate every \(np_i\).
Treating a rule of thumb as an absolute theorem. Passing a guideline is evidence in favor of the approximation, not a guarantee. Failing it signals caution, not an automatic conclusion that every possible analysis is invalid.
Reporting a nominal p-value without qualification. A number from a chi-square table or software is only as trustworthy as the reference distribution used to obtain it. Check the expected counts before interpreting that number.
Thinking that adjusted degrees of freedom solve sparsity. Accounting for estimated probabilities is important, but it does not change the fact that a small expected count can undermine the reference approximation.

Key takeaway: Compute the null expected count in every category before relying on the Pearson chi-square p-value. Sparse expectations warn that its chi-square reference distribution may be inaccurate; when that warning matters, choose an appropriate exact or simulation-based analysis, or state the limitation clearly.

Check Your Understanding

Answer these questions using the distinction between expected counts and observed counts.

A null model assigns probability \(0.01\) to one category in a sample of 80. What is that category’s expected count, and why might a chi-square approximation be questionable?
Why does a zero observed count not, by itself, establish that the expected-count condition has failed?
A goodness-of-fit test has 10 categories and \(n=500\), but one null probability is \(0.001\). Calculate that category’s expected count and explain why the total sample size is not enough to assess the approximation.
What is the difference between an exact test and a chi-square-reference test when expected counts are sparse?
Does meeting a common expected-count guideline guarantee a trustworthy p-value? Explain what else must be checked.