Sparse Data and Unreliable Chi-Square Fit Tests

Why Sparse Counts Matter

A chi-square fit statistic can be computed even when some fitted counts are very small. The harder question is whether its usual chi-square reference distribution is then a credible way to judge the statistic. Sparse counts matter because the approximation relies on residuals behaving sufficiently like normal variables; when expected counts are small, their distributions can be strongly discrete and asymmetric.

This issue arises in grouped-binomial and Poisson GLMs. For a grouped-binomial observation \(Y_i\sim\operatorname{Binomial}(m_i,\pi_i)\), the fitted expected numbers of successes and failures are \(m_i\hat{\pi}_i\) and \(m_i(1-\hat{\pi}_i)\). For a Poisson observation \(Y_i\sim\operatorname{Poisson}(\mu_i)\), the fitted expected count is \(\hat{\mu}_i\). If one or more of these quantities are small, the ordinary chi-square approximation may be inaccurate even when the statistic and its nominal residual degrees of freedom have been calculated correctly.

Definition: A fit assessment is sparse when fitted expected counts relevant to its reference approximation are small enough that the count distributions remain noticeably discrete or skewed. Sparsity is a warning about the calibration of the test, not by itself evidence that the model fits poorly.

Rules such as “all expected counts should be at least five” are useful warnings, not universal theorems. Approximation quality depends on the model, the number and arrangement of observations, and how parameters were estimated. An expected count just above a rule-of-thumb cutoff does not guarantee a reliable test; a few small counts do not by themselves prove that every possible approximation fails. The appropriate response is to examine the design and, when needed, calibrate the statistic under the fitted model.

What the Expected Counts Reveal

In a grouped-binomial model, the fitted variance is \(m_i\hat{\pi}_i(1-\hat{\pi}_i)\). The usual Pearson contribution for group \(i\) is therefore

X_i^2=\frac{(y_i-m_i\hat{\pi}_i)^2}{m_i\hat{\pi}_i(1-\hat{\pi}_i)}.

This expression is equivalent to adding the Pearson contributions for the success and failure cells. To see why, write \(e_i=m_i\hat{\pi}_i\) and \(f_i=m_i(1-\hat{\pi}_i)\). The observed cells are \(y_i\) and \(m_i-y_i\), so their contributions sum to

\frac{(y_i-e_i)^2}{e_i}+ \frac{((m_i-y_i)-f_i)^2}{f_i} = (y_i-e_i)^2\left(\frac{1}{e_i}+\frac{1}{f_i}\right) = \frac{(y_i-e_i)^2}{m_i\hat{\pi}_i(1-\hat{\pi}_i)}.

The last equality uses \(e_i+f_i=m_i\). Thus, a very small expected number of successes or failures signals that the corresponding binomial distribution may be poorly approximated by a normal distribution. In a Poisson model, a small \(\hat{\mu}_i\) likewise means that zero counts are common and the distribution is far from symmetric.

Result: The chi-square reference for a Pearson or deviance fit statistic is an approximation, not an automatic consequence of computing the statistic. Its standard large-sample justification requires a correctly specified model, regular parameter estimation, and a design and count regime in which the relevant residual contributions are adequately approximated by their limiting behavior. Sparse expected counts are a reason to doubt that approximation; the nominal degrees of freedom do not repair it.

The usual residual-degrees-of-freedom calculation still describes the dimension of the residual comparison under the stated model structure. It does not certify that the statistic has a chi-square distribution in a sparse sample. Nor does a large residual degrees of freedom guarantee accuracy: many small, discrete contributions do not necessarily behave like the sum of many squared standard normal variables.

Worked Example: Sparse Grouped-Binomial Counts

Suppose eight independent groups each have \(m_i=10\) trials and share an intercept-only binomial probability. The observed success counts are \(0,0,1,0,0,0,0,1\). Estimate the common probability and calculate the Pearson statistic.

There are \(2\) successes among \(80\) trials, so the maximum-likelihood estimate is \(\hat{\pi}=2/80=0.025\). Each group has fitted expected successes \(10(0.025)=0.25\) and expected failures \(10(0.975)=9.75\). The expected success counts are sparse in every group.

For a group with zero observed successes, the contribution is \( (0-0.25)^2/[10(0.025)(0.975)]=0.0625/0.24375\approx0.2564\). There are six such groups, contributing about \(1.5385\). For a group with one success, the contribution is \( (1-0.25)^2/0.24375=0.5625/0.24375\approx2.3077\). There are two such groups, contributing about \(4.6154\). Hence \(X^2\approx6.1539\). With eight groups and one fitted parameter, the conventional residual degrees of freedom are \(8-1=7\).

The arithmetic is valid, but comparing \(6.1539\) with a chi-square distribution on seven degrees of freedom may not be. The fitted expected successes are only \(0.25\) per group, so the observed outcomes are mostly zeros with occasional ones. The calculation alone cannot establish that the nominal tail probability is well calibrated.

Worked Example: Sparse Poisson Counts

Consider twelve independent Poisson observations in an intercept-only model. The observed counts are ten zeros and two ones. Find the fitted mean and Pearson statistic, and assess the expected-count warning.

The maximum-likelihood estimate is the sample mean, \(\hat{\mu}=2/12=1/6\). Each observation therefore has fitted expected count \(1/6\), well below five. A zero contributes \( (0-1/6)^2/(1/6)=1/6\); a one contributes \( (1-1/6)^2/(1/6)=(25/36)(6)=25/6\).

The total is \(10(1/6)+2(25/6)=60/6=10\). The conventional residual degrees of freedom are \(12-1=11\). But under the fitted model, the probability of zero for one observation is \(e^{-1/6}\approx0.8465\). Zeros are therefore expected to dominate, and a normal-residual interpretation of the individual contributions is questionable. A chi-square tail area should not be treated as reliable merely because eleven residual degrees of freedom are available.

Simulation as a Reference Distribution

When the usual reference approximation is doubtful, simulate datasets under the fitted null model and compare their fit statistics with the observed statistic. The simulation must reproduce the design: for example, retain the binomial trial counts and covariates, or the Poisson covariates and exposures. It must also repeat the parameter estimation used in the original analysis. Otherwise, simulated statistics and the observed statistic do not represent the same procedure.

Fit the null model.
Record fitted probabilities or means, the design and offsets, and the chosen discrepancy statistic, such as Pearson \(X^2\) or deviance.

Generate replicate data.
For each replicate, draw binomial or Poisson outcomes from the fitted model while keeping the observed design fixed.

Repeat the fitting procedure.
Refit the same model to each replicate, then recompute the same discrepancy. This accounts for the fact that the statistic in the observed data uses estimated parameters.

Compare the statistics.
Estimate the upper-tail probability by the fraction of replicate statistics at least as large as the observed statistic, and report the number of replicates and the resulting Monte Carlo uncertainty.

The logic of the comparison is exact when the null parameter is known and the observed statistic and simulated statistics are exchangeable under that null. For \(B\) simulated statistics \(T_1,\ldots,T_B\) and observed statistic \(T_0\), with no ties, the rank of \(T_0\) among the \(B+1\) exchangeable values is uniform. Therefore

\frac{1+\sum_{b=1}^{B} I(T_b\geq T_0)}{B+1}

is a valid Monte Carlo upper-tail p-value under that known-parameter null. With ties, this non-randomized upper-tail calculation is conservative. In an ordinary GLM, parameters are estimated, so simulation from the fitted values with refitting is generally a plug-in parametric bootstrap, not an exact test. It can provide a more appropriate calibration than a questionable chi-square approximation, but its accuracy still depends on the model and the quality of the fitted null. A small number of simulations also adds Monte Carlo error.

When Aggregation Is Defensible

Aggregation can help when it follows from the model and scientific question, rather than being chosen solely to push expected counts over a cutoff. For example, independent Poisson variables with means \(\mu_1,\ldots,\mu_k\) have a Poisson sum with mean \(\sum_{j=1}^k\mu_j\). This follows from multiplying their probability generating functions: the function for \(Y_j\) is \(\exp\{\mu_j(t-1)\}\), so the sum has function

\prod_{j=1}^{k}\exp\{\mu_j(t-1)\} = \exp\left\{\left(\sum_{j=1}^{k}\mu_j\right)(t-1)\right\}.

That is the probability generating function of a Poisson variable with mean \(\sum_j\mu_j\), establishing the claimed distribution. Similarly, independent binomial counts with the same success probability sum to a binomial count with the combined number of trials and that probability. These identities justify some aggregations; they do not imply that every pooling scheme makes a useful goodness-of-fit test.

Worked Example: Pooling Homogeneous Poisson Observations

Suppose fifty independent observations have the same fitted Poisson mean \(0.6\). Their expected counts are sparse. Before examining the outcomes, partition them into five sets of ten observations and sum within each set. Suppose the five totals are \(4,7,5,8,6\).

Each total has a Poisson distribution with mean \(10(0.6)=6\), by the sum result above. The fitted common mean for the five totals is \((4+7+5+8+6)/5=6\). Their Pearson statistic is \( [(4-6)^2+(7-6)^2+(5-6)^2+(8-6)^2+(6-6)^2]/6 =(4+1+1+4+0)/6=10/6\approx1.667\).

The aggregation has produced five count totals with expected value six each, but it changes what can be detected: it hides differences among the ten observations inside each set. The partition must therefore be substantively and statistically defensible, and the loss of within-set information must be acknowledged. Pooling is not a way to manufacture evidence of fit.

Common Mistakes and Practical Interpretation

Treating a cutoff as a guarantee. Having every expected count above five does not prove that the chi-square approximation is accurate. Use the cutoff as a diagnostic prompt, not a certificate.
Confusing a small expected count with poor fit. Sparsity concerns the reference distribution. It can occur when the model is correct, particularly for rare outcomes.
Simulating without refitting. If the observed statistic uses estimated parameters but simulated statistics use the original estimates as if known, the calibration does not match the analysis. Refit each simulated dataset.
Aggregating after seeing inconvenient counts. Data-driven pooling can conceal a local lack of fit and make the assessment difficult to interpret. Specify groups from the design or scientific question, and state what the grouping loses.
Assuming a nominal p-value settles the issue. When counts are sparse, report the limitation and consider a parametric simulation check alongside plots, residuals, or other model diagnostics.

The practical question is not simply “Are the expected counts small?” It is “Could the discreteness and skewness of these counts materially change the reference distribution for this statistic?” If the answer is plausible, simulation under the fitted model is often the clearest way to assess the approximation. Aggregation may be useful when supported by the model, but should preserve an interpretable question and should not obscure meaningful patterns.

Check Your Understanding

Answer each question by connecting the expected counts to the interpretation of the fit assessment.

In a grouped-binomial GLM, why can a small fitted number of successes or failures make a chi-square reference questionable?
A Poisson GLM has several fitted means below one, but its Pearson statistic and residual degrees of freedom are readily computed. What can and cannot be concluded from that calculation?
Why should a parametric simulation refit the model to every generated dataset?
Under what condition does the sum of independent binomial counts have a binomial distribution, and what condition supports summing independent Poisson counts?
Give one important feature of the data that aggregation may hide, even when the summed distribution is valid.