Sparse Data and Small-Sample Deviance Calibration

Why Sparse Data Need a Calibration Check

A deviance difference can be calculated exactly from fitted likelihoods, yet its reference distribution under the null is usually approximate. In regular, sufficiently informative problems, Wilks’ theorem gives a chi-square limit. Sparse data can make that approximation poor: the likelihood may be far from quadratic near its maximum, some fitted probabilities may be close to zero or one, or a few observations may carry most of the information.

This distinction matters in practice. A reported deviance difference is a statistic; its p-value also depends on a reference distribution. If the chi-square approximation is inaccurate, the p-value can suggest more or less evidence against the null than the data warrant. There is no universal expected-count threshold that guarantees a good approximation in every GLM. The structure of the model, the number of sparse groups, and the fitted values all matter.

For a nested-model likelihood-ratio test, the statistic is the difference in deviances, equivalently twice the difference in maximized log-likelihoods. The usual chi-square calibration is justified asymptotically when the null is true, the models are regular and identifiable, the null parameter lies in the interior of the larger model’s parameter space, the relevant information grows, and the dimensions being compared remain fixed. Under those conditions, the statistic converges in distribution to a chi-square random variable with degrees of freedom equal to the dimension difference. Sparse samples can make this large-sample approximation inaccurate even when the statistic and its nominal degrees of freedom are computed correctly.

Why Small Expected Counts Are a Warning

One reason chi-square approximations arise is that, near its maximum, a log-likelihood can often be approximated by a quadratic function. A quadratic approximation is most useful when the data contain enough information for the likelihood to be locally smooth and well concentrated. In sparse settings, the observed counts may be far from their expected values relative to the expected values themselves. Then the local quadratic description can be poor.

The Poisson deviance contribution for an observed count \(y\) and fitted mean \(\mu>0\) is \(2\{y\log(y/\mu)-(y-\mu)\}\), with \(0\log 0=0\). Its relation to the Pearson contribution can be seen by writing \(y=\mu(1+u)\), where \(u=(y-\mu)/\mu\). For \(u\) near zero, a Taylor expansion gives \(2\mu\{(1+u)\log(1+u)-u\}\approx \mu u^2=(y-\mu)^2/\mu\). The approximation relies on \(u\) being small; when \(y=0\), \(u=-1\), which is not a small deviation.

Worked Comparison: Exact and Chi-Square Calibration

A small binomial problem gives a direct way to see how calibration can change a p-value. Consider \(X\sim\operatorname{Binomial}(10,p)\) and test \(H_0:p=0.5\) against the unrestricted alternative. The saturated model estimates \(p\) by \(\hat p=X/10\). For an observation \(x\), the likelihood-ratio deviance is

where terms with a zero count are interpreted by continuity.

Extreme Fitted Probabilities

In binomial and binary-response GLMs, extreme fitted probabilities are another warning sign. If a fitted probability is near zero or one, the likelihood can be highly asymmetric, and a coefficient estimate may depend heavily on a small number of events. In some data sets, an unrestricted logistic model has no finite maximum-likelihood coefficient because a predictor separates successes from failures. The likelihood-ratio statistic may still have a finite limiting value, but the usual regularity conditions behind the chi-square approximation do not automatically hold.

A boundary estimate is not by itself a proof that a particular p-value is badly wrong. It is a reason to examine the data and the assumptions behind the reference distribution rather than treating the chi-square result as exact. The relevant question is whether the sample supplies the regular, growing information that makes the asymptotic approximation credible.

What a Small-Sample P-Value Says

A p-value is a tail probability computed under a specified null model and a specified reference distribution. With a chi-square calibration, it is the probability, under that chi-square distribution, of a statistic at least as large as the observed statistic. It is not the probability that the null hypothesis is true, nor does it express the probability that the observed data are “due to chance.”

In a sparse GLM, the chi-square p-value should be understood as an asymptotic approximation whose accuracy needs assessment. A small value does not repair a poor approximation, and a large value does not establish that the null model is correct. If calibration is questionable, useful checks include examining observed and fitted counts, identifying groups with fitted probabilities near zero or one, and comparing with an appropriate exact or null parametric simulation calculation where feasible. Such checks assess the reference distribution; they do not change the observed deviance statistic itself.

Common Mistake

A common mistake is to treat a chi-square p-value as exact because the software reports a numerical value with several decimal places. For example, seeing \(p=0.0086\) in the two-group example does not mean that the null probability of a result this extreme is known to four decimal places. The displayed precision belongs to the calculation under the chosen chi-square reference distribution; it does not measure the accuracy of that approximation. The correction is to distinguish the computed statistic from its calibration, inspect the sparsity and fitted values, and describe the resulting p-value as approximate when regularity is doubtful.