Recognizing Underdispersion

What Underdispersion Means

A generalized linear model (GLM) specifies both a relationship between predictors and the response mean and a variance structure for the response. A residual pattern with less variation than that structure predicts may indicate underdispersion. Recognizing it matters because inference and goodness-of-fit procedures depend on the assumed variability, not only on the fitted means.

The comparison is relative to the model’s variance function. For example, a Poisson GLM assumes that the conditional variance equals the conditional mean. A binomial GLM has a variance determined by its binomial size and success probability. If the responses vary less than the relevant model structure predicts, residual statistics may be systematically small. A small statistic is a diagnostic signal, not by itself an explanation of why the variation is low.

The dispersion convention depends on the response family. In a quasi-likelihood model, \(\phi\) may be estimated rather than fixed. In standard Poisson and binomial likelihood GLMs, the family typically fixes \(\phi=1\); a residual ratio below one is therefore a diagnostic comparison with that assumed value, not an automatic replacement of the model’s variance specification. Independence, a suitable mean structure, and an appropriate variance function are important assumptions for interpreting these diagnostics.

Residual Ratios as Evidence

The Pearson residual for observation \(i\), when the fitted dispersion is not included in the denominator, is

The Pearson statistic is the sum of the squared Pearson residuals. If a model has \(n\) observations and \(p\) estimated regression coefficients, its residual degrees of freedom are commonly \(\nu=n-p\). Dividing the statistic by \(\nu\) gives a residual dispersion ratio:

In a model with a free dispersion parameter, \(\hat{\phi}_P\) is often used as an estimate of that parameter. In a fixed-dispersion model, it is still useful as a diagnostic summary. A value below 1 indicates that the total squared Pearson residual is below the rough reference level \(\nu\). A value far below 1 is more suggestive of underdispersion than a value just below 1, but there is no universal cutoff that proves it.

The residual deviance provides a related diagnostic. Its ratio to residual degrees of freedom is \(D/\nu\). Under regularity conditions, the residual deviance and Pearson statistic can both be approximately compared with residual degrees of freedom. These are approximations, not identities: their values need not match, particularly with small samples, sparse outcomes, or influential observations.

Why the Residual Ratio Is Near the Dispersion

A short justification explains why \(X_P^2/\nu\) is informative. If the means were known and the variance model were correct, each standardized residual would have variance approximately \(\phi\). Fitting regression coefficients uses information from the responses, however, so the fitted residuals are constrained and their total variation is reduced. In a local weighted least-squares approximation to a GLM, this reduction is represented by the projection matrix \(H\), whose trace is \(p\) when \(p\) coefficients are fitted.

The first approximation uses the fitted model’s local linearization and assumes independent observations with the specified variance structure. The trace calculation follows from \(\operatorname{tr}(I-H)=n-\operatorname{tr}(H)\). Thus \(E[X_P^2/\nu]\) is approximately \(\phi\). This does not make the ratio an exact unbiased estimate in every GLM; it explains its diagnostic role and why degrees of freedom matter.

Calculating the Statistic From Observations

Each observation contributes its squared difference from the fitted mean divided by the model variance function evaluated at that mean. For a Poisson response, \(V(\mu_i)=\mu_i\), so its contribution is \((y_i-\hat{\mu}_i)^2/\hat{\mu}_i\). For a binomial response, the appropriate variance depends on the number of trials and fitted success probability; using a Poisson denominator for binomial data would be incorrect.

Effects on Standard Errors and Goodness-of-Fit Conclusions

The effect on standard errors depends on the inferential procedure. In a model that estimates a common dispersion parameter, the covariance estimate for the regression coefficients is often proportional to \(\hat{\phi}\). Relative to using \(\phi=1\), a value below 1 reduces estimated standard errors by the factor \(\sqrt{\hat{\phi}}\), if the common-scale variance model is appropriate. A fixed-dispersion Poisson or binomial likelihood model does not automatically make this adjustment just because a diagnostic ratio is small. Its standard errors remain those from the specified likelihood model unless the analyst deliberately adopts a different variance model or method.

Consequently, an uncritical scale adjustment can create overconfident inference if the low ratio is caused by misspecification, dependence, or an inappropriate dispersion estimate. Conversely, if the assumed variance is genuinely too large and a suitable model supports a lower dispersion, leaving that discrepancy unaddressed can make fixed-\(\phi=1\) standard errors larger than warranted. The ratio alone cannot decide which interpretation is justified.

Goodness-of-fit interpretation also requires care. A low residual deviance or Pearson statistic is not evidence of lack of fit in the same way that an unusually large statistic can be evidence against a model. If the assumed variability is greater than the actual variability, an unscaled upper-tail test can be conservative: it may produce large p-values more often than its nominal calibration suggests. In a quasi-model, dividing a statistic by a dispersion estimate below 1 increases the scaled statistic. But when the same data estimate that scale, the simple chi-square reference may not apply exactly. The chosen model and inferential framework determine the calibration.

Common Causes and Exam Traps

A ratio below 1 should lead to questions about the model and the observations, not to an automatic declaration that the GLM is “better than expected.” Possible explanations include a response mechanism with less variation than the chosen family, dependence among observations, data aggregation, constrained outcomes, or a mean model that uses the data unusually closely. These are possibilities to investigate, not diagnoses made from the ratio alone. A single global scale factor can be especially misleading if variability differs across observations.

• Confusing low variation with correct fit. A model can have small residual variation and still have a wrong mean relationship or other structural problem.
• Ignoring the response family. A Poisson model’s reference variance is its mean; a binomial model’s variance is not generally the mean.
• Using the wrong degrees of freedom. For the common residual calculation, use \(n-p\), not \(n\), when \(p\) coefficients were estimated.
• Applying a scale adjustment automatically. A diagnostic ratio does not change the assumptions of a fixed-dispersion likelihood model by itself.
• Treating a low goodness-of-fit statistic as proof. Low statistics can make conventional upper-tail goodness-of-fit tests conservative; they do not certify the model.