Diagnosing Overdispersion in Poisson Models

Why Diagnose Overdispersion?

Poisson regression is often used for a count outcome \(Y_i\) conditional on covariates \(x_i\). Its conditional mean is typically written \(\mu_i=\exp(x_i'\beta)\). The Poisson model also imposes a variance restriction: conditional on the covariates, the variance equals the mean. If the observed counts vary more than this restriction allows, standard Poisson likelihood-based uncertainty estimates can be too small. But a diagnostic suggesting extra variation does not, by itself, identify its cause.

A key distinction is between excess conditional variance and a misspecified conditional mean. A fitted mean that systematically misses different parts of the data can create large residuals, even when the actual variance around the true conditional mean is Poisson. This tutorial uses dispersion summaries, residual patterns, and a score diagnostic to assess the issue without treating any single statistic as a definitive verdict.

For the usual diagnostic formulas, assume observations are conditionally independent, counts are nonnegative integers, and fitted means \(\hat{\mu}_i\) are positive. Let \(k\) be the number of estimated regression parameters, including an intercept if present. The residual degrees of freedom are commonly approximated by \(n-k\). This approximation does not automatically account for clustering, complex sampling, or other dependence.

Pearson Dispersion and Deviance

The Pearson residual for observation \(i\) is the observed-minus-fitted count scaled by its Poisson standard deviation. Squaring and summing these residuals gives the Pearson statistic. Dividing by residual degrees of freedom produces a dispersion ratio: values appreciably above one are a warning that the fitted residual variation exceeds the Poisson benchmark. Values below one can occur, especially in small samples or with a flexible fitted model; they do not establish that the model is correct.

A second summary is the Poisson deviance. It compares the fitted model with a saturated model, which assigns a separate fitted mean to each observation. The convention \(Y_i\log(Y_i/\hat{\mu}_i)=0\) when \(Y_i=0\) makes the formula defined at zero counts. The deviance ratio \(D/(n-k)\) is another rough dispersion check. Neither ratio has a universal cutoff: their reference behavior depends on adequate mean specification, sample size, and the design.

Residual Patterns and the Conditional Mean

A single dispersion ratio discards information about where the discrepancies occur. Inspect Pearson residuals against fitted means and important covariates. A systematic curve, trend, or grouping of positive and negative residuals can indicate that \(\hat{\mu}_i\) is wrong for part of the covariate space. Residuals that are unusually large across otherwise well-fitted regions may instead be consistent with excess conditional variation, although these explanations can coexist.

The reason mean misspecification can inflate a dispersion diagnostic follows from a conditional variance identity. Write the true conditional mean as \(m_i=E[Y_i\mid x_i]\), while the fitted mean is \(\mu_i^*\). Then

The second term is nonnegative. Consequently, squared residuals around the wrong mean can be large even if \(\operatorname{Var}(Y_i\mid x_i)=m_i\), the Poisson variance relation at the true mean. A dispersion ratio alone cannot separate this mean-error contribution from excess variance.

For independent Poisson observations with a correctly specified mean and suitable regularity conditions, deviance and Pearson residual summaries are often compared with their residual degrees of freedom as approximate reference benchmarks. The approximation can be poor in small samples, with sparse counts, or when some fitted means are very small. Inference also needs adjustment when observations are dependent; treating clustered counts as independent can make the diagnostic and its apparent precision misleading.

A Score-Based Check Under an Explicit Alternative

A score diagnostic asks whether the likelihood would increase, to first order, by moving away from the Poisson model in a specified direction. It must specify a probability model for that alternative: a mean and variance alone do not determine a likelihood score. One common choice is the NB2 family, parameterized by conditional mean \(\mu_i\) and \(\alpha\geq 0\), with variance \(\mu_i+\alpha\mu_i^2\). At \(\alpha=0\), its limiting distribution is Poisson. Here the purpose of the family is to define the score check, not to claim that it is the only explanation for extra variation.

The resulting observation-level score at the null is \(q_i=[(Y_i-\mu_i)^2-Y_i]/2\), and its null information is \(\mu_i^2/2\). These formulas are specific to the NB2 likelihood. To see the score, expand the log likelihood near \(\alpha=0\). The gamma-function ratio contributes \(y\log(1/\alpha)+\alpha y(y-1)/2+O(\alpha^2)\); the term \(y\log(\alpha\mu)\) cancels the first part. Expanding \(-(y+1/\alpha)\log(1+\alpha\mu)\) gives \(-\mu+\alpha(\mu^2/2-y\mu)+O(\alpha^2)\). Combining this with the gamma-ratio and \(y\log(\alpha\mu)\) terms gives the coefficient \(y(y-1)/2-y\mu+\mu^2/2=[(y-\mu)^2-y]/2\). Equivalently, differentiating the full NB2 log likelihood and taking its limit at zero yields this coefficient.

Under the Poisson null, the score has expectation zero. Its variance, equal to the expected information for this scalar parameter, is \(\mu_i^2/2\). For example, this follows by taking the variance of \(q_i\) under a Poisson distribution; the Poisson central moments give \(\operatorname{Var}(q_i)=\mu_i^2/2\). The expected cross-information between \(\alpha\) and the mean parameters is zero at the null, since the covariance of \(q_i\) with the Poisson mean score \(Y_i-\mu_i\) is zero. This supports evaluating the score and information at the fitted Poisson means under the usual regularity conditions.

Because the alternative allows \(\alpha>0\), evidence for overdispersion is in the positive direction: a one-sided score test uses the upper tail of the standard normal reference distribution, with approximate p-value \(1-\Phi(Z_{\text{score}})\). A negative score is not evidence for this positive-dispersion alternative. The approximation is asymptotic, so a large score from a tiny or highly influential sample should not be treated as reliable merely because the calculation is possible.

Putting the Diagnostics Together

Dispersion diagnostics matter because a Poisson variance restriction affects more than a descriptive fit: it underlies conventional likelihood-based uncertainty calculations. However, detecting a discrepancy is different from establishing its source. A practical assessment combines the scale summaries with the structure of the residuals and a clear account of the assumptions behind any test.

Common mistake: “The Pearson ratio is above one, so the Poisson conditional variance must be too small.” This conclusion does not follow. A misspecified conditional mean contributes the nonnegative squared-bias term \((m_i-\mu_i^*)^2\) to expected squared residuals, and dependence can also distort the usual benchmark. The correction is to describe an elevated ratio as a warning, examine residual patterns and model structure, and state the assumptions required to interpret a formal test.