Worked Diagnostic for Overdispersed Poisson Counts

From a Dispersion Ratio to a Diagnosis

A large Pearson or deviance ratio is a reason to investigate a Poisson regression, not a diagnosis by itself. The same numerical signal can accompany extra variation around a sensible mean function or arise because the fitted means miss a systematic pattern. Residual plots help separate these explanations: a broadly changing spread suggests one concern, while an ordered pattern in residual averages points toward another.

This tutorial works through both situations. The examples use independent counts $Y_i$ with fitted Poisson means $\hat{\mu}_i$. For a model with $n$ observations and $p$ fitted regression coefficients, the residual degrees of freedom are $n-p$. The Pearson and deviance ratios divide their respective residual statistics by those degrees of freedom. These ratios are useful summaries, but neither should be interpreted without the model, the residual plots, and the amount of information in the data.

Definition: For an observed count $y_i$ and fitted mean $\hat{\mu}_i>0$, the Pearson residual is $r_{Pi}=(y_i-\hat{\mu}_i)/\sqrt{\hat{\mu}_i}$. The Poisson deviance residual is $r_{Di}=\operatorname{sign}(y_i-\hat{\mu}_i)\sqrt{d_i}$, where $d_i=2\{y_i\log(y_i/\hat{\mu}_i)-(y_i-\hat{\mu}_i)\}$, with $0\log(0/\hat{\mu}_i)$ defined as 0. The Pearson statistic is $X_P^2=\sum_i r_{Pi}^2$, and the residual deviance is $D=\sum_i r_{Di}^2=\sum_i d_i$. Their dispersion ratios are $X_P^2/(n-p)$ and $D/(n-p)$, respectively.

Worked Example: A Poisson Fit With Large Residual Variation

Suppose 12 independent observations are fitted with a log-link Poisson regression containing an intercept and a linear predictor $x$. The fitted means are $\hat{\mu}_i=\exp(\hat{\beta}_0+\hat{\beta}_1x_i)$, with $\hat{\mu}=2,4,8$ at $x=0,1,2$, respectively. There are four observations at each predictor value:

Predictor $x$	Observed counts $y$	Fitted mean for each count
0	0, 1, 3, 4	2
1	1, 3, 5, 7	4
2	2, 5, 10, 15	8

The fitted means are consistent with a fitted Poisson regression, rather than being arbitrary values. The observed totals at the three predictor levels are 8, 16, and 32, exactly the corresponding totals of the fitted means. Thus the score contributions for the intercept and slope are both zero: the total residual is $0$, and the predictor-weighted residual total is $0(0)+1(0)+2(0)=0$. There are $n-p=12-2=10$ residual degrees of freedom.

For $x=0$, the four residuals $y-\hat{\mu}$ are $-2,-1,1,2$. Their squared values sum to $4+1+1+4=10$; dividing by the fitted mean 2 gives a Pearson contribution of 5. For $x=1$, the squared residuals sum to $9+1+1+9=20$, giving $20/4=5$. For $x=2$, they sum to $36+9+4+49=98$, giving $98/8=12.25$. Therefore

X_P^2=5+5+12.25=22.25,\qquad \frac{X_P^2}{n-p}=\frac{22.25}{10}=2.225.

The deviance can also be calculated level by level. Because the residuals sum to zero within each level, the terms $-(y-\hat{\mu})$ sum to zero there. For $x=0$, the deviance contribution is $2\{1\log(1/2)+3\log(3/2)+4\log(4/2)\}$, approximately $6.592$. For $x=1$, it is $2\{1\log(1/4)+3\log(3/4)+5\log(5/4)+7\log(7/4)\}$, approximately $5.567$. For $x=2$, it is $2\{2\log(2/8)+5\log(5/8)+10\log(10/8)+15\log(15/8)\}$, approximately $13.076$. Hence

D\approx 6.592+5.567+13.076=25.235,\qquad \frac{D}{n-p}\approx\frac{25.235}{10}=2.524.

Both ratios are appreciably above 1, although they are not identical. The residuals at each predictor level have both signs and sum to zero, so these totals do not show a consistent upward or downward shift in the fitted mean at those levels. But their magnitudes, especially at $x=2$, are substantial. A plot of Pearson residuals against fitted means would show increasing residual spread as the means rise; the plot against $x$ would show the same observations in predictor order. This pattern raises concern about variation, but should still be checked for influential observations and other structure.

Why Residual Patterns Can Identify Mean-Structure Problems

For a Poisson regression with log link, write the fitted mean as $\mu_i(\beta)=\exp(x_i^\mathsf{T}\beta)$, where $x_i$ is the predictor vector for observation $i$. At an interior maximum-likelihood estimate, the residuals obey a useful score identity.

Result: If the independent responses have Poisson means $\mu_i(\beta)=\exp(x_i^\mathsf{T}\beta)$, and a finite interior maximum-likelihood estimate $\hat{\beta}$ exists, then $$ \sum_{i=1}^n x_i(y_i-\hat{\mu}_i)=0. $$ That is, the ordinary residual vector is orthogonal to each fitted predictor column.

To prove the identity, the log-likelihood, up to terms independent of $\beta$, is $\ell(\beta)=\sum_i\{y_i x_i^\mathsf{T}\beta-\exp(x_i^\mathsf{T}\beta)\}$. Differentiating with respect to $\beta$ gives

\frac{\partial\ell(\beta)}{\partial\beta} =\sum_{i=1}^n x_i\{y_i-\exp(x_i^\mathsf{T}\beta)\}.

At a finite interior maximum, the score is zero. Substituting $\hat{\beta}$ therefore gives $\sum_i x_i(y_i-\hat{\mu}_i)=0$, as claimed. For an intercept, this implies that the residuals sum to zero overall. For a predictor column, it implies a corresponding weighted sum is zero. These constraints do not imply that residual averages must be zero in every subgroup or at every predictor value: the fitted regression imposes only the constraints represented by its columns. A plot can reveal patterns those constraints do not remove. $\square$

Worked Example: A Large Ratio With an Ordered Residual Pattern

Now consider eight observations at $x=0,1,2,3$, with two observations at each level. Fit the log-linear Poisson model whose fitted means at those levels are $2,4,8,16$, respectively. The observed counts are:

$x$	Observed counts	Fitted mean for each count
0	5, 3	2
1	0, 0	4
2	10, 10	8
3	15, 17	16

The observed totals by level are $8,0,20,32$, while the corresponding fitted totals are $4,8,16,32$. Thus the levelwise residual totals are $4,-8,4,0$. Their sum is $0$, and their predictor-weighted sum is $0(4)+1(-8)+2(4)+3(0)=0$. The score equations are satisfied for an intercept and slope, so these fitted means are consistent with the maximum-likelihood fit. Here $n-p=8-2=6$.

The Pearson contributions at $x=0,1,2,3$ are, respectively, $(3^2+1^2)/2=5$, $(4^2+4^2)/4=8$, $(2^2+2^2)/8=1$, and $(1^2+1^2)/16=0.125$. The total and ratio are

X_P^2=5+8+1+0.125=14.125,\qquad \frac{X_P^2}{6}\approx2.354.

For the deviance, the contributions at $x=0$ are $2\{5\log(5/2)-3+3\log(3/2)-1\}\approx3.596$. At $x=1$, each zero count contributes $2\{0-(0-4)\}=8$, for a total of 16. At $x=2$, each count of 10 contributes $2\{10\log(10/8)-2\}\approx0.463$, for a total of $0.926$. At $x=3$, the counts 15 and 17 contribute approximately $0.064$ and $0.061$, for a total of $0.125$. Thus $D\approx20.647$, and $D/6\approx3.441$.

The ratios are large, but the plot supplies important additional evidence. The average Pearson residual at each predictor level is approximately $1.414,-2,0.707,0$, in order. The positive, negative, positive pattern across the first three levels is systematic: the fitted means are too low, too high, and too low in those regions, respectively. Plotting against fitted means shows the same pattern, since the fitted means increase with $x$. This points to a possible mean-structure problem, even though the score equations are satisfied. A dispersion ratio alone would not reveal that distinction.

Deviance and Pearson Contributions on a Common Scale

The Pearson and deviance statistics are not identical. Their contributions nevertheless have a precise local relationship when an observed count is close to its fitted mean.

Result: For $y>0$ and $\mu>0$, let $t=y/\mu$. The Poisson deviance contribution is nonnegative, and as $t$ approaches 1, $$ \frac{2\{y\log(y/\mu)-(y-\mu)\}}{(y-\mu)^2/\mu}\longrightarrow 1. $$ Thus, for counts close to their fitted means, an individual deviance contribution is approximately its Pearson contribution.

For nonnegativity, define $g(t)=t\log t-t+1$ for $t>0$. Its derivative is $g'(t)=\log t$, which is negative below 1 and positive above 1. Therefore $g$ is minimized at $t=1$, where $g(1)=0$. The deviance contribution equals $2\mu g(t)$, so it is nonnegative. At $y=0$, its defined value is $2\mu$, also nonnegative.

For the limiting relationship, put $h(t)=t\log t-t+1$. Since $y=\mu t$, the deviance contribution is $2\mu h(t)$ and the Pearson contribution is $\mu(t-1)^2$. At $t=1$, $h(1)=h'(1)=0$ and $h''(1)=1$. The second-order Taylor limit gives $h(t)/(t-1)^2\to1/2$, so $2h(t)/(t-1)^2\to1$, proving the result. For large deviations, this local approximation need not be accurate; the two aggregate ratios can consequently differ. $\square$

Worked Example: Reading Two Ratios in a Small Poisson Fit

Fit an intercept-only Poisson model to six independent counts $1,2,2,3,3,1$. The fitted mean is their sample average, $\hat{\mu}=(1+2+2+3+3+1)/6=2$. There is one fitted coefficient, so the residual degrees of freedom are $6-1=5$.

For the Pearson statistic, the two counts of 1 each contribute $(1-2)^2/2=0.5$, the two counts of 2 contribute zero, and the two counts of 3 each contribute $(3-2)^2/2=0.5$. Hence $X_P^2=2$, and $X_P^2/5=0.4$.

For the deviance, each count of 1 contributes $2\{1\log(1/2)+1\}=0.614$, while each count of 3 contributes $2\{3\log(3/2)-1\}=0.433$. Counts of 2 contribute zero. Therefore $D\approx2(0.614)+2(0.433)=2.093$, and $D/5\approx0.419$. The ratios are close, consistent with the counts being near the fitted mean. They are below 1 in this particular sample; that does not establish that the Poisson model is correct or that variation is lower in the underlying process. Residual evidence is inherently sample-dependent.

How to Use the Diagnostic Plots

For a practical Poisson-regression check, plot Pearson or deviance residuals against fitted means and against each important predictor. Examine both the locations and the spread of the residuals. A smooth trend, curvature, or recurring positive and negative regions can suggest that the mean function does not capture the relationship. A spread that changes with fitted mean, without a clear shift in residual averages, is more compatible with a variation concern. Either pattern can occur alongside the other, and neither plot establishes a cause on its own.

Read the plots together with the Pearson and deviance ratios. Agreement between large ratios indicates that both summaries are flagging residual discrepancy, not that its source has been identified. If the plots show a clear mean pattern, investigate that structure before interpreting the ratio as evidence of excess variation. Also inspect whether a small number of cases dominate the statistics. A single large contribution can raise a ratio while most observations look unremarkable.

Common Pitfall

Wrong reasoning: “The Pearson ratio is greater than 1, so the Poisson variance assumption is the problem.” This conclusion treats the ratio as if it identified the source of lack of fit. It does not: residuals caused by an inadequate mean function can also inflate the statistic.

Correction: Report the ratio with its residual degrees of freedom, compare it with the deviance ratio, and inspect residuals against fitted means and predictors. A pattern in residual averages calls the mean structure into question; broadly greater residual spread may motivate concern about variation. Consider both explanations when the evidence is mixed, rather than treating one ratio as a verdict.

Key takeaway: Pearson and deviance ratios summarize the size of Poisson-regression residuals. Residual plots help diagnose their pattern. Use both forms of evidence to distinguish a mean-structure problem from excess variation, and do not equate a large ratio with a confirmed cause.

Check Your Understanding

Use the definitions and diagnostic reasoning in this tutorial to answer the following.

For a Poisson regression with 18 observations and 3 fitted coefficients, what are the residual degrees of freedom?
Why can a Poisson regression satisfy its score equations and still show a pattern in residual averages across predictor values?
A large Pearson ratio is accompanied by residual averages that alternate from positive to negative across ordered predictor values. What concern should be investigated before attributing the ratio to excess variation?
Why can the deviance and Pearson ratios differ more when some observations are far from their fitted means?
In the six-count intercept-only example, what fitted mean is used for every observation, and what is the Pearson statistic?