Why Check Dispersion?
A binomial or Poisson GLM specifies more than a relationship between predictors and a conditional mean. It also specifies how the response variance relates to that mean. For a binomial model, the variance follows from the fitted success probability and number of trials; for a Poisson model, the conditional variance equals the conditional mean. If the data vary more than these assumptions allow, standard errors and tests from the ordinary GLM may be too optimistic.
A Pearson-based dispersion check compares the observed residual variation with the amount expected under the model. It is a diagnostic, not a definitive test of why variability is high. This tutorial develops the calculation, its interpretation, and the limits of a simple dispersion adjustment. These checks are aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Variance Assumptions and Pearson Residuals
The stated variance assumptions are conditional on the predictors and the model being used. The binomial formula assumes the trials within a row have the stated common probability and are conditionally independent. The Poisson formula assumes the conditional count variance equals its mean. If the response structure does not support those assumptions, the usual Pearson check can flag the discrepancy, but it does not identify its cause.
For a fitted model, let \(\hat{\mu}_i\) be the fitted mean. The Pearson residual divides the response-minus-fitted-mean difference by the square root of the variance specified by the model. The sum of squared Pearson residuals is Pearson’s statistic, \(X_P^2\).
Here, \(m\) is the number of response rows and \(r\) is the rank of the fitted model matrix, so \(\nu\) is the residual degrees of freedom. For Poisson data, \(V(\hat{\mu}_i)=\hat{\mu}_i\). For grouped binomial counts, \(V(\hat{\mu}_i)=n_i\hat{p}_i(1-\hat{p}_i)\). If a grouped binomial response is recorded as a proportion \(y_i/n_i\), the corresponding contribution is \(n_i(y_i/n_i-\hat p_i)^2/[\hat p_i(1-\hat p_i)]\). The factor \(n_i\) matters: treating a proportion as one unweighted observation discards its trial count.
Why the Statistic Is Compared With Residual Degrees of Freedom
Under a correctly specified model with the stated variance function, Pearson residuals should have roughly unit variance. Before fitting, \(m\) independent standardized residuals would contribute about \(m\) to their sum of squares. Fitting \(r\) parameters uses information from the responses and imposes approximately \(r\) constraints on the residuals. The remaining residual variation therefore has approximately \(m-r\) degrees of freedom.
This gives the standard large-sample justification for comparing \(X_P^2\) with \(\nu=m-r\): under regularity conditions and a suitable model, the statistic is approximately chi-squared with \(\nu\) degrees of freedom. Since a chi-squared variable with \(\nu\) degrees of freedom has mean \(\nu\), the ratio \(X_P^2/\nu\) should be around 1. This is an approximation, not an exact identity for every finite data set. It can be less reliable with small samples, fitted probabilities near 0 or 1, or other violations of the model assumptions.
Worked Calculations
Worked Example: Grouped Binomial Dispersion
A hypothetical insurer groups policyholders into four rating cells. Each cell contains 10 policies, and the response is the number with a specified event during the year. An intercept-only binomial GLM is fitted. The observed event counts are 1, 4, 6, and 9. Calculate the Pearson dispersion estimate and interpret it.
Method. For an intercept-only binomial model, the fitted probability is the overall event proportion. There are \(1+4+6+9=20\) events among \(4(10)=40\) policies, so \(\hat p=20/40=0.5\). The fitted mean in each cell is \(10(0.5)=5\), and the assumed variance in each cell is \(10(0.5)(0.5)=2.5\).
The four Pearson contributions are:
Thus \(X_P^2=6.4+0.4+0.4+6.4=13.6\). The model has rank \(r=1\), so \(\nu=4-1=3\), and \(\hat\phi=13.6/3=4.5333\). To four decimal places, the estimate is 4.5333.
This is substantially above 1, indicating excess residual variability relative to the ordinary binomial variance assumption. The calculation avoids a common mistake: using only the variance \(p(1-p)=0.25\) for each cell instead of the grouped-count variance \(n_i p(1-p)=2.5\). The result is a diagnostic signal, not proof of a specific cause.
Worked Example: Poisson Dispersion With an Intercept-Only Fit
A hypothetical service team records the number of urgent requests received on each of four comparable shifts. The counts are 0, 4, 0, and 8. An intercept-only Poisson GLM is fitted. Calculate the Pearson dispersion estimate.
Method. The fitted mean is the sample mean, \(\hat\mu=(0+4+0+8)/4=3\), for every shift. Under the Poisson variance assumption, each fitted variance is also 3. The Pearson contributions are:
Adding gives \(X_P^2=3+1/3+3+25/3=44/3=14.6667\). The intercept-only model has rank 1, so \(\nu=4-1=3\). Therefore \(\hat\phi=(44/3)/3=44/9=4.8889\), to four decimal places.
The check indicates considerably more variation than the fitted Poisson mean alone predicts. The common error is to divide each squared residual by a single variance shared by all rows. Here the specified Poisson variance is the row-specific fitted mean, which happens to be 3 for each row in this intercept-only fit.
Worked Example: Adjusting a Standard Error
A fitted binomial GLM has a coefficient estimate of 0.30 with an ordinary-model standard error of 0.08. A separate Pearson check gives \(\hat\phi=2.25\). Under a quasi-binomial adjustment, calculate the adjusted standard error, the resulting Wald statistic, and a 95% Wald interval on the coefficient scale.
Method. A scalar dispersion adjustment multiplies the estimated coefficient covariance matrix by \(\hat\phi\). Consequently, standard errors are multiplied by \(\sqrt{\hat\phi}=\sqrt{2.25}=1.5\). The adjusted standard error is \(0.08(1.5)=0.12\). The Wald statistic for testing a zero coefficient is \(0.30/0.12=2.5\).
Using 1.96 as the approximate 95% normal critical value, the interval is \(0.30\pm1.96(0.12)=0.30\pm0.2352\), or \((0.0648,0.5352)\). The ordinary standard error would have given \(0.30/0.08=3.75\), overstating the strength of evidence if the extra variability is genuine.
This adjustment addresses uncertainty at the coefficient level under a scalar variance inflation assumption. It does not establish that the conditional mean is correct, and it does not explain the excess variation. The common mistake is to multiply the standard error by \(\hat\phi\) rather than by its square root; it is the variance, not the standard error, that is multiplied by \(\hat\phi\).
Interpreting Excess Variation and Its Limits
A high dispersion estimate says that the observed residual variation is larger than the fitted binomial or Poisson variance function anticipates. Possible reasons include unmodeled differences between observations, omitted predictors, dependence among responses, or a variance relationship that is unsuitable for the data. A single Pearson statistic cannot distinguish these explanations. It is important to investigate the data and model structure rather than treating the estimate as an automatic correction.
For a quasi-Poisson model, the variance is specified as \(\operatorname{Var}(Y_i\mid x_i)=\phi\mu_i\). For a quasi-binomial model, it is specified as \(\operatorname{Var}(Y_i\mid x_i)=\phi n_i p_i(1-p_i)\). These models allow a scale factor beyond 1. In R, the corresponding GLM family choices are `quasipoisson` and `quasibinomial`. When the same mean structure and link are used, these choices retain the same estimating equations for the mean coefficients as their ordinary Poisson or binomial counterparts, while adjusting their estimated uncertainty for dispersion.
The adjustment has limits. Inflating standard errors does not fix biased fitted means, omitted structure, or dependence between observations. A single scale factor also assumes the extra variability can be represented by the same multiplier across the observations. In addition, quasi-likelihood is not a full probability likelihood, so likelihood-based quantities such as AIC are not available in the usual way for these quasi families. Do not interpret a larger standard error as evidence that all model problems have been resolved.
A value of \(\hat\phi\) below 1 is also possible in a finite sample. It indicates less residual variation than the variance function predicts; it is not evidence of overdispersion. The value should be considered alongside the residual degrees of freedom, the data structure, and other model diagnostics. In particular, the approximation behind the Pearson ratio is not a guarantee that the ratio will be close to 1 in every correctly specified small sample.
Common Exam Traps
- Using the wrong variance. For a Poisson row, divide by \(\hat\mu_i\). For a grouped binomial count, divide by \(n_i\hat p_i(1-\hat p_i)\), not just \(\hat p_i(1-\hat p_i)\).
- Using the wrong degrees of freedom. Use the number of response rows minus the rank of the fitted model matrix. The rank, not a casual count of named predictors, determines the number of fitted dimensions.
- Calling the estimate a cause. A high ratio identifies excess residual variation relative to the assumed variance function; it does not by itself show why it occurred.
- Inflating the standard error incorrectly. Multiply the variance by \(\hat\phi\), or equivalently multiply the standard error by \(\sqrt{\hat\phi}\).
- Assuming a scale adjustment repairs the model. It can adjust uncertainty under a scalar dispersion assumption, but cannot repair a wrong mean structure or dependence.
Check Your Understanding
For each question, calculate carefully and distinguish a dispersion signal from a conclusion about its cause.
- A Poisson GLM has three response rows with observed counts 1, 3, and 6 and fitted means 2, 3, and 4. The model rank is 1. Calculate \(X_P^2\), the residual degrees of freedom, and \(\hat\phi\). (Answer: \(X_P^2=1.5\), \(\nu=2\), and \(\hat\phi=0.7500\).)
- Three binomial groups each contain 8 trials. Their observed success counts are 2, 4, and 6, and an intercept-only fit gives \(\hat p=0.5\). Calculate the Pearson dispersion estimate. (Answer: \(X_P^2=4\), \(\nu=2\), and \(\hat\phi=2.0000\).)
- A Poisson model has Pearson statistic 24 and residual degrees of freedom 6. An ordinary-model coefficient standard error is 0.15. Under a scalar quasi-Poisson adjustment, what are the dispersion estimate and adjusted standard error? (Answer: \(\hat\phi=4\) and adjusted standard error \(0.30\).)
- Two binomial rows have trial counts 20 and 40, observed proportions 0.4 and 0.6, and fitted probabilities 0.5 and 0.5. The model rank is 1. Calculate the Pearson statistic and dispersion estimate. (Answer: \(X_P^2=2.4\), \(\nu=1\), and \(\hat\phi=2.4000\).)
- A Poisson fit has a Pearson dispersion estimate well above 1. State what the result supports and give one conclusion it does not establish. (Answer: It supports concern that residual variability exceeds the ordinary Poisson variance assumption; it does not establish the particular cause or show that a scalar adjustment repairs the model.)