Why Dispersion Matters
A GLM specifies how the response mean depends on predictors and how response variance relates to that mean. Some GLM families also include a dispersion parameter that scales the variance. Estimating that parameter matters because it affects standard errors and the uncertainty attached to fitted coefficients. The central distinction is that dispersion is estimated in many non-binomial GLMs, while ordinary binomial and Poisson models fix it at one.
This tutorial develops two common estimates for models with an estimable dispersion parameter: the Pearson estimate and the deviance-based estimate. It also explains their degrees-of-freedom adjustment and shows why a ratio calculated for a Poisson or binomial model is not, by itself, an estimate of that model’s dispersion parameter.
The variance function \(V(\mu_i)\) is determined by the GLM family. For a Gaussian model it is \(1\), so \(\phi\) is the residual variance. For a Gamma model it is \(\mu_i^2\), so \(\phi\) scales variance in proportion to the squared mean. The parameter does not replace the mean model: it scales the variability around that mean.
The usual Pearson and deviance calculations below assume independent responses, a correctly specified variance function, and a fitted model with \(n\) response observations and design-matrix rank \(r\). The residual degrees of freedom are \(n-r\). Rank, rather than the number of written predictor terms, matters if some columns are linearly dependent. The estimates are intended for families in which \(\phi\) is estimated; they do not alter the fixed-dispersion convention of ordinary Poisson and binomial models.
Pearson and Deviance Estimates
For each observation, compare its response \(y_i\) with its fitted mean \(\hat{\mu}_i\). The Pearson residual is the response residual scaled by the model’s variance function. Its squared values form the Pearson statistic \(X^2\). The deviance \(D\) is a separate measure of discrepancy: it compares the fitted model’s likelihood with the likelihood of a saturated model, which fits each observed response as closely as the family permits.
The Pearson estimate uses squared residuals directly, scaled by the fitted variance function. The deviance estimate uses the family-specific likelihood discrepancy. Both divide by residual degrees of freedom, not by \(n\), because fitting model parameters uses information from the same responses and reduces the remaining residual variation.
These are generally approximate estimates, not identities guaranteed to hold exactly in every finite sample. They can differ noticeably in small samples or when a few observations have substantial influence. The Pearson statistic also depends on the fitted means through \(V(\hat{\mu}_i)\); it is not simply the unscaled sum of squared residuals except when the variance function is constant.
Why the Degrees-of-Freedom Adjustment Works
A short justification comes from the weighted least-squares approximation used locally in GLM fitting. After standardizing residual variation by its variance, the fitting step projects the response onto the model space. The residuals are the component left outside that space.
Under the model, the standardized response errors have variance approximately \(\phi\) in each independent direction. In a local weighted least-squares representation, write the standardized error vector as \(z\), with covariance matrix \(\phi I\).
The fitted model projects onto a space of dimension \(r\). The residual projection is approximately \(I-H\), where \(H\) is the projection matrix and has rank \(r\). Thus the residual vector is approximately \((I-H)z\).
The expected squared length of the residual vector is approximately \(\phi\operatorname{tr}(I-H)=\phi(n-r)\). The Pearson statistic estimates that squared length, giving \(E[X^2]\approx\phi(n-r)\) and hence \(\hat{\phi}_{P}=X^2/(n-r)\).
For a well-behaved fitted model, deviance is approximately equivalent to the Pearson statistic in large samples. Therefore \(D/(n-r)\) provides another approximate estimate of \(\phi\), though it need not equal the Pearson estimate in a particular data set.
The projection argument is a local approximation for GLMs, not a claim that every residual is independent or that either ratio is exactly unbiased in every finite sample. It explains why the residual degrees-of-freedom denominator is appropriate and why the two estimates are often close when the model fits adequately.
Worked Calculations
Worked Example: Pearson Dispersion in a Gamma GLM
A hypothetical model of positive claim amounts uses a Gamma response with \(V(\mu)=\mu^2\), unit prior weights, and five independent observations. The fitted model has design-matrix rank \(r=2\). The observed values and fitted means are \(y=(12,16,11,24,7)\) and \(\hat{\mu}=(10,20,10,20,10)\). Calculate the Pearson dispersion estimate.
Method. For a Gamma GLM, the Pearson contribution for observation \(i\) is \((y_i-\hat{\mu}_i)^2/\hat{\mu}_i^2\), since \(V(\hat{\mu}_i)=\hat{\mu}_i^2\). The residual degrees of freedom are \(5-2=3\).
The five contributions are $$ \frac{(12-10)^2}{10^2}=0.04,\quad \frac{(16-20)^2}{20^2}=0.04,\quad \frac{(11-10)^2}{10^2}=0.01,\quad \frac{(24-20)^2}{20^2}=0.04,\quad \frac{(7-10)^2}{10^2}=0.09. $$
Therefore \(X^2=0.04+0.04+0.01+0.04+0.09=0.22\), and $$ \hat{\phi}_{P}=\frac{0.22}{3}=0.073333\ldots\approx 0.0733. $$
Check. The five contributions sum to \(0.22\), and dividing by three residual degrees of freedom gives \(0.0733\) to four decimal places. A common mistake is to divide by five observations instead: \(0.22/5=0.044\) uses the wrong denominator and ignores the two fitted dimensions.
Worked Example: Deviance Dispersion in a Gaussian GLM
A hypothetical Gaussian model has six independent observations and design-matrix rank \(r=2\). Its fitted residuals, calculated as \(y_i-\hat{\mu}_i\), are \((-2,1,0,2,-1,0)\). Find the deviance-based dispersion estimate.
Method. For a Gaussian GLM with identity link and unit weights, the deviance is the sum of squared residuals. Thus $$ D=(-2)^2+1^2+0^2+2^2+(-1)^2+0^2=4+1+0+4+1+0=10. $$
The residual degrees of freedom are \(6-2=4\). Therefore $$ \hat{\phi}_{D}=\frac{D}{n-r}=\frac{10}{4}=2.5. $$
Check. The result is \(2.5\) response-units squared, as expected for the Gaussian dispersion. In this family the Pearson statistic is also \(10\), because \(V(\hat{\mu}_i)=1\), so the Pearson estimate is \(10/4=2.5\). The common trap is to use the square root of the estimate, \( \sqrt{2.5}\), as the dispersion; that is a standard deviation, whereas \(\phi\) here is a variance.
Why Standard Poisson and Binomial Models Fix Dispersion
In the ordinary Poisson GLM, the response variance is the mean: \(\operatorname{Var}(Y_i)=\mu_i\). In the ordinary binomial GLM, the variance is determined by the number of trials and the success probability. These are fixed family relationships, with dispersion conventionally set to \(\phi=1\); it is not an additional freely estimated parameter in the standard model.
A Pearson or deviance ratio can still be calculated for a fitted Poisson or binomial model. Such a ratio is a diagnostic comparison with the model’s assumed variability; it does not change the standard model’s specified dispersion. Distinguishing a diagnostic ratio from an estimated parameter prevents a common interpretation error.
Worked Example: A Poisson Ratio Is Not a Fitted Dispersion Parameter
A hypothetical intercept-only Poisson model is fitted to four counts \(y=(0,1,2,3)\). The fitted mean is \(\hat{\mu}_i=1.5\) for each observation, and the design-matrix rank is \(r=1\). Calculate the Pearson ratio and interpret it under the standard Poisson model.
Method. A Poisson GLM has \(V(\hat{\mu}_i)=\hat{\mu}_i=1.5\). The squared residuals are \(2.25,0.25,0.25,2.25\), so $$ X^2=\frac{2.25+0.25+0.25+2.25}{1.5} =\frac{5}{1.5}=3.3333\ldots. $$
There are \(4-1=3\) residual degrees of freedom, so the diagnostic ratio is $$ \frac{X^2}{n-r}=\frac{3.3333\ldots}{3}=1.1111\ldots. $$
Interpretation and check. The standard Poisson model still has \(\phi=1\). The calculated \(1.1111\ldots\) is the Pearson statistic per residual degree of freedom, not a newly estimated Poisson dispersion parameter. The fitted mean is also checked by the data: \((0+1+2+3)/4=1.5\). A common mistake is to report \(1.1111\) as the dispersion estimate for an ordinary Poisson GLM; the correction is to label it a diagnostic ratio and retain the standard model’s fixed value \(\phi=1\).
Worked Example: Fixed Dispersion in a Grouped Binomial Model
A hypothetical grouped binomial observation records 6 successes in 20 trials. Suppose its success probability is \(p=0.3\). State the model variance and the dispersion value in the ordinary binomial GLM.
Method. If \(Y\) is the number of successes in \(m\) independent Bernoulli trials with common success probability \(p\), then \(Y\sim\operatorname{Binomial}(m,p)\) and \(\operatorname{Var}(Y)=mp(1-p)\). Here \(m=20\) and \(p=0.3\), so $$ E[Y]=20(0.3)=6,\qquad \operatorname{Var}(Y)=20(0.3)(0.7)=4.2. $$
The observed count of 6 agrees with the stated mean, and the variance is \(4.2\) count-squared units. In the ordinary binomial GLM, the dispersion is fixed at \(\phi=1\); the value \(4.2\) is the response variance implied by the trial count and probability, not the dispersion parameter.
Common mistake. Dividing the variance \(4.2\) by \(m\), or treating \(4.2\) as an estimated \(\phi\), confuses the binomial variance with the dispersion multiplier. The standard variance formula already accounts for the 20 trials.
Choosing and Reporting an Estimate
For a non-binomial family with estimable dispersion, report which method was used and give the residual degrees of freedom. Pearson and deviance estimates answer closely related questions but need not match. A material difference can be a reason to inspect the fitted model, residuals, influential observations, and the suitability of the variance specification; neither ratio alone establishes why the difference occurred.
| Situation | Dispersion treatment | Useful calculation |
|---|---|---|
| Gaussian, Gamma, inverse Gaussian, or a suitable Tweedie model | Estimate \(\phi\), subject to the model’s assumptions | \(X^2/(n-r)\) or \(D/(n-r)\) |
| Ordinary Poisson model | Fixed at \(\phi=1\) | Pearson or deviance ratio can be used as a diagnostic |
| Ordinary binomial model | Fixed at \(\phi=1\) | Pearson or deviance ratio can be used as a diagnostic |
Before calculating a dispersion estimate, verify that the family allows an estimable \(\phi\), identify the correct variance function, and count independent response observations. For grouped binomial data, for example, a row representing a group of trials is one grouped response observation for the residual degrees-of-freedom calculation; the number of trials is not substituted for the number of rows.
Check Your Understanding
For each question, identify the relevant degrees of freedom or variance relationship before using a ratio.
- A Gaussian GLM has 7 independent observations, design-matrix rank 3, and residual sum of squares 24. What is the deviance-based dispersion estimate? (Answer: \(24/(7-3)=6\).)
- A Gamma GLM has four observations, rank 2, unit weights, and observed-to-fitted-mean ratios \(1.1,0.9,1.2,0.8\). What is the Pearson dispersion estimate? (Answer: \((0.01+0.01+0.04+0.04)/(4-2)=0.05\).)
- A Poisson model has 8 observations, rank 2, and Pearson statistic \(9\). What is the Pearson ratio, and what is the dispersion in the ordinary Poisson model? (Answer: The ratio is \(9/(8-2)=1.5\); the model dispersion is fixed at \(1\).)
- A grouped binomial response has 20 trials and fitted success probability \(0.3\). What are its mean and variance under the ordinary binomial model, and what is \(\phi\)? (Answer: Mean \(=6\), variance \(=20(0.3)(0.7)=4.2\), and \(\phi=1\).)
- A non-binomial GLM has deviance \(20\), 12 independent observations, and design-matrix rank 4. Calculate the deviance-based dispersion estimate and state its status. (Answer: \(20/(12-4)=2.5\), an approximate deviance-based estimate.)