Why Estimate Dispersion From Residuals?
A regression mean model describes how the expected response changes with predictors. A variance model describes the response’s variability around that mean. In many quasi-likelihood settings, the variance is specified up to a common positive multiplier, the dispersion parameter \(\phi\). Estimating that multiplier is important because it calibrates how large the observed residuals are relative to the variability the mean–variance relationship predicts.
The Pearson approach compares squared residuals with their model-based variances. It applies to independent observations whose variances have the form \(\phi V(\mu_i)/w_i\), where \(V\) is a specified variance function and \(w_i\) is a known positive weight. The estimate is a moment-based scale estimate: it does not require a full probability distribution for the responses.
For a Poisson mean–variance relationship, \(V(\mu_i)=\mu_i\). For grouped binomial data with \(m_i\) trials and success probability \(p_i\), the baseline variance is \(m_i p_i(1-p_i)\). Thus the Pearson residuals in these two cases divide the observed-minus-fitted response by, respectively, the square root of the fitted mean or the square root of the fitted binomial variance. In an ordinary binomial model the dispersion is fixed at one; a common multiplier is introduced when using a quasi-binomial variance model.
Deriving the Pearson Moment Estimate
The first derivation explains why Pearson residual squares contain information about \(\phi\). Suppose for the moment that the means \(\mu_i\) are known, the observations are independent, and each has finite variance \(\phi V(\mu_i)/w_i\), with \(V(\mu_i)>0\) and \(w_i>0\). Define the standardized errors \(Z_i=(Y_i-\mu_i)/\sqrt{V(\mu_i)/w_i}\).
Proof. For each \(i\), \(E[Y_i-\mu_i]=0\), so $$ E[Z_i^2] =\frac{E[(Y_i-\mu_i)^2]}{V(\mu_i)/w_i} =\frac{\operatorname{Var}(Y_i)}{V(\mu_i)/w_i} =\phi. $$ By linearity of expectation, which does not require independence, $$ E\left[\sum_{i=1}^n Z_i^2\right] =\sum_{i=1}^n E[Z_i^2] =n\phi. $$ Dividing by \(n\) proves the unbiasedness statement. \(\square\)
In regression, the means are estimated from the same responses used to form residuals. The residuals are then constrained by the fitted model and are not simply \(n\) independent standardized errors. A residual-degrees-of-freedom denominator accounts, approximately, for the parameters used in fitting. The following calculation gives the exact motivation in a locally linear, whitened model.
Proof. Since \(I_n-H\) is also symmetric and idempotent, \(e^\mathsf{T}e=Z^\mathsf{T}(I_n-H)Z\). For any mean-zero vector with covariance \(\phi I_n\), expanding the quadratic form gives $$ E[Z^\mathsf{T}AZ] =\sum_{i,j}A_{ij}E[Z_iZ_j] =\phi\sum_i A_{ii} =\phi\operatorname{tr}(A). $$ Take \(A=I_n-H\). The trace of a projection matrix equals its rank, so \(\operatorname{tr}(H)=q\), and \(\operatorname{tr}(I_n-H)=n-q\). Therefore \(E[e^\mathsf{T}e]=\phi(n-q)\), as claimed. \(\square\)
For a generalized regression model, standardizing by fitted variances and linearizing the fitted means leads to this projection calculation as a local approximation, not as an exact finite-sample identity in every model. The conventional estimate \(X_P^2/(n-q)\) uses that residual-dimension rationale. How its finite-sample bias depends on the model is a separate issue; the formula alone does not guarantee an exactly unbiased estimate.
Calculating the Estimate
Worked Example: Poisson Regression With a Log Link
Suppose a Poisson log-link regression with an intercept and one predictor has four fitted means \((2,4,8,16)\), and the observed counts are \((4,0,10,16)\). The predictor values are \(x=(0,1,2,3)\). Verify that these fitted means satisfy the Poisson score equations, then calculate the Pearson dispersion estimate.
For a Poisson log-link model, the score equations for the intercept and slope require \(\sum_i(y_i-\hat{\mu}_i)=0\) and \(\sum_i x_i(y_i-\hat{\mu}_i)=0\). The response-minus-fitted residuals are \((2,-4,2,0)\), so $$ \sum_i(y_i-\hat{\mu}_i)=2-4+2+0=0, \qquad \sum_i x_i(y_i-\hat{\mu}_i)=0(2)+1(-4)+2(2)+3(0)=0. $$ The fitted means have the required log-linear form: \(\hat{\mu}_i=2(2)^{x_i}\), so the intercept and slope are \(\log 2\) and \(\log 2\). The score check confirms that these fitted values solve the likelihood equations; the Poisson log-likelihood is concave in these coefficients.
For Poisson data the variance function is \(V(\mu_i)=\mu_i\). The four squared Pearson residuals are $$ \frac{(4-2)^2}{2}=2,\qquad \frac{(0-4)^2}{4}=4,\qquad \frac{(10-8)^2}{8}=\frac12,\qquad \frac{(16-16)^2}{16}=0. $$ Thus \(X_P^2=2+4+\frac12+0=6.5\). There are \(n=4\) observations and \(q=2\) fitted mean parameters, giving \(n-q=2\). Therefore $$ \hat{\phi}_P=\frac{6.5}{2}=3.25. $$ The Pearson estimate is substantially above one: the squared residuals are large relative to the fitted Poisson variances, on this residual-scale calculation.
Worked Example: Grouped-Binomial Logistic Regression
Four groups each contain \(m_i=10\) trials. Let the predictor values be \(x_i=\operatorname{logit}(p_i)\), where the fitted probabilities are \((0.2,0.4,0.6,0.8)\). A logistic regression with intercept and slope \((0,1)\) therefore gives these fitted probabilities. The observed success counts are \((4,2,4,10)\). Calculate the Pearson dispersion estimate.
The fitted success counts are \(m_i\hat p_i=(2,4,6,8)\), so the response residuals are \((2,-2,-2,2)\). The intercept score is their sum, \(2-2-2+2=0\). For the slope score, the predictors are \((-a,-b,b,a)\), where \(a=\log 4\) and \(b=\log(2/3)\). Hence the score is $$ (-a)(2)+(-b)(-2)+b(-2)+a(2)=0. $$ The specified fit satisfies both logistic score equations.
The baseline grouped-binomial variance for group \(i\) is \(m_i\hat p_i(1-\hat p_i)\). For the four groups these variances are \(1.6,2.4,2.4,1.6\). The Pearson statistic is therefore $$ X_P^2=\frac{2^2}{1.6}+\frac{(-2)^2}{2.4} +\frac{(-2)^2}{2.4}+\frac{2^2}{1.6} =2.5+\frac{5}{3}+\frac{5}{3}+2.5 =\frac{25}{3}. $$ There are four independent grouped responses and two fitted regression parameters, so the residual degrees of freedom are \(4-2=2\). The common-dispersion estimate is $$ \hat{\phi}_P=\frac{25/3}{2}=\frac{25}{6}\approx4.17. $$ This calculation uses the binomial variance as the baseline and estimates a common multiplier relative to it. It does not claim that the ordinary binomial model itself has a free dispersion parameter.
Worked Example: Known Means Versus Fitted Means
Suppose three independent Poisson responses have known means \((2,4,6)\), and the observed counts are \((3,2,9)\). First calculate the known-mean moment estimate of dispersion. Then suppose instead that the means came from a fitted intercept-only Poisson regression, for which the fitted mean is the sample average.
With known means, the Pearson terms are $$ \frac{(3-2)^2}{2}=\frac12,\qquad \frac{(2-4)^2}{4}=1,\qquad \frac{(9-6)^2}{6}=\frac32. $$ Thus \(X_P^2=3\), and the known-mean moment estimate is \(3/3=1\).
If the mean is estimated by an intercept-only Poisson regression, its fitted value is \(\bar y=(3+2+9)/3=14/3\) for each observation. This is a different fitted-mean calculation, so the Pearson statistic must be recomputed: $$ X_P^2 =\frac{(3-14/3)^2}{14/3} +\frac{(2-14/3)^2}{14/3} +\frac{(9-14/3)^2}{14/3} =\frac{25}{42}+\frac{32}{21}+\frac{169}{42} =\frac{129}{21}. $$ Here \(q=1\), leaving \(3-1=2\) residual degrees of freedom. The conventional estimate is \((129/21)/2=43/14\approx3.07\). The contrast illustrates that fitting the mean changes the residuals and the denominator; the known-mean unbiasedness result cannot simply be applied to fitted residuals.
Interpretation and a Common Pitfall
The estimate \(\hat{\phi}_P\) compares total squared Pearson residual variation with the amount expected per residual degree of freedom under the chosen variance model. Values near one indicate that the observed residual scale is broadly consistent with that baseline, while larger values indicate more residual variation than the baseline predicts. A value below one indicates less variation by this summary. These interpretations are descriptive: the estimate alone is not a formal test, and a large value does not identify why the variance model and data differ.
Common mistake: “The Pearson statistic is \(X_P^2\), so the dispersion estimate must be \(X_P^2/n\), even after fitting a regression.” The \(X_P^2/n\) argument is the direct moment calculation when the means are known. With fitted means, residuals are affected by parameter estimation; the conventional regression estimate divides by \(n-q\). A second mistake is treating \(\hat{\phi}_P>1\) as proof that the data are overdispersed. It is evidence relative to the selected mean and variance specification, not a diagnostic explanation or a calibrated hypothesis test.
The calculation also depends on the chosen baseline variance. For grouped binomial data, for example, using the individual-trial variance rather than the grouped-count variance would omit the factor \(m_i\) and produce the wrong Pearson residuals. Always write down the response scale, fitted mean, and corresponding baseline variance before summing residual squares.
Check Your Understanding
Use the definitions and derivations above to answer these questions.
- Under what assumptions is \(X_P^2/n\) an unbiased moment estimate of the common dispersion?
- For a Poisson regression with \(n=12\) observations and \(q=3\) fitted mean parameters, what denominator is used in the conventional Pearson dispersion estimate?
- Why does the projection calculation produce \(n-q\), rather than \(n\), as the expected residual sum-of-squares factor?
- In grouped-binomial regression, what baseline variance belongs in the denominator of each Pearson residual?
- Why does a Pearson estimate greater than one not, by itself, constitute a formal test for overdispersion?