Uncertainty in Dispersion-Adjusted Fit Statistics

Why an Estimated Scale Changes the Interpretation

A goodness-of-fit statistic is often compared with a chi-square reference distribution. When the model has a dispersion parameter \(\phi\), however, its variability is on the scale set by \(\phi\). Replacing an unknown \(\phi\) with an estimate \(\hat{\phi}\) is not the same as knowing the scale: the estimate itself varies from sample to sample. That distinction matters especially when the residual degrees of freedom are few.

This tutorial focuses on the uncertainty introduced by estimating dispersion when interpreting dispersion-adjusted Pearson and deviance statistics. The chi-square reference distributions used below are the usual large-sample approximations for fitted GLMs, not exact finite-sample laws in general. Any further reference-distribution calculation will state its additional assumptions explicitly.

A useful working approximation is \(Q/\phi \approx \chi^2_\nu\), where \(\chi^2_\nu\) denotes a chi-square random variable with \(\nu\) degrees of freedom. This approximation motivates the Pearson scale estimate \(Q/\nu\). It also shows why a small \(\nu\) is consequential: a chi-square variable with few degrees of freedom is highly variable, so its ratio to \(\nu\) need not be close to one. In practice, the approximation may itself be imperfect, particularly with sparse data or a poorly fitting model.

How Much Uncertainty Can the Scale Estimate Have?

Under the working approximation \(Q/\phi \sim \chi^2_\nu\), the Pearson estimate is unbiased for \(\phi\), and its relative standard error can be calculated directly. This calculation describes the uncertainty under the approximation; it does not make that approximation exact for every fitted GLM.

Derivation. A chi-square random variable with \(\nu\) degrees of freedom has mean \(\nu\) and variance \(2\nu\). Since \(Q=\phi(Q/\phi)\),

Dividing the standard deviation \(\phi\sqrt{2/\nu}\) by the mean \(\phi\) gives \(\sqrt{2/\nu}\). Thus, at \(\nu=4\), the relative standard error is about \(0.707\), or 71%. An estimate reported as a single number can therefore conceal considerable uncertainty.

The same approximation gives an interval for \(\phi\). Write \(\chi^2_{\nu,\alpha}\) for the \(\alpha\) quantile of a chi-square distribution with \(\nu\) degrees of freedom. Since \(P(\chi^2_{\nu,.025} \le Q/\phi \le \chi^2_{\nu,.975})\) is approximately 0.95, reversing the inequalities when solving for the positive quantity \(\phi\) gives the approximate interval

This interval is based on the scaled chi-square approximation for \(Q\). It is not a universal confidence interval for dispersion in every GLM; its calibration depends on the adequacy of that approximation.

When an F Reference Is Justified

A familiar adjustment is to divide a fit statistic by an estimated scale and compare the result with a chi-square distribution. That reference is not automatically valid. One setting in which the uncertainty can instead be represented explicitly is an idealized one with an independent scale estimate.

Proof. Set \(U=Q/\phi\) and \(V=\nu_2\hat{\phi}/\phi\). By assumption, \(U\sim\chi^2_{\nu_1}\), \(V\sim\chi^2_{\nu_2}\), and \(U\) and \(V\) are independent. The definition of an \(F\) distribution is that the ratio of two independent chi-square variables, each divided by its degrees of freedom, has that distribution. Algebra gives

Therefore the stated statistic has an \(F_{\nu_1,\nu_2}\) distribution. \(\square\)

This result is a conditional lesson, not a rule that every dispersion-adjusted GLM statistic has an \(F\) distribution. In ordinary model fitting, \(Q\) and a scale estimate computed from the same residuals are not independent. The assumptions must be checked, not inferred merely from the fact that a denominator is called a dispersion estimate.

The Same-Sample Pearson Adjustment Can Be a Tautology

A particularly important case occurs when the scale estimate is \(\hat{\phi}=Q/\nu\), calculated from the same Pearson statistic being adjusted. Then the adjusted Pearson statistic is not a new diagnostic at all:

provided \(Q>0\). It equals the residual degrees of freedom by construction. A result that is forced to equal \(\nu\) cannot test whether the Pearson discrepancy is unusually large relative to that same estimated scale. Likewise, if a deviance is divided by this estimate, its distribution cannot simply be declared chi-square on the basis of the usual fixed-scale approximation: the deviance and Pearson-based estimate are computed from the same fitted data and may be dependent.

Practical Interpretation and Common Pitfalls

A dispersion-adjusted statistic can still be useful as a descriptive comparison: it expresses a discrepancy relative to an estimated scale. But a descriptive ratio and a calibrated hypothesis test are different things. Before interpreting an adjusted statistic as evidence for or against absolute fit, identify how \(\hat{\phi}\) was obtained, how many residual degrees of freedom support it, and what reference distribution is justified for the particular construction.

• Pitfall: treating \(\hat{\phi}\) as known. The reasoning “I estimated dispersion, so I can now use the known-dispersion chi-square reference” omits uncertainty in the estimate. With few residual degrees of freedom, the relative standard error can be large. The correction is to state the reference-distribution assumptions and account for scale uncertainty when those assumptions support it.
• Pitfall: using an F distribution automatically. The F derivation requires independent chi-square quantities with specified degrees of freedom. A scale estimate based on the same fitted residuals as the statistic generally does not satisfy the independence condition simply by being an estimate of scale.
• Pitfall: interpreting \(Q/(Q/\nu)=\nu\) as a successful fit check. This equality is an algebraic identity, not evidence from the data. Use it as a warning that the statistic has been normalized by itself, not as a goodness-of-fit conclusion.
• Pitfall: treating the chi-square approximation as exact. The scaled chi-square calculations are useful for showing the consequences of uncertainty, but sparse data, model misspecification, or other failures of the approximation can undermine their calibration as well.