Residual Degrees of Freedom and Reference Distributions

Why Degrees of Freedom Matter

A goodness-of-fit statistic measures how far observed responses are from what a fitted model predicts. A reference distribution turns that discrepancy into a calibrated assessment: it tells us how unusual the statistic would be if the model were adequate. The degrees of freedom in that reference distribution depend on how many independent response units were modeled and how many locally distinct mean directions were fitted.

For a standard GLM with \(n\) response units and a full-rank design matrix with \(r\) columns, the familiar residual degrees of freedom are \(n-r\). The count is simple, but its interpretation needs care. It is an algebraic count that motivates a chi-square reference distribution under suitable conditions; it does not by itself guarantee that the approximation is accurate.

Here, a response unit is one independently modeled response or one grouped response treated as a single observation. For grouped binomial data, for example, a row containing the number of successes out of a specified number of trials is one response unit; the trials do not each add a row to the design matrix. Likewise, when a count is recorded for each of several cells, the cell counts are the response units.

Counting the Degrees of Freedom

The rank, rather than the number of coefficient symbols, is the important quantity. A design can contain redundant columns, and a nonlinear mean model can have fewer locally distinct directions than its parameter notation might suggest. In a regular GLM, the derivative of the mean vector describes the local directions in which small parameter changes move the fitted means. If that derivative has rank \(r\), its local tangent space has dimension \(r\); the orthogonal complement of that tangent space has dimension \(n-r\).

This local description is not a claim that all fitted mean vectors form a linear subspace. In a nonlinear GLM, the set of possible fitted means may be curved. The tangent-space calculation describes the model near the fitted value, which is the relevant geometry for the usual local approximation.

Why a Chi-Square Reference Can Arise

The local argument is most transparent after weighting response deviations by their variances. Suppose the model is correct, observations are independent, the design is fixed, the model has fixed rank \(r\), and the information in each modeled response unit increases so that standardized errors have a joint normal limit. Assume also that the likelihood is smooth near the true parameter, the true parameter is an interior point, and the fitted means are identifiable locally. These are substantive conditions, not automatic properties of every GLM data set.

Write \(W\) for the diagonal matrix of inverse response variances at the true means, and \(J\) for the \(n\)-by-\(r\) derivative matrix of the mean vector with respect to its locally identifiable parameters. The weighted tangent directions are the columns of \(W^{1/2}J\). Under the stated regularity conditions, the leading part of fitting removes the projection of the standardized random error onto those directions. The remaining standardized error is in the orthogonal complement, whose dimension is \(n-r\).

Here is the projection argument for the reference distribution. Let \(P_{\mathcal S}\) be the orthogonal projection onto \(\mathcal S\), and let \(Z\) be the limiting standardized error, so \(Z\sim N(0,I_n)\). The leading residual is \((I_n-P_{\mathcal S})Z\), and its squared length is \(\|(I_n-P_{\mathcal S})Z\|^2\). The matrix \(I_n-P_{\mathcal S}\) is symmetric and idempotent, with rank \(n-r\). By the spectral theorem, an orthogonal change of coordinates makes this matrix diagonal with \(n-r\) ones and \(r\) zeros. An orthogonal transformation of \(Z\sim N(0,I_n)\) is still standard normal, so the squared length is the sum of squares of \(n-r\) independent standard normal variables. By definition, that sum has a \(\chi^2_{n-r}\) distribution. The assumed quadratic expansions and convergence of the fitted residual then give the stated limiting reference distribution. \(\square\)

The deviance enters this argument through a local likelihood expansion. Around the true mean, the log likelihood is locally quadratic in standardized deviations; twice the loss in log likelihood from fitting the model rather than the saturated means has the same leading weighted residual sum of squares as the Pearson statistic. This is why the two statistics can share a chi-square limit under regular conditions. It is a local, asymptotic statement—not an identity between the statistics in finite samples.

A useful special case is a normal linear model with known variance and a linear mean. There the standardized error is exactly multivariate normal, and fitting is exactly orthogonal projection onto the design space. The residual sum of squares divided by the known variance therefore has an exact \(\chi^2_{n-r}\) distribution. In many non-normal GLMs, by contrast, the chi-square reference is an approximation whose quality depends on whether the local normal and quadratic descriptions are adequate.

When the Approximation Is Unreliable

A residual degrees-of-freedom count is not a diagnostic of approximation quality. The chi-square argument can be poor when response distributions are highly discrete at the available information level, expected counts are small, fitted probabilities are near 0 or 1, or the model is close to a boundary or is weakly identified. It can also be misleading when there are many fitted directions relative to the number of response units: the nominal residual degrees of freedom may be small even though individual residual contributions are far from their large-sample behavior.

The setting matters. If there are many independent observations but each has a distinct covariate pattern and a binary response, the formal count \(n-r\) may be positive while many fitted cell probabilities are supported by little or no replication. Counting degrees of freedom does not create the repeated information needed for a reliable chi-square approximation. Similarly, if the mean model is misspecified, a nominal reference distribution under the model does not turn the statistic into a valid test of that misspecified model.

The same caution applies to geometric explanations. It is correct to say that the local tangent space has dimension \(r\), leaving an orthogonal complement of dimension \(n-r\). It is generally incorrect to claim that the fitted means in a GLM form a global \(r\)-dimensional linear subspace, or that every observed residual is exactly an orthogonal projection into a fixed complement. For nonlinear mean models, the tangent space changes with the parameter, and the projection is part of a local approximation.