Degrees of Freedom and Bias in Scale Estimation

Why Adjust a Pearson Scale Estimate?

A Pearson scale estimate compares observed variation with the variation predicted by a mean-variance model. In a model with \(n\) observations, the Pearson sum of squares is typically divided by \(n\). But the means in that sum are usually estimated from the same data. Fitting those means absorbs some residual variation, so dividing by \(n\) can underestimate the scale. The familiar correction is to divide by residual degrees of freedom, often \(n-p\) when \(p\) mean parameters have been fitted.

That correction has an exact justification in a linear mean model with a known variance pattern. It is not a universal finite-sample result for nonlinear models: there, fitted means and variance weights may depend on the data in ways that do not yield a simple projection of fixed rank. The distinction matters when interpreting Pearson scale estimates as unbiased estimates rather than as conventional adjustments.

For a general mean-variance model, \(\hat\mu_i\) and \(v(\hat\mu_i)\) are estimated quantities. Thus \(Q_P\) is not simply a sum of \(n\) independent, mean-zero, unit-variance terms. The adjustment by \(n-p\) is exact under particular linear-model conditions described next; elsewhere it is a conventional approximation whose performance depends on the model and sample.

Exact Adjustment in a Linear Mean Model

Let \(Y=(Y_1,\ldots,Y_n)^\mathsf{T}\) have mean \(X\beta\), where \(X\) is a fixed \(n\times p\) matrix of rank \(p\). Assume \(\operatorname{Var}(Y)=\phi D\), where \(D=\operatorname{diag}(v_1,\ldots,v_n)\) is known and positive definite. The generalized least-squares estimator fits the mean, and its Pearson sum of squares is the weighted residual sum of squares.

Whitening the observations makes the geometry explicit. Define \(Z=D^{-1/2}Y\) and \(A=D^{-1/2}X\). Then \(E[Z]=A\beta\), \(\operatorname{Cov}(Z)=\phi I_n\), and the least-squares projection onto the columns of \(A\) is \(P=A(A^\mathsf{T}A)^{-1}A^\mathsf{T}\). The residual-maker matrix is \(M=I_n-P\).

Proof. The whitened residual vector is \(MZ\), so \(Q_P=Z^\mathsf{T}MZ\). Since \(MA=0\), the mean component of \(Z=A\beta+(Z-A\beta)\) is removed, giving \(Q_P=(Z-A\beta)^\mathsf{T}M(Z-A\beta)\). For any random vector \(U\) with mean zero and covariance matrix \(\Sigma\), \(E[U^\mathsf{T}MU]=\operatorname{tr}(M\Sigma)\) when \(M\) is fixed and symmetric. Here \(\Sigma=\phi I_n\), so

The first equality follows by expanding the quadratic form into entries and using \(E[U_iU_j]=\Sigma_{ij}\). The rank of \(P\) is \(p\), so \(M\) is a symmetric idempotent matrix of rank \(n-p\). If \(Z-A\beta\) is multivariate normal with covariance \(\phi I_n\), diagonalizing \(M\) by an orthogonal transformation expresses \(Q_P/\phi\) as the sum of squares of \(n-p\) independent standard normal variables. That is a chi-square distribution with \(n-p\) degrees of freedom. \(\square\)

This result identifies the source of the correction: fitting \(p\) linearly independent mean parameters leaves a residual projection of rank \(n-p\). Normality is not needed for unbiasedness; it is needed for the stated exact chi-square distribution. Also, \(p\) means the rank of the fitted design, not necessarily the number of coefficient labels in a redundant parameterization.

Weighted Residuals and the Role of Model Rank

The same correction applies when observations have different known variance weights. The appropriate Pearson residual is divided by the base variance \(v_i\), and the fitted mean is obtained by generalized least squares. Whitening converts that problem to ordinary least squares, so the exact expected residual sum of squares is still \(\phi(n-p)\), provided the whitened design has rank \(p\).

The last expectation follows directly from the theorem: \(E[Q_P/n]=\phi(n-p)/n\), whereas \(E[Q_P/(n-p)]=\phi\). The unadjusted estimate is downward-biased in this setting when \(p>0\). The correction is not a claim that every observed adjusted estimate is closer to \(\phi\), nor does it guarantee that an estimate in a nonlinear model is unbiased.

Why the Adjustment Is Limited in Nonlinear Models

In a nonlinear mean model, fitted values generally do not arise by projecting the data onto a fixed \(p\)-dimensional linear subspace. The fitting map depends on the observations, and the Pearson weights may also change with the fitted means. Consequently, the residual sum of squares need not be a fixed quadratic form with a residual-maker matrix of trace \(n-p\). Dividing by \(n-p\) remains common, but the exact linear-model proof does not establish its unbiasedness.

A simple Bernoulli example shows why caution is needed. Let \(Y_1,\ldots,Y_4\) be independent Bernoulli observations with common success probability \(p\), and fit a logistic intercept \(\alpha\), with \(p=\exp(\alpha)/\{1+\exp(\alpha)\}\). For a sample with \(k\) successes and \(0<k<4\), the fitted probability is \(\hat p=k/4\), and the Pearson sum of squares is

To verify the equality, substitute \(k=4\hat p\): the numerator becomes \(4\hat p(1-\hat p)^2+4(1-\hat p)\hat p^2=4\hat p(1-\hat p)\), which cancels the denominator. For \(k=1\), for example, the fitted probability is \(1/4\). The one success contributes \((3/4)^2/\{(1/4)(3/4)\}=3\); each of the three failures contributes \((1/4)^2/\{(1/4)(3/4)\}=1/3\). Hence \(Q_P=3+3(1/3)=4\). The adjusted estimate with \(n-p=3\) is \(4/3\), although the Bernoulli variance corresponds to scale \(\phi=1\).

In regular nonlinear models, \(n-p\) is often used as a practical residual-degrees-of-freedom adjustment, and large-sample arguments may support it as an approximation. But nonlinear curvature, leverage, estimated variance weights, and boundary behavior can all affect the finite-sample expectation. A residual degrees-of-freedom count is therefore a useful convention, not a substitute for checking the assumptions behind an exact result.

Common Mistake: Treating \(n-p\) as a Universal Identity

• Incorrect reasoning: “There are \(p\) estimated parameters, so the Pearson sum of squares always has expectation \(\phi(n-p)\).”
• Correction: The exact result above depends on a fixed linear mean space, known positive variance weights, and a full-rank design. In a nonlinear model, the fitting and weighting operations need not produce the fixed projection used in the proof.
• Another mistake: “The adjusted estimate must be closer to the true scale in this sample.” Unbiasedness concerns an average over repeated samples, not the accuracy of every realized estimate.