Residual Mean Square and Residual Degrees of Freedom

Why Residual Degrees of Freedom Matter

The residual mean square is the standard estimate of the common error variance in a fixed-design linear model. Its numerator is the residual sum of squares, but its denominator is not automatically “the sample size minus the number of named coefficients.” The correct denominator is determined by how many independent directions in the data are used by the fitted model: the rank of the design matrix.

That distinction matters whenever a model contains redundant columns, as can happen with an over-parameterized representation. We will define the residual projection, derive its degrees of freedom, and establish the expectation and exact distribution of the residual sum of squares under the relevant assumptions.

Definition: In the fixed-design linear model \(y=X\beta+\varepsilon\), \(y\) is an \(n\)-vector of responses, \(X\) is a fixed \(n\times p\) design matrix of rank \(r\), and \(\beta\) is a \(p\)-vector. Suppose \(E[\varepsilon]=0\) and \(\operatorname{Var}(\varepsilon)=\sigma^2 I_n\), where \(\sigma^2>0\). Let \(P_X\) be the orthogonal projection onto the column space of \(X\), and let \(M_X=I_n-P_X\) be the residual projection. The residual vector is \(e=M_Xy\), the residual sum of squares is \(\operatorname{RSS}=e'e=y'M_Xy\), and, when \(n>r\), the residual mean square is \(\operatorname{RMS}=\operatorname{RSS}/(n-r)\).

The Projection and Its Residual Degrees of Freedom

The fitted vector is \(P_Xy\), the projection of the observed data onto the space of vectors the model can fit. Therefore \(e=y-P_Xy=M_Xy\) is perpendicular to every column of \(X\), so \(X'e=0\). The residual projection is symmetric and idempotent:

M_X'=M_X,\qquad M_X^2=M_X,\qquad M_XX=0.

The residuals consequently lie in the orthogonal complement of the column space of \(X\). That complement has dimension \(n-r\), not necessarily \(n-p\). Equivalently, \(M_X\) has rank \(n-r\): it retains exactly the data directions that the fitted model does not use.

If \(X\) has full column rank, then \(r=p\), and the familiar expression \(n-p\) is correct. If columns are linearly dependent, however, they do not add independent fitted directions. Counting them separately would subtract too many degrees of freedom.

Theorem: If \(X\) has rank \(r\), then \(M_X=I_n-P_X\) is a symmetric idempotent matrix with rank and trace \(n-r\). Thus the residual space has \(n-r\) dimensions. In particular, \(n-r\) is the residual degrees of freedom, and the residual mean square is defined only when \(n>r\).

To see the dimension claim, \(P_X\) projects onto a space of dimension \(r\), while \(M_X\) projects onto its orthogonal complement. The two spaces are perpendicular and together span \(\mathbb{R}^n\); hence the complement has dimension \(n-r\). A projection has eigenvalues only 0 and 1, so its trace equals its rank. Applying this fact to \(M_X\) gives \(\operatorname{tr}(M_X)=n-r\).

Worked Calculations

Worked Example: An Orthogonal Design With Three Residual Degrees of Freedom

Let \(n=6\), and use an intercept together with the columns \(x=(-1,-1,0,0,1,1)'\) and \(z=(-1,1,-1,1,-1,1)'\). Take \(y=(1,2,2,4,4,5)'\). The columns \(1,x,z\) are mutually orthogonal, with squared lengths \(6,4,6\), respectively. Thus the projection coefficients can be calculated separately:

\frac{1'y}{1'1}=\frac{18}{6}=3,\qquad \frac{x'y}{x'x}=\frac{6}{4}=\frac32,\qquad \frac{z'y}{z'z}=\frac{4}{6}=\frac23.

In particular, \(z'z=6\), because all six entries of \(z\) have square 1. The fitted values are \(3+\frac32x+\frac23z=(5/6,13/6,7/3,11/3,23/6,31/6)'\). Subtracting them from \(y\) gives \(e=(1/6,-1/6,-1/3,1/3,1/6,-1/6)'\). Therefore

\operatorname{RSS} =\frac{1+1+4+4+1+1}{36} =\frac{12}{36} =\frac13.

The design rank is \(r=3\), so the residual degrees of freedom are \(6-3=3\), and \(\operatorname{RMS}=(1/3)/3=1/9\). The displayed residuals also sum to zero and have zero inner product with both \(x\) and \(z\), as required for least-squares residuals.

Worked Example: Redundant Columns Do Not Use Extra Degrees of Freedom

Let \(n=5\), \(x=(-1,-1,0,1,1)'\), and \(y=(1,2,2,4,6)'\). Suppose \(X\) has three columns, \(1,x,2x\). The last column is twice the second, so the three columns span only the two-dimensional space generated by \(1\) and \(x\). Consequently \(p=3\), but \(r=2\).

The vectors \(1\) and \(x\) are orthogonal, with \(1'1=5\), \(x'x=4\), \(1'y=15\), and \(x'y=7\). The fitted vector is therefore \(3+\frac74x=(5/4,5/4,3,19/4,19/4)'\). The residual vector is \((-1/4,3/4,-1,-3/4,5/4)'\), and

\operatorname{RSS} =\frac{1+9+16+9+25}{16} =\frac{60}{16} =\frac{15}{4}.

There are \(n-r=5-2=3\) residual degrees of freedom, so \(\operatorname{RMS}=(15/4)/3=5/4\). Subtracting the number of columns would incorrectly give \(5-3=2\). The redundant column changes neither the fitted space nor the residuals, so it cannot consume an additional degree of freedom.

Worked Example: Residual Mean Square in a One-Way Layout

Suppose two groups have observations \(1,3\) and \(5,7,9\), respectively. Represent the group means with a design matrix whose columns indicate membership in each group. Those two indicator columns are linearly independent, so \(n=5\), \(r=2\), and the residual degrees of freedom are \(5-2=3\).

The fitted values are the group means: \(2\) for the first two observations and \(7\) for the last three. The residuals are \((-1,1,-2,0,2)'\). Thus \(\operatorname{RSS}=1+1+4+0+4=10\), and \(\operatorname{RMS}=10/3\). If the errors in this model are independent \(N(0,\sigma^2)\), then the exact distribution result below gives \(\operatorname{RSS}/\sigma^2\sim\chi^2_3\).

Why the Residual Mean Square Estimates \(\sigma^2\)

The unbiasedness calculation requires that the model mean lie in the column space of \(X\), as it does when \(E[y]=X\beta\), and that the error covariance be \(\sigma^2I_n\). Gaussian errors are not required for this expectation calculation.

Theorem: Under the fixed-design model \(y=X\beta+\varepsilon\), with \(E[\varepsilon]=0\) and \(\operatorname{Var}(\varepsilon)=\sigma^2 I_n\), if \(r=\operatorname{rank}(X)<n\), then \(\operatorname{E}[\operatorname{RSS}]=\sigma^2(n-r)\). Consequently, \(\operatorname{E}[\operatorname{RMS}]=\sigma^2\).

Proof. Because \(M_XX=0\), the residual vector is \(e=M_X(X\beta+\varepsilon)=M_X\varepsilon\). Hence \(\operatorname{RSS}=\varepsilon'M_X\varepsilon\). For any random vector with mean zero and covariance matrix \(\Sigma\), expanding the quadratic form entry by entry gives \(\operatorname{E}[\varepsilon'A\varepsilon]=\sum_{i,j}A_{ij}\operatorname{E}[\varepsilon_i\varepsilon_j]=\operatorname{tr}(A\Sigma)\) when \(A\) is symmetric. Here \(A=M_X\), \(\Sigma=\sigma^2I_n\), and \(\operatorname{tr}(M_X)=n-r\). Therefore

\operatorname{E}[\operatorname{RSS}] =\operatorname{tr}(M_X\sigma^2I_n) =\sigma^2(n-r), \qquad \operatorname{E}[\operatorname{RMS}] =\frac{\sigma^2(n-r)}{n-r} =\sigma^2.

This proves unbiasedness. \(\square\)

Exact Distribution Under Gaussian Errors

When the errors are Gaussian as well as spherical, the residual sum of squares has an exact scaled chi-square distribution. The degrees of freedom in that distribution are the same \(n-r\) dimensions of the residual space.

Theorem: If \(y=X\beta+\varepsilon\) for fixed \(X\) of rank \(r<n\), and \(\varepsilon\sim N(0,\sigma^2I_n)\), then \(\operatorname{RSS}/\sigma^2\sim\chi^2_{n-r}\). Equivalently, \(\operatorname{RMS}/\sigma^2\sim\chi^2_{n-r}/(n-r)\).

Proof. As above, \(\operatorname{RSS}=\varepsilon'M_X\varepsilon\). The symmetric projection \(M_X\) has \(n-r\) eigenvalues equal to 1 and \(r\) equal to 0. By the spectral theorem, there is an orthogonal matrix \(Q\) such that \(Q'M_XQ=\operatorname{diag}(I_{n-r},0_r)\). Since \(\varepsilon/\sigma\) is a standard multivariate normal vector, the orthogonally transformed vector \(Q'\varepsilon/\sigma\) also has independent standard normal components. Therefore

\frac{\operatorname{RSS}}{\sigma^2} =\left(\frac{Q'\varepsilon}{\sigma}\right)' \operatorname{diag}(I_{n-r},0_r) \left(\frac{Q'\varepsilon}{\sigma}\right) =\sum_{j=1}^{n-r}Z_j^2,

where \(Z_1,\ldots,Z_{n-r}\) are independent \(N(0,1)\) random variables. By the definition of the chi-square distribution, this sum has a \(\chi^2_{n-r}\) distribution. Dividing by \(n-r\) gives the stated distribution of the residual mean square. \(\square\)

Interpretation and Common Pitfall

The residual degrees of freedom count the independent directions left after fitting the mean structure. They are not a correction chosen merely to make the estimate look plausible: they are the rank of the residual projection, and they determine both the expectation of the residual sum of squares and its exact Gaussian sampling distribution.

Wrong: “There are \(p\) columns, so the residual degrees of freedom are \(n-p\).”
Correction: Use \(n-r\), where \(r=\operatorname{rank}(X)\). The expression \(n-p\) applies only when all \(p\) columns are linearly independent.
Wrong: “The residual mean square is unbiased because the residuals are independent.”
Correction: Unbiasedness follows from the covariance calculation above and does not require normality or independent residual components. Exact chi-square sampling does require Gaussian errors; the residuals themselves generally obey \(X'e=0\) and so are not freely varying independent observations.

If the mean is not in the column space of \(X\), the residual projection also retains part of the mean, and the stated unbiasedness conclusion need not hold. Likewise, if the error covariance is not \(\sigma^2I_n\), the trace calculation changes. These conditions are part of the result, not optional details.

Key takeaway: For a fixed-design model with rank \(r\), residual degrees of freedom are \(n-r\). Under zero-mean errors with covariance \(\sigma^2I_n\), \(\operatorname{RSS}/(n-r)\) is unbiased for \(\sigma^2\); under Gaussian errors, its scaled chi-square distribution is exact.

Check Your Understanding

Use the design rank, not just the number of columns, when answering.

A design matrix has \(n=12\), \(p=5\), and rank \(r=4\). What are the residual degrees of freedom?
State the assumptions needed for the residual mean square to be unbiased for \(\sigma^2\).
Why does the exact chi-square result have \(n-r\) degrees of freedom?
A model adds a column that is an exact linear combination of existing columns. What changes about the fitted space and residual degrees of freedom?
For the one-way example, verify the residual sum of squares and residual mean square from the listed residuals.