Why the Residual Sum of Squares Has a Sampling Distribution
In simple linear regression, the residual sum of squares measures the variation left after fitting an intercept and a slope. Its observed value changes from sample to sample, even when the underlying error variance is fixed. Under normal errors, that sampling variation has an exact chi-square distribution. This result explains why the error-variance estimate uses \(n-2\) degrees of freedom and establishes that the residual variation is independent of the fitted intercept and slope.
The result requires more than the least-squares fitting equations. In particular, the errors must be independent and normally distributed with a common variance. The predictor values are treated as fixed; for random predictors, the same result applies conditionally on their observed values when the assumptions hold given those values.
The result concerns the true error variance \(\sigma^2\), not an estimated variance in the denominator. The familiar estimator \(s^2=RSS/(n-2)\) therefore satisfies \((n-2)s^2/\sigma^2\sim\chi^2_{n-2}\). Dividing by \(n-2\) makes \(s^2\) unbiased for \(\sigma^2\); the chi-square distribution also describes how it varies from sample to sample.
Why There Are \(n-2\) Degrees of Freedom
Least squares chooses two coefficients, an intercept and a slope, to fit the data. Consequently, the residual vector cannot vary freely in all \(n\) coordinate directions. It must satisfy two constraints: its components sum to zero, and its inner product with the predictor values is zero. When the predictor values are not all equal, those constraints are independent, leaving \(n-2\) directions for residual variation.
The degrees-of-freedom count alone does not prove the chi-square distribution or independence. Those stronger results follow from the geometry of a normal error vector. Let \(Y\) be the vector of observations, let \(\boldsymbol{\varepsilon}\) be the vector of errors, and write the design matrix as \(X\), with a column of ones and a column of predictor values. The least-squares fitted vector is the projection of \(Y\) onto the two-dimensional column space of \(X\). The residual vector is the projection onto the orthogonal complement of that space.
The column space of \(X\) has dimension 2. Its orthogonal complement has dimension \(n-2\), because the design matrix has rank 2.
Use an orthonormal basis whose first two directions span the column space of \(X\) and whose remaining directions span its orthogonal complement.
The error vector has distribution \(N(0,\sigma^2 I)\). An orthogonal change of coordinates preserves this distribution, so the transformed error coordinates remain independent \(N(0,\sigma^2)\) variables.
The fitted coefficients depend on the first two coordinates, while the residual sum of squares is the sum of squares of the remaining \(n-2\) coordinates. These two groups of independent normal coordinates are independent.
Derivation of the Chi-Square Result and Independence
The mean vector of \(Y\) is \(X\boldsymbol{\beta}\), where \(\boldsymbol{\beta}=(\beta_0,\beta_1)^\mathsf{T}\). It lies entirely in the column space of \(X\). Thus, after projecting onto the orthogonal complement, the mean contributes nothing: the residual vector is the projection of the error vector onto that complement. In the orthonormal coordinates described above, let \(Z_1,\ldots,Z_{n-2}\) be the error coordinates in the complement, each divided by \(\sigma\). They are independent \(N(0,1)\) variables. Therefore,
The same coordinates explain independence. The least-squares coefficient estimates are determined by the projection of \(Y\) onto the column space of \(X\), so they depend only on the first two transformed coordinates. The residual sum of squares depends only on the remaining coordinates. Since the two groups are independent under the normal model, \(RSS\) is independent of the pair \((\hat{\beta}_0,\hat{\beta}_1)\). It is consequently independent of each estimate separately and of any function of the two estimates.
Normality is essential for this exact independence argument: uncorrelated components of a general error vector need not be independent. The chi-square distribution is also an exact finite-sample result, not a large-sample approximation. If the errors are not normal, the residual sum of squares may still be useful, but this exact chi-square distribution and the stated independence do not generally follow.
Worked Applications
Worked Example: Computing a Residual Sum of Squares and Its Distribution
A hypothetical laboratory studies the relation between a fixed instrument setting \(x\) and a measured response \(Y\). The four settings are \(0,1,2,3\), and the recorded responses are \(2,4,5,9\). Assume the simple regression model with independent errors \(\varepsilon_i\sim N(0,0.9)\). Find the least-squares line, the observed residual sum of squares, and the probability that a repeated sample under this model would have residual sum of squares at least \(3.6\).
Method. Calculate the least-squares coefficients and residuals, then standardize the residual sum of squares by the stated error variance. There are \(n=4\) observations and two fitted coefficients, so the reference distribution has \(4-2=2\) degrees of freedom.
First, \(\bar{x}=(0+1+2+3)/4=1.5\) and \(\bar{y}=(2+4+5+9)/4=5\). The centered predictor sum of squares is
The centered cross-product is
Thus \(\hat{\beta}_1=11/5=2.2\), and \(\hat{\beta}_0=5-(2.2)(1.5)=1.7\). The fitted values at the four settings are \(1.7,3.9,6.1,8.3\), giving residuals \(0.3,0.1,-1.1,0.7\). Hence,
As a check, the residuals sum to \(0.3+0.1-1.1+0.7=0\), and their predictor-weighted sum is \(0(0.3)+1(0.1)+2(-1.1)+3(0.7)=0\), as required by the least-squares equations.
For repeated samples, \(RSS/0.9\sim\chi^2_2\). The requested threshold standardizes to \(3.6/0.9=4\), so the probability is \(P(\chi^2_2\ge4)=e^{-4/2}=e^{-2}=0.1353\), to four decimal places. The observed sample has \(RSS/0.9=1.8/0.9=2\); that observed value is not the threshold used in the probability calculation.
Common mistake avoided: Using \(n=4\) degrees of freedom would ignore the two fitted coefficients. The correct degrees of freedom are 2, and the calculation standardizes by the model variance \(0.9\), not by the observed \(RSS\).
Worked Example: A Tail Probability and the Error-Variance Estimate
In a hypothetical calibration study, six fixed settings are used in a simple linear regression with independent normal errors of variance \(\sigma^2=5\). For one sample, the residual sum of squares is \(30\). Find the observed value of \(RSS/\sigma^2\), the degrees of freedom, the probability under the model that a repeated sample has \(RSS\ge30\), and the estimate \(s^2\).
Method. Apply the chi-square result with \(n-2\) degrees of freedom, then use the chi-square survival probability for four degrees of freedom. Since two coefficients are fitted, \(n-2=6-2=4\).
The observed standardized residual sum of squares is \(30/5=6\). Therefore, \(RSS/5\sim\chi^2_4\), and
The estimate of the error variance is \(s^2=RSS/(n-2)=30/4=7.5\). The value \(7.5\) is the estimate from this sample; the model variance specified in the problem remains \(5\). As a check, the observed standardized value also satisfies \((n-2)s^2/\sigma^2=(4)(7.5)/5=6\), matching \(RSS/\sigma^2\).
Common mistake avoided: The tail probability is not \(P(\chi^2_6\ge6)\). Six is the sample size, but estimating the intercept and slope leaves four residual degrees of freedom.
Worked Example: Using Independence to Combine Probabilities
A hypothetical instrument-response study uses the fixed settings \(x=0,1,2,3,4\), under a simple regression model with independent normal errors of variance \(4\). The observed-data procedure produces a slope estimate \(\hat{\beta}_1\) and residual sum of squares \(RSS\). Find the probability that both \(\hat{\beta}_1\) is within two of its own standard deviations of \(\beta_1\) and \(RSS\ge24\).
Method. Compute the marginal probability for each event, then multiply them. This multiplication is justified by the independence of the least-squares coefficient estimates and \(RSS\) under normal errors.
The predictor mean is \(\bar{x}=2\), so
Under the stated model, the slope estimate has mean \(\beta_1\) and variance \(\sigma^2/S_{xx}=4/10=0.4\). Thus the event that it is within two standard deviations of \(\beta_1\) has probability \(P(-2\le Z\le2)=0.9545\), where \(Z\sim N(0,1)\).
There are \(5-2=3\) residual degrees of freedom. The event \(RSS\ge24\) is equivalent to \(RSS/4\ge6\), so its probability is \(P(\chi^2_3\ge6)=0.1116\). By independence, the probability of both events is
The result is approximately \(0.1065\). The two marginal probabilities concern different parts of the orthogonal decomposition: the slope estimate is determined by the fitted subspace, while \(RSS\) is determined by its orthogonal complement.
Common mistake avoided: Multiplying marginal probabilities is not justified merely because two statistics have different formulas. Here it is justified by the normal-error model and the resulting independence.
When the Result Does Not Apply
The exact statement relies on a full-rank two-parameter regression and independent \(N(0,\sigma^2)\) errors. If every predictor value is identical, the slope and intercept cannot be separately identified. The design matrix then has rank 1, and the two-parameter result with \(n-2\) residual degrees of freedom does not apply. If a different model is fitted, its rank and number of estimated parameters determine the residual dimension; one should not automatically reuse \(n-2\).
Equal error variances and independence matter as well. With correlated errors or unequal variances, the residual coordinates generally do not become independent, equal-variance normal coordinates under the ordinary least-squares projection. With nonnormal errors, an orthogonal transformation preserves zero covariance but does not generally ensure independence. These conditions are why the result is stated for the model, rather than as a property of residuals from any fitted line.
Check Your Understanding
For each problem, use the stated regression assumptions and distinguish the observed residual sum of squares from its sampling distribution.
- A simple regression has \(n=9\), and its observed \(RSS\) is 28. If the error variance is 4, state the distribution and observed value of \(RSS/\sigma^2\). (Answer: \(\chi^2_7\); observed value \(28/4=7\).)
- A simple regression has \(n=5\) and independent normal errors with variance 2. Find \(P(RSS>8)\). (Answer: \(P(\chi^2_3>4)\approx0.2615\).)
- A simple regression has \(n=7\) and independent normal errors with variance 3. Find \(P(RSS\ge30)\). (Answer: \(P(\chi^2_5\ge10)\approx0.0752\).)
- In a proposed simple regression, all six fixed predictor values equal 12. Can the standard two-parameter result \(RSS/\sigma^2\sim\chi^2_{n-2}\) be used? (Answer: No; the intercept and slope are not separately identifiable because the design matrix has rank 1.)
- In a full-rank simple regression, the errors are independent, have mean zero and common variance, but are not normally distributed. Is it guaranteed that \(RSS\) is independent of the coefficient estimates and that \(RSS/\sigma^2\sim\chi^2_{n-2}\)? (Answer: No; the exact chi-square distribution and independence are not guaranteed without normal errors.)