Estimating Residual Variance and Degrees of Freedom

Why Divide the Residual Sum of Squares?

In simple linear regression, the residuals describe the observed vertical differences between the data and the fitted line. Squaring and adding those residuals gives a measure of the unexplained variation. But that sum is not itself an estimate of the error variance: its expected value depends on how many observations there are and how many regression coefficients were estimated from them. The denominator must account for both.

This tutorial develops that denominator. In a regression with an intercept and one slope, two degrees of freedom are used to fit the line, leaving \(n-2\) residual degrees of freedom. Dividing the residual sum of squares by \(n-2\) gives an unbiased estimate of the error variance under the model assumptions.

Model, Residuals, and Residual Sum of Squares

Suppose there are \(n\) observations, with predictor values \(x_i\) and response values \(y_i\), for \(i=1,\ldots,n\). The simple linear regression model is

Y_i=\beta_0+\beta_1x_i+\varepsilon_i.

Here \(\beta_0\) and \(\beta_1\) are the population intercept and slope, and \(\varepsilon_i\) is the random error for observation \(i\). To estimate the line, least squares produces fitted coefficients \(b_0\) and \(b_1\), and fitted values \(\hat{y}_i=b_0+b_1x_i\). The residual is the observed response minus its fitted value: \(e_i=y_i-\hat{y}_i\).

Definition: The residual sum of squares is \(SSE=\sum_{i=1}^{n}e_i^2\). If the model errors have common variance \(\sigma^2\), the residual variance estimate is \(\hat{\sigma}^2=SSE/(n-2)\), provided the predictor values are not all identical. Its units are the square of the response units.

The model assumptions relevant to this estimate are that the conditional mean response is linear in \(x_i\), the errors have mean zero and a common variance \(\sigma^2\), and errors from different observations are independent. The predictor values must vary, so that an intercept and slope can both be estimated. These conditions justify treating \(\hat{\sigma}^2\) as an unbiased estimate of \(\sigma^2\). A small set of residuals cannot establish that the assumptions hold; they must be considered in light of how the data were collected and the model’s fit.

Why the Residual Degrees of Freedom Are \(n-2\)

There are \(n\) observed responses, but the fitted residuals are not \(n\) freely varying quantities. Least squares chooses the intercept and slope to minimize \(SSE\). With an intercept, the fitted residuals satisfy two constraints:

\sum_{i=1}^{n}e_i=0 \qquad\text{and}\qquad \sum_{i=1}^{n}x_i e_i=0.

The first says the residuals sum to zero. The second says the residuals have zero sample cross-product with the predictor values. When the predictor varies, these are two independent constraints. Thus only \(n-2\) residual directions can vary freely. This remaining number is the residual degrees of freedom.

The same reasoning explains why dividing by \(n-2\), rather than \(n\), corrects the average squared residual. A compact derivation uses the geometry of least squares. Let \(X\) be the \(n\)-by-2 matrix whose columns contain the values of the intercept term (all ones) and the predictor. Since the predictor varies, \(X\) has rank 2. The matrix \(H=X(X'X)^{-1}X'\) projects onto the fitted-line space, and \(M=I-H\) projects onto its orthogonal complement. The residual vector is \(e=M\varepsilon\), so \(SSE=e'e=\varepsilon'M\varepsilon\).

Use the error assumptions.
The error vector has mean zero and covariance matrix \(\sigma^2 I\), because errors are independent and have the same variance.

Find the expected residual sum of squares.
For a fixed matrix \(M\), \(E(\varepsilon'M\varepsilon)=\operatorname{tr}(M E(\varepsilon\varepsilon'))=\sigma^2\operatorname{tr}(M)\).

Count the residual dimensions.
\(M=I-H\), and \(H\) projects onto a two-dimensional space, so \(\operatorname{tr}(M)=n-2\). Therefore \(E(SSE)=(n-2)\sigma^2\).

Consequently,

E\left(\frac{SSE}{n-2}\right)=\sigma^2.

This shows why the residual variance estimate is unbiased under the stated assumptions. The two degrees of freedom are used because the fitted line estimates two quantities: the intercept and the slope. This argument does not require normally distributed errors for unbiasedness.

Worked Example: Calculating Residual Variance From Five Observations

Suppose a hypothetical data set has predictor values \(x=-2,-1,0,1,2\), and the fitted regression line is \(\hat{y}=10+2x\). The response is measured in points. The observed responses are \(7,6,12,10,15\), respectively. Calculate the residual variance estimate.

Check the fitting setup. The predictor values vary, so the intercept and slope are identifiable. The stated line is the least-squares fit: its residuals below sum to zero and their cross-product with \(x\) is zero. For interpreting the variance estimate as unbiased, we also assume the linear model and independent, mean-zero errors with a common variance. The hypothetical data alone do not verify those assumptions.

Find each residual. The fitted values are \(6,8,10,12,14\). Subtracting each fitted value from its observed response gives residuals \(1,-2,2,-2,1\). Therefore,

$$ SSE=1^2+(-2)^2+2^2+(-2)^2+1^2=14. $$

There are \(n=5\) observations, so the residual degrees of freedom are \(5-2=3\), not 5. The residual variance estimate is

\hat{\sigma}^2=\frac{SSE}{n-2}=\frac{14}{3}=4.667\ \text{points}^2.

Under the assumptions, this is an unbiased estimate of the error variance in squared points.

Worked Calculations and Interpretation

Worked Example: Residual Variance With Six Observations

In a hypothetical analysis, \(x=0,1,2,3,4,5\), and the fitted line is \(\hat{y}=5+0.5x\). The responses, in millimetres, are \(6,3.5,7,7.5,5,8.5\). Calculate \(SSE\), the residual degrees of freedom, and the residual variance estimate.

The fitted values are \(5,5.5,6,6.5,7,7.5\), so the residuals are \(1,-2,1,1,-2,1\). These residuals sum to zero, and their cross-product with the predictor values is \(0-2+2+3-8+5=0\), consistent with the stated least-squares fit. The predictor varies. As before, interpreting the estimate as unbiased additionally assumes independent errors with mean zero and common variance, along with a linear conditional mean.

The residual sum of squares is

SSE=1^2+(-2)^2+1^2+1^2+(-2)^2+1^2=12\ \text{mm}^2.

With six observations and two fitted coefficients, the residual degrees of freedom are \(6-2=4\). Thus,

\hat{\sigma}^2=\frac{12}{4}=3.000\ \text{mm}^2.

This number estimates the variance of the regression errors, not the variance of the measured responses overall. Its units are square millimetres because the residuals are in millimetres and are squared.

Worked Example: Why the Denominator Is Not the Sample Size

Consider seven hypothetical observations with predictor values \(x=-3,-2,-1,0,1,2,3\) and fitted line \(\hat{y}=20+3x\). Suppose the observed responses, in points, are \(13,13,16,20,22,25,31\). Find the residual variance estimate and compare it with dividing \(SSE\) by \(n\).

The fitted values are \(11,14,17,20,23,26,29\). The residuals are therefore \(2,-1,-1,0,-1,-1,2\). Their sum is zero, and the cross-product with \(x\) is \((-3)(2)+(-2)(-1)+(-1)(-1)+0+(1)(-1)+(2)(-1)+(3)(2)=0\), so the given line satisfies the least-squares residual conditions. The predictor varies; the usual error assumptions are presumed for the variance interpretation, not proven by this example.

The squared residuals add to

SSE=2^2+(-1)^2+(-1)^2+0^2+(-1)^2+(-1)^2+2^2=12\ \text{points}^2.

Here \(n=7\), so \(df=7-2=5\), and

\hat{\sigma}^2=\frac{12}{5}=2.400\ \text{points}^2.

Dividing by \(n\) would give \(12/7=1.714\ \text{points}^2\), which is not the residual variance estimate for this model. The residuals are constrained by the two fitted coefficients; using \(n\) ignores the two degrees of freedom spent fitting the line.

Common Mistakes and Practical Meaning

The residual variance estimate summarizes the typical squared error around a fitted line, adjusted for estimating that line from the same data. It matters because later calculations of regression coefficient uncertainty use an estimate of the error variance. The estimate itself is not a measure of whether the slope is important, nor does it establish that the linear model is appropriate.

Using \(n\) in the denominator. This overlooks that least squares estimated both \(b_0\) and \(b_1\). For a simple regression with an intercept, use \(n-2\).
Subtracting only one degree of freedom. The denominator \(n-1\) is not the residual degrees of freedom here. The fitted intercept and slope use two degrees of freedom.
Calling \(SSE\) the variance estimate. \(SSE\) is a sum of squared residuals and grows with the number of observations. Divide by the residual degrees of freedom to estimate the error variance.
Ignoring units. If the response is measured in millimetres, the residual variance estimate is in square millimetres, not millimetres.
Assuming the formula proves the model is suitable. The calculation is always possible when the line can be fitted, but its interpretation relies on the model assumptions. A numerical result does not establish linearity, independence, or constant variance.

Key takeaway: In simple linear regression with an intercept, \(SSE\) is divided by \(n-2\) because fitting the intercept and slope imposes two independent constraints on the residuals. Under independent, mean-zero errors with common variance, \(SSE/(n-2)\) estimates \(\sigma^2\) without bias.

Check Your Understanding

Answer each question using the definitions and reasoning in this tutorial.

A simple regression has 12 observations, an intercept, and one slope. What are its residual degrees of freedom?
Residuals are \(1,-1,0,2,-2\). Calculate \(SSE\) and the residual variance estimate for a simple regression fitted to these five observations.
Explain why a fitted intercept and slope use two degrees of freedom rather than one.
If responses are recorded in centimetres, what are the units of the residual variance estimate?
Does calculating \(SSE/(n-2)\) demonstrate that the errors are independent and have constant variance? Explain.