Why Estimate the Error Variance?
A fitted regression line describes the estimated mean response at each predictor value. The observations do not usually fall exactly on that line. Their deviations from it provide information about the variability of the unobserved errors, which is summarized by the error variance. Estimating that variance is also a necessary step in later calculations of uncertainty for regression estimates.
The key detail is that the residuals are calculated after estimating the regression line from the same data. Consequently, their squared values are not averaged over \(n\) unrestricted deviations. Fitting an intercept and a slope uses two degrees of freedom, so the appropriate denominator is \(n-2\).
Model and Assumptions
Use the simple linear regression model with an intercept, \(Y_i=\beta_0+\beta_1x_i+\varepsilon_i\), for \(i=1,\ldots,n\). The predictor values \(x_i\) are treated as fixed, and they must not all be equal, so that a slope can be estimated. For the usual variance estimate to be unbiased, assume that, conditional on the predictor values, the errors have mean zero, common variance \(\sigma^2\), and zero covariance between different observations. Thus \(E[\varepsilon_i\mid x_1,\ldots,x_n]=0\), \(\operatorname{Var}(\varepsilon_i\mid x_1,\ldots,x_n)=\sigma^2\), and \(\operatorname{Cov}(\varepsilon_i,\varepsilon_j\mid x_1,\ldots,x_n)=0\) for \(i\ne j\).
These conditions do not require a particular distributional shape for the errors. In particular, normality is not needed for the unbiasedness result below. If the error variances differ across observations or the errors are correlated, \(\operatorname{SSE}/(n-2)\) is not generally an unbiased estimate of a single common error variance.
The fitted line estimates two parameters, \(\beta_0\) and \(\beta_1\). Its residuals obey two constraints: they sum to zero, and their products with the centered predictor values sum to zero. These restrictions express that the fitted line has already used information in the data to select its level and slope.
Why the Denominator Is \(n-2\)
The adjustment can be justified by finding the expected residual sum of squares under the stated model. Write \(\bar{x}=\sum_i x_i/n\) and \(S_{xx}=\sum_i(x_i-\bar{x})^2\). The least-squares slope is \(\hat{\beta}_1=\beta_1+\sum_i(x_i-\bar{x})\varepsilon_i/S_{xx}\), and the intercept is estimated using the sample means. Subtracting the fitted line from the observations removes the part of the errors that determines the estimated intercept and slope.
In particular, the residual sum of squares can be written as the total squared errors minus the squared component used to estimate the intercept and the squared component used to estimate the slope:
Now take conditional expectations given the predictor values. The first term has expectation \(n\sigma^2\), because it is a sum of \(n\) squared, mean-zero errors with variance \(\sigma^2\). The sum of the errors has variance \(n\sigma^2\), so the expected second term is \(\sigma^2\). The weighted sum in the third term has variance \(\sigma^2S_{xx}\), so the expected third term is also \(\sigma^2\). Therefore:
Dividing by \(n-2\) therefore gives an unbiased estimate of \(\sigma^2\) under these assumptions. This is the residual degrees-of-freedom adjustment: two degrees of freedom are used to fit the intercept and slope. The residual standard error \(s\) is the square root of this variance estimate; it is not itself an estimate of variance.
Calculating and Interpreting the Estimate
For each observation, subtract its fitted response from its observed response: \(e_i=y_i-\hat{y}_i\).
Square each residual and add the squares: \(\operatorname{SSE}=\sum_i e_i^2\).
For an intercept-and-slope fit, calculate \(s^2=\operatorname{SSE}/(n-2)\).
The residual standard error is \(s=\sqrt{s^2}\), expressed in the same units as the response.
The units provide a useful check. If the response is measured in minutes, each residual is in minutes, \(\operatorname{SSE}\) and \(s^2\) are in minutes squared, and \(s\) is in minutes. If the response is recorded in thousands of dollars, then \(s\) is in thousands of dollars unless it is converted to dollars.
Worked Example: Estimating Variability in Hypothetical Claim Amounts
A hypothetical analyst regresses five claim amounts, recorded in thousands of dollars, on a risk score. The fitted line is \(\hat{y}=20+3x\). For \(x=1,2,3,4,5\), the observed responses are \(24,24,29,34,34\), respectively. Estimate the error variance and residual standard error. Assume the errors have conditional mean zero, common variance \(\sigma^2\), and zero covariance across observations; no particular distribution is assumed. Interest and mortality assumptions do not apply to this regression calculation.
Method and residuals. Subtract each fitted value from the observed response. The fitted values are \(23,26,29,32,35\), so the residuals are \(1,-2,0,2,-1\) thousand dollars.
Check the fitted residuals. Their sum is \(1-2+0+2-1=0\). Their predictor-weighted sum is \(1(1)+2(-2)+3(0)+4(2)+5(-1)=0\), consistent with an intercept-and-slope least-squares fit.
Calculate the estimate. The residual sum of squares is \(1^2+(-2)^2+0^2+2^2+(-1)^2=10\) in squared thousands of dollars. There are \(n-2=5-2=3\) residual degrees of freedom, so \(s^2=10/3=3.3333\) in squared thousands of dollars. Taking the square root gives \(s=\sqrt{10/3}=1.8257\) thousand dollars.
Interpretation and check. Rounded to two decimals, the residual standard error is \(1.83\) thousand dollars, or \(\$1{,}825.74\). It describes the estimated scale of response deviations from the fitted line under the model. The common mistake is to divide by five: that would give \(10/5=2\), not the residual degrees-of-freedom estimate. No interest rate or mortality assumption enters this hypothetical example.
Worked Example: Computing the Estimate From Residuals in Minutes
A hypothetical service team fits a simple linear regression of response time, in minutes, on a workload measure for six days. The residuals, in day order, are \(1,-2,1,-1,2,-1\) minutes, from an intercept-and-slope least-squares fit. Estimate the error variance and residual standard error. Assume conditional mean-zero, uncorrelated errors with a common variance; no distributional form is specified. There are no interest or mortality assumptions in this setting.
Method and residual check. The residuals sum to \(1-2+1-1+2-1=0\). For predictor values \(1,2,3,4,5,6\), their predictor-weighted sum is \(1-4+3-4+10-6=0\), as required for these fitted residuals.
Calculate the residual sum of squares. Squaring and adding gives \(1+4+1+1+4+1=12\) minutes squared.
Apply the degrees-of-freedom adjustment. There are \(6-2=4\) residual degrees of freedom. Thus \(s^2=12/4=3\) minutes squared, and \(s=\sqrt{3}=1.7321\) minutes.
Interpretation and check. Rounded to three decimals, the residual standard error is \(1.732\) minutes. Squaring it returns approximately \(3\) minutes squared, the estimated error variance. A common mistake is to report \(3\) minutes as the residual standard error: that is the variance estimate, and its units are squared.
Worked Example: Converting From Thousands of Dollars
A hypothetical model of repair costs, recorded in thousands of dollars, has four observations and residuals \(0.4,-0.4,-0.4,0.4\). The fitted model includes an intercept and slope. Assume errors with conditional mean zero, common variance, and zero covariance; do not assume a specific distribution. Interest and mortality assumptions are not relevant. Find the residual standard error in both recorded units and dollars.
Method and residual check. The residuals sum to \(0.4-0.4-0.4+0.4=0\). For predictor values \(1,2,3,4\), the weighted sum is \(0.4-0.8-1.2+1.6=0\), consistent with the least-squares fit.
Calculate the variance estimate. The residual sum of squares is \(4(0.4^2)=4(0.16)=0.64\) squared thousands of dollars. The residual degrees of freedom are \(4-2=2\). Therefore \(s^2=0.64/2=0.32\) squared thousands of dollars.
Take the square root and convert units. The residual standard error is \(s=\sqrt{0.32}=0.565685\) thousand dollars. Since one thousand dollars is \(\$1{,}000\), this equals \(\$565.685\), or \(\$565.69\) rounded to the nearest cent.
Interpretation and check. The answer in thousands is \(0.57\) to two decimals; the more precise dollar answer is \(\$565.69\). The common mistake is to convert the variance estimate \(0.32\) directly into dollars. Take the square root first: the variance is in squared thousands of dollars, while the standard error is in thousands of dollars.
Common Traps and Limits
- Using \(n\) in the denominator. The residuals come from a line that estimated two parameters. For a simple regression with an intercept, use \(n-2\), not \(n\).
- Confusing variance and standard error. \(s^2\) estimates \(\sigma^2\) and has squared response units. \(s\) is its square root and has response units.
- Forgetting the model being fitted. The \(n-2\) denominator applies to a simple linear regression that estimates an intercept and a slope. It is not a universal denominator for every model.
- Ignoring the assumptions. The usual unbiasedness result relies on a common error variance and zero covariance across observations. If these conditions fail, the formula can still be calculated, but its usual interpretation as an unbiased estimate of a common \(\sigma^2\) is not justified.
The residual standard error is a measure of response-scale scatter around the fitted line under the model. It is not the error variance, and it does not by itself establish that the model assumptions hold. For calculations, keep the sequence clear: square residuals, sum them, divide by \(n-2\), and take a square root only when a response-unit measure is requested.
Check Your Understanding
For each problem, assume a simple linear regression with an intercept and slope unless stated otherwise. When a distributional form is not specified, no normality assumption is needed for the variance estimate.
- A fitted regression has \(n=9\) observations and \(\operatorname{SSE}=56\) response-units squared. Find \(s^2\) and \(s\). (Answer: \(s^2=8\); \(s=2.828\) response units.)
- A fitted model has 12 observations and residuals whose squared values sum to 44. Find the residual degrees of freedom, the estimated error variance, and the residual standard error. (Answer: 10 degrees of freedom; \(s^2=4.4\); \(s=2.098\) response units.)
- A regression has five observations and \(\operatorname{SSE}=18\) minutes squared. A candidate divides by five and reports a residual standard error of \(\sqrt{3.6}\) minutes. Correct the calculation. (Answer: divide by \(5-2=3\); \(s^2=6\) minutes squared and \(s=\sqrt{6}=2.449\) minutes.)
- A hypothetical repair-cost regression uses seven observations, with \(\operatorname{SSE}=24\) squared thousands of dollars. Find the estimated variance and express the residual standard error in dollars to the nearest cent. (Answer: \(s^2=24/5=4.8\) squared thousands of dollars; \(s=\sqrt{4.8}\) thousand dollars, or \(\$2{,}190.89\).)
- A simple regression with an intercept and slope has \(n=10\) and \(\operatorname{SSE}=27\). State the estimated error variance and residual standard error, and identify the condition needed for the usual unbiasedness interpretation. (Answer: \(s^2=27/8=3.375\); \(s=\sqrt{3.375}=1.837\) response units; the errors must have a common variance and zero covariance across observations, as well as conditional mean zero.)