Why Residuals Are Not the Unobserved Errors

Introduction

A residual plot can help reveal whether a regression model fits the observed data well. But its points are not the underlying errors themselves. The distinction matters: errors are defined relative to the true regression mean, while residuals are calculated relative to a line estimated from the same data. Since that fitted line is not generally the true line, a residual and its corresponding error need not be equal.

This tutorial defines both quantities, works through their relationship, and explains what that relationship means when interpreting regression diagnostics. The examples use hypothetical data; in an actual analysis, the model errors are unobserved.

The Model Error and the Fitted Residual

Consider simple linear regression with a quantitative response \(Y_i\) and a predictor value \(x_i\). For the distinction in this tutorial, assume the conditional mean of the response is linear:

Y_i=\beta_0+\beta_1x_i+\varepsilon_i, \qquad E(\varepsilon_i\mid x_i)=0.

Here, \(\beta_0+\beta_1x_i\) is the true conditional mean of \(Y_i\) at \(x_i\), and \(\varepsilon_i\) is the model error for observation \(i\). The mean-zero condition says that, at a given predictor value, the errors have average zero. This model statement does not require errors to be normally distributed for the definitions that follow.

Definition: The model error for observation \(i\) is \(\varepsilon_i=Y_i-(\beta_0+\beta_1x_i)\), the deviation from the true conditional mean. Given estimated coefficients \(b_0\) and \(b_1\), the fitted value is \(\hat{y}_i=b_0+b_1x_i\), and the residual is \(e_i=y_i-\hat{y}_i\), the deviation from the fitted line.

The model error depends on the true coefficients, which are usually unknown. The residual is calculable from the observed response and the fitted regression line. It is therefore available for diagnostics, but it is not a direct measurement of \(\varepsilon_i\).

The definitions already show why the two quantities can differ. A residual compares an observation with an estimated mean; an error compares it with the true mean. If the estimated line differs from the true line, the observation is being compared with two different reference values.

A Worked Comparison

Worked Example: Comparing Errors and Residuals

Suppose a hypothetical data set comes from the model \(Y_i=10+2x_i+\varepsilon_i\). For illustration, the model errors are known to be \(2,-1,-1,0\). In a real data analysis, these errors would not be known. Calculate the observed responses, fit a least-squares line, and compare the residuals with the errors.

The predictor values are \(0,1,2,3\). Adding each stated error to the model mean gives the responses \(12,11,13,16\), respectively. Thus \(\bar{x}=1.5\) and \(\bar{y}=13\). The least-squares slope is

b_1= \frac{\sum (x_i-\bar{x})(y_i-\bar{y})} {\sum (x_i-\bar{x})^2} = \frac{7}{5}=1.4.

For example, the four products in the numerator are \(1.5,1,0,4.5\), which sum to \(7\); the denominator is \(2.25+0.25+0.25+2.25=5\). The intercept is

b_0=\bar{y}-b_1\bar{x} =13-(1.4)(1.5)=10.9.

The fitted line is \(\hat{y}_i=10.9+1.4x_i\). Its fitted values at \(x=0,1,2,3\) are \(10.9,12.3,13.7,15.1\). Subtracting each fitted value from the corresponding observed response gives residuals \(1.1,-1.3,-0.7,0.9\). For the first observation, for instance, the model error is \(12-10=2\), while the residual is \(12-10.9=1.1\). They differ because the fitted intercept and slope are not the true intercept and slope.

The squared residuals sum to \(1.1^2+(-1.3)^2+(-0.7)^2+0.9^2=4.20\). The squared model errors sum to \(2^2+(-1)^2+(-1)^2+0^2=6\). These are sums for two different sets of deviations, so there is no reason for them to match. In this example, the fitted line produces a smaller sum of squared deviations for these observations than the stated true line does.

Why Estimating the Line Changes the Residuals

The difference can be written exactly. Start with the definition of the model error, rearrange it to express the response, and substitute that expression into the residual:

\begin{aligned} Y_i&=\beta_0+\beta_1x_i+\varepsilon_i,\\ e_i&=Y_i-(b_0+b_1x_i)\\ &=\varepsilon_i+(\beta_0-b_0)+(\beta_1-b_1)x_i. \end{aligned}

The last two terms account for differences between the true coefficients and their estimates. Unless their combined contribution is zero at observation \(i\), the residual is not the model error. This identity also clarifies why a residual can be closer to zero than the corresponding error: estimating the line can move the fitted value toward the observed response.

Key result: For a linear model with true conditional mean \(\beta_0+\beta_1x_i\) and fitted line \(b_0+b_1x_i\), \(e_i=\varepsilon_i+(\beta_0-b_0)+(\beta_1-b_1)x_i\). A residual contains both the model error and the effect of coefficient-estimation differences.

This result is not a way to recover the error from the residual in ordinary data analysis: the true coefficients in the correction are unknown. It is a way to understand why treating a residual as if it were the corresponding error is generally unjustified.

Worked Example: A Second Hypothetical Data Set

Suppose \(Y_i=5+3x_i+\varepsilon_i\), and a hypothetical sample has \(x=0,1,2\) with errors \(2,-1,0\). Calculate the residuals and compare them with the stated errors.

Adding the errors to the true model means gives responses \(7,7,11\). Here, \(\bar{x}=1\) and \(\bar{y}=25/3\). The least-squares slope is

b_1= \frac{(-1)(7-25/3)+(0)(7-25/3)+(1)(11-25/3)} {(-1)^2+0^2+1^2} = \frac{4}{2}=2.

The intercept is \(b_0=25/3-2(1)=19/3\), so the fitted line is \(\hat{y}=19/3+2x\). The fitted values are \(19/3,25/3,31/3\). Subtracting them from the observed responses gives residuals \(2/3,-4/3,2/3\), approximately \(0.667,-1.33,0.667\).

The corresponding model errors were \(2,-1,0\). For the third observation, the error is zero because its response equals the true mean \(11\), yet its residual is \(11-31/3=2/3\). The fitted line does not pass through the true mean at every predictor value. Conversely, the second residual is \(-4/3\), not its model error of \(-1\). Both differences are consistent with the identity above.

What Residuals Can Tell Us

Residuals remain useful because they show how the fitted model differs from the observed responses. Their patterns can provide evidence about whether the chosen mean structure describes the data adequately. For example, systematic curvature in residuals can indicate that a straight-line mean is inadequate, while a changing spread can indicate that the model's error variability is not constant. These are diagnostic clues, not direct observations of the errors.

Several limits follow from the distinction:

A residual's sign describes whether the observed response is above or below the fitted value. It does not, by itself, say whether the response is above or below the true conditional mean.
A residual near zero means the observation is close to the fitted line. It does not establish that the model error is near zero.
A residual pattern is evidence about model fit, but it is not a plot of the unobserved errors. Interpreting it requires remembering that the line was estimated from the data being examined.

In the earlier material on residual diagnostics, plots were used to look for patterns in fitted residuals. That practice remains appropriate. The careful conclusion is that a plot shows how observed responses depart from an estimated model—not that it reveals the exact errors that would be seen relative to the true mean.

Worked Example: A New Observation and Its Prediction Deviation

Return to the first hypothetical model, whose true mean is \(10+2x\), and whose fitted line from the four observations is \(10.9+1.4x\). Suppose a new observation at \(x=4\) has response \(18.5\). For illustration, assume the true model error for this new observation is \(0.5\), since its true mean is \(10+2(4)=18\).

The prediction from the fitted line is \(10.9+1.4(4)=16.5\). The new observation's deviation from that prediction is \(18.5-16.5=2.0\). Its model error, relative to the true mean, is \(18.5-18=0.5\). The difference is \(2.0-0.5=1.5\), which is exactly the difference between the true mean and fitted value at \(x=4\): \(18-16.5=1.5\).

Because this new point was not used to estimate the displayed line, its deviation is best described here as a prediction error or prediction deviation, rather than a fitted residual from that four-observation regression. The calculation makes the same central distinction: deviation from a fitted prediction need not equal deviation from the true mean.

Common Mistake

Incorrect reasoning: “The residual for an observation is its true error, because both are the observed response minus a regression line.”

The correction is that the lines are different reference values. A model error subtracts the true conditional mean, while a residual subtracts the estimated fitted value. The true mean is generally unknown, so the error is not directly available. A residual is an observable estimate-like diagnostic quantity, not the hidden error with a different name.

Another related mistake is to interpret a small residual as proof that an observation had a small model error. The first worked example shows why this need not hold: the residual \(1.1\) accompanies a model error of \(2\). The fitted line has absorbed part of the departure by shifting its estimated coefficients.

Key takeaway: Model errors are deviations from the true conditional mean; residuals are deviations from the fitted regression line. Residuals let us examine the fit, but coefficient estimation means they are not direct measurements of the unobserved errors.

Check Your Understanding

Answer each question using the distinction between the true mean and the fitted line.

In one sentence each, define a model error and a fitted residual.
If an observation has a residual of zero, what does that establish about its observed response and fitted value? What does it not establish about its model error?
In the identity \(e_i=\varepsilon_i+(\beta_0-b_0)+(\beta_1-b_1)x_i\), which terms represent coefficient-estimation differences?
Why is a residual plot not literally a plot of the model errors?
For an observation not used to fit a line, explain why its deviation from the fitted prediction can differ from its model error.