Residuals Versus Unobserved Errors

Why Residuals and Errors Are Different

In regression, the observed response for an individual is described using a systematic part and a random part. The random part is the model error, which is not directly observed: the true regression coefficients and the error are not separately revealed by the data. After estimating the coefficients, we can calculate a residual for each observation. Residuals are observable, but they are not generally the same as the errors.

This distinction matters when interpreting a regression fit. A residual measures the difference between an observed response and its fitted value. A model error measures the difference between that response and its true conditional mean. Since the fitted value uses estimated coefficients, the two differences need not agree.

Definition: For observations \(i=1,\ldots,n\), the simple linear regression model is \(Y_i=\beta_0+\beta_1x_i+\varepsilon_i\), where \(x_i\) is the predictor value, \(\beta_0+\beta_1x_i\) is the true conditional mean, and \(\varepsilon_i\) is the model error. Typically, the model assumes \(E[\varepsilon_i\mid x_i]=0\). The observed response is \(y_i\). Given least-squares estimates \(b_0\) and \(b_1\), the fitted value is \(\hat{y}_i=b_0+b_1x_i\), and the residual is \(e_i=y_i-\hat{y}_i\). The errors \(\varepsilon_i\) are unobserved; the residuals \(e_i\) can be calculated from the data.

The model describes a response around the true regression line, while the residual is calculated around the estimated line. Once the data have been observed, each \(e_i\) is a known number. The corresponding \(\varepsilon_i\) remains unknown because the true coefficients are not known. The zero conditional mean assumption describes the errors across repeated observations at a predictor value; it does not make any particular realized error observable.

How Estimation Connects Them

Assume the stated linear model applies and least squares is used to estimate its intercept and slope from the observed data. Since \(y_i=\beta_0+\beta_1x_i+\varepsilon_i\), subtracting the fitted value \(b_0+b_1x_i\) gives the relation between the residual and the model error.

\begin{aligned} e_i &=y_i-(b_0+b_1x_i)\\ &=\beta_0+\beta_1x_i+\varepsilon_i-b_0-b_1x_i\\ &=\varepsilon_i-(b_0-\beta_0)-(b_1-\beta_1)x_i. \end{aligned}

This is a decomposition, not a way to recover the unknown error from the residual: the coefficient differences \(b_0-\beta_0\) and \(b_1-\beta_1\) are also unknown. If the fitted coefficients happened to equal the true coefficients, the residual and error would coincide for every observation. In general, however, coefficient estimation shifts the fitted line, and that shift changes each residual.

The shift depends on \(x_i\). Therefore, even if the difference between the fitted and true intercepts is the same for every observation, the slope estimation difference contributes differently at different predictor values. This is why there is no general rule that a residual is the corresponding error plus one common adjustment.

Start with the model.
The observed response is the true mean \(\beta_0+\beta_1x_i\) plus the unobserved error \(\varepsilon_i\).

Calculate the fitted value.
Use the estimated coefficients to obtain \(\hat{y}_i=b_0+b_1x_i\).

Subtract the fitted value from the observation.
The result \(e_i=y_i-\hat{y}_i\) is the observable residual. It differs from \(\varepsilon_i\) by the amount the fitted line differs from the true line at \(x_i\).

Worked Example: Calculating Residuals and Comparing Them With Errors

A hypothetical pricing study records response \(y_i\) at predictor values \(x_i=0,1,2\). Suppose the true model is \(Y_i=1+2x_i+\varepsilon_i\), and the realized errors for these three observations are \(1,0,1\), respectively. Calculate the data, fit the least-squares line, and compare the residuals with the errors. The true coefficients and errors are supplied only to make the comparison possible.

Step 1: Calculate the observed responses. From \(y_i=1+2x_i+\varepsilon_i\):

At \(x_1=0\), \(y_1=1+2(0)+1=2\).
At \(x_2=1\), \(y_2=1+2(1)+0=3\).
At \(x_3=2\), \(y_3=1+2(2)+1=6\).

Step 2: Find the least-squares coefficients. The means are \(\bar{x}=(0+1+2)/3=1\) and \(\bar{y}=(2+3+6)/3=11/3\). The centered sums are

\begin{aligned} S_{xx}&=(0-1)^2+(1-1)^2+(2-1)^2=2,\\ S_{xy}&=(0-1)(2-11/3)+(1-1)(3-11/3)+(2-1)(6-11/3)\\ &=5/3+0+7/3=4. \end{aligned}

Thus \(b_1=S_{xy}/S_{xx}=4/2=2\), and \(b_0=\bar{y}-b_1\bar{x}=11/3-2=5/3\). The fitted line is \(\hat{y}=5/3+2x\).

Step 3: Calculate fitted values and residuals. The fitted values at \(x=0,1,2\) are \(5/3,11/3,17/3\). Subtract each fitted value from its observed response:

\(e_1=2-5/3=1/3\).
\(e_2=3-11/3=-2/3\).
\(e_3=6-17/3=1/3\).

The errors were \(1,0,1\), whereas the residuals are \(1/3,-2/3,1/3\). For a direct check, \(b_0-\beta_0=5/3-1=2/3\) and \(b_1-\beta_1=2-2=0\). The relation \(e_i=\varepsilon_i-(b_0-\beta_0)-(b_1-\beta_1)x_i\) gives residuals \(1-2/3=1/3\), \(0-2/3=-2/3\), and \(1-2/3=1/3\), as calculated. Method: calculate the observations, fit by least squares, then compare. Mistake avoided: calling the supplied model errors the residuals.

Why the Formula Holds

The identity above follows directly from the two definitions: the model writes the observation relative to the true line, and the residual writes it relative to the fitted line. Subtracting the fitted line from the model equation leaves the error minus the difference between the fitted and true lines. The fitted-minus-true difference at \(x_i\) is \((b_0-\beta_0)+(b_1-\beta_1)x_i\). Therefore the exact relationship is

e_i=\varepsilon_i-\left[(b_0-\beta_0)+(b_1-\beta_1)x_i\right].

This identity does not require the estimates to be close to the true coefficients. It holds for any values of \(b_0\) and \(b_1\), provided the stated model describes \(y_i\). If the model is not appropriate, the identity as a representation of the model error is not justified: in particular, it relies on the model's true conditional mean being \(\beta_0+\beta_1x_i\).

Worked Example: Applying the Error–Residual Identity

A hypothetical study has true model \(Y_i=10+3x_i+\varepsilon_i\). For one observation, \(x_i=2\) and the realized error is \(-1\). A least-squares fit to the study gives \(b_0=11.5\) and \(b_1=2.5\). Calculate the response, fitted value, and residual, then verify the identity.

Step 1: Calculate the observed response. The true mean at \(x_i=2\) is \(10+3(2)=16\). Adding the error gives \(y_i=16-1=15\).

Step 2: Calculate the fitted value and residual. The fitted value is \(\hat{y}_i=11.5+2.5(2)=16.5\). Hence \(e_i=15-16.5=-1.5\).

Step 3: Check using the coefficient differences. Here \(b_0-\beta_0=11.5-10=1.5\), and \(b_1-\beta_1=2.5-3=-0.5\). Thus

$$ e_i=-1-1.5-(-0.5)(2)=-1-1.5+1=-1.5. $$

The result agrees with the direct calculation. The error is \(-1\), but the residual is \(-1.5\) because the fitted line is 0.5 above the true line at \(x_i=2\). Method: verify the residual using both definitions. Mistake avoided: assuming that a known error alone determines the residual without accounting for coefficient estimation.

A Zero Residual Does Not Mean a Zero Error

A residual of zero says that the observed response lies exactly on the estimated line at that predictor value. It says nothing by itself about whether the response lies on the true regression line. The estimated line can differ from the true line in a way that absorbs part, or all, of the realized errors.

Worked Example: Nonzero Errors With Zero Residuals

Suppose a hypothetical dataset has \(x=0,1,2\), and the true model is \(Y=5+2x+\varepsilon\). The realized errors are \(1,2,3\), respectively. Find the observations and least-squares residuals. Explain what the result shows.

Step 1: Calculate the observations. At \(x=0\), \(y=5+0+1=6\). At \(x=1\), \(y=5+2+2=9\). At \(x=2\), \(y=5+4+3=12\). The data are therefore \((0,6),(1,9),(2,12)\).

Step 2: Fit a line. The three points lie on \(y=6+3x\), so the least-squares line has \(b_0=6\) and \(b_1=3\). At \(x=0,1,2\), the fitted values are \(6,9,12\).

Step 3: Calculate the residuals. Subtracting fitted values from observations gives \(6-6=0\), \(9-9=0\), and \(12-12=0\). Yet the model errors are \(1,2,3\), each nonzero. Indeed, \(b_0-\beta_0=1\) and \(b_1-\beta_1=1\), so the fitted line has absorbed the realized error pattern into its estimated coefficients. Method: compare observations with both lines. Mistake avoided: interpreting a perfect fit to this small dataset as proof that every model error was zero.

Common Mistakes and Interpretation

Using “error” and “residual” as interchangeable words. The error is defined relative to the true conditional mean; the residual is calculated relative to the fitted value.
Claiming a residual is directly observed error. The response and fitted coefficients are observed or calculated, but the true coefficients are unknown. The model error therefore cannot generally be isolated.
Assuming a small residual guarantees a small error. The fitted line can be close to an observation even when the true line is not, or vice versa. The residual alone does not reveal the coefficient-estimation difference.
Forgetting the sign convention. Here \(e_i=y_i-\hat{y}_i\). Reversing the subtraction changes the sign.

Residuals are useful observable quantities for assessing how responses differ from the estimated regression line. But conclusions about the unobserved errors require care: a residual is only an estimate-related representation of an error, not the error itself. The distinction is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Check Your Understanding

For each problem, distinguish the observed residual from the model error rather than treating them as identical.

A fitted line is \(\hat{y}=4+1.2x\). For an observation with \(x=3\) and \(y=8\), calculate its fitted value and residual. (Answer: fitted value \(7.6\); residual \(0.4\).)
The true model is \(Y=20+2x+\varepsilon\). At \(x=3\), the realized error is \(-1\), and the fitted line is \(\hat{y}=19+2.5x\). Calculate \(y\), \(\hat{y}\), and \(e\), and verify the error–residual identity. (Answer: \(y=25\), \(\hat{y}=26.5\), \(e=-1.5\); identity gives \(-1-(-1+1.5)=-1.5\).)
For three observations, \(x=1,2,3\) and \(y=4,6,11\). Calculate the least-squares line and all three residuals. (Answer: \(\hat{y}=3.5x\); residuals \(0.5,-1,0.5\).)
A true model is \(Y=5+2x+\varepsilon\). At \(x=1\), the realized error is \(2\), and the fitted line is \(\hat{y}=6+3x\). Find the observation and residual. Does a zero residual imply a zero error? (Answer: \(y=9\), \(\hat{y}=9\), \(e=0\); no, the error is \(2\).)
In the simple linear model, write \(e_i\) in terms of \(\varepsilon_i\), \(b_0-\beta_0\), and \(b_1-\beta_1\). State the condition under which the residual equals the error for every observation. (Answer: \(e_i=\varepsilon_i-(b_0-\beta_0)-(b_1-\beta_1)x_i\); they coincide for every observation if \(b_0=\beta_0\) and \(b_1=\beta_1\).)

What You'll Learn