Introduction
A regression line gives a fitted value for each observed predictor value. The difference between an observed response and that fitted value can be calculated directly, so it is tempting to treat this difference as the observation’s error. The distinction matters: a fitted value comes from estimated coefficients, while the model error is defined relative to the unknown regression relationship.
This tutorial separates the two quantities. A residual is observable once the data and fitted line are available. A model error is an unobserved component in the model for the response. Residuals provide information about errors, but they are not generally the errors themselves.
Model Errors and Fitted Residuals
For simple linear regression, let \(x_i\) be the predictor value for observation \(i\), and let \(Y_i\) be its response. The model is
Here \(\beta_0\) and \(\beta_1\) are the fixed, generally unknown population coefficients, and \(\varepsilon_i\) is the model error for observation \(i\). A standard regression model assumes that, conditional on the predictor values, the errors have mean zero and a common variance \(\sigma^2\); in the usual model they are also independent. Normality is an additional assumption used for some inference, but it is not needed to define an error or a residual.
The response \(Y_i\), predictor \(x_i\), fitted coefficients, fitted value, and residual can all be obtained from the observed sample. The model error is not directly observed because its definition uses the unknown population coefficients. The model’s assumptions describe the errors; they do not make the errors visible in the data.
| Quantity | Defined relative to | Observable from the sample? |
|---|---|---|
| Model error, \(\varepsilon_i\) | Population regression line using \(\beta_0,\beta_1\) | No, in general |
| Residual, \(e_i\) | Fitted line using \(b_0,b_1\) | Yes |
| Fitted value, \(\hat{Y}_i\) | Estimated regression relationship | Yes |
Why a Residual Is an Estimate of an Error
The distinction follows by substituting the model for \(Y_i\) into the definition of the residual. This also shows exactly how estimating the coefficients changes the residual from the corresponding error.
\(e_i=Y_i-(b_0+b_1x_i)\).
Since \(Y_i=\beta_0+\beta_1x_i+\varepsilon_i\), the residual becomes \(\(\beta_0+\beta_1x_i+\varepsilon_i\)-(b_0+b_1x_i)\).
Collecting terms gives \(e_i=\varepsilon_i-\{(b_0-\beta_0)+(b_1-\beta_1)x_i\}\).
The expression in braces is the difference between the fitted line and the population line at \(x_i\). If that difference is zero for an observation, its residual equals its error. Otherwise the residual is the error adjusted for the fitted line’s estimation difference at that predictor value. Because the true coefficients are unknown, this identity explains the relationship conceptually; it does not usually let an analyst calculate the actual error.
This is why residuals are used as estimates or diagnostic stand-ins for errors: they are computable and reflect departures from a fitted relationship. But the fitted relationship is estimated, so a residual need not equal the underlying error. This definition and identity apply to the ordinary simple linear regression model with an intercept. If a different model is fitted, its fitted value and residual must be defined using that model.
Worked Example: Fitting a Line and Comparing It with Known Errors
In a hypothetical simulation, three observations have predictor values \(x=0,1,2\) and responses \(Y=3,4,8\). The simulation used the true line \(Y=2+3x+\varepsilon\). Fit the ordinary least-squares line, calculate each residual, and compare the results with the model errors. The true line is supplied only because this is a simulation; in an ordinary data analysis it would be unknown.
Method and fitted line. The predictor mean is \(\bar{x}=(0+1+2)/3=1\), and the response mean is \(\bar{Y}=(3+4+8)/3=5\). The centered cross-product sum is
The centered predictor sum of squares is \(S_{xx}=(0-1)^2+(1-1)^2+(2-1)^2=1+0+1=2\). Thus \(b_1=S_{xy}/S_{xx}=5/2=2.5\), and \(b_0=\bar{Y}-b_1\bar{x}=5-(2.5)(1)=2.5\). The fitted line is \(\hat{Y}=2.5+2.5x\).
Residuals. At \(x=0\), the fitted value is \(2.5+2.5(0)=2.5\), so \(e=3-2.5=0.5\). At \(x=1\), the fitted value is \(2.5+2.5(1)=5\), so \(e=4-5=-1\). At \(x=2\), the fitted value is \(2.5+2.5(2)=7.5\), so \(e=8-7.5=0.5\).
Errors under the stipulated true model. The true line gives responses \(2+3(0)=2\), \(2+3(1)=5\), and \(2+3(2)=8\). Subtracting these true-line values from the observed responses gives errors \(3-2=1\), \(4-5=-1\), and \(8-8=0\). The residuals \(0.5,-1,0.5\) therefore differ from the errors \(1,-1,0\) at two of the three predictor values.
Check using the identity. Here \(b_0-\beta_0=2.5-2=0.5\) and \(b_1-\beta_1=2.5-3=-0.5\). At \(x=0\), the identity gives \(e=1-[0.5+(-0.5)(0)]=0.5\). At \(x=1\), it gives \(e=-1-[0.5-0.5]=-1\). At \(x=2\), it gives \(e=0-[0.5-1]=0.5\), matching all the calculated residuals. This example avoids the common mistake of calling the observable residual the exact model error.
Interpreting Residuals in Practice
A residual’s sign describes the observed response relative to the fitted line. A positive residual means the observation is above that line; a negative residual means it is below. Its magnitude is measured in the same units as the response. These statements concern the fitted line, not the unobserved population line.
For example, a hypothetical insurer might fit a line to relate a property characteristic to annual claim cost, measured in thousands of dollars. A residual of \(1.5\) means the observed cost is \(1.5\) thousand dollars above the fitted cost for that property. It does not establish that the property’s true model error is exactly \(1.5\) thousand dollars, because the fitted coefficients may differ from the population coefficients.
Worked Example: Residual for a Hypothetical Claim-Cost Model
A hypothetical claim-cost model uses roof age \(x\), measured in years, and annual cost \(Y\), measured in thousands of dollars. A fitted line from a sample is \(\hat{Y}=10.5+1.25x\). For a property with roof age \(4\), the observed annual cost is \(17\) thousand dollars. In the simulation that generated the data, the true line is \(Y=10+1.5x+\varepsilon\). Calculate the residual and, for comparison, the error.
Method and residual. The fitted value at \(x=4\) is \(\hat{Y}=10.5+1.25(4)=10.5+5=15.5\) thousand dollars. Therefore the observable residual is \(e=17-15.5=1.5\) thousand dollars.
Error under the known simulation truth. The true line’s mean response at \(x=4\) is \(10+1.5(4)=16\) thousand dollars. Thus the model error is \(\varepsilon=17-16=1\) thousand dollars. The residual exceeds the error by \(0.5\) thousand dollars because the fitted value is \(15.5\), while the true-line value is \(16\).
Identity check. The coefficient differences are \(b_0-\beta_0=10.5-10=0.5\) and \(b_1-\beta_1=1.25-1.5=-0.25\). At \(x=4\), their combined difference is \(0.5+(-0.25)(4)=-0.5\). Hence \(e=\varepsilon-(-0.5)=1+0.5=1.5\) thousand dollars. In real analysis, the residual can be reported, but the true error cannot be calculated this way because the population coefficients are not known.
The common mistake is to report \(1.5\) thousand dollars as the model error. It is the response’s deviation from the fitted line; in the stipulated simulation, the error is \(1\) thousand dollars.
Worked Example: When Residual and Error Coincide by Chance
In another hypothetical simulation, the true regression line is \(Y=50+4x+\varepsilon\), while the fitted line is \(\hat{Y}=47+4.5x\). Two observed responses are \(68\) at \(x=4\) and \(76\) at \(x=6\). For each observation, calculate the residual and the error under the known simulation model. Explain whether equality for one observation implies equality for the other.
At \(x=4\). The fitted value is \(47+4.5(4)=47+18=65\), so the residual is \(68-65=3\). The true-line value is \(50+4(4)=66\), so the error is \(68-66=2\). They differ by \(1\).
At \(x=6\). The fitted value is \(47+4.5(6)=47+27=74\), so the residual is \(76-74=2\). The true-line value is \(50+4(6)=74\), so the error is \(76-74=2\). Here the residual and error happen to be equal.
Why the equality is not a general rule. The fitted-minus-true line difference at \(x\) is \((47-50)+(4.5-4)x=-3+0.5x\). At \(x=6\), this difference is \(-3+3=0\), so the fitted and true lines coincide at that predictor value. At \(x=4\), it is \(-3+2=-1\), so the residual is the error minus \(-1\), or \(2+1=3\). The equality at \(x=6\) is a consequence of the particular lines crossing there, not a general property of residuals.
This example avoids the opposite mistake: assuming residuals and errors must always differ. They can coincide at a particular predictor value, but that coincidence does not make the two definitions interchangeable.
Common Mistakes and Why the Distinction Matters
- Calling a residual the actual error. A residual is calculated from the fitted line. Unless the population coefficients are known, the actual model error is not observed.
- Using the wrong reference line. Subtracting the fitted value gives a residual; subtracting the population mean response gives an error. Always identify which line is being used.
- Ignoring units. A residual has the response’s units. If responses are measured in thousands of dollars, a residual of \(1.5\) is \(1.5\) thousand dollars, not \(1.5\) dollars.
- Inferring too much from a single residual. A residual describes one observed response relative to the fitted line. It does not reveal the exact unobserved error or, by itself, establish why the response differs from the fit.
The distinction is useful whenever regression results are interpreted. Residuals are available for examining how observations compare with the fitted relationship. The model errors belong to the underlying statistical description and are not directly available for individual observations. Treating residuals as estimates, rather than identities, preserves that difference. This treatment of least-squares residuals and model errors is aligned with the SOA VEE Mathematical Statistics syllabus.
Check Your Understanding
For each problem, distinguish what can be calculated from the fitted line from what depends on the stipulated true model.
- A simulation has true line \(Y=5+2x+\varepsilon\). At \(x=4\), the observed response is \(14\), and the fitted line is \(\hat{Y}=4.5+2.2x\). Find the residual and the model error under the simulation truth. (Answer: Residual \(=0.7\); model error \(=1\).)
- A fitted line for a hypothetical response is \(\hat{Y}=12-0.8x\). An observation has \(x=5\) and \(Y=9\). Calculate its residual and state what its sign means. (Answer: Residual \(=1\); the observation is one response unit above the fitted value.)
- At \(x=6\), the fitted coefficients differ from the true coefficients by \(b_0-\beta_0=0.4\) and \(b_1-\beta_1=-0.1\). If the residual is \(-0.2\), find the model error using the residual-error identity. (Answer: \(\varepsilon=-0.4\).)
- A simulation has true line \(Y=20+3x+\varepsilon\) and fitted line \(\hat{Y}=18+3.4x\). At \(x=5\), the observed response is \(36\). Calculate the residual and error, and explain their relationship at this predictor value. (Answer: Residual \(=1\); error \(=1\); they coincide because the fitted and true lines both give \(35\) at \(x=5\).)
- An analyst knows only that \(x=2\), \(Y=11\), and the fitted value is \(10.5\); the population coefficients are unknown. What can the analyst calculate, and can the model error be determined exactly? (Answer: The residual is \(0.5\); the model error cannot be determined exactly from the supplied information.)