Why Look at Residuals?
A regression model summarizes how an outcome tends to vary with one or more predictors. A fitted relationship can look plausible on average yet still miss important structure in the data. Looking at the differences between observed outcomes and fitted values helps reveal where the model may be inadequate. These differences are called observed residuals.
Residuals are useful because they are available for every observation used to fit the model. Their patterns can prompt questions about the fitted conditional mean, the amount of variation around that mean, or the shape of the outcome distribution given the predictors. But residuals are calculated using the fitted model, so they are not direct observations of the model’s unobserved errors. A residual plot or summary can raise a concern; it cannot, by itself, prove which assumption is false or which replacement model is correct.
Errors, Fitted Values, and Residuals
Suppose observation \(i\) has predictor value \(x_i\) and observed outcome \(y_i\). Write the conditional mean of the outcome at that predictor value as \(m(x_i)\). A basic regression model represents the outcome as this mean plus an unobserved error:
The error \(\varepsilon_i\) is the difference between the outcome and the true conditional mean, which is not known. A standard mean assumption is \(E(\varepsilon_i\mid X_i=x_i)=0\). When constant conditional variance is assumed, the model also has \(\operatorname{Var}(\varepsilon_i\mid X_i=x_i)=\sigma^2\). The fitted model supplies a fitted value \(\hat{y}_i=\hat{m}(x_i)\), an estimate of the conditional mean at \(x_i\).
Residuals are expressed in the outcome’s units. For example, if the outcome is a waiting time measured in minutes, a residual of \(-3\) means the observed wait was three minutes below the fitted wait. It does not mean the unobserved error is known to be \(-3\).
What a Residual Pattern Can Suggest
A useful diagnostic question is whether residuals look like unstructured deviations around zero, or whether their behavior changes in a way the fitted model has not accounted for. The question depends on the model assumptions and the data being examined.
| What is examined | Possible concern if a pattern appears | What the pattern does not establish |
|---|---|---|
| Residuals in relation to predictors or fitted values | The fitted conditional mean may be missing structure | Which alternative mean function is correct |
| Size of residuals across the data | The conditional variance may not be constant | The precise variance pattern or its cause |
| Overall distribution of residuals | The assumed conditional error distribution may be questionable | The distribution of the unobserved errors with certainty |
These are clues rather than automatic verdicts. Random variation can produce apparent patterns, especially in a small sample. Conversely, a modest sample can fail to display a real departure clearly. Diagnostics are most informative when considered alongside the fitted model, the data collection process, and the substantive context.
Why Residuals Are Not the Errors
The distinction can be seen directly by subtracting the fitted value from the observed outcome. Since \(y_i=m(x_i)+\varepsilon_i\), we have:
Thus, a residual combines the unobserved error with the difference between the true conditional mean and the fitted mean. If the fitted mean is close to the true mean at \(x_i\), the residual may be a useful clue about the error there. If the fitted mean is inaccurate, the residual also reflects that inaccuracy. In addition, the same data are used to estimate the fitted model and to calculate the residuals. The residuals therefore depend on the fitting process; they are not independent, untouched measurements of the errors.
This identity also explains why residual diagnostics address different questions. A pattern tied to predictor values may point to a problem in the mean function; changing residual magnitudes may point to changing variability; and an unusual overall shape may raise questions about a distributional assumption. None of these observations alone separates every possible explanation.
Worked Example: Calculating Residuals From a Fitted Line
Suppose a hypothetical model predicts a clinic’s daily waiting time, in minutes, from a predictor \(x\). For four days, the observed and fitted waiting times are as follows. The fitted values are already calculated from the regression model.
| Day | Observed \(y_i\) (minutes) | Fitted \(\hat{y}_i\) (minutes) |
|---|---|---|
| 1 | 12 | 13 |
| 2 | 15 | 16 |
| 3 | 17 | 19 |
| 4 | 22 | 22 |
For each day, calculate \(e_i=y_i-\hat{y}_i\). For day 1, \(e_1=12-13=-1\) minute. For day 2, \(e_2=15-16=-1\) minute. For day 3, \(e_3=17-19=-2\) minutes. For day 4, \(e_4=22-22=0\) minutes.
The day 3 outcome was two minutes below its fitted value. That calculation describes the observed discrepancy for that day. It does not identify the unobserved error, because the true conditional mean is not given. Nor can four residuals alone show whether the fitted mean or another model assumption is generally adequate.
Worked Example: Residual Averages as a Clue About the Mean
A hypothetical analyst divides observations into two predictor ranges and calculates residuals from one fitted regression. In the lower range, the residuals are \(-2,-1,0,1,2\). In the higher range, they are \(3,4,5,6,7\). Calculate each range’s average residual and explain what the contrast suggests.
The average residual in the lower range is
The average residual in the higher range is
So the observations in the higher range are, on average, five outcome units above their fitted values, while the lower-range residuals average zero. This contrast is a warning that the fitted conditional mean may not describe all predictor ranges equally well. It is not proof of a particular alternative mean function: the pattern could also be affected by sampling variation, omitted information, or features of the observations not represented by these averages.
Spread and Distribution Are Separate Questions
The conditional mean describes the center of the outcome distribution at given predictor values; the conditional variance describes how much outcomes vary around that center. Residuals can provide indirect evidence about each, but they answer different diagnostic questions. A model might describe the average response reasonably while its residuals vary more in some parts of the data than in others.
The distributional question is also distinct. Some regression procedures for inference use a normal-error assumption, typically stated as a conditional distribution such as \(\varepsilon_i\mid X_i=x_i\sim N(0,\sigma^2)\). A markedly asymmetric residual distribution or an unusually extreme residual can be a reason to examine that assumption. A residual distribution is only an indirect check: it is based on fitted residuals, and a departure may reflect an unusual observation, a mean-model problem, changing variance, or a genuinely non-normal error distribution.
Worked Example: Comparing Residual Spread
A hypothetical model is fitted to observations in two predictor ranges. The residuals in the lower range are \(-1,0,1,-1,1\); those in the higher range are \(-4,0,4,-3,3\). Compare the root mean squared residual within each range, calculated here as the square root of the average squared residual. This is a descriptive comparison, not a formal variance test.
For the lower range, the sum of squared residuals is \(1+0+1+1+1=4\). Thus, its root mean squared residual is
For the higher range, the sum is \(16+0+16+9+9=50\), giving
The residuals are substantially more spread out in the higher range in this small hypothetical example. That difference may signal that a constant-variance assumption deserves scrutiny. It does not establish that the true conditional variances are exactly \(0.8\) and \(10\), or identify why the spread differs: these calculations use only five fitted residuals in each range.
Worked Example: A Residual Distribution With an Extreme Value
For a hypothetical fitted regression, consider the five residuals \(-4,-2,-1,1,6\), measured in the outcome’s units. Their average is
Although the average is zero, the residual \(6\) is much larger than the others in the positive direction. This small set gives a reason to investigate whether an extreme observation or an unusual distributional feature is present. The residuals alone cannot tell which explanation applies, and five values are not enough to characterize the error distribution reliably. In particular, an average of zero does not imply that the residual distribution is symmetric or normal.
Using Diagnostic Evidence Carefully
Residual diagnostics matter because the quality of a regression analysis is not determined solely by how well a fitted curve summarizes the observed outcomes. A model may give useful predictions in one region and perform poorly in another; its uncertainty calculations may also rely on assumptions about variability or the error distribution. Residuals help identify questions worth investigating before results are interpreted too confidently.
A disciplined reading of a diagnostic begins with the model and its assumptions, then describes what the residuals do, and finally states the limited inference supported. For example, “residuals are more dispersed in the higher predictor range, which raises a concern about constant conditional variance” is more accurate than “the model is wrong.” Additional analysis and subject-matter reasoning are needed to decide what, if anything, to change.
Identify whether the question concerns the conditional mean, variance, or error distribution.
Report the observed pattern or calculation without treating residuals as known errors.
Explain what the evidence suggests, what it cannot establish, and why context or further analysis may matter.
A common mistake is to say, “The residual is the error, so it tells us the true error for this observation.” The correction is that the residual is \(y_i-\hat{y}_i\), whereas the error is \(y_i-m(x_i)\). Their difference is \(m(x_i)-\hat{m}(x_i)\), which is generally unknown. Another mistake is to treat one irregular residual as proof that a model assumption fails. A single unusual value may warrant investigation, but it does not settle the diagnosis.
Check Your Understanding
Answer these questions using the distinction between observed residuals and unobserved errors.
- An observation has outcome \(y_i=18\) and fitted value \(\hat{y}_i=21\). Calculate its residual and interpret its sign in context.
- In words, distinguish \(e_i\) from \(\varepsilon_i\). Why can they differ?
- Residuals appear more spread out in one predictor range than another. Which assumption might merit scrutiny, and what can this observation not establish?
- Why does an unusual residual distribution provide only indirect evidence about the error distribution?
- Give one reason a residual pattern should be treated as a clue rather than as proof that a regression model is inadequate.