Why Plot Residuals Against the Predictor?
A straight-line regression summarizes the relationship between a response and a predictor. But a high correlation or a plausible fitted line does not guarantee that a straight-line model describes the response well across the whole predictor range. The residual-versus-predictor plot gives a direct visual check: it places each prediction error against the predictor value at which that error occurred.
A pattern in this plot can point to a missed curve, a region where predictions are systematically too high or too low, or a change in how widely responses vary. These are clues for further assessment, not automatic proof of a particular cause. In particular, a plot with only a few observations can make random variation look structured.
A positive residual means the observed response is above the fitted value, so the line underpredicts that observation. A negative residual means the line overpredicts it. A satisfactory plot need not have residuals exactly equal to zero or alternate regularly in sign. Instead, the points should be scattered around zero without a clear systematic shape.
What the Pattern Can Reveal
| Visual feature | Possible interpretation | What to keep in mind |
|---|---|---|
| U-shape or upside-down U-shape | The mean response may bend rather than follow a straight line. | Check the pattern across the full predictor range; a few points may create an apparent curve. |
| Residuals mostly negative in one predictor region and positive in another | The line may systematically overpredict in one region and underpredict in another. | State the direction in terms of observed minus fitted, and identify the region. |
| Wider vertical scatter at some predictor values | The variability may change with the predictor. | This is a warning to investigate spread; a visual cue alone does not establish the cause. |
| No obvious structure around zero | The plot does not show a clear predictor-related failure of the fitted mean pattern. | A clean plot does not prove every model assumption or guarantee accurate predictions. |
The shape of the mean response and the spread of responses are different questions. A curve in the residual centers suggests the straight-line mean may be inadequate. A fan-like change in vertical spread instead suggests variability may differ across predictor values. These patterns can coexist, and a small sample may make either one difficult to judge. Here the main task is to recognize the clue and describe it carefully; a detailed assessment of nonconstant variance is a separate diagnostic question.
Worked Example: Detecting Curvature in a Fitted Line
A hypothetical calibration study records five predictor values \(x=0,1,2,3,4\), with responses \(y=1,2,5,10,17\). Fit the ordinary least-squares line, calculate the residuals, and interpret the plot. Treat the five pairs as the complete data set used to fit the line; no probability distribution is needed to calculate or inspect these residuals.
Method. Calculate the least-squares slope and intercept, then use observed minus fitted to get each residual.
The means are \(\bar{x}=(0+1+2+3+4)/5=2\) and \(\bar{y}=(1+2+5+10+17)/5=7\). The sum of squared predictor deviations is \(S_{xx}=(-2)^2+(-1)^2+0^2+1^2+2^2=10\). The sum of cross-products is \(S_{xy}=(-2)(-6)+(-1)(-5)+0(-2)+1(3)+2(10)=40\).
Thus the fitted line is \(\hat{y}=-1+4x\). At the five predictor values, the fitted responses are \(-1,3,7,11,15\). Subtract each fitted response from its observed response:
The residuals are positive at both ends and negative in the middle. A residual plot would therefore show a U-shaped pattern around zero, which suggests that the straight-line mean may miss curvature. For this data set the responses happen to follow \(y=x^2+1\) exactly, but the diagnostic conclusion is about the plotted pattern, not a claim that every real data set with a U-shape follows this precise equation.
Check. The residuals sum to \(2-1-2-1+2=0\), as expected for least-squares residuals when an intercept is fitted. The common mistake is to call the positive residual at a high \(x\) an overprediction: because residual equals observed minus fitted, that positive value means underprediction.
Why Least-Squares Residuals Have No Linear Trend
For a least-squares line with an intercept, residuals have two useful algebraic properties: their sum is zero, and their cross-product with the centered predictor values is zero. These properties explain why a residual plot can reveal curvature even though it cannot show an overall linear residual trend in the fitted sample.
The coefficients minimize \(S(b_0,b_1)=\sum_i[y_i-(b_0+b_1x_i)]^2=\sum_i e_i^2\).
At the minimizing coefficients, \(\partial S/\partial b_0=-2\sum_i e_i=0\), so \(\sum_i e_i=0\).
At the minimizing coefficients, \(\partial S/\partial b_1=-2\sum_i x_i e_i=0\), so \(\sum_i x_i e_i=0\).
Since \(\sum_i(x_i-\bar{x})e_i=\sum_i x_i e_i-\bar{x}\sum_i e_i=0-\bar{x}(0)=0\), the residuals have no linear trend with centered \(x\).
This result does not make the residual plot uninformative. A curved sequence can have no overall linear association: its positive and negative parts can balance while still tracing a clear shape. The result also depends on fitting least squares with an intercept to the same observations being plotted. It need not hold for residuals calculated on a separate validation sample or for a fitted model without an intercept.
Worked Example: Checking a Possible Change in Spread
A hypothetical maintenance study fits a least-squares line to five observations with predictor values \(x=1,2,3,4,5\) and responses \(y=12,16,20,24,23\). Determine the fitted line and residuals, then assess whether the plot raises a concern about changing spread. Treat the observations as the fitted sample and make no normal-distribution assumption for this visual assessment.
Method. Find the least-squares coefficients and compare the vertical residual scatter at lower and higher predictor values.
The means are \(\bar{x}=3\) and \(\bar{y}=95/5=19\). The predictor sum of squares is \(S_{xx}=(-2)^2+(-1)^2+0^2+1^2+2^2=10\). The response deviations from 19 are \(-7,-3,1,5,4\), so \(S_{xy}=(-2)(-7)+(-1)(-3)+0(1)+1(5)+2(4)=30\). Therefore \(b_1=30/10=3\), and \(b_0=19-3(3)=10\). The fitted line is \(\hat{y}=10+3x\).
The fitted values for \(x=1,2,3,4,5\) are \(13,16,19,22,25\). The residuals are therefore \(12-13=-1\), \(16-16=0\), \(20-19=1\), \(24-22=2\), and \(23-25=-2\), giving \((-1,0,1,2,-2)\).
The two observations at the largest predictor values have residual magnitudes 2, while the first three have magnitudes 1, 0, and 1. That modest increase in spread could be a reason to look more closely for changing variability, but five observations are not enough to establish a variance pattern. There is also a positive residual at \(x=4\) followed by a negative residual at \(x=5\), so the plot does not show a simple, steadily increasing mean error.
Check. The residuals sum to zero, and their predictor-weighted sum is \((-1)(1)+0(2)+1(3)+2(4)+(-2)(5)=0\). The common mistake is to treat the larger high-\(x\) residual magnitudes as proof of changing variance. They are a diagnostic clue that requires context and further assessment.
Reading Systematic Regions Carefully
A residual plot can also reveal that the fitted line performs differently in different regions of the predictor range. For example, residuals that are mostly negative at low \(x\) and mostly positive at high \(x\) mean the line tends to overpredict at low \(x\) and underpredict at high \(x\). A curved pattern may produce just such a change in sign. The direction and location should be stated explicitly rather than described vaguely as “a bad fit.”
For a least-squares fit to the same observations, residuals cannot show a simple overall linear trend, as the derivation above demonstrates. However, they can show structured departures such as a U-shape or a local run of positive residuals. For residuals on new observations, even the no-linear-trend property does not apply, so a trend may be visible. In either setting, describe the observed pattern first and avoid claiming more than the plot supports.
Worked Example: Locating a Region of Underprediction
A hypothetical equipment study uses a line fitted on an earlier training sample, \(\hat{y}=10+2x\), to assess five new validation observations. The new predictor values are \(1,2,3,4,5\), and the observed responses are \(12,14,18,23,24\). Calculate the validation residuals and state what region shows systematic misfit. These are new observations, not the sample used to estimate the line; no distributional assumption is required for the calculation.
Method. Compute each prediction from the supplied line, subtract it from the observed response, and compare residual signs across predictor regions.
The fitted values are \(10+2(1)=12\), \(10+2(2)=14\), \(10+2(3)=16\), \(10+2(4)=18\), and \(10+2(5)=20\). Subtracting gives residuals \(12-12=0\), \(14-14=0\), \(18-16=2\), \(23-18=5\), and \(24-20=4\). Thus the residual sequence is \((0,0,2,5,4)\).
The residuals at \(x=3,4,5\) are all positive, and their values are relatively large compared with those at \(x=1,2\). In this validation set the line underpredicts in the higher-predictor region. The run of positive residuals could indicate that the fitted relationship misses some systematic feature, but these five observations do not by themselves identify what feature caused it.
Check. For example, at \(x=4\), the prediction is 18 and the observed response is 23, so the residual is \(23-18=5\), not \(-5\). The common mistake is to reverse the subtraction and then reverse the interpretation of overprediction and underprediction.
Residuals Against the Predictor and Common Traps
In simple linear regression, a residual-versus-fitted plot and a residual-versus-predictor plot usually carry the same horizontal information when \(b_1\ne 0\), because \(\hat{y}=b_0+b_1x\) is a one-to-one linear transformation of \(x\). If \(b_1\) is positive, the horizontal order is the same; if it is negative, the order is reversed. If \(b_1=0\), all fitted values are equal, so a residual-versus-fitted plot cannot display how residuals vary across predictor values. The predictor plot remains useful for locating patterns in \(x\).
Do not confuse a plot of \(y\) against \(x\) with a residual plot. The raw-response plot displays the fitted trend and the response values together; the residual plot displays deviations from the fitted line around a zero reference. A rising raw-response plot may simply reflect the fitted linear relationship and is not, by itself, evidence of a diagnostic problem.
- Keep the residual sign straight. Use \(e_i=y_i-\hat{y}_i\): positive means underprediction; negative means overprediction.
- Look for structure, not a perfect pattern. Curvature or a run of residuals on one side of zero is more informative than whether individual points are close to zero.
- Do not infer a cause from one plot. A visible pattern signals that the line deserves scrutiny; it does not alone establish why the pattern occurs.
- Do not overinterpret small samples. With few observations, isolated points or random variation can resemble curvature or changing spread.
- Separate mean shape from variability. A changing center and a changing vertical spread are distinct features, even if they appear together.
Check Your Understanding
For each problem, calculate or interpret the residual pattern before checking the answer.
- A supplied line is \(\hat{y}=2+3x\). For observations \((x,y)=(1,6),(2,7),(3,12)\), calculate the residuals. (Answer: \(1,-1,1\).)
- A residual plot from a fitted line has mostly negative residuals at small predictor values and mostly positive residuals at large values. State what this means about the line in each region. (Answer: It tends to overpredict at small predictor values and underpredict at large predictor values.)
- For a least-squares line with an intercept fitted to the same observations, can the residuals have a nonzero sample linear trend with the predictor? Give the relevant property. (Answer: No; \(\sum_i(x_i-\bar{x})e_i=0\).)
- A plot shows residual magnitudes that are generally small at low \(x\) and larger at high \(x\), with few observations in each region. What is a careful interpretation? (Answer: The changing spread is a warning to investigate, not proof of nonconstant variance.)
- A fitted line is \(\hat{y}=8-2x\). As \(x\) increases, how does the horizontal order compare between a residual-versus-predictor plot and a residual-versus-fitted plot? (Answer: The order is reversed because the fitted value decreases as \(x\) increases; the two plots contain the same horizontal information up to this reversal.)