Why Plot Residuals Against Individual Predictors?
A residual-versus-predictor plot places a fitted-model residual on the vertical axis and one predictor’s observed value on the horizontal axis. It asks a focused question: does the model leave a systematic pattern in the response as this predictor changes? A curved band, separated groups, or another organized shape can indicate that the fitted mean relationship has not captured some structure in the data.
This plot complements a residual-versus-fitted plot. A fitted value combines the effects of several predictors; a plot against one predictor makes it easier to notice whether the model’s treatment of that particular predictor may be inadequate. In a multiple regression, however, the predictors can be related. A pattern against one predictor is a diagnostic clue, not by itself proof that the predictor caused the pattern or that a particular model change is correct.
For orientation, consider the multiple linear regression model
The usual regression assumptions include a correctly specified conditional mean, errors with mean zero conditional on the predictors, and independent errors with constant variance for the standard OLS inference framework. Normal errors are additionally used for exact small-sample \(t\)- and \(F\)-inference. A residual plot is mainly a visual check on whether the mean structure has left a pattern; it does not establish all these assumptions.
What Patterns Can Signal
If the model includes a straight-line effect for a quantitative predictor, a residual plot with no obvious structure is broadly consistent with that choice. A smooth U-shape or inverted U-shape suggests that a straight line may be insufficient. A systematic rise or fall in the residual center can indicate that an effect has been omitted or modeled incorrectly. Separated bands may suggest an unmodeled group effect. These are prompts to investigate the model and data, not automatic instructions to add a term.
A plot should be read with the model in mind. For example, if a categorical predictor is included correctly, its fitted group differences should already be reflected in the fitted values. If it is omitted, group-specific residuals may remain. In multiple regression, a plot against one predictor can also reflect a related omitted predictor or an interaction. Follow-up analysis should consider the variables jointly and the scientific or business context.
Worked Example: Detecting Curvature in a Hypothetical Claim Model
A hypothetical insurer fits a straight-line model for claim cost \(y\), in thousands of dollars, using a single rating score \(x\). The five observed pairs are \((-2,8),(-1,8.5),(0,10),(1,12.5),(2,16)\). Calculate the OLS fitted line and residuals, then interpret the residual-versus-\(x\) plot.
Method. Fit an intercept and a linear predictor. The predictor values sum to zero, so \(\bar{x}=0\). The response mean is \( \bar{y}=(8+8.5+10+12.5+16)/5=55/5=11 \).
The OLS slope is \( \hat{\beta}_1=\sum_i(x_i-\bar{x})(y_i-\bar{y})/\sum_i(x_i-\bar{x})^2 \). Since \(\bar{x}=0\), the numerator is \(\sum x_i y_i\): \( (-2)(8)+(-1)(8.5)+(0)(10)+(1)(12.5)+(2)(16)=-16-8.5+0+12.5+32=20 \). The denominator is \(4+1+0+1+4=10\), so \(\hat{\beta}_1=20/10=2\). The intercept is \(\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}=11\). Thus, \(\hat{y}=11+2x\).
The fitted values and residuals are:
| \(x_i\) | \(y_i\) | \(\hat{y}_i=11+2x_i\) | \(e_i=y_i-\hat{y}_i\) |
|---|---|---|---|
| -2 | 8 | 7 | 1 |
| -1 | 8.5 | 9 | -0.5 |
| 0 | 10 | 11 | -1 |
| 1 | 12.5 | 13 | -0.5 |
| 2 | 16 | 15 | 1 |
The residuals are positive at both ends and negative in the middle, forming a U-shaped pattern. That structure is consistent with curvature that the straight-line model has not captured. A quadratic term is one possible model to investigate, but the plot alone does not prove that it is the right change. Check: the residuals sum to \(1-0.5-1-0.5+1=0\), and \(\sum x_i e_i=(-2)(1)+(-1)(-0.5)+0(-1)+1(-0.5)+2(1)=0\), as expected for this OLS fit with an intercept and linear predictor. The common mistake is to see that the residuals have no overall linear trend and conclude that the straight-line relationship must be adequate; the U-shape is the important evidence here.
Why OLS Can Have Zero Linear Association and Still Miss a Pattern
In an OLS fit with an intercept, the residuals are orthogonal to every included predictor column. Orthogonality here means that the sum of the products of residuals and predictor values is zero. This is a property of how the fitted coefficients are calculated, not a guarantee that the residuals are randomly arranged.
OLS chooses the coefficients to minimize \(S=\sum_i e_i^2=\sum_i(y_i-\hat{y}_i)^2\).
For predictor \(x_j\), \(\partial S/\partial\hat{\beta}_j=-2\sum_i x_{ij}e_i\). At an OLS minimum, this derivative is zero, so \(\sum_i x_{ij}e_i=0\).
The residuals have no remaining linear association with that included predictor in the sample. The equality does not require residuals to be patternless: a curved or grouped arrangement can still be present.
The same minimization with respect to the intercept gives \(\sum_i e_i=0\). These equalities are useful checks, but they should not replace visual diagnosis. A scatter of residuals can have zero sum and zero linear association with \(x_j\) while still displaying an unmistakable curve.
Worked Example: Recognizing an Omitted Group Effect
A hypothetical pricing team models claim cost \(y\), in thousands of dollars, using only a quantitative score \(x\), although each policy also belongs to one of two product groups \(z=0\) and \(z=1\). The data follow \(y=10+2x+6z\), with both groups observed at each \(x=0,1,2,3\). Fit the model with only an intercept and \(x\), and describe its residual plot.
The eight responses, listed by \(x\), are: at \(x=0\), \(10\) for \(z=0\) and \(16\) for \(z=1\); at \(x=1\), \(12\) and \(18\); at \(x=2\), \(14\) and \(20\); at \(x=3\), \(16\) and \(22\). The mean predictor is \(\bar{x}=(0+1+2+3+0+1+2+3)/8=12/8=1.5\). The mean response is \(\bar{y}=(10+16+12+18+14+20+16+22)/8=128/8=16\).
For each \(x\), the two responses average to \(13,15,17,19\), respectively. The slope from the full eight observations is therefore the slope of those equally repeated group averages. Its numerator is \(2[(-1.5)(13-16)+(-0.5)(15-16)+(0.5)(17-16)+(1.5)(19-16)]\) \(=2[4.5+0.5+0.5+4.5]=20\). The denominator is \(2[(-1.5)^2+(-0.5)^2+(0.5)^2+(1.5)^2]=2[2.25+0.25+0.25+2.25]=10\). Thus the slope is \(20/10=2\), and the intercept is \(16-2(1.5)=13\). The fitted model is \(\hat{y}=13+2x\).
At every \(x\), the fitted value equals the midpoint of the two group responses. The residual is \(10-13=-3\) for group \(z=0\) at \(x=0\), and \(16-13=3\) for group \(z=1\). The same calculation at \(x=1,2,3\) gives residuals \(-3\) and \(3\) at each value of \(x\). The plot has two horizontal bands. This pattern suggests that group structure remains after fitting \(x\), so including or otherwise accounting for group may be worth investigating. Check: at each \(x\), the two residuals sum to zero, so the residuals have no linear trend with \(x\); nevertheless, the two bands reveal a systematic omission. The common mistake is to inspect only the average residual at each \(x\), see that it is zero, and overlook the separated group pattern.
Patterns That a Single-Predictor Plot Can Hide
A residual-versus-predictor plot is not a complete diagnostic for all model structure. In particular, an interaction can produce a pattern that disappears when the residuals are viewed against either predictor separately. The following small example shows why a plot that looks centered at zero is not proof of a correct mean model.
Worked Example: An Interaction Hidden in Separate Plots
Suppose four observations have predictors \(x,z\in\{-1,1\}\), one at each combination, and responses follow \(y=5+2x+3z+4xz\). Fit the additive model with an intercept, \(x\), and \(z\), but omit the interaction \(xz\). Calculate the fitted values and residuals.
The four rows are:
| \(x\) | \(z\) | \(xz\) | \(y=5+2x+3z+4xz\) | \(\hat{y}=5+2x+3z\) | \(e=y-\hat{y}\) |
|---|---|---|---|---|---|
| -1 | -1 | 1 | 4 | 0 | 4 |
| -1 | 1 | -1 | 2 | 6 | -4 |
| 1 | -1 | -1 | 0 | 4 | -4 |
| 1 | 1 | 1 | 14 | 10 | 4 |
The residuals are \(4,-4,-4,4\). Against \(x=-1\), they are \(4\) and \(-4\); against \(x=1\), they are also \(-4\) and \(4\). Thus, each \(x\) level has residuals centered at zero. The same happens against \(z\): for each \(z\) level, one residual is \(4\) and one is \(-4\). A plot against either predictor alone does not display a changing group average in this balanced example. But the residual equals \(4xz\), so plotting residuals by the combinations of \(x\) and \(z\), or against the interaction \(xz\), reveals the omitted structure. Check: the residuals sum to zero and their products with each of \(x\) and \(z\) sum to zero, while their products with \(xz\) sum to \(16\). The common mistake is to treat two unpatterned separate plots as proof that no interaction is missing.
How to Use the Plot Carefully
A useful interpretation proceeds from the fitted model, through the visible shape, to a cautious conclusion. The plot identifies where to investigate; it does not select the final model on its own. When predictors are correlated, apparent structure against one predictor may reflect another predictor’s omission or misspecification. Also consider whether the plotted predictor is quantitative or categorical: for a categorical predictor, compare residual distributions across levels rather than looking for a smooth curve.
- Curvature: a smooth arc may indicate that a linear term does not capture the mean relationship. Consider whether a nonlinear term is defensible and whether the pattern persists after fitting it.
- Separated bands or level shifts: residuals grouped by a category may indicate an omitted group effect or an incorrectly encoded category.
- Different patterns across groups: a predictor’s effect may differ by group, suggesting that an interaction could be relevant. Check combinations of predictors when separate plots are inconclusive.
- No visible pattern: this is reassuring but limited. Small samples, overlapping points, weak effects, or a structure involving multiple predictors can make a real defect hard to see.
Do not confuse a diagnostic clue with a causal conclusion. A residual pattern can show that the model has not explained some systematic variation in the observed data; it does not establish why the pattern exists. A proposed change should be assessed against the model’s purpose, the data collection, and whether the revised specification improves the analysis without creating an unsupported complication.
Check Your Understanding
For each problem, interpret the residual information in the stated model context.
- A fitted claim-frequency model includes an intercept and a straight-line term for risk score \(x\). The residuals at \(x=-2,-1,0,1,2\) are \(2,-1,-2,-1,2\). Describe the pattern and state one type of model feature worth investigating. (Answer: A U-shaped pattern suggests curvature; investigate a defensible nonlinear mean term, such as a quadratic, rather than concluding that the linear fit is adequate.)
- A model with an intercept and predictor \(x\) has residuals \(-5\) and \(5\) at each of four observed \(x\) values. The negative residuals all belong to product group A and the positive residuals all belong to group B. What does the plot suggest, and why is checking only the average residual at each \(x\) insufficient? (Answer: The separated group bands suggest omitted group structure; the average residual at each \(x\) is zero and hides the systematic difference between groups.)
- In an OLS model with an intercept and predictor \(x\), a candidate says that because \(\sum_i x_i e_i=0\), a residual-versus-\(x\) plot cannot show a pattern. Is the claim correct? (Answer: No. The equality rules out sample linear association with \(x\), not curvature, bands, or other nonlinear organization.)
- A balanced two-predictor data set has residuals \(4,-4,-4,4\) for \((x,z)=(-1,-1),(-1,1),(1,-1),(1,1)\), respectively. Each separate plot against \(x\) or \(z\) is centered at zero within each level. What omitted structure should be checked? (Answer: An \(x\)-by-\(z\) interaction; the residuals equal \(4xz\), so the interaction pattern is visible when combinations or the product \(xz\) are examined.)
- A residual-versus-predictor plot looks like an unstructured cloud in a small sample. What conclusion is justified, and what conclusion is not? (Answer: There is no obvious pattern in that plot, which is limited evidence in favor of the specified mean structure; it does not prove the model is correctly specified or rule out patterns involving other predictors.)