Why Look for Residual Associations?
A regression model can describe a systematic relationship between an outcome and the predictors it includes while still leaving a pattern in its residuals. If those residuals vary with a scientifically plausible variable that the model omits, the pattern is a reason to examine the model specification. For example, a model predicting recovery time from baseline severity might leave residuals associated with a patient’s age. That association could indicate that age helps describe the conditional mean, but it does not, by itself, show that age causes recovery time to change.
The key question is not simply whether residuals and a candidate predictor are associated in the raw data. It is whether the candidate has a relationship with the residuals that is not already explained by the predictors in the fitted model, and whether including it reduces systematic residual structure in a meaningful way.
Assess the Candidate Conditionally
Suppose the current model includes an intercept and predictors collected in \(X\), and \(W\) is a candidate predictor. A plot of \(e_i\) against \(W_i\), or a comparison of residuals across categories of \(W\), can reveal a pattern. But an unadjusted pattern can be misleading when \(W\) is related to predictors already in the model. The relevant question is whether \(W\) adds information after accounting for those predictors.
One way to express this distinction is to regress \(W\) on the current predictors and retain the residuals from that regression. Call these residuals \(z_i\). They represent the part of \(W\) not linearly explained by the current predictors. For a linear model that adds \(W\), the added coefficient is determined by the association between \(z_i\) and the current response residuals \(e_i\):
This expression helps explain why a raw residual-versus-\(W\) pattern is only a clue. If \(W\) is strongly related to existing predictors, its residualized version \(z\) may differ substantially from \(W\) itself. The added coefficient describes the candidate’s linear association with the outcome after adjusting for the current predictors; it does not establish what would happen under an intervention on \(W\).
Why the Added-Coefficient Formula Works
Here is a short derivation. Let the fitted values from the current model be \(\hat{y}\), and write the model with the added term as \(\hat{y}+\gamma z\). Because \(z\) is the residual from regressing \(W\) on the current predictors, it is orthogonal to those predictors. Thus, adding \(\gamma z\) to the current fit gives the same least-squares improvement as adding the part of \(W\) not already explained by those predictors. The residual sum of squares as a function of \(\gamma\) is
Differentiate with respect to \(\gamma\), set the derivative to zero, and solve. If \(\sum z_i^2>0\), the minimizing value is the formula above. Substituting it back into the sum of squares shows that the reduction in residual sum of squares is
This reduction describes improvement in the fitted sample, not proof that the candidate is a cause or that the revised model is correct. A useful diagnostic assessment also asks whether the residual pattern has become less systematic, whether the functional form is appropriate, and whether the candidate is scientifically defensible.
Worked Example: A Grouped Candidate Predictor Reduces Residual Structure
Consider hypothetical data on nine independently observed patients. The current model predicts an outcome from a quantitative predictor \(X\), and its fitted equation is \(\hat{y}=22+2X\). A researcher has a scientific reason to consider candidate predictor \(W\), which has values 0 and 1. The table gives the outcome, fitted value, and residual from the current model.
| Patient | \(X\) | \(W\) | \(y\) | \(\hat{y}=22+2X\) | \(e=y-\hat{y}\) |
|---|---|---|---|---|---|
| 1 | 1 | 0 | 23 | 24 | -1 |
| 2 | 0 | 0 | 19 | 22 | -3 |
| 3 | 2 | 0 | 23 | 26 | -3 |
| 4 | 1 | 0 | 25 | 24 | 1 |
| 5 | 1 | 0 | 22 | 24 | -2 |
| 6 | 1 | 1 | 27 | 24 | 3 |
| 7 | 1 | 1 | 25 | 24 | 1 |
| 8 | 1 | 1 | 25 | 24 | 1 |
| 9 | 1 | 1 | 27 | 24 | 3 |
For patient 1, the fitted value is \(22+2(1)=24\), so the residual is \(23-24=-1\). In particular, the observation with \(y=23\) and \(X=1\) has residual \(-1\), not \(1\). The displayed equation is indeed the least-squares fit with an intercept: the residuals sum to zero, and \(\sum X_i e_i=0\), the two normal-equation conditions for an intercept and \(X\).
The residual mean for \(W=0\) is \((-1-3-3+1-2)/5=-1.6\). For \(W=1\), it is \((3+1+1+3)/4=2.0\). The two groups have the same mean \(X\), namely 1, so \(W\) is not linearly associated with \(X\) in this sample. Consequently, the candidate’s adjusted coefficient is the difference in residual means: \(2.0-(-1.6)=3.6\) response units. The residuals show a clear group pattern: the current model tends to underpredict for \(W=1\) and overpredict for \(W=0\).
The current residual sum of squares is \(1+9+9+1+4+9+1+1+9=44\). After adding \(W\), the residuals are the old residuals minus their group means. For \(W=0\), they are \(0.6,-1.4,-1.4,2.6,-0.4\); for \(W=1\), they are \(1,-1,-1,1\). Their squared sum is \(0.36+1.96+1.96+6.76+0.16+1+1+1+1=15.2\). The adjusted model therefore removes a substantial group-level residual pattern, while leaving residual variation within the groups.
For illustration, suppose the observations were independent and the errors were approximately normal with constant variance under both models. The current model has 7 residual degrees of freedom and the model with \(W\) has 6. The partial \(F\) statistic for adding this one term is \((44-15.2)/(15.2/6)=11.37\), with 1 and 6 degrees of freedom; its p-value is about 0.015. Under those assumptions, this is evidence against the current model’s conditional mean being adequate relative to the model that also includes \(W\). The small hypothetical sample cannot establish that the assumptions hold, and neither the residual pattern nor this test establishes that \(W\) causes the outcome to change.
Raw and Partial Associations Can Differ
When candidate \(W\) and current predictors are associated, comparing raw group means of the residuals may not equal the coefficient from the model that adds \(W\). The next example makes this distinction explicit. Residualizing \(W\) first isolates the candidate variation that is not already represented by the current model.
Worked Example: Adjusting a Candidate Before Interpreting Its Association
Suppose the current model contains an intercept and quantitative predictor \(X\). In a hypothetical six-observation data set, \(X=(-2,-1,0,0,1,2)\), the current-model residuals are \(e=(1,-2,1,-1,2,-1)\), and a binary candidate is \(W=(0,1,0,1,0,1)\). The residuals sum to zero and \(\sum X_i e_i=0\), so they satisfy the least-squares orthogonality conditions for the current model.
The raw residual means are \(4/3\) for \(W=0\) and \(-4/3\) for \(W=1\), a difference of \(-8/3\). But the groups do not have the same distribution of \(X\). Regressing \(W\) on an intercept and \(X\) gives mean \(W=0.5\) and slope \(\sum X_iW_i/\sum X_i^2=1/10=0.1\). Thus the residualized candidate values are \(z_i=W_i-0.5-0.1X_i\), or \((-0.3,0.6,-0.5,0.5,-0.6,0.3)\).
The adjusted coefficient is \(\hat{\gamma}=\sum z_i e_i/\sum z_i^2=-4/1.4=-2.857\). The raw group difference and adjusted coefficient are not identical because \(W\) and \(X\) are related. The current residual sum of squares is 12; adding \(W\) reduces it by \((-4)^2/1.4=11.43\), leaving approximately 0.5714. The new residuals are \(e_i-\hat{\gamma}z_i=(1/7,-2/7,-3/7,3/7,2/7,-1/7)\). In this constructed example the remaining residuals are much smaller, but that improvement in sample fit alone is not a reason to claim a causal effect or to generalize beyond the data.
A Residual Pattern Is Not a Causal Conclusion
A residual association can suggest that the current model does not capture some aspect of the conditional mean. It cannot identify why the pattern exists. A candidate predictor might be genuinely useful for describing the conditional response, but it might instead be a proxy for a missing variable, a feature of how observations were selected, or a consequence of measurement or coding problems. A residual pattern can also arise by chance.
Worked Example: A Group Difference Does Not Establish a Cause
Imagine a hypothetical observational analysis of recovery time, measured in days. The current model includes baseline severity. A clinician proposes candidate predictor \(W\), indicating whether patients received a particular support service. Assume the residual means are \(-1.5\) days for \(W=0\) and \(1.5\) days for \(W=1\), and that the two groups have comparable distributions of baseline severity. The adjusted residual contrast is \(1.5-(-1.5)=3.0\) days: in this sample, the model underpredicts recovery time by an average of 3.0 days for patients receiving the service relative to those not receiving it.
Adding the indicator would allow the model’s fitted conditional means to account for that group-level residual difference. It would not show that the service lengthens recovery time. For example, patients with more complex needs may be more likely to receive the service, and baseline severity may not capture all of that complexity. The residual association is therefore a reason to investigate the model, the data-collection process, and relevant subject-matter explanations—not a causal estimate.
What to Check Before Adding a Candidate
A useful assessment begins with a scientific reason to consider the candidate, then checks whether the residual pattern is coherent with that reason. The form of the pattern matters: a roughly linear trend in a continuous candidate suggests a different mean structure from a step pattern across categories. Also check whether one or two unusual observations dominate the apparent association, and whether the candidate is measured consistently and is available for the intended use of the model.
After fitting a candidate term, compare residual plots and summaries with those from the current model. A reduction in residual sum of squares is guaranteed when a predictor is added by least squares, so that reduction alone does not demonstrate a useful specification change. Ask whether the new residuals show less systematic structure, whether important patterns remain, and whether the added term is scientifically interpretable. If the candidate is closely related to current predictors, interpret its adjusted association rather than relying on raw group means or a simple residual-versus-candidate plot.
Common Mistakes
- Treating a residual association as proof of causation. Residuals can reveal a pattern not captured by the current model, but they cannot rule out confounding, selection, or measurement explanations.
- Using raw residual group means when the candidate is related to current predictors. Residualize the candidate against the current predictors or fit the expanded model to assess its adjusted association.
- Calling any reduction in residual sum of squares persuasive evidence. Least squares cannot increase that sum when terms are added. Consider the pattern removed, the assumptions, and the scientific rationale.
- Choosing a candidate only because its residual plot looks striking. A plausible reason to investigate should come from subject-matter knowledge, not solely from searching many variables for an appealing pattern.
Check Your Understanding
Use these questions to check how you would interpret a candidate predictor.
- What does a residual association with a candidate predictor suggest, and why is it not by itself a causal conclusion?
- Why might raw residual group means differ from the coefficient for a candidate added to a multiple regression model?
- In the nine-observation example, why does the difference in group residual means equal the adjusted coefficient for \(W\)?
- What does the reduction in residual sum of squares establish, and what does it not establish?
- If the candidate is associated with a predictor already in the model, what quantity should be examined to describe its added linear association?