Why Residualize Both Variables?
A multiple-regression slope describes the relationship between an outcome and one predictor after accounting for the other predictors in the model. Residualization gives a numerical way to see that adjustment at work: first remove the part associated with a control variable from the outcome, then remove the part associated with that same control from the focal predictor. The slope relating the two sets of remaining values is the adjusted slope.
The phrase the same control matters. Regressing only the outcome on a control does not, by itself, give the adjusted slope for a focal predictor. The predictor also needs to be residualized against the control. The worked calculations below use one control so that each step can be checked directly.
The numerical identity we will use is that the least-squares coefficient of \(X\) in the model \(Y_i=\beta_0+\beta_1X_i+\beta_2Z_i+\varepsilon_i\) equals the slope from regressing \(r_Y\) on \(r_X\) through the origin:
The condition in the denominator says that the focal predictor must have some variation left after its linear relationship with the control is removed. If the denominator is zero, \(X\) is an exact linear function of the intercept and \(Z\) in these data, so its separate adjusted slope cannot be estimated.
A Numerical Calculation
Consider the following hypothetical data for five students. Let \(Y\) be a test score in points, \(X\) be hours spent studying, and \(Z\) be a prior-score measure. The purpose is only to illustrate the calculation; the numbers do not describe a real study.
| Student | \(Z\) | \(X\) (hours) | \(Y\) (points) |
|---|---|---|---|
| A | 1 | 5 | 25 |
| B | 2 | 2.5 | 22 |
| C | 3 | 5 | 28 |
| D | 4 | 7.5 | 38 |
| E | 5 | 5 | 37 |
Worked Example: Residualizing the Outcome Against the Control
Fit the least-squares regression of test score \(Y\) on prior-score measure \(Z\), including an intercept. Calculate the fitted scores and outcome residuals.
The means are \(\bar{z}=3\) and \(\bar{y}=30\). For a simple linear regression with an intercept, the fitted slope is \(\sum (z_i-\bar{z})(y_i-\bar{y})/\sum (z_i-\bar{z})^2\). Here the deviations of \(Z\) are \(-2,-1,0,1,2\), and those of \(Y\) are \(-5,-8,-2,8,7\). Thus:
The fitted line is \(\widehat{Y}=30+4(Z-3)=18+4Z\). Subtracting each fitted score from its observed score gives:
| Student | Fitted score \(\widehat{Y}\) | Outcome residual \(r_Y=Y-\widehat{Y}\) |
|---|---|---|
| A | 22 | 3 |
| B | 26 | -4 |
| C | 30 | -2 |
| D | 34 | 4 |
| E | 38 | -1 |
For example, student A's residual is \(25-22=3\) points. A positive residual means the observed score is above the score predicted by the regression on \(Z\); it does not mean the student has a positive residual in the full multiple regression.
Worked Example: Residualizing the Focal Predictor and Finding the Adjusted Slope
Now regress study time \(X\) on the same control \(Z\), with an intercept. Then calculate the slope from the outcome and predictor residuals.
The sample mean of \(X\) is 5 hours, and its deviations from the mean are \(0,-2.5,0,2.5,0\) hours. Using the same \(Z\) deviations as before, the fitted slope is:
The fitted line is \(\widehat{X}=5+0.5(Z-3)=3.5+0.5Z\). Subtracting these fitted values from the observed study times gives predictor residuals \(1,-2,0,2,-1\) hours. The residuals have nonzero variation, since their sum of squares is \(1+4+0+4+1=10\).
Pair each predictor residual with the outcome residual from the first example. The residual cross-product sum is \(3(1)+(-4)(-2)+(-2)(0)+4(2)+(-1)(-1)=20\). Therefore the adjusted slope is:
In this fitted additive model, a one-hour difference in study time is associated with a difference of 2 points in the predicted score when the prior-score measure is held fixed. The calculation does not establish that studying an additional hour causes a two-point increase.
Why the Residual Slope Is the Multiple-Regression Slope
The calculation has a general least-squares justification. In the full regression, the fitted value is a linear combination of an intercept, \(X\), and \(Z\). For any proposed coefficient \(b\) on \(X\), the intercept and \(Z\) coefficients are chosen to fit \(Y-bX\) using those two terms. Removing the fitted intercept-and-\(Z\) parts from \(Y\) and \(X\) leaves \(r_Y\) and \(r_X\). The remaining least-squares criterion for \(b\) is therefore the sum of squared differences between \(r_Y\) and \(b r_X\).
To see the minimizing value, write that criterion as \(Q(b)=\sum (r_{Y,i}-b r_{X,i})^2\). Differentiating and setting the derivative to zero gives:
Rearranging yields \(b\sum r_{X,i}^2=\sum r_{X,i}r_{Y,i}\). When the residual sum of squares for \(X\) is positive, division gives the residual-slope formula. This is why both variables must be residualized against the control, and why the residual regression for this identity is through the origin: the residuals already have zero sample mean because the first-stage regressions included intercepts.
Worked Example: Checking the Result Against the Full Regression
Verify the adjusted slope by writing the fitted multiple-regression equation for the same hypothetical data and checking its residuals.
The control-only fitted line for \(Y\) was \(18+4Z\), so the outcome residuals were \(3,-4,-2,4,-1\). The fitted line for \(X\) was \(3.5+0.5Z\), so \(X=3.5+0.5Z+r_X\). Substituting this expression into the outcome decomposition \(Y=(18+4Z)+r_Y\), and using the numerical relationship between the residuals, gives a full fitted equation with coefficients 2 for \(X\) and 3 for \(Z\):
For example, student A's fitted score is \(11+2(5)+3(1)=24\), leaving a full-model residual of \(25-24=1\). Applying the equation to all five students gives fitted scores \(24,22,30,38,36\) and full-model residuals \(1,0,-2,0,1\).
These full-model residuals are not the same as the earlier outcome residuals \(r_Y\): those came from regressing \(Y\) on \(Z\) alone. As a least-squares check, the full-model residuals sum to zero, their products with the centered \(Z\) values sum to \((-2)(1)+(-1)(0)+(0)(-2)+(1)(0)+(2)(1)=0\), and their products with the centered \(X\) values sum to \(0(1)+(-2.5)(0)+0(-2)+(2.5)(0)+0(1)=0\). They are orthogonal to the intercept and both predictors, as required for the full least-squares fit.
Assumptions, Scope, and a Common Pitfall
The residual-slope identity is a least-squares calculation. For it to represent the adjusted coefficient in the stated model, the fitted model must use the same observations and include an intercept, the focal predictor, and the control with an additive linear form. With more controls, residualize both \(Y\) and \(X\) on the same set of controls, including the intercept, and apply the same cross-product formula. If the model includes interactions or nonlinear terms, those terms must also be handled as part of the adjustment; the one-control calculation above is not a substitute for specifying the intended model.
No assumptions about normally distributed errors or a particular standard error are needed to calculate this coefficient. Such assumptions may matter for inference, but this example concerns the fitted slope only. Also, an adjusted coefficient remains an association in the model. Residualization alone does not show that the control is sufficient to support a causal interpretation.
- Common mistake: Regress \(Y\) on \(Z\), take those outcome residuals, and regress them on the original \(X\). This does not generally give the adjusted slope, because the original \(X\) still contains variation associated with \(Z\).
- Correction: Regress both \(Y\) and \(X\) on the same control set, then regress the outcome residuals on the predictor residuals through the origin. Check that the predictor residuals have positive sum of squares.
- Units: The adjusted slope has units of outcome per unit of focal predictor. In the example, it is points per study hour; the control's units do not become the slope's units.
Check Your Understanding
Use the definitions and calculations above to answer these questions.
- Why must the focal predictor, as well as the outcome, be residualized against the control?
- In the numerical example, what does student B's outcome residual of \(-4\) points mean?
- What condition on the predictor residuals is needed to calculate the residual slope, and what does failure of that condition imply?
- Why are the residual-slope regression and the full multiple regression different regressions, even though they share the same adjusted slope?
- Does a residualized slope of 2 points per study hour show that an extra hour of study causes a two-point score increase? Explain.