Why Residualize Both Variables?
A predictor and response may move together because both are associated with other predictors in a regression model. To examine the association that remains after linear adjustment for those other predictors, remove the part of each variable explained by the adjustment variables. The slope relating what remains is the predictor’s partial regression coefficient.
This is useful when a multiple-regression coefficient needs a geometric or computational interpretation. Rather than treating the coefficient as an opaque output, residualization shows which part of the response and predictor determines it. The method is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
The definition assumes that \(Z\) has full column rank, so \(P_Z\) is defined by the displayed formula. It also requires \(x^*\) to be nonzero: if \(x^*=0\), then \(x\) is already an exact linear combination of the columns of \(Z\), and its separate coefficient cannot be identified. The same rank and identifiability conditions are needed for the corresponding ordinary least-squares coefficient in the full model.
The residuals \(y^*\) and \(x^*\) are not generally the original response and predictor. They are the parts left after each has been linearly projected onto the same adjustment variables. With an intercept in \(Z\), each residual vector sums to zero.
The Partial Regression Identity
Consider the multiple linear regression model \(y=Z\gamma+x\beta+\varepsilon\), where \(\beta\) is the coefficient of interest and \(\gamma\) contains the coefficients for \(Z\). The following identity is an algebraic least-squares result; it does not require normally distributed errors.
The denominator is positive when \(x^*\ne 0\). The formula is the usual through-origin least-squares slope: choose the value of \(\beta\) that best approximates \(y^*\) by \(\beta x^*\). A short derivation explains why this slope equals the coefficient from the full regression.
For a fixed value of \(\beta\), the least-squares residual vector after fitting \(Z\gamma\) to \(y-x\beta\) is \(M_Z(y-x\beta)\). Thus the minimized sum of squares for this fixed \(\beta\) is \(\lVert M_Z(y-x\beta)\rVert^2\).
Since \(M_Z\) is symmetric and idempotent, \(M_Z^\mathsf{T}M_Z=M_Z\). Therefore the objective is \((y^*-\beta x^*)^\mathsf{T}(y^*-\beta x^*)\).
The derivative is \(-2(x^*)^\mathsf{T}y^*+2\beta(x^*)^\mathsf{T}x^*\). Setting it to zero gives \(\widehat{\beta}=(x^*)^\mathsf{T}y^*/(x^*)^\mathsf{T}x^*\). The positive denominator makes this the unique minimizing value.
This argument is the partial regression identity, also called the Frisch–Waugh–Lovell theorem. It shows that the coefficient of \(x\) in the full regression is exactly the slope obtained after removing the linear effects of \(Z\) from both \(x\) and \(y\). Residualizing only the response or only the predictor does not produce this identity.
Worked Examples
Worked Example: Finding a Partial Slope from Residuals
In a hypothetical study of claim processing time, analysts adjust linearly for claim complexity. Suppose the residuals from regressing processing time \(y\) and a staffing measure \(x\) on complexity are, respectively,
Find the partial regression slope of processing time on staffing after adjustment for complexity. Then verify the least-squares condition.
Method. Use the through-origin slope of \(y^*\) on \(x^*\), as required by the partial regression identity.
Calculate the cross-product. Multiply corresponding entries and add: \( (x^*)^\mathsf{T}y^*=(-2)(-3)+(-1)(-1)+(0)(1)+(1)(1)+(2)(2)=6+1+0+1+4=12\).
Calculate the sum of squared predictor residuals. \( (x^*)^\mathsf{T}x^*=(-2)^2+(-1)^2+0^2+1^2+2^2=4+1+0+1+4=10\).
Compute the slope. \( \widehat{\beta}=12/10=1.2\). For each one-unit increase in the staffing measure’s residual after adjustment for complexity, the fitted processing-time residual increases by 1.2 units.
Check. The fitted residualized response is \(1.2x^*=(-2.4,-1.2,0,1.2,2.4)\). The remaining residuals are \(y^*-1.2x^*=(-0.6,0.2,1,-0.2,-0.4)\). Their inner product with \(x^*\) is \(1.2-0.2+0-0.2-0.8=0\), as required for a least-squares fit through the origin.
Common mistake avoided. The coefficient is not calculated from the original staffing and processing-time values. It is calculated from the two sets of residuals after the same adjustment.
Worked Example: Updating Residual Sum of Squares
For a hypothetical regression of monthly service cost on a new workload measure, suppose the existing model contains an intercept and other adjustment variables. Its residual sum of squares is 84. After residualizing both cost and workload on those variables, suppose the residualized workload has sum of squares 20 and its cross-product with the residualized cost is 12. Find the new partial slope and the residual sum of squares after adding workload.
Method. First calculate the partial slope. Then calculate how much the baseline residual sum of squares falls when the residualized workload is added.
Partial slope. The coefficient is \(12/20=0.6\).
Reduction in residual sum of squares. The residualized response is projected onto the residualized workload. Its fitted component is \(0.6x^*\), so the reduction is the squared length of that fitted component: \( (0.6)^2(20)=0.36(20)=7.2\). Equivalently, the reduction is \(12^2/20=144/20=7.2\).
Updated residual sum of squares. Subtract the reduction from the baseline value: \(84-7.2=76.8\).
Check. The reduction is nonnegative and no larger than the baseline residual sum of squares; \(7.2\le84\). The updated residual sum of squares, 76.8, is therefore smaller than 84, as least squares requires.
Common mistake avoided. The reduction is not the square of the slope alone. It is \(\widehat{\beta}^{\,2}(x^*)^\mathsf{T}x^*\), which accounts for the scale of the residualized predictor.
Worked Example: Distinguishing Partial Slope from Partial Correlation
In a hypothetical portfolio analysis, response residuals and predictor residuals have cross-product 12. The sum of squared predictor residuals is 18, and the sum of squared response residuals is 32. These residuals were obtained by adjusting both variables for the same set of underwriting variables. Calculate the partial slope and partial correlation, then find the remaining residual sum of squares after adding the predictor.
Partial slope. The slope is the cross-product divided by the predictor residual sum of squares: \(12/18=2/3\), or 0.6667 to four decimal places.
Partial correlation. The correlation of the two residual vectors is \(12/\sqrt{18(32)}=12/\sqrt{576}=12/24=0.5\).
Remaining residual sum of squares. Before adding the predictor, the residualized response sum of squares is 32. The reduction is \(12^2/18=144/18=8\), so the remaining sum of squares is \(32-8=24\).
Check. The partial correlation is dimensionless and lies between \(-1\) and \(1\). The slope is measured in response units per predictor unit, so it need not equal the partial correlation. The residual sum of squares falls from 32 to 24.
Common mistake avoided. Reporting 0.5 as the partial slope confuses a standardized measure of linear association with the slope in the original units.
Interpretation, Conditions, and Exam Traps
Residualization gives a precise meaning to “holding other predictors fixed” in a linear model. The coefficient describes the association between the response and predictor after removing the parts each can linearly explain using the adjustment variables. It is a statement about a conditional linear association, not by itself a causal effect. A causal interpretation requires additional assumptions about how the data were generated.
The calculation is an algebraic property of least squares. For inference about \(\beta\), further assumptions are needed. In the classical model, errors have conditional mean zero and constant variance given the predictors; normal errors give exact finite-sample \(t\)-inference under the usual full-rank conditions. Residualization alone does not make an observational association causal, nor does it establish that the linear model is appropriate.
| Quantity | Calculation | Interpretation |
|---|---|---|
| Partial slope | \((x^*)^\mathsf{T}y^*/(x^*)^\mathsf{T}x^*\) | Response units per predictor unit after linear adjustment |
| Partial correlation | \((x^*)^\mathsf{T}y^*/\sqrt{[(x^*)^\mathsf{T}x^*][(y^*)^\mathsf{T}y^*]}\) | Unit-free association between the two residual vectors |
| Reduction in residual sum of squares | \([(x^*)^\mathsf{T}y^*]^2/[(x^*)^\mathsf{T}x^*]\) | Decrease in least-squares error from adding this predictor |
A frequent exam trap is to use the unadjusted slope \(x^\mathsf{T}y/x^\mathsf{T}x\) when the question asks for an adjusted coefficient. For example, if the original cross-products are \(x^\mathsf{T}y=50\) and \(x^\mathsf{T}x=100\), that calculation gives 0.5. If residualizing both variables instead gives \((x^*)^\mathsf{T}y^*=12\) and \((x^*)^\mathsf{T}x^*=10\), the adjusted coefficient is \(12/10=1.2\), not 0.5. The other common trap is to residualize only \(y\). The theorem requires applying the same adjustment matrix \(M_Z\) to both variables.
Check Your Understanding
Use the partial regression identity and show the essential arithmetic for each problem.
- After adjusting two variables for the same covariates, suppose the residualized predictor sum of squares is 15 and the residual cross-product is 9. Find the partial slope. (Answer: 0.6)
- The residualized predictor has sum of squares 25, and its cross-product with the residualized response is \(-10\). The baseline residual sum of squares is 60. Find the partial slope and the updated residual sum of squares. (Answer: slope \(-0.4\); updated RSS \(56\))
- For two residual vectors, the predictor sum of squares is 8, the response sum of squares is 18, and the cross-product is 6. Find their partial correlation and partial slope. (Answer: partial correlation \(0.5\); partial slope \(0.75\))
- A residualized predictor has sum of squares 12 and cross-product with the residualized response 6. The baseline residual sum of squares is 20. Calculate the partial slope, the reduction in residual sum of squares, and the updated residual sum of squares. (Answer: slope \(0.5\); reduction \(3\); updated RSS \(17\))
- Suppose \(x^*=0\) for every observation after regressing \(x\) on the adjustment matrix \(Z\). Can the partial slope for \(x\) be computed? Explain the identifiability issue. (Answer: No; the denominator \((x^*)^\mathsf{T}x^*\) is zero because \(x\) is an exact linear combination of the adjustment variables, so its separate coefficient is not identifiable.)