Why Remove the Existing Predictors?
Suppose a regression model already includes several predictors, and you want to assess whether another variable may add systematic structure to the outcome. A plot of the outcome against that candidate can be misleading: some of their apparent association may simply reflect how both vary with predictors already in the model. An added-variable plot addresses this by removing the linear associations with the existing predictors from both variables before plotting.
The plot is useful for asking a focused diagnostic question: after accounting linearly for the current predictors, is there a systematic relationship between the remaining part of the outcome and the remaining part of the candidate predictor? It can help identify a potentially useful omitted predictor or term. It does not, by itself, establish that the candidate causes the outcome or that adding it is appropriate.
Definition and Model Conditions
Let \(y\) be the vector of \(n\) observed outcomes, let \(X\) be the design matrix for the current model, and let \(z\) be the observed values of a candidate predictor. The columns of \(X\) include the intercept and all current predictors. The current model is fitted by ordinary least squares. To interpret a possible extension, consider the additive model \(E(Y\mid X,z)=X\beta+\gamma z\), where \(\gamma\) is the candidate predictor’s coefficient after accounting for the existing predictors.
The construction below uses the same observations in all regressions, an intercept in \(X\), and a full-column-rank \(X\). For the candidate coefficient to be separately estimable, \(z\) must not be an exact linear combination of the columns of \(X\). These conditions make the required least-squares fits well-defined. The algebraic identity for the plotted slope does not require normally distributed errors. Any broader interpretation of the fitted model still depends on the substantive suitability of its mean structure and data.
Each plotted coordinate is therefore a residual, not the original observation. The vertical coordinate records how much of the outcome remains after its linear association with the current predictors is removed. The horizontal coordinate records how much of the candidate remains after its linear association with those same predictors is removed. Both residualizations must use the same \(X\).
Why the Plotted Slope Is the Added Coefficient
Write \(P_X\) for the projection onto the column space of \(X\), and \(M_X=I-P_X\) for the residual-maker matrix. Thus \(r_y=M_Xy\) and \(r_z=M_Xz\). In the expanded model, the least-squares coefficient on \(z\), denoted \(\hat{\gamma}\), minimizes the residual sum of squares over both the coefficients on \(X\) and the coefficient on \(z\).
For any fixed value \(\gamma\), the best-fitting contribution from \(X\) is obtained by regressing \(y-\gamma z\) on \(X\). The residual after that fit is \(M_X(y-\gamma z)=r_y-\gamma r_z\). Consequently, the remaining sum of squared errors as a function of \(\gamma\) is
Expanding this expression gives \(r_y'r_y-2\gamma r_z'r_y+\gamma^2r_z'r_z\). Its derivative with respect to \(\gamma\) is \(-2r_z'r_y+2\gamma r_z'r_z\). Setting the derivative equal to zero and solving, when \(r_z'r_z>0\), gives
This is exactly the least-squares slope when \(r_y\) is regressed on \(r_z\) with the intercept fixed at zero. The residuals from both regressions are orthogonal to the columns of \(X\), including the intercept, so both residual vectors have mean zero. The through-origin line is the relevant slope calculation; adding a freely estimated intercept to the plotted regression is unnecessary and does not express this identity in the same way.
Worked Example: Calculating the Adjusted Slope
In hypothetical data on four locations, suppose the current model uses an intercept and a location-level predictor \(x\). The recorded candidate \(z\) and outcome \(y\) are
| Location | \(x\) | \(z\) | \(y\) |
|---|---|---|---|
| 1 | 0 | 2 | 15 |
| 2 | 1 | -1 | 3 |
| 3 | 2 | 4 | 9 |
| 4 | 3 | 9 | 13 |
First regress \(y\) on an intercept and \(x\). The mean outcome is \(10\), and the slope on \(x\) is zero: the outcome deviations \(5,-7,-1,3\) have zero sum and have zero inner product with the centered \(x\) values \(-1.5,-0.5,0.5,1.5\). Thus the fitted values are all \(10\), and \(r_y=(5,-7,-1,3)'\).
Next regress \(z\) on an intercept and \(x\). The mean of \(z\) is \(3.5\). Its centered values are \(-1.5,-4.5,0.5,5.5\). Their inner product with the centered \(x\) values is \(13\), and the sum of squared centered \(x\) values is \(5\), so the fitted slope is \(13/5=2.6\). Subtracting the fitted values gives \(r_z=(2.4,-3.2,-0.8,1.6)'\).
The slope in the added-variable plot is
Therefore, in the model including \(x\) and \(z\), the estimated coefficient on \(z\) is \(25/12\) outcome units per unit of \(z\), holding \(x\) fixed in the linear model. This is an adjusted association, not a causal effect. The calculation uses all four observations and a full-rank current design; \(z\)'s residual vector is nonzero, so the candidate coefficient is estimable.
Reading the Plot
The direction and shape of the plotted points give diagnostic information about the candidate’s remaining relationship with the outcome. A clear upward or downward trend indicates positive or negative adjusted linear association, respectively. A diffuse cloud without a discernible trend provides little visual evidence of a linear association in these residualized variables. A curve or other systematic shape may suggest that the candidate’s contribution is not adequately represented by a straight-line term.
The plot’s fitted line is constrained to pass through \((0,0)\), and its slope has the exact coefficient interpretation just derived. The individual points, however, are residualized values; they should not be interpreted as raw outcomes or raw candidate measurements. The slope summarizes a linear association in the residualized data and does not guarantee that the model’s other assumptions are satisfied.
Worked Example: A Candidate With an Adjusted Linear Pattern
Suppose a hypothetical study records an outcome \(y\), an existing predictor \(x\), and a binary candidate \(z\) for five locations. Let \(x=(0,1,2,3,4)'\), \(z=(1,0,0,0,1)'\), and let the observed outcomes be \(y=(11.2,9.7,10.2,10.7,13.2)'\). The values are chosen to make the residualization and slope easy to verify.
The mean of \(z\) is \(0.4\), and its centered values are \((0.6,-0.4,-0.4,-0.4,0.6)'\). Their inner product with the centered \(x\) values \((-2,-1,0,1,2)'\) is zero. Thus the fitted regression of \(z\) on an intercept and \(x\) is the constant \(0.4\), and \(r_z=(0.6,-0.4,-0.4,-0.4,0.6)'\).
The mean outcome is \(11\). Its centered values are \((0.2,-1.3,-0.8,-0.3,2.2)'\). Their inner product with the centered \(x\) values is \(5\), so the fitted slope on \(x\) is \(5/10=0.5\). The current-model residuals are therefore \(r_y=(1.2,-0.8,-0.8,-0.8,1.2)'\).
The added-variable slope is
The points lie exactly on the through-origin line with slope \(2\). The expanded model estimates an outcome difference of \(2\) units per one-unit increase in the binary candidate, conditional on the linear adjustment for \(x\). For a binary \(z\), this coefficient describes the adjusted difference between its two coded groups under this model. The plot and calculation establish neither causation nor generalizability beyond the data and model context.
When the Candidate Is Hard to Assess
An added-variable plot can reveal a serious limitation as well as a possible missing predictor. If \(z\) is almost explained by the existing predictors, its residual vector \(r_z\) has little variation. The denominator \(r_z'r_z\) is then small, so small changes in the data can substantially change the estimated slope. The plot may be horizontally compressed, making a trend difficult to judge. This is a warning about limited information for separating the candidate’s association from those of the existing predictors.
Worked Example: An Exactly Redundant Candidate
In hypothetical data, suppose the current model contains an intercept and \(x=(0,1,2,3)'\), while a proposed candidate is \(z=(5,7,9,11)'\). Since \(z=5+2x\), its regression on the intercept and \(x\) fits every value exactly. Hence \(r_z=(0,0,0,0)'\), and \(r_z'r_z=0\).
The added-variable slope formula would require division by zero, so no slope exists. This is not a case of “no adjusted association”; rather, the candidate is perfectly redundant with the current predictors, and its separate coefficient cannot be identified. A plot with every horizontal coordinate equal to zero cannot show how the outcome varies with a distinct contribution from \(z\).
Common Mistakes and Practical Limits
- Plotting raw values instead of residuals. A raw \(y\)-versus-\(z\) plot does not remove the current predictors’ linear contributions. Regress each variable on the same \(X\) first.
- Using an unconstrained intercept for the key slope. The coefficient identity is for the through-origin regression of \(r_y\) on \(r_z\). The residuals are already centered because the current model includes an intercept.
- Calling the slope causal. The slope is an adjusted regression association under the specified model. Adjustment alone does not rule out confounding or justify a causal conclusion.
- Treating visual evidence as a formal test. A plot is a diagnostic display, not a p-value or confidence interval. Its appearance can be affected by a few influential observations, so investigate unusual points rather than letting them silently determine the conclusion.
- Assuming a weak-looking pattern proves the candidate is irrelevant. If \(z\) has little residual variation, the data may provide limited information about its separate contribution. Check how much variation remains after residualization.
The central purpose of an added-variable plot is to isolate the candidate’s linear association with the outcome beyond the predictors already included. It is most informative when the candidate retains meaningful variation after adjustment and the points show a reasonably interpretable pattern. Use it as evidence for reviewing model specification, not as an automatic rule for adding or rejecting a term.
Check Your Understanding
Answer in terms of residualized variables and the current regression model.
- What does the horizontal coordinate of an added-variable plot represent, and which regression produces it?
- Why does the through-origin slope equal the coefficient of the candidate in the expanded model?
- If the candidate predictor is an exact linear combination of the existing predictors, what happens to its residual vector and why is its separate coefficient not estimable?
- A plot has a clear positive slope. What can you conclude about the adjusted association, and what can you not conclude about causation?
- Why can a candidate with very little residual variation be difficult to assess, even if a numerical slope can be calculated?