Why Correlated Predictors Can Substitute
Suppose an insurer studies claim cost using two predictors that measure similar characteristics, such as an account’s recent claim activity and a summary score built from that activity. Each predictor may track the outcome well. Yet once one is included, the other may contribute little additional information. A subset-selection procedure can therefore find models with similar fit but different variables.
This is not just a matter of two predictors having large correlations with the response. The key question is how much a candidate predictor adds after accounting for the predictors already in the model. That conditional contribution can be small even when the candidate has a strong marginal relationship with the response.
For a linear regression with an intercept, let \(y\) denote the response vector and let \(X_S\) be the matrix of predictor columns included in subset \(S\). The residual sum of squares, or RSS, is the sum of squared differences between observed and fitted responses. In the examples below, predictor and response columns are centered, so the intercept is fitted separately and the calculations can be shown with inner products.
With exact collinearity, the included columns do not identify a unique set of regression coefficients: multiple coefficient vectors give the same fitted values. With strong but imperfect collinearity, the coefficients are identifiable if the design matrix has full column rank, but small changes in the data can lead to large coefficient changes. For subset selection, the practical issue is that different subsets can yield nearly the same fitted values and RSS.
Comparing a Predictor With Its Substitute
A useful way to see substitution is to compare one-predictor models directly. Assume a fixed, centered data set and fit each model by ordinary least squares. If \(x\) and \(y\) are centered, and \(x\) is a nonzero predictor column, the fitted slope is \(x^\mathsf{T}y/(x^\mathsf{T}x)\). The fitted values are the projection of \(y\) onto the direction of \(x\). When \(x\) has length one, the reduction in RSS from the intercept-only model is \((x^\mathsf{T}y)^2\).
Worked Example: Two Correlated Predictors With Equal Fit
Consider centered, unit-length predictor columns \(x_1=u\) and \(x_2=\frac45u+\frac35v\), where \(u\) and \(v\) are orthogonal unit vectors. Let the centered response be \(y=\frac{3u+v}{\sqrt{10}}\). Compare the two one-predictor subsets and then fit both predictors together. Identify the selection issue.
Method. Use inner products to find each one-predictor fitted value and its RSS. The predictor correlation is \(x_1^\mathsf{T}x_2=\frac45=0.8\), so the predictors are strongly correlated but not exactly dependent.
First, \(y^\mathsf{T}y=(9+1)/10=1\). For \(x_1\), \(x_1^\mathsf{T}y=3/\sqrt{10}\). Its fitted slope is therefore \(3/\sqrt{10}\), and its RSS is \(1-(3/\sqrt{10})^2=1-9/10=0.1\).
For \(x_2\), \(x_2^\mathsf{T}y=\frac45\frac3{\sqrt{10}}+\frac35\frac1{\sqrt{10}}=\frac{12+3}{5\sqrt{10}}=\frac3{\sqrt{10}}\). Since \(x_2\) also has unit length, its fitted slope is \(3/\sqrt{10}\) and its RSS is again \(1-9/10=0.1\). Each model has \(R^2=1-0.1/1=0.9\).
To check the joint model, set both coefficients to \(5/(3\sqrt{10})\). The resulting fitted response is \[ \frac{5}{3\sqrt{10}}x_1+\frac{5}{3\sqrt{10}}x_2 =\frac{5}{3\sqrt{10}}u+\frac{5}{3\sqrt{10}}\left(\frac45u+\frac35v\right) =\frac{3u+v}{\sqrt{10}}=y. \] Thus the joint RSS is zero. Both single-predictor subsets fit equally well, although they select different variables; together the predictors recover the response exactly.
Check. The two single-predictor RSS values are both \(0.1\), while the joint fit has RSS \(0\). The common mistake is to infer from this exact joint fit that either single predictor was uniquely responsible for the response pattern. The calculations show otherwise.
The example isolates the selection issue. A criterion that chooses one predictor from the two single-predictor subsets cannot distinguish them by RSS. A deterministic tie rule may select one, but that rule does not establish that the chosen variable is substantively more important. If the two RSS values are merely close rather than equal, the same caution applies: a small fit difference may not justify a strong distinction between the variables.
Conditional RSS Reduction
In a search, a candidate is usually considered after some predictors are already included. Its contribution must then be measured relative to the current model, not by its association with the response on its own.
Here the current model includes an intercept; equivalently, all vectors can be treated as centered. The formula assumes ordinary least squares and that the candidate is not already in the span of the current model. If \(z_\perp=0\), the candidate is exactly explained by the existing columns and cannot add a new fitted direction.
Write \(z=z_{\parallel}+z_\perp\), where \(z_{\parallel}\) is in the span of the current model and \(z_\perp\) is orthogonal to that span.
The residual \(e\) is orthogonal to the current model’s columns, so \(z_{\parallel}^\mathsf{T}e=0\). Consequently \(z^\mathsf{T}e=z_\perp^\mathsf{T}e\).
Adding the candidate amounts to fitting a multiple \(b z_\perp\) to \(e\). The squared error is \((e-bz_\perp)^\mathsf{T}(e-bz_\perp)\), minimized at \(b=z_\perp^\mathsf{T}e/(z_\perp^\mathsf{T}z_\perp)\).
Substituting that value of \(b\) gives a decrease in RSS of \((z_\perp^\mathsf{T}e)^2/(z_\perp^\mathsf{T}z_\perp)\), as claimed.
This derivation explains why a raw correlation with the response can give the wrong impression during a search. The relevant predictor is its residualized direction \(z_\perp\): the portion that is new after accounting for the current subset.
Worked Example: A Search Path Produces Different Subsets
Let \(u,v,w\) be mutually orthogonal unit vectors. Consider \(x_1=u\), \(x_2=\frac45u+\frac35v\), \(x_3=w\), and \(y=3u+v+3w\). A forward search is to select two predictors. Show how different first choices can lead to different two-predictor subsets with the same RSS.
Method. Compare marginal RSS reductions at the first step, then calculate the residual and RSS after each possible first choice. The three predictors are unit length; \(x_1^\mathsf{T}x_2=0.8\), and \(x_3\) is orthogonal to both.
The response has squared length \(y^\mathsf{T}y=9+1+9=19\), so the intercept-only RSS is \(19\). The inner products are \(x_1^\mathsf{T}y=3\), \(x_2^\mathsf{T}y=\frac45(3)+\frac35(1)=3\), and \(x_3^\mathsf{T}y=3\). Each possible first predictor has a fitted slope of \(3\), reduces RSS by \(3^2=9\), and leaves RSS \(19-9=10\). Thus the first step is tied.
If the search first selects \(x_1\), the residual is \(y-3x_1=v+3w\), with RSS \(1+9=10\). Adding \(x_3=w\) fits \(3w\), leaving residual \(v\) and RSS \(1\). The resulting subset is \(\{x_1,x_3\}\).
If the search first selects \(x_2\), the residual is \(y-3x_2=3u+v+3w-3(\frac45u+\frac35v)=\frac35u-\frac45v+3w\). Its squared length is \(9/25+16/25+9=10\). Adding \(x_3\) again fits \(3w\), leaving \(\frac35u-\frac45v\), whose squared length is \(9/25+16/25=1\). The resulting subset is \(\{x_2,x_3\}\), also with RSS \(1\).
For either two-predictor subset, \(R^2=1-1/19=18/19\). The remaining proxy can reduce RSS by one more: after \(x_1,x_3\), the residual \(v\) has inner product \(3/5\) with \(x_2\)’s part orthogonal to \(x_1\), which has squared length \(9/25\); the decrease is \((3/5)^2/(9/25)=1\). After \(x_2,x_3\), adding \(x_1\) gives the same decrease. Both three-predictor fits therefore have RSS zero.
Check. The two possible two-predictor subsets have the same RSS, \(1\), and the same \(R^2\), \(18/19\). The first tied choice determines which proxy appears in the subset. Choosing one by a tie rule is a search decision, not evidence that the alternative predictor could not substitute.
Why Search Order Matters
Forward selection and backward elimination examine subsets through different sequences of additions and removals. With correlated predictors, the subset reached at a given step can depend on what is already in the model. Once one proxy is included, the other may have little conditional contribution, even if either could have represented much of the response pattern when considered alone. A search can therefore favor one variable partly because of its route through the candidate subsets.
This does not mean every search will choose different variables, or that every difference in subset is harmless. It means that a near-equal fit alone is weak evidence for a unique choice. A careful comparison reports both the model size and the fit criterion, checks whether competing subsets have close scores, and describes the selected variables in context without treating selection as proof of unique importance.
Worked Example: Correcting a Marginal RSS Calculation
A current regression model has residual \(e=2q+r\), where \(q\) and \(r\) are orthogonal unit vectors. Its RSS is \(2^2+1=5\). A unit-length candidate predictor is \(z=0.8x_1+0.6q\), where \(x_1\) is a unit vector in the current model’s span and is orthogonal to \(q\). Find the conditional RSS reduction and the new RSS. Explain why a calculation using the unadjusted predictor can mislead.
Method. Residualize the candidate on the current model. The \(0.8x_1\) component is already represented, so \(z_\perp=0.6q\). Then \(z_\perp^\mathsf{T}e=1.2\) and \(z_\perp^\mathsf{T}z_\perp=0.36\). The conditional reduction is \(1.2^2/0.36=1.44/0.36=4\), leaving RSS \(5-4=1\).
Common mistake. If the fitted part of the response along \(x_1\) is \(10x_1\), then the full response is \(y=10x_1+e\), and \(z^\mathsf{T}y=8+1.2=9.2\). Using \((z^\mathsf{T}y)^2=84.64\) as the reduction for adding \(z\) to the current model is wrong: it uses the response component already explained by \(x_1\), and it even exceeds the current RSS of \(5\). The correction is to use \(z_\perp\) and the current residual, which gives a reduction of \(4\) and a valid new RSS of \(1\).
Check. The fitted candidate adds \(b z_\perp\), with \(b=1.2/0.36=10/3\), so it removes \((10/3)(0.6q)=2q\) from the residual. The remaining residual is \(r\), with squared length \(1\), confirming the RSS calculation.
Common Interpretation Traps
- Calling the selected proxy uniquely important. A similar substitute may have been excluded because another correlated predictor entered first. Selection identifies a useful subset under a particular procedure and criterion; it does not by itself establish unique importance.
- Comparing marginal scores after the first step. Once predictors are in the model, compare conditional RSS reductions or the stated selection criterion for the competing subsets. A raw response correlation does not measure the remaining contribution.
- Confusing similar fit with identical models. Two subsets can have nearly equal RSS but different predictors. Their coefficients and interpretations can differ even when their fitted values are close.
- Assuming collinearity always prevents fitting. Exact dependence prevents unique coefficient estimates for the redundant columns. Strong but imperfect dependence can still allow a fit, though coefficient estimates may be sensitive.
Check Your Understanding
For each problem, focus on what the candidate adds beyond the current subset.
- Two centered, unit-length predictors have correlations \(0.90\) and \(0.89\), respectively, with a unit-length response. Find the one-predictor RSS for each model and state which has the smaller RSS. (Answer: \(0.1900\) and \(0.2079\), respectively; the first predictor has the smaller RSS.)
- Two same-size subsets have RSS values \(42\) and \(43\) on the same analysis sample. State the absolute RSS difference and one reason not to treat the lower-RSS subset as automatically the uniquely important model. (Answer: The difference is \(1\); correlated substitute predictors may yield nearly equal fit, so the small difference alone does not establish unique importance.)
- A candidate’s component orthogonal to the current model has squared length \(1.5\) and inner product \(-3\) with the current residual. Find the RSS reduction from adding it. (Answer: \(9/1.5=6\).)
- Two predictors have correlation \(0.98\). Either one, when included alone, gives nearly the same RSS; after one is selected, the other has little conditional contribution. Give the appropriate interpretation. (Answer: The selected predictor may be acting as a substitute for the other; the search result does not show that it is uniquely important.)
- At the first step of a forward search, two candidates have equal RSS reductions. At the next step, each candidate’s substitute gives little additional reduction after the first choice. What can be concluded about the selected variable? (Answer: The first-step tie or tie rule can determine which variable appears; selection alone does not distinguish the substitutes as uniquely important.)