Predictor Substitution Under Strong Collinearity

Why Correlated Predictors Can Substitute

Suppose a regression includes two measures of nearly the same quantity, such as two hypothetical indicators of the operating scale of an insurance portfolio. A model-selection procedure might retain one measure rather than the other, or the fitted coefficients might change substantially when the response data change slightly. Neither result automatically means that the model’s predictions have changed substantially. The key is to separate the stability of individual coefficients from the stability of fitted values.

The discussion below assumes an ordinary least squares regression with an intercept. The predictors are observed on the same cases, and the response and predictors are measured on consistent scales. For coefficient-uncertainty statements, the classical linear model assumes errors with conditional mean zero and constant variance, and independent observations. Normal errors are additionally needed for the usual exact small-sample t and F inference, but not for the coefficient-instability explanation itself.

Definition: Predictors are strongly collinear when one or more predictors are closely related to linear combinations of other predictors. With two predictors, a correlation near $1$ or $-1$ indicates that they contain much overlapping linear information. Predictor substitution occurs when competing models use one of these predictors in place of another and obtain similar fits over the observed data.

Collinearity does not by itself make ordinary least squares biased under the stated conditional-mean assumption. Instead, it makes some coefficient combinations difficult to estimate precisely. The earlier material on VIF provides a diagnostic for inflated coefficient variance; here the focus is what that instability means when models compete.

What Happens to the Least Squares Coefficients

For two centered predictor columns $x_1$ and $x_2$, let each column have squared length $n$, and let their correlation be $\rho$. If the design has full column rank, the least squares coefficient vector is $(X^\mathsf{T}X)^{-1}X^\mathsf{T}y$, with the intercept handled separately. The predictor part of the cross-product matrix is

X^\mathsf{T}X = n \begin{pmatrix} 1 & \rho\\ \rho & 1 \end{pmatrix}.

The directions $x_1+x_2$ and $x_1-x_2$ are its principal directions: multiplying the matrix by the corresponding coefficient directions gives eigenvalues $n(1+\rho)$ and $n(1-\rho)$. When $\rho$ is near $1$, the difference direction has very little information. Thus many different coefficient pairs can give nearly the same fitted values: changes in the two coefficients can largely cancel when the predictors move together.

Result: Under the homoskedastic linear model with error variance $\sigma^2$, the conditional covariance matrix of the two least squares slopes is $$ \operatorname{Cov}(\hat{\boldsymbol{\beta}}\mid X) = \frac{\sigma^2}{n(1-\rho^2)} \begin{pmatrix} 1 & -\rho\\ -\rho & 1 \end{pmatrix}. $$ As $|\rho|$ approaches $1$, the slope variances increase. If $|\rho|=1$, the predictor columns are exactly linearly dependent, the matrix is singular, and the two coefficients cannot be uniquely estimated.

This formula follows by inverting the predictor cross-product matrix: the inverse of the $2$ by $2$ correlation matrix is $\frac{1}{1-\rho^2}\begin{pmatrix}1&-\rho\\-\rho&1\end{pmatrix}$. Multiplying by $\sigma^2$ gives the coefficient covariance under the model assumptions. The variance inflation concerns coefficient estimates; it does not imply that every fitted value is equally unstable. Predictions for cases whose predictors follow the same relationship as the training data can remain stable, while predictions for unusual combinations may be sensitive.

Comparing Competing Fits

A reduced model containing only $x_1$ and a reduced model containing only $x_2$ need not have identical coefficients or residuals. If the predictors track each other closely, however, their fitted values and overall fit can be similar. The following constructed examples isolate the calculations; their numbers are hypothetical, not empirical findings.

Worked Example: Two Reduced Models With Similar Fit

There are four cases. The centered predictor vectors are $x_1=(-1,1,-1,1)$ and $z=(-1,-1,1,1)$, and $x_2=x_1+0.1z$. The response is $y=2x_2$. Compare regressions of $y$ on $x_1$ alone and on $x_2$ alone, each with an intercept.

Method. Since both predictors are centered, each fitted intercept is the response mean, which is zero. Also $x_1^\mathsf{T}x_1=z^\mathsf{T}z=4$ and $x_1^\mathsf{T}z=0$. Therefore $x_2^\mathsf{T}x_2=4+0.01(4)=4.04$, and $x_1^\mathsf{T}x_2=4$. Their correlation is $4/\sqrt{4(4.04)}=1/\sqrt{1.01}$, approximately $0.9950$.

For the $x_1$-only model, the slope is $x_1^\mathsf{T}y/(x_1^\mathsf{T}x_1)$. Since $y=2x_1+0.2z$, the numerator is $2(4)+0.2(0)=8$, so the slope is $8/4=2$. The residual vector is $y-2x_1=0.2z$, giving residual sum of squares $0.2^2(4)=0.16$.

For the $x_2$-only model, $y=2x_2$, so its slope is $x_2^\mathsf{T}y/(x_2^\mathsf{T}x_2)=2(4.04)/4.04=2$. Its residual sum of squares is zero. The total sum of squares is $y^\mathsf{T}y=4(4.04)=16.16$. Thus the $x_1$-only model has $R^2=1-0.16/16.16$, approximately $0.9901$; the $x_2$-only model has $R^2=1$.

Check. Both models use slope 2, and the first leaves only $0.16$ of residual sum of squares compared with total variation $16.16$. The fits are close in this constructed example, even though only the $x_2$-only model is exact. The common mistake is to treat a small difference in fit as proof that the predictors are interchangeable for every purpose; that conclusion requires considering the intended predictions and the range of predictor values.

Why Coefficient Signs Can Change

A multiple-regression coefficient describes a conditional comparison: it is the change in fitted response associated with changing that predictor while holding the other included predictors fixed. When two predictors almost always move together in the data, holding one fixed while changing the other may describe a poorly supported comparison. That is one reason why a coefficient sign can be sensitive even when the fitted response pattern is not.

Worked Example: A Small Response Change Reverses a Slope Sign

Use the centered vectors $x_1=(-1,1,-1,1)$, $z=(-1,-1,1,1)$, and $x_2=x_1+0.1z$ from the prior calculation. Consider two response vectors, $y_A=2x_1+0.19z$ and $y_B=2x_1+0.21z$. Fit each response on both predictors with an intercept.

Method. Because $z=10(x_2-x_1)$, the first response can be rewritten as $y_A=2x_1+1.9(x_2-x_1)=0.1x_1+1.9x_2$. Thus its fitted slopes are $\hat\beta_1=0.1$ and $\hat\beta_2=1.9$, with intercept zero. Similarly, $y_B=2x_1+2.1(x_2-x_1)=-0.1x_1+2.1x_2$, so its slopes are $-0.1$ and $2.1$.

The response change is $y_B-y_A=0.02z$. Its squared length is $0.02^2(4)=0.0016$, and each case’s fitted value changes by only $0.02$ in magnitude. Yet the coefficient of $x_1$ changes from $0.1$ to $-0.1$. Both two-predictor models fit their respective constructed responses exactly because each response is a linear combination of the included predictors.

Check. The sign reversal is consistent with a very small change in response values in a direction that distinguishes the two highly correlated predictors. It does not establish that the underlying conditional effect truly changed sign. The common mistake is to interpret a coefficient sign in isolation, without noting the high correlation and the uncertainty in the conditional comparison.

When Similar Training Fits Do Not Mean Similar Predictions

Substitution is most defensible when the use of the model concerns cases whose predictor combinations resemble those in the fitting data. If a future case has an unusual combination—such as one scale measure being high while a closely related measure is low—the data may provide little basis for determining its fitted response. Coefficient differences that cancel along the observed pattern need not cancel for that new case.

Worked Example: Comparing Predictions Away From the Observed Pattern

The two coefficient pairs in the preceding example are $(0.1,1.9)$ and $(-0.1,2.1)$, with intercept zero. For a new case with $x_1=10$ and $x_2=9$, calculate both fitted responses and compare their difference with the small response change in the training cases.

Method. The first model predicts $0.1(10)+1.9(9)=1+17.1=18.1$. The second predicts $-0.1(10)+2.1(9)=-1+18.9=17.9$. The predictions differ by $18.1-17.9=0.2$.

In the constructed training cases, $x_2-x_1$ is either $-0.1$ or $0.1$, whereas the new case has $x_2-x_1=-1$. The models’ coefficient difference is $(0.2,-0.2)$, so their prediction difference at any case is $0.2x_1-0.2x_2$. At the new case this is $0.2(10)-0.2(9)=0.2$; for a training case it has magnitude only $0.02$.

Check. The arithmetic agrees with the direct predictions, and the new case is farther from the observed relationship between predictors. The common mistake is to infer reliable extrapolation from similar in-sample fitted values. Similar fit within the data does not establish stable predictions for unsupported predictor combinations.

A Practical Interpretation Checklist

When competing regression models select different members of a strongly correlated predictor group, evaluate the comparison in context rather than treating the variable name or coefficient sign as decisive. In particular:

Compare fitted values and residual summaries on the same observations, not just the slopes.
Check whether the predictors are highly correlated and whether the coefficient estimates are correspondingly uncertain.
Interpret a multiple-regression slope conditionally: it describes changing one predictor while holding the other included predictors fixed.
Ask whether that conditional comparison, or the intended future predictor combination, is represented in the observed data.
Do not infer that a predictor has no useful information solely because another correlated predictor can substitute for it in one candidate model.

A further calculation trap is to confuse a marginal slope with a multiple-regression coefficient. In the sign-change example, for response $y_A$, the slope from regressing on $x_2$ alone is $x_2^\mathsf{T}y_A/(x_2^\mathsf{T}x_2)$. Here $x_2^\mathsf{T}y_A=8+0.076=8.076$ and $x_2^\mathsf{T}x_2=4.04$, giving approximately $1.9990$. That is not the multiple-regression coefficient $1.9$ for $x_2$, because the single-predictor fit does not hold $x_1$ fixed. Substituting the marginal slope for the conditional coefficient answers a different question.

Key takeaway: Strong collinearity can make individual coefficients, their signs, and variable selection unstable while leaving fitted values similar over the observed predictor pattern. Assess both the purpose of the model and whether the predictions of interest stay within that pattern.

Check Your Understanding

Work each problem before checking the answer shown at its end.

Two centered predictors have equal squared lengths $n$ and correlation $0.98$. Is their coefficient covariance matrix nonsingular? What is the smaller eigenvalue of their predictor cross-product matrix? (Answer: Yes; $0.02n$.)
Suppose $x_1^\mathsf{T}x_1=z^\mathsf{T}z=4$, $x_1^\mathsf{T}z=0$, $x_2=x_1+0.25z$, and $y=3x_1+0.5z$. Find the exact two-predictor slopes and the correlation of $x_1$ and $x_2$. (Answer: Slopes $1$ and $2$; correlation $1/\sqrt{1.0625}$, approximately $0.9701$.)
For standardized centered predictors with correlation $0.96$, the two eigenvalues of their cross-product matrix are $n(1+\rho)$ and $n(1-\rho)$. Find the ratio of the larger eigenvalue to the smaller. (Answer: $1.96/0.04=49$.)
Let $x_2=x_1+0.2z$, where $x_1$ and $z$ are orthogonal, centered vectors with squared lengths 4. If $y=1.5x_2$, find the slope and residual sum of squares for the $x_1$-only model, and its $R^2$. (Answer: Slope $1.5$; residual sum of squares $0.36$; $R^2=1-0.36/9.36$, approximately $0.9615$.)
Two fitted models have coefficient pairs $(0.3,1.7)$ and $(-0.2,2.2)$, both with intercept zero. Calculate their predictions at $(x_1,x_2)=(8,7)$, and state their difference. (Answer: $14.3$ and $13.8$; difference $0.5$.)

What You'll Learn