Tutorials › Regression Analysis (Undergraduate) › Correlated Predictors and Path Instability

Home  /  Tutorials  /  Subset and Stepwise Selection

Subset and Stepwise Selection · Tutorial 654 of 991

Correlated Predictors and Path Instability

Learn why correlated predictors can produce unstable selection paths, and how to distinguish changes in predictor order from changes in fitted predictions.

Advanced 10 min read

What You'll Learn

  • Explain why highly correlated predictors can have similar marginal associations with a response.
  • Calculate a one-predictor forward-selection comparison from sample correlations.
  • Show how a small change in data summaries can reverse the first selected predictor.
  • Compare fitted values from competing one-predictor models using a root-mean-square difference.
  • Explain how starting from a forced predictor can change which candidate has the strongest partial association.
  • Avoid interpreting selection order as proof of unique importance or causation.

Why Correlated Predictors Can Change Places

When two predictors carry much of the same information, a selection procedure may have difficulty distinguishing between them. One predictor can enter first because its sample association with the response is just a little stronger. A small change in the observed data—or a change in which predictors are already in the model—can then make the other predictor enter first. This change in order need not mean that the resulting fitted predictions are very different.

The point is not that selection is arbitrary: a specified procedure applies a specified rule to the observed data. Rather, when predictors are strongly correlated, the evidence separating their contributions can be slight. It is useful to keep three questions distinct: which predictor enters first, how the fitted predictions compare, and whether either predictor has a uniquely important or causal role. The first two can be studied from the candidate models; selection order alone does not answer the third.

Definition: Predictors are correlated when their observed values tend to vary together. When two predictors have a sample correlation near 1 or −1, they are strongly linearly associated in the sample. Path instability occurs when a small change in data or in the selection setup changes the sequence of models selected, such as which predictor enters first.

What Forward Selection Compares First

Suppose the response is \(Y\), and two candidate predictors are \(X_1\) and \(X_2\). In forward selection, the procedure begins with an intercept-only model and considers adding eligible predictors. For a first step that uses the partial \(F\) statistic, adding either one predictor is a comparison of an intercept-only model with a model containing that predictor and an intercept. With the same sample size and one added predictor, the larger absolute sample correlation with \(Y\) gives the larger \(F\) statistic.

For a one-predictor regression, let \(r\) be the sample correlation between the predictor and response. The regression \(F\) statistic is \(F = (n-2)r^2/(1-r^2)\), with \(1\) and \(n-2\) degrees of freedom. This statistic increases as \(r^2\) increases. Thus, if the entry rule chooses the eligible predictor with the largest partial \(F\), the first step is determined by which predictor has the larger absolute sample correlation with the response. A rule may also require that the proposed addition meet a specified entry threshold.

These comparisons concern the observed sample. To use the usual exact \(F\) reference distribution for a candidate-model test, assume a linear mean model for the response, independent errors with mean zero and constant variance, and normally distributed errors; predictors may be treated as fixed or the analysis may be conditional on their observed values. The algebraic ranking of the observed \(F\) statistics can still be calculated without normal errors, but the exact small-sample \(F\) inference relies on those assumptions. A ranking is not itself evidence that the chosen predictor is uniquely important.

Worked Example: A Narrow First-Step Advantage

Consider hypothetical data on \(n=30\) observations. Suppose the sample correlations are \(r_{X_1,Y}=0.800\), \(r_{X_2,Y}=0.795\), and \(r_{X_1,X_2}=0.990\). The candidate models both contain an intercept, and the entry rule selects the predictor with the larger one-predictor \(F\) statistic. Which predictor enters first?

For \(X_1\), substitute \(n=30\) and \(r=0.800\):

$$ F_1=\frac{(30-2)(0.800)^2}{1-(0.800)^2} =\frac{28(0.6400)}{0.3600} =49.78. $$

For \(X_2\):

$$ F_2=\frac{(30-2)(0.795)^2}{1-(0.795)^2} =\frac{28(0.632025)}{0.367975} \approx48.09. $$

Thus \(X_1\) enters first under this rule because \(49.78>48.09\). The advantage is modest relative to the overall size of both statistics, and the predictors themselves are extremely strongly correlated. The result identifies the first step for these particular data and this particular rule; it does not establish that \(X_1\) is substantively more important than \(X_2\).

A Small Data Change Can Reverse the First Step

The difference between the two response correlations in the example is only \(0.005\). If small changes to the observations shift those correlations by a few thousandths, the ordering can reverse. The following comparison uses a second hypothetical set of 30 observations with summaries close to the first set. It illustrates how a small data change can change the selected path without making one predictor’s relationship with the response dramatically different from the other’s.

Worked Example: A Slightly Different Sample Selects the Other Predictor

Suppose a slightly altered hypothetical sample has \(r_{X_1,Y}=0.796\), \(r_{X_2,Y}=0.799\), and \(r_{X_1,X_2}=0.990\). Keep the same one-predictor entry rule and \(n=30\). Calculate the two statistics.

For \(X_1\):

$$ F_1=\frac{28(0.796)^2}{1-(0.796)^2} =\frac{28(0.633616)}{0.366384} \approx48.42. $$

For \(X_2\):

$$ F_2=\frac{28(0.799)^2}{1-(0.799)^2} =\frac{28(0.638401)}{0.361599} \approx49.43. $$

Now \(X_2\) enters first because \(49.43>48.42\). The response correlations have moved by only \(0.004\) for \(X_1\) and \(0.004\) for \(X_2\), in opposite directions. Since the predictors are highly correlated, their separate rankings can depend on small differences in the observed sample.

A changed first step matters for the selected model sequence, but it does not automatically imply a large change in fitted values. The next calculation compares the two one-predictor fits from the first sample. It gives a scale for their difference on the observations used to fit them, rather than a guarantee about prediction for new observations.

Similar Fitted Values Can Accompany Different First Choices

Standardize each variable in the first hypothetical sample by subtracting its sample mean and dividing by its sample standard deviation. Denote the resulting values by \(x_1,x_2,y\); each has sample mean zero and sample variance one. In a regression of \(y\) on one standardized predictor, with an intercept, the fitted values are \(r_{X_j,Y}x_j\). This follows because the intercept is zero when both sample means are zero, and the least-squares slope is the sample covariance divided by the predictor’s sample variance, which here equals the sample correlation.

The fitted-value difference for observation \(i\) is therefore \(0.800x_{1i}-0.795x_{2i}\). Because sample-standardized variables have sums of squares \(n-1\), not \(n\), the mean squared difference across the \(n\) observations is:

$$ \begin{aligned} \frac{1}{30}\sum_{i=1}^{30}(0.800x_{1i}-0.795x_{2i})^2 &=\frac{29}{30}\left[0.800^2+0.795^2-2(0.990)(0.800)(0.795)\right]\\ &=\frac{29}{30}(0.012745)\\ &\approx0.01232. \end{aligned} $$

Taking the square root gives the root-mean-square difference in fitted values across these 30 observations:

$$ \sqrt{0.01232}\approx0.1110 $$

The fitted-value differences therefore have a root-mean-square size of about \(0.1110\) standardized response units on this sample. The quantity \(0.012745\) inside the calculation is the sample variance of the fitted-value differences, not their mean squared difference across the 30 observations. The factor \(29/30\) converts that sample variance to the requested average over observations.

This comparison explains how two models can choose different predictors and still produce similar fitted values: \(x_1\) and \(x_2\) tend to move together, so multiplying either by a similar slope can produce similar fitted responses. Similar fitted values do not prove that the models have identical residuals, that they will predict equally well on new data, or that their coefficients have the same interpretation. Those questions require separate evaluation.

The Starting Model Can Change the Comparison

The first-step comparison above starts from an intercept-only model. A different setup may require a predictor to be included before considering additions—for example, because it is required by the analysis plan. Once a predictor is already in the model, the procedure compares the candidates’ contributions after accounting for that predictor. These are conditional, or partial, associations; they need not have the same order as the marginal associations from the intercept-only model.

For a candidate \(X_j\) added to a model that already contains \(Z\), its partial correlation with \(Y\) is the correlation between the residuals from regressing \(X_j\) on \(Z\) and from regressing \(Y\) on \(Z\). For standardized variables, it can be calculated from their pairwise correlations:

$$ r_{X_j,Y\mid Z} = \frac{r_{X_j,Y}-r_{X_j,Z}r_{Y,Z}} {\sqrt{(1-r_{X_j,Z}^2)(1-r_{Y,Z}^2)}}. $$

When the two candidates have equal numbers of added parameters, the candidate with the larger squared partial correlation has the larger partial \(F\) statistic for the addition. This comparison is now conditional on \(Z\), so it answers a different question from the first-step comparison.

Worked Example: A Forced Predictor Changes the First Addition

Return to the first hypothetical sample’s correlations, and suppose a predictor \(Z\) is required in the starting model. For illustration, let \(r_{X_1,Z}=0.700\), \(r_{X_2,Z}=0.650\), and \(r_{Y,Z}=0.600\). The remaining correlations are \(r_{X_1,Y}=0.800\), \(r_{X_2,Y}=0.795\), and \(r_{X_1,X_2}=0.990\). These hypothetical summaries describe strongly associated predictors. Calculate each candidate’s partial correlation with \(Y\), controlling for \(Z\).

For \(X_1\):

$$ r_{X_1,Y\mid Z} = \frac{0.800-(0.700)(0.600)} {\sqrt{(1-0.700^2)(1-0.600^2)}} = \frac{0.380}{\sqrt{(0.510)(0.640)}} \approx0.6651. $$

For \(X_2\):

$$ r_{X_2,Y\mid Z} = \frac{0.795-(0.650)(0.600)} {\sqrt{(1-0.650^2)(1-0.600^2)}} = \frac{0.405}{\sqrt{(0.5775)(0.640)}} \approx0.6662. $$

The partial correlation for \(X_2\) is slightly larger, even though \(X_1\) had the larger marginal correlation with \(Y\) in the intercept-only comparison. With \(n=30\), adding one predictor to a model already containing \(Z\) leaves \(27\) residual degrees of freedom in the full model. The corresponding partial \(F\) statistics are approximately:

$$ F_{X_1\mid Z} =\frac{27(0.6651)^2}{1-(0.6651)^2} \approx21.42, \qquad F_{X_2\mid Z} =\frac{27(0.6662)^2}{1-(0.6662)^2} \approx21.54. $$

Under a rule that adds the candidate with the larger partial \(F\), \(X_2\) is the first addition after \(Z\). This does not contradict the earlier selection of \(X_1\): the two procedures compare different quantities, because one starts with no predictor and the other accounts for \(Z\) before evaluating candidates. The very small difference between the partial statistics is another sign that the ordering is sensitive to the precise comparison.

How to Read an Unstable Path

When correlated predictors compete, a selection path is best read as the output of a specified search rule applied to a particular sample. The path may be useful for constructing a compact model, but the order is not a ranking of causal effects or an unconditional measure of importance. In particular, a selected predictor can partly stand in for another predictor that conveys similar information.

Keep the following distinctions in view when reporting or interpreting the result:

  • Entry order: which candidate had the strongest qualifying contribution under the rule at that step.
  • Fitted predictions: the values produced by the model; competing models can have similar fits even if they contain different correlated predictors.
  • Conditional contribution: what a predictor contributes after accounting for predictors already in the model. This can differ from its marginal association.
  • Substantive interpretation: what the variables mean in context. A selection path alone does not establish causation or settle which correlated measure is the uniquely correct one.

Common Mistake

Wrong reasoning: “\(X_1\) entered first, so it is clearly more important than \(X_2\), and the fitted model using \(X_1\) must be much better.” This treats a narrow, rule-dependent difference in sample statistics as a decisive scientific conclusion.

Correction: State which rule and starting model produced the entry order, and describe the observed difference in the relevant statistic. When predictors are strongly correlated, check whether the competing models’ fitted predictions are materially different and interpret the selected variables cautiously. A small change in the data or in the conditional comparison may change the first entry without a comparable change in fitted values.

Key takeaway: Strongly correlated predictors can compete to represent similar information. Their entry order may be sensitive to small sample changes or to which predictors are already included, while their fitted predictions remain close. Selection order is a property of the data and the specified rule, not proof of unique importance.

Check Your Understanding

Use the calculations and distinctions in this tutorial to answer the following questions.

  1. In the first example, why does the predictor with the larger absolute response correlation have the larger one-predictor \(F\) statistic?
  2. In the second example, which predictor enters first, and what small change in the correlation summaries explains the reversal?
  3. Why is the mean squared fitted-value difference multiplied by \(29/30\) when the predictors are standardized using their sample standard deviations?
  4. How can controlling for a forced predictor \(Z\) change which candidate has the stronger association with the response?
  5. Give one conclusion that entry order supports and one conclusion about importance or causation that entry order alone does not support.

Previous

Next

Back to Tutorials