Bidirectional Stepwise Selection

Why Reconsider Predictors After Every Change?

A predictor’s contribution to a regression model is conditional on which other predictors are included. A variable may look useful when considered alone, yet add little once another variable enters. Conversely, a variable that is weak on its own can become useful after adjustment for predictors already in the model. This is why a selection procedure that only adds variables can miss changes in the role of those already included.

Bidirectional stepwise selection addresses this by allowing both additions and deletions during a model-building path. After a change, it reassesses the available moves using the updated model. The procedure is a search strategy: it does not guarantee that the final model is the best model among all possible subsets.

Definition: Bidirectional stepwise selection is a sequential procedure that begins with a specified current regression model, considers eligible additions from the excluded predictors, and also considers eligible deletions from the included predictors. After a predictor enters or leaves, the procedure reassesses eligibility conditional on the new model. A rule for choosing among eligible moves and a stopping rule must be specified.

The exact procedure depends on the chosen entry and removal rules. For a transparent illustration, this tutorial uses two-sided coefficient tests with a common threshold of \(0.05\): an excluded predictor is eligible to enter when its conditional p-value is below \(0.05\), and an included predictor is eligible for removal when its conditional p-value exceeds \(0.05\). This is one possible convention, not the only one.

All models in an ordinary comparison should use the same response, the same observations, and the same treatment of the intercept. The usual exact coefficient t tests assume a correctly specified linear mean model, independent errors with mean zero and constant variance, normally distributed errors for exact small-sample t inference, and no perfect collinearity among predictors. The hypothetical calculations below are understood to meet these conditions.

Conditional Tests Drive the Search

An addition test asks whether an excluded predictor has a nonzero coefficient after adjustment for the predictors currently in the model. A deletion test asks the corresponding question for an included predictor, conditional on the other included predictors. These are not generally the same questions as testing each predictor’s marginal relationship with the response.

For a useful calculation, let \(r\) be the sample partial correlation between a predictor and the response after controlling for the other predictors in the model. If the test has \(\nu\) residual degrees of freedom, the coefficient t statistic can be written as

t = r\sqrt{\frac{\nu}{1-r^2}}.

For an addition to a model with \(m\) predictors and an intercept, \(\nu=n-m-2\). For deleting one predictor from a model with \(q\) predictors and an intercept, the coefficient test has \(\nu=n-q-1\), since that is the residual degrees of freedom in the full model.

The formula follows by considering the response and candidate predictor after each has been residualized against the predictors being controlled for. In that residualized simple regression, the squared t statistic is the regression sum of squares divided by the residual mean square. The regression sum of squares is proportional to \(r^2\), and the residual sum of squares to \(1-r^2\), giving

t^2 = \nu\frac{r^2}{1-r^2}.

The sign of \(t\) is the sign of \(r\), which yields the stated formula. The p-value is calculated from the t distribution with \(\nu\) degrees of freedom. For a two-sided test it is the probability, under a zero coefficient, of a t statistic at least as far from zero as the observed statistic.

A Typical Bidirectional Update

One common implementation follows a cycle. It adds the strongest eligible excluded predictor, then checks the included predictors for possible removal. If a deletion occurs, it reassesses excluded predictors using the reduced model. It continues to update the model until neither an addition nor a deletion is eligible. “Strongest” must be defined by the chosen rule, such as the smallest eligible p-value; other implementations can order moves differently.

Assess additions.
Test excluded predictors conditional on the current model. Add an eligible predictor according to the specified rule, if one is available.

Reassess included predictors.
After the addition, test each included predictor conditional on the others. Remove an eligible predictor according to the specified rule, if one is available.

Update and repeat.
After any addition or deletion, reassess eligibility for both additions and deletions under the updated model. Stop only when the specified procedure finds no eligible move in either direction.

Worked Example: An Addition Makes an Included Predictor Removable

Suppose a hypothetical data set contains \(n=50\) observations and a response \(Y\) with two candidate predictors, \(A\) and \(B\). The sample correlations are \(r_{YA}=0.50\), \(r_{YB}=0.65\), and \(r_{AB}=0.88\). Imagine the current model contains \(A\), perhaps because it was included in an earlier step or in the specified starting model. Use the \(0.05\) entry and removal threshold described above.

First, consider adding \(B\). Its partial correlation with \(Y\), controlling for \(A\), is

r_{YB\cdot A} =\frac{r_{YB}-r_{YA}r_{AB}} {\sqrt{(1-r_{YA}^2)(1-r_{AB}^2)}} =\frac{0.65-(0.50)(0.88)} {\sqrt{(1-0.50^2)(1-0.88^2)}} =\frac{0.21}{\sqrt{(0.75)(0.2256)}} \approx 0.5105.

The addition test controls for one predictor, so its degrees of freedom are \(50-1-2=47\). The t statistic is

t=0.5105\sqrt{\frac{47}{1-0.5105^2}} \approx 4.070.

The two-sided p-value is approximately \(0.0001784\), below \(0.05\), so \(B\) is eligible to enter. The model now contains both \(A\) and \(B\).

Next, reconsider \(A\). Its partial correlation with \(Y\), now controlling for \(B\), is

r_{YA\cdot B} =\frac{r_{YA}-r_{YB}r_{AB}} {\sqrt{(1-r_{YB}^2)(1-r_{AB}^2)}} =\frac{0.50-(0.65)(0.88)} {\sqrt{(1-0.65^2)(1-0.88^2)}} =\frac{-0.072}{\sqrt{(0.5775)(0.2256)}} \approx -0.1995.

The full model has two predictors, so the coefficient test has \(50-2-1=47\) residual degrees of freedom. Thus

t=-0.1995\sqrt{\frac{47}{1-(-0.1995)^2}} \approx -1.396.

The two-sided p-value is approximately \(0.169\), which exceeds \(0.05\). Under the stated rule, \(A\) is eligible for removal. This is the key change: \(A\)'s usefulness was evaluated again after \(B\) entered, and its conditional evidence is now different. The procedure does not conclude that \(A\) is unrelated to \(Y\) in every sense; it finds that \(A\) does not meet the chosen conditional retention rule in the model containing \(B\).

After deleting \(A\), the model contains \(B\) alone. The search must then reassess possible moves using this updated model rather than treating the earlier test results as permanent.

Worked Example: Rechecking Both Directions After a Deletion

Continue with the same hypothetical correlations. After \(A\) is removed, check whether \(B\) should remain and whether \(A\) can now re-enter. In the one-predictor model containing \(B\), the marginal correlation is \(r_{YB}=0.65\). The corresponding t statistic is

t=0.65\sqrt{\frac{50-2}{1-0.65^2}} =0.65\sqrt{\frac{48}{0.5775}} \approx 5.926.

This test has \(48\) degrees of freedom, and its two-sided p-value is approximately \(3.252\times 10^{-7}\). Therefore, \(B\) does not meet the stated removal condition.

To test whether \(A\) can re-enter, use its partial correlation controlling for \(B\), which was calculated above as approximately \(-0.1995\). This is an addition to a model with one predictor, so \(df=47\):

t=-0.1995\sqrt{\frac{47}{1-(-0.1995)^2}} \approx -1.396, \qquad p\approx 0.169.

Because this p-value exceeds \(0.05\), \(A\) is not eligible to re-enter. At this point the model containing \(B\) has no eligible deletion and no eligible addition among the two candidates, so the procedure stops under the stated rule. Checking both directions matters: a deletion changes the conditioning set for every remaining addition test.

A Different Model Can Produce a Different Path

The next small hypothetical example illustrates the arithmetic for an addition followed by a deletion in a model with more predictors. Suppose \(n=40\), the current model contains \(P\) and \(Q\), and the candidate \(R\) has partial correlation \(0.34\) with the response after controlling for \(P\) and \(Q\). Its addition test has \(40-2-2=36\) degrees of freedom:

Worked Example: Applying the Update Rule With Three Predictors

For the hypothetical regression output, suppose the partial correlation for adding \(R\) is \(0.34\). Then

t=0.34\sqrt{\frac{36}{1-0.34^2}} =0.34\sqrt{\frac{36}{0.8844}} \approx 2.169.

The two-sided p-value with \(36\) degrees of freedom is approximately \(0.037\), so \(R\) is eligible to enter at the \(0.05\) threshold. After it enters, suppose the updated model output gives a partial correlation of \(0.12\) for \(Q\), controlling for \(P\) and \(R\). The test has \(40-3-1=36\) residual degrees of freedom:

t=0.12\sqrt{\frac{36}{1-0.12^2}} =0.12\sqrt{\frac{36}{0.9856}} \approx 0.725, \qquad p\approx 0.473.

Since \(0.473>0.05\), \(Q\) is eligible for removal under the example’s rule. The procedure must then evaluate the reduced model afresh. These values illustrate a sequence of conditional tests; they are not marginal correlations and should not be reused as tests for a different model.

What the Search Does—and Does Not—Establish

Bidirectional stepwise selection is useful when a candidate set is too large for a person to compare every possible model by hand. Its distinctive feature is the repeated reassessment: after an addition, earlier predictors may be deleted; after a deletion, excluded predictors may become eligible under the revised conditioning set. A one-way forward procedure does not remove variables, while a one-way backward procedure does not add them back.

The path is not a guarantee of a globally best subset. Each decision is made from the current model, so a different starting model, candidate set, entry/removal rule, or tie-breaking rule can lead to a different path. The selected model should therefore be understood as the result of a specified search procedure, not as proof that the variables it contains are uniquely important.

Do not treat a predictor’s first p-value as permanent. Its conditional test changes when the other predictors in the model change.
Do not confuse marginal and conditional evidence. A strong relationship considered alone can weaken after adjustment for another predictor.
Do not stop after a deletion without reassessing additions. The reduced model defines new conditional tests.
Do not interpret the final model as an exhaustive optimum. Stepwise selection visits only a particular sequence of models.

A common mistake is to reason, “\(A\) entered the model, so it must stay.” Entry only means that \(A\) met the entry rule at the point it was tested. The correction is to apply the stated removal rule after each relevant change and judge \(A\) conditional on the other predictors currently included.

Key takeaway: Bidirectional stepwise selection is a sequence of conditional decisions. Every addition or deletion changes the model, so eligibility must be reconsidered after each change; the resulting path is a search outcome, not an exhaustive comparison of all subsets.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

In a bidirectional procedure, why can an included predictor become eligible for removal after another predictor enters?
A model contains three predictors and an intercept. For a coefficient test in that full model, what are the residual degrees of freedom in terms of \(n\)?
In the first worked example, why is the test for adding \(B\) conditional on \(A\), rather than based only on \(r_{YB}\)?
After a predictor is deleted, which two types of eligibility should be reassessed?
Why does following a bidirectional stepwise path not guarantee that the final model is the best subset among all candidate models?