Why Selection Paths Matter
When a regression has several candidate predictors, the question is not only which model fits best, but also which models a selection procedure actually considers. Best-subset selection searches every predictor combination. Forward and backward stepwise procedures build a path by changing one predictor at a time. Because their candidate sets differ, they can favor different models even when they use the same response, predictors, and fitting method.
The examples below compare these procedures using ordinary least squares and residual sum of squares (RSS). The calculations illustrate candidate generation, not a universal rule for choosing a final model. A separate selection rule is needed to decide where to stop or which candidate to prefer.
For \(p\) predictors, there are \(2^p\) predictor subsets, including the empty subset that gives the intercept-only model. Thus best-subset selection considers all \(2^p\) models, while a forward or backward run follows only one chain through the collection. Each step in a basic stepwise procedure compares the possible single additions or deletions from the current model and chooses the one favored by the stated step rule. If a procedure uses a stopping threshold, it may stop before reaching the full or intercept-only model.
The comparisons here assume the same response, observations, intercept convention, and ordinary least-squares fitting for every candidate. RSS is the sum of squared residuals for a fitted model. These calculations do not require normally distributed errors; distributional assumptions matter when conducting inference, not for computing the least-squares RSS values shown here.
A Useful RSS Calculation
The RSS reduction from adding a predictor depends on what that predictor explains after accounting for predictors already in the model. A large marginal relationship with the response does not guarantee a large additional contribution once other predictors are included.
To justify the formula, after fitting the current model the remaining problem is to fit \(y_{\perp}\) with a multiple \(b x_{\perp}\). The residual sum of squares as a function of \(b\) is \[ \|y_{\perp}-b x_{\perp}\|^2 =\|y_{\perp}\|^2-2b x_{\perp}'y_{\perp}+b^2 x_{\perp}'x_{\perp}. \] This quadratic is minimized at \(b=(x_{\perp}'y_{\perp})/(x_{\perp}'x_{\perp})\). Substituting that value shows that the minimized RSS is smaller by \((x_{\perp}'y_{\perp})^2/(x_{\perp}'x_{\perp})\). This also explains why an addition cannot increase the least-squares RSS: the expanded model can always set the new coefficient to zero and reproduce the old fit.
Worked Example: A Correlated Pair Changes the Best Two-Predictor Model
A hypothetical pricing analysis uses three centered predictors \(A,B,C\) and a centered response \(y\). The intercept-only RSS is \(30\). The predictor cross-products and response cross-products are \[ A'A=B'B=C'C=1,\quad A'B=0.9,\quad A'C=B'C=0, \] \[ A'y=0.2,\quad B'y=-0.2,\quad C'y=0.3. \] Trace best-subset, forward, and backward selection by model size. Assume the procedures continue through every size and ties may be resolved either way.
Best subset, size one. With each predictor scaled to have sum of squares one, its one-predictor RSS reduction is its squared response cross-product. The reductions are \(0.2^2=0.04\) for \(A\), \((-0.2)^2=0.04\) for \(B\), and \(0.3^2=0.09\) for \(C\). The best one-predictor model is therefore \(C\), with RSS \(30-0.09=29.91\).
Best subset, size two. Predictors \(A\) and \(B\) are correlated, so their joint reduction is not just the sum of their individual reductions. Their two-predictor Gram matrix is \[ \begin{pmatrix}1&0.9\\0.9&1\end{pmatrix}. \] Solving the two-predictor least-squares equations, or using the equivalent expression below, gives their joint reduction: \[ \frac{(A'y)^2+(B'y)^2-2(A'B)(A'y)(B'y)}{1-(A'B)^2} = \frac{0.04+0.04-2(0.9)(0.2)(-0.2)}{1-0.9^2} = \frac{0.152}{0.19} =0.80. \] Thus model \(AB\) has RSS \(30-0.80=29.20\). Since \(C\) is orthogonal to both \(A\) and \(B\), model \(AC\) has reduction \(0.04+0.09=0.13\), and model \(BC\) also has reduction \(0.13\). Each has RSS \(29.87\). Best subset therefore selects \(AB\) as its best size-two model.
Forward path. From the intercept-only model, forward selection adds \(C\), since its reduction \(0.09\) exceeds the \(0.04\) reductions from \(A\) and \(B\). From model \(C\), adding either \(A\) or \(B\) reduces RSS by \(0.04\), because each is orthogonal to \(C\). Resolve the tie by adding \(A\). The size-two forward model is \(CA\), with RSS \(29.87\), not the best-subset model \(AB\). Adding \(B\) next yields the full-model RSS \(29.11\).
Backward path. The full-model reduction is \(0.80+0.09=0.89\), so its RSS is \(30-0.89=29.11\). Removing \(C\) gives model \(AB\), whose RSS is \(29.20\); the deletion cost is \(29.20-29.11=0.09\). Removing \(A\) leaves \(BC\), with RSS \(29.87\), a cost of \(0.76\); removing \(B\) likewise costs \(0.76\). Backward selection first removes \(C\), leaving \(AB\), the best size-two subset.
Check. The best-subset size-two RSS, \(29.20\), is lower than the forward size-two RSS, \(29.87\), and the backward path reaches \(AB\). All three procedures reach the full model, whose RSS \(29.11\) is no greater than any smaller model's RSS.
How the Candidate Paths Differ
The example shows the central limitation of a stepwise path: a predictor that is weak on its own may be valuable in combination with another predictor. Here \(A\) and \(B\) have a strong positive correlation, but their response cross-products have opposite signs. Their combination captures a direction that neither predictor captures well alone. Forward selection chooses \(C\) first, and its subsequent size-two candidate is constrained to contain \(C\). It never considers \(AB\) as a size-two candidate. Best subset does consider it, while backward selection reaches it by starting from the full model and deleting \(C\).
This does not mean backward selection always finds the best subset at each size. It means only that its candidate path differs from the forward path and, in this example, passes through the best size-two model. The same issue can arise in reverse: a predictor that appears useful in the full model may be removed early, even though it would have been useful in a different combination. A stepwise procedure does not generally revisit every earlier decision.
The distinction is easiest to keep straight by separating two questions:
- Candidate generation: Which models are fitted by the procedure?
- Model preference: How are the fitted candidates compared, and when does the procedure stop?
Best subset answers the first question by exhaustive search. Forward and backward selection answer it with a greedy sequence of one-variable changes. A stopping rule or model-ranking criterion answers the second question. The procedures can therefore disagree because they search different candidates, because they stop at different points, or both.
Worked Example: Tracing Both Directions From a Supplied RSS Table
For a hypothetical reserve study, suppose ordinary least-squares fits using predictors \(A,B,C\) give the following RSS values. The intercept is included in every model.
| Model | RSS |
|---|---|
| Intercept only | 120 |
| A | 108 |
| B | 111 |
| C | 109 |
| AB | 100 |
| AC | 102 |
| BC | 104 |
| ABC | 96 |
Forward selection. From the intercept-only model, the RSS reductions are \(120-108=12\) for \(A\), \(120-111=9\) for \(B\), and \(120-109=11\) for \(C\). Add \(A\). From \(A\), adding \(B\) gives reduction \(108-100=8\), while adding \(C\) gives \(108-102=6\). Add \(B\), producing \(AB\). Adding the remaining predictor \(C\) gives reduction \(100-96=4\). The forward path is intercept only, \(A\), \(AB\), \(ABC\).
Backward selection. Starting from \(ABC\), deleting \(A\) leaves \(BC\) and increases RSS by \(104-96=8\); deleting \(B\) leaves \(AC\) and costs \(102-96=6\); deleting \(C\) leaves \(AB\) and costs \(100-96=4\). Remove \(C\), the deletion with the smallest RSS cost. From \(AB\), deleting \(A\) leaves \(B\), increasing RSS by \(111-100=11\); deleting \(B\) leaves \(A\), increasing it by \(108-100=8\). Remove \(B\). The backward path is \(ABC\), \(AB\), \(A\), intercept only.
Comparison. Reversing the backward path gives the same sequence as the forward path in this particular table. At size two, both generate \(AB\), which is also the lowest-RSS pair among \(AB\), \(AC\), and \(BC\). This agreement is a feature of these values, not a guarantee of the procedures. In the preceding example, forward and backward selection produced different size-two candidates.
Check. The RSS never increases when moving from a smaller model to a larger nested model: for example, \(120\ge108\ge100\ge96\) along the forward path. In the reverse direction, each deletion increases or leaves unchanged the RSS.
Deletion Costs and a Common Sign Trap
When a predictor is removed, the reduced model is nested within the original model. Its least-squares RSS cannot be smaller, because it has fewer available fitted directions. The deletion cost should therefore be calculated as \[ \text{RSS after deletion}-\text{RSS before deletion}. \] For a valid nested least-squares comparison, this value is nonnegative. It can be zero if the deleted predictor contributes no additional fitted direction.
Worked Example: Correctly Ranking Deletion Candidates
A full model for hypothetical claim counts has predictors \(P,Q,R,S\) and RSS \(40\). Refitting after deletion gives RSS \(41\) without \(P\), \(43\) without \(Q\), \(42\) without \(R\), and \(41\) without \(S\). Identify the first backward-selection deletion, then calculate its RSS cost.
Compare the reduced-model RSS values. Deleting \(P\) costs \(41-40=1\); deleting \(Q\) costs \(43-40=3\); deleting \(R\) costs \(42-40=2\); and deleting \(S\) costs \(41-40=1\). The smallest cost is \(1\), tied between \(P\) and \(S\). Either is a valid first deletion under the stated tie rule.
Avoid the sign reversal. For deletion of \(P\), the subtraction \(40-41=-1\) is the change in the opposite direction, not the RSS increase. Calling \(-1\) the deletion cost and then reporting \(+1\) is internally inconsistent. The correct cost is \(41-40=1\). The same check applies to every deletion: if a purported least-squares RSS increase is negative, first check which model's RSS was subtracted from which.
Check. Both candidate reduced-model RSS values, \(41\), exceed the full-model RSS \(40\), so the two tied deletion costs are positive and equal to \(1\).
Exam-Style Checks
In each problem, distinguish the models a procedure can generate from the rule used to stop or prefer among them. Treat each listed RSS as an ordinary least-squares fit on the same observations with an intercept.
Check Your Understanding
Work through the candidate paths and verify each RSS comparison.
- A regression has predictors \(D,E,F\). The intercept-only RSS is \(50\), and the one-predictor RSS values are \(D:44\), \(E:45\), and \(F:42\). Which predictor does forward selection enter first? (Answer: \(F\), since its RSS reduction is \(50-42=8\), the largest.)
- In a second fit, the one-predictor RSS values are \(D:44\), \(E:45\), and \(F:42\); the two-predictor RSS values are \(DE:39\), \(DF:40\), and \(EF:38\). Which model is best among size-two subsets, and which size-two model does the forward path generate after first entering \(F\)? (Answer: \(EF\) is best, and forward selection generates \(F\) followed by \(EF\).)
- A full model \(GHI\) has RSS \(25\). The reduced RSS values are \(HI:29\), \(GI:27\), and \(GH:28\). Which predictor does backward selection remove first, and what is the RSS deletion cost? (Answer: remove \(H\), leaving \(GI\), at cost \(27-25=2\).)
- For four predictors, best-subset selection is run with all subsets considered. How many models are fitted in the exhaustive search, including the intercept-only model? How many models are on a complete forward path from the intercept-only model through the full model? (Answer: \(2^4=16\) models in the exhaustive search and \(4+1=5\) models on the forward path.)
- A full model has RSS \(62\); deleting \(J\) gives RSS \(64\), deleting \(K\) gives RSS \(63\), and deleting \(L\) gives RSS \(67\). A candidate reports a deletion cost of \(-1\) for removing \(K\). Correct the cost and identify the first deletion. (Answer: the cost for \(K\) is \(63-62=1\), so remove \(K\) first.)