Why Forward Selection Needs a Stopping Rule
Forward selection starts with an intercept-only model and adds predictors one at a time. At each step, the algorithm compares the eligible additions to the current model and chooses the best-scoring candidate. Because adding a predictor cannot increase training residual sum of squares (RSS), a decrease in RSS by itself is not a useful stopping rule: even a very small decrease is still a decrease.
A stopping rule supplies a separate decision. One approach sets a target model size in advance, such as three predictors, and continues until that size is reached. Another approach requires each addition to meet an entry criterion, such as a partial F score above a specified threshold. These approaches answer different questions: the first asks for the best sequence of additions up to a chosen size, while the second asks whether the best available addition is sufficiently strong under the chosen rule.
Scoring a Candidate Addition
For a numerical comparison, consider the partial F score for adding a candidate predictor to the current model. Let \(RSS_C\) be the current model’s residual sum of squares and \(RSS_E\) the RSS after adding the candidate. For a single added predictor, let \(n\) be the number of observations and \(k\) the number of predictors already in the current model. Both models include an intercept.
The denominator degrees of freedom are \(n-k-2\): the expanded model estimates an intercept, \(k\) existing coefficients, and one new coefficient. The expanded model must have positive residual degrees of freedom for this calculation. A candidate with a larger partial F score is preferred at that step. If the entry rule is \(F \geq c\), the candidate enters only when the highest score reaches \(c\). Equivalently, a rule may require the candidate’s partial-test p-value to be at most a chosen significance level; the corresponding F cutoff depends on the test’s degrees of freedom.
The usual partial F test uses the classical linear model assumptions: the conditional mean is linear in the included predictors, errors are independent with mean zero and constant variance, and errors are normally distributed for the exact finite-sample F reference distribution. The calculation can still be used as a selection score in other settings, but its classical test interpretation and p-value calibration may not apply.
Why the Partial F Formula Works
The current and expanded models are nested: the current model is recovered by setting the new coefficient to zero. The reduction \(RSS_C-RSS_E\) measures the extra variation explained by the added coefficient. Dividing by the one added degree of freedom gives the explained sum of squares per added parameter. The expanded model’s residual mean square, \(RSS_E/(n-k-2)\), estimates the error variance under the classical model.
Under the null hypothesis that the added coefficient is zero, each quantity, after scaling by the error variance, has the appropriate chi-square distribution. The added-model quantity has one degree of freedom, and the residual quantity has \(n-k-2\) degrees of freedom. Their ratio is therefore compared with an F distribution under the stated assumptions. This gives the formula above and explains why the score assesses the candidate’s improvement relative to the remaining residual variation, not merely its raw RSS decrease.
Worked Example: Entry Rule Stops Before the Preset Size
Worked Example: Entry Rule Stops Before the Preset Size
A hypothetical pricing analysis has \(n=24\) observations and an intercept-only RSS of 100. Three predictors, A, B, and C, are eligible. At each step, choose the largest partial F score, and enter a predictor only if \(F \geq 4\). Compare this result with a search required to add two predictors.
Step 1: score the initial candidates. The current model has \(k=0\), so each expanded model has \(24-0-2=22\) residual degrees of freedom. The candidate RSS values are A: 80, B: 85, and C: 90.
For A, \(F=(100-80)/(80/22)=20/(40/11)=5.500\). For B, \(F=(100-85)/(85/22)=330/85=3.882\). For C, \(F=(100-90)/(90/22)=220/90=2.444\). A has the highest score and meets the threshold, so A enters.
Step 2: rescore the remaining candidates. The current model now contains one predictor, A, and has RSS 80. Each expanded model has \(24-1-2=21\) residual degrees of freedom. Suppose adding B gives RSS 70, while adding C gives RSS 68.
For B, \(F=(80-70)/(70/21)=10/(10/3)=3.000\). For C, \(F=(80-68)/(68/21)=252/68=3.706\). C is the better addition, but its score is below 4. The entry-rule search stops with A alone.
A preset-size search required to add two predictors instead adds C as the second predictor, because C is the best remaining candidate, even though it fails the entry threshold. The stopping decisions differ: the entry rule returns one predictor; the preset-size rule returns two.
Check. At the second step, the entry rule stops because the largest score is \(3.706<4\). Under the preset-size rule, the final model contains A and C. The common mistake avoided here is to assume that “best remaining candidate” automatically means “eligible to enter.”
Preset Size, Entry Rule, and a Maximum-Size Cap
A preset size does not test whether the final addition passes a statistical threshold. An entry rule does not promise to reach a particular size. These distinctions matter when reporting a selected model: “two predictors were requested” is not the same decision as “each entered predictor met the stated criterion.”
A practical procedure may combine both ideas. For example, it may add a predictor only when \(F \geq c\), but stop in any case once a maximum of \(K\) predictors has entered. The entry rule can stop the process before \(K\); the size cap can stop it even when another candidate would pass the entry rule. State both conditions when describing such a procedure.
Worked Example: A Maximum Size Is a Cap, Not a Target
A hypothetical claims study has \(n=30\) observations and intercept-only RSS 120. Use the rule “enter the best candidate if \(F \geq 3.5\), but add no more than two predictors.” The initial candidate RSS values are A: 90, B: 96, and C: 100.
Step 1. With \(k=0\), the residual degrees of freedom for each expanded model are \(30-0-2=28\). The scores are A: \((120-90)/(90/28)=840/90=9.333\); B: \((120-96)/(96/28)=672/96=7.000\); and C: \((120-100)/(100/28)=560/100=5.600\). A has the largest score and exceeds 3.5, so A enters.
Step 2. The current model has \(k=1\), RSS 90, and candidate-expanded residual degrees of freedom \(30-1-2=27\). Suppose adding B gives RSS 82 and adding C gives RSS 78. B’s score is \((90-82)/(82/27)=216/82=2.634\). C’s score is \((90-78)/(78/27)=324/78=4.154\). C is the best candidate and exceeds 3.5, so C enters.
The maximum size of two is now reached, so the procedure stops with A and C. The cap is not what caused the second predictor to enter: C met the entry rule as well. If instead the best second-step score had been below 3.5, the entry rule would have stopped the search with only A, notwithstanding the cap of two.
Check. The initial scores use 28 residual degrees of freedom, while the second-step scores use 27. Reusing the initial degrees of freedom at the second step would be incorrect.
Candidate Scores Must Be Recomputed
Each score is conditional on the predictors already in the model. A predictor can look useful when assessed alone but add little once another predictor has entered. Therefore, do not rank all predictors once using their initial scores and then simply take the first few. Refit each eligible expanded model after every accepted addition and calculate the candidate scores relative to the new current model.
Worked Example: An Initial Score Does Not Guarantee Later Entry
A hypothetical risk study has \(n=25\) observations, intercept-only RSS 100, and an entry rule \(F \geq 2\). Initially, adding A gives RSS 80, while adding B gives RSS 82 and adding C gives RSS 88.
Step 1. The initial residual degrees of freedom are \(25-0-2=23\). A’s score is \((100-80)/(80/23)=460/80=5.750\). B’s score is \((100-82)/(82/23)=414/82=5.049\). C’s score is \((100-88)/(88/23)=276/88=3.136\). A has the highest score and passes the threshold, so A enters.
Step 2. After A enters, the current RSS is 80 and \(k=1\). The residual degrees of freedom for each expanded model are \(25-1-2=22\). Suppose adding B now gives RSS 77 and adding C gives RSS 74. B’s new score is \((80-77)/(77/22)=66/77=0.857\). C’s new score is \((80-74)/(74/22)=132/74=1.784\). Neither reaches 2, so the entry-rule search stops with A.
A preset-size search for two predictors would add C, the best remaining candidate, despite its second-step score being below the entry threshold. The score for C at step 1, \(3.136\), cannot be reused at step 2: it was calculated against the intercept-only model, not against the model containing A.
Common Mistakes and Interpretation
- Treating every RSS decrease as sufficient. A candidate may reduce RSS and still fail the entry rule. Compare its score with the specified cutoff.
- Using the wrong reference model. At a later step, calculate \(RSS_C-RSS_E\) relative to the current selected model, not relative to the intercept-only model.
- Forgetting that the degrees of freedom change. For a single added predictor, the denominator degrees of freedom are \(n-k-2\), where \(k\) is the number already in the model.
- Confusing a target size with a cap. A preset target continues to that size when candidates are available; a maximum-size cap merely prohibits further additions after that size.
- Interpreting a selection p-value as an ordinary final-model test. Repeatedly screening candidates affects the interpretation of the selected model’s p-values. An entry rule describes the selection procedure; it does not by itself establish that the final model has the usual unadjusted inferential properties.
Check Your Understanding
For each problem, use the stated partial F entry rule and recalculate scores against the current model after each accepted addition.
- An intercept-only model has \(n=18\) and RSS 60. Candidate A gives RSS 48 and candidate B gives RSS 54. Calculate both initial scores and apply \(F \geq 3.5\). Which predictor enters? (Answer: A enters; \(F_A=4.000\), \(F_B=1.778\).)
- A current model has \(n=20\), \(k=1\), and RSS 50. Adding X gives RSS 44 and adding Y gives RSS 40. Which candidate enters under \(F \geq 4\), and what is its score? (Answer: Y enters; \(F_Y=4.250\), compared with \(F_X=2.318\).)
- An intercept-only model has \(n=22\) and RSS 90. Initially, adding A gives RSS 72 and adding B gives RSS 75. After A enters, adding B gives RSS 70. Apply \(F \geq 3\). Does B enter at the second step? (Answer: No; initially \(F_A=5.000\) and \(F_B=4.000\), so A enters; then \(F_B=(72-70)/(70/19)=0.543\), and selection stops with A.)
- An intercept-only model has \(n=26\) and RSS 100. Adding A gives RSS 80 and adding B gives RSS 84. After A enters, the best remaining candidate C would give RSS 74. Use \(F \geq 2\) and a maximum size of two. Does the procedure reach size two? (Answer: No; initially \(F_A=6.000\) and \(F_B=4.571\), so A enters; then \(F_C=(80-74)/(74/23)=1.865<2\), so the entry rule stops the search with one predictor.)