Two Different Questions for a Search Procedure
A forward search can add predictors one at a time, but it still needs a rule for when to stop. One rule answers, “How many predictors should the final model contain?” It specifies a target size and keeps adding the highest-ranked available predictor until that size is reached. Another answers, “Is the best available addition strong enough?” It specifies a score threshold and stops when no candidate reaches it. These rules can select different models, even when they use the same search path and score.
This distinction matters because a target size does not check whether the last added predictor has a worthwhile score. Conversely, a threshold-based rule can stop before reaching a desired size. Neither rule, by itself, guarantees that the selected model is best for prediction. The rule determines where the search stops; it does not remove the limitations of a greedy search.
A Conditional F Score for One-Predictor Additions
Use the following score to make the comparison concrete. Suppose there are \(n\) observations and the current linear regression model has an intercept and \(p\) predictors. Let \(\operatorname{RSS}_{\text{current}}\) be its residual sum of squares. For an eligible predictor \(X_j\), refit the model with \(X_j\) added, and call the resulting residual sum of squares \(\operatorname{RSS}_{+j}\). The score for adding \(X_j\) is the partial F statistic for comparing these two nested models.
The numerator measures the reduction in residual sum of squares from adding one predictor. The denominator estimates the error variance using the augmented model, which has \(p+1\) predictors and an intercept, so its residual degrees of freedom are \(n-(p+1)-1=n-p-2\). The formula requires the augmented model to have full column rank and positive residual degrees of freedom.
The usual exact F reference distribution has additional assumptions: the regression errors are independent normal random variables with mean zero and a common variance, and the predictors are treated as fixed (or the calculation is conditional on them). Under the null hypothesis that the added predictor’s coefficient is zero given the current model, \(F_j\) has an \(F(1,n-p-2)\) distribution. In forward selection, however, the candidate is chosen after comparing several scores. The nominal single-candidate reference distribution does not, by itself, account for that search.
Why the Partial F Formula Has This Form
Under the null hypothesis, adding one predictor does not improve the population regression mean once the current predictors are included. With normal errors and full-rank nested models, the reduction in RSS divided by the error variance has a chi-square distribution with one degree of freedom. The augmented model’s RSS divided by the error variance has a chi-square distribution with \(n-p-2\) degrees of freedom. For nested normal linear models, these quantities are independent under the null. Dividing each chi-square quantity by its degrees of freedom gives the ratio of mean squares, which has an F distribution:
This also explains why the score is conditional: the RSS reduction is measured after accounting for the predictors already in the model. A predictor’s marginal association with the response is not necessarily its incremental contribution after other predictors have entered.
Worked Examples: Tracing the Stopping Rules
Worked Example: A Fixed Target Can Require a Weak Addition
A forward search currently contains one predictor. There are \(n=20\) observations, and the current model has \(\operatorname{RSS}_{\text{current}}=100\). The candidate one-predictor additions give RSS values of 90 for \(X_2\) and 95 for \(X_3\). Compare a rule that targets three total predictors with a score-threshold rule using cutoff \(F=4.0\). If the fixed-size rule has a tie, it chooses either tied candidate.
Method. Calculate each partial F score using the residual degrees of freedom for the augmented model. Then follow each stopping rule separately.
First step. The current model has \(p=1\), so either augmented model has \(20-1-2=17\) residual degrees of freedom.
For \(X_2\), the RSS reduction is \(100-90=10\), giving \(F_2=10/(90/17)=170/90=1.8889\) to four decimals.
For \(X_3\), the reduction is \(100-95=5\), giving \(F_3=5/(95/17)=85/95=0.8947\) to four decimals.
The highest score is \(1.8889\), for \(X_2\). The fixed-size rule adds \(X_2\), because it must reach three total predictors. The threshold rule stops immediately because \(1.8889<4.0\); it selects no additional predictor.
Continue the fixed-size rule. Its current model now contains two predictors and has RSS 90. Adding the remaining candidate \(X_3\) gives RSS 88. The augmented model has \(20-2-2=16\) residual degrees of freedom, so \(F_3=(90-88)/(88/16)=2/(88/16)=32/88=0.3636\) to four decimals.
The fixed-size rule adds \(X_3\) anyway and reaches three predictors. The threshold rule’s selected model remains at one predictor. Check: each addition reduces RSS (100 to 90 to 88), but the scores are both below the cutoff of 4.0.
Common mistake avoided: Do not treat “the next predictor has the largest score” as sufficient reason to add it under a threshold rule. The largest score can still fall short of the cutoff.
Worked Example: The Search Reorders Candidates After an Addition
An intercept-only model has RSS 120 for \(n=30\) observations. Adding \(X_1\), \(X_2\), or \(X_3\) individually gives RSS values 100, 108, and 110, respectively. After \(X_1\) is added, the RSS values for models that also include \(X_2\) or \(X_3\) are 96 and 94. Compare a target of two predictors with a threshold rule using cutoff \(F=4.0\).
Method. At each step, compare the candidates conditional on the current model. Do not assume their initial ranking remains valid after an addition.
First step. With \(p=0\), the residual degrees of freedom after an addition are \(30-0-2=28\). The scores are:
\(F_1=(120-100)/(100/28)=20/(100/28)=5.6000\); \(F_2=(120-108)/(108/28)=12/(108/28)=3.1111\); and \(F_3=(120-110)/(110/28)=10/(110/28)=2.5455\).
The best candidate is \(X_1\). Both rules add it: the fixed-size rule needs another predictor, and \(5.6000\geq4.0\) meets the threshold.
Second step. Now \(p=1\), current RSS is 100, and an augmented model has \(30-1-2=27\) residual degrees of freedom. The conditional scores are \(F_2=(100-96)/(96/27)=4/(96/27)=108/96=1.1250\) and \(F_3=(100-94)/(94/27)=6/(94/27)=162/94=1.7234\).
A target of two predictors adds \(X_3\), the higher-scoring candidate, and stops. The threshold rule stops without adding either candidate because the maximum score, \(1.7234\), is below 4.0. Check: the initial score for \(X_3\) was \(2.5455\), but its score conditional on \(X_1\) is \(1.7234\); the conditional score, not the initial one, governs the next step.
Common mistake avoided: Do not keep using scores calculated against the intercept-only model after a predictor has entered. Refit each candidate addition against the current model.
Worked Example: A Score Just Below the Cutoff Still Stops the Search
A current model has one predictor, \(n=25\), and RSS 80. Adding \(X_2\) gives RSS 76; adding \(X_3\) gives RSS 70. If \(X_3\) is added, the RSS for the model with \(X_3\) and \(X_4\) is 64. Compare a fixed target of three total predictors with a threshold rule using cutoff \(F=2.0\).
First step. Here \(p=1\), so the augmented models have \(25-1-2=22\) residual degrees of freedom. The scores are \(F_2=(80-76)/(76/22)=88/76=1.1579\) and \(F_3=(80-70)/(70/22)=220/70=3.1429\). Both rules add \(X_3\), since its score meets the cutoff and is the largest.
Second step. The current model now has \(p=2\), RSS 70, and the augmented models have \(25-2-2=21\) residual degrees of freedom. For \(X_2\), the score is \(F_2=(70-68)/(68/21)=42/68=0.6176\). For \(X_4\), it is \(F_4=(70-64)/(64/21)=126/64=1.96875\).
Because \(1.96875<2.0\), the threshold rule stops. The fixed-size rule adds \(X_4\) to reach three total predictors. The decision does not change by rounding the score: its unrounded value is below 2.0. Check: RSS falls from 80 to 70 to 64 along the fixed-size path, but a lower RSS alone does not establish that the last addition meets the chosen threshold.
Common mistake avoided: Do not round a score near the cutoff before applying the stopping rule. Compare the unrounded score with the cutoff, then round only when reporting it.
Interpreting and Applying the Rules
A threshold must be defined on the scale of the score being used. With a partial F score, a rule such as “add the best candidate if \(F\geq4.0\)” is a clear algorithmic cutoff. A rule based on a significance level instead uses a critical value that depends on the numerator and denominator degrees of freedom. Those are different specifications; a single fixed F cutoff is not automatically the same as a fixed significance level at every step.
A raw reduction in RSS is not interchangeable with a threshold on a partial F score. The F score scales the reduction by the augmented model’s estimated error variance and uses the relevant residual degrees of freedom. Also, the score used to rank candidates and the rule used to stop are logically separate: ranking identifies the best available addition, while the threshold decides whether that best addition is acceptable to the search rule.
- A fixed-size rule is useful when a specified model dimension is required for a comparison, but it may add a predictor with a very small conditional score.
- A threshold rule can stop early, but its outcome depends on the score definition and cutoff.
- Conditional scores can change after each addition because the current model changes.
- A greedy stopping rule examines available additions along its search path; it does not establish that a different subset or a different search path could not perform better.
The partial F reference distribution describes a test for a specified candidate under stated regression assumptions. When the same data are used to screen many candidates and select the largest score, the nominal single-test interpretation requires caution. Treat the cutoff as a selection rule unless the inferential procedure accounts for the selection process. A model selected by either rule should still be assessed for its intended use; stopping at a threshold is not proof of good out-of-sample prediction.
Check Your Understanding
For each problem, apply the stated rule to the conditional scores. Use unrounded values when comparing with a cutoff.
- A current model has \(n=20\), \(p=1\), and RSS 80. Candidate additions give RSS 72 and 74. What is the largest partial F score, and does a threshold of 2.0 add a predictor? (Answer: The largest score is \(8/(72/17)=1.8889\); stop because it is below 2.0.)
- An intercept-only model has \(n=16\) and RSS 60. Candidate additions give RSS 50 and 54. Which candidate is added by a threshold rule with cutoff 2.5? (Answer: Add the candidate with RSS 50; its score is \(10/(50/14)=2.8000\), while the other score is \(6/(54/14)=1.5556\).)
- A current model has \(n=25\), \(p=2\), and RSS 90. Candidate additions give RSS 84 and 80. What does a cutoff of 3.0 do, and what does a fixed-size rule targeting three total predictors do? (Answer: The best score is \(10/(80/21)=2.6250\), so the threshold rule stops; the fixed-size rule adds the candidate with RSS 80.)
- A current model has \(n=30\), \(p=1\), and RSS 120. Two additions give RSS 110 and 108. Which candidate is added by a rule that includes the best candidate when \(F\geq3.0\)? (Answer: Add the candidate with RSS 108; its score is \(12/(108/27)=3.0000\), which meets the cutoff.)
- An intercept-only model has \(n=22\) and RSS 100. Candidate additions give RSS 80 and 85. After the candidate with RSS 80 enters, the other candidate gives RSS 76 and a third candidate gives RSS 74. Under a cutoff of 4.0, how many predictors are added? (Answer: One. The first score is \(20/(80/20)=5.0000\); after that addition, the two scores are \(4/(76/19)=1.0000\) and \(6/(74/19)=1.5405\), so the search stops.)