Why the Rules Matter
A stepwise procedure does more than specify which predictors are considered. It also needs rules for deciding when a candidate predictor is eligible to enter, when an included predictor should be removed, and when the search should stop. Those choices can change the selected model, even when the response, candidate predictors, and fitting method stay the same.
This tutorial compares three kinds of rules: fixed p-value thresholds, rules requiring a specified improvement in fit, and rules that stop at a chosen model size. These are algorithmic choices. They are not, by themselves, proof that the selected model is the best possible model or that its coefficients have a particular scientific interpretation.
Stepwise Rules and Their Ingredients
Suppose the response is quantitative and there is a fixed set of candidate predictors. A model is a chosen subset of those predictors, together with an intercept. For a given model, least squares chooses the coefficient estimates that minimize the residual sum of squares, abbreviated RSS. A stepwise search moves between models by adding or removing predictors according to a specified rule.
When a rule uses coefficient-test p-values, the usual regression model is assumed to have independent errors with mean zero and constant variance. For the usual exact t tests, the errors are also assumed to be normally distributed. The p-value for a predictor is computed conditional on the other predictors currently in the model. It can therefore change after another predictor is added or removed.
For a p-value entry rule with threshold \(\alpha_{\text{enter}}\), a candidate is eligible when its current conditional p-value is less than \(\alpha_{\text{enter}}\). If more than one candidate is eligible, a common convention is to add the one with the smallest p-value. For a p-value removal rule with threshold \(\alpha_{\text{remove}}\), an included predictor is eligible for removal when its current conditional p-value is greater than \(\alpha_{\text{remove}}\); a common convention is to remove the predictor with the largest p-value. The procedure then refits the model and reassesses the remaining candidates.
The entry and removal thresholds need not be equal. For example, \(\alpha_{\text{enter}}=0.05\) and \(\alpha_{\text{remove}}=0.10\) require stronger evidence for entry than for retention. This gap can help avoid an immediate reversal of a recent decision, but it does not ensure that the algorithm finds a uniquely appropriate model.
Worked Example: Changing the Entry Threshold
Suppose a hypothetical bidirectional search uses the removal threshold \(\alpha_{\text{remove}}=0.10\), and we compare two entry thresholds, 0.05 and 0.10. The following p-values are assumed to come from repeated fits on the same observations. At each addition, the eligible candidate with the smallest p-value is considered first.
With only the intercept in the model, the candidate p-values are A: 0.012, B: 0.038, and C: 0.064. Both entry thresholds admit A, whose p-value is the smallest. The model becomes intercept plus A.
After fitting that model, suppose the p-values are B: 0.057, C: 0.021, and D: 0.14. Candidate C has the smallest p-value, 0.021, so it enters under either rule. In the model containing A and C, suppose their p-values are 0.028 and 0.021. Both are below the removal threshold of 0.10, so neither is removed.
Now suppose the remaining candidates have p-values B: 0.082 and D: 0.14. Under the 0.05 entry rule, neither candidate qualifies, because \(0.082>0.05\) and \(0.14>0.05\). The search stops with A and C. Under the 0.10 entry rule, B qualifies because \(0.082<0.10\), so B enters. If, after refitting, all three included predictors have p-values below 0.10, none is removed by the stated removal rule.
Thus the stricter entry threshold selects A and C, while the more permissive threshold can select A, C, and B. The values are hypothetical; they illustrate that the p-values used at each step are conditional on the current model, rather than fixed characteristics of the predictors.
Fit-Improvement Rules
An alternative is to require a specified improvement in fit, rather than a p-value below a fixed threshold. One simple rule uses the decrease in RSS when a predictor is added. Because RSS has squared response units, an absolute RSS threshold is meaningful only relative to the response scale and the data being analyzed. Other procedures can instead set a minimum percentage decrease or require improvement in a chosen penalized fit criterion.
A rule based on raw RSS needs a threshold or another stopping condition. Without one, adding predictors to a least-squares model cannot increase RSS, so the rule “add any predictor that reduces RSS” tends to keep adding predictors until no candidates remain. A minimum required reduction makes the decision explicit: select the eligible addition with the greatest decrease, but add it only if that decrease meets the specified minimum.
To justify this result, let \(\mathcal{B}_1\) be the set of coefficient choices allowed by the smaller model and \(\mathcal{B}_2\) the choices allowed by the larger model. Setting the added predictor coefficients to zero shows that every choice in \(\mathcal{B}_1\) is also available in \(\mathcal{B}_2\). Therefore \(\mathcal{B}_1\subseteq\mathcal{B}_2\). Minimizing the same RSS function over a set that contains all the original choices cannot give a larger minimum:
This explains why raw in-sample RSS alone does not provide a reason to stop adding predictors. It also explains why comparisons of fit need to specify what counts as enough improvement, or include a stopping rule or penalty.
Worked Example: Requiring a Minimum RSS Reduction
Consider a hypothetical search using the same observations throughout. Start with an intercept-only model whose RSS is 240 squared response units. At the first step, adding A would reduce RSS by 18 units, adding B by 12, and adding C by 7. The rule is to add the candidate with the largest reduction only if that reduction is at least 10.
The largest first-step reduction is 18, from A, and \(18\geq10\). Add A. The new RSS is \(240-18=222\) squared response units.
After refitting with A, suppose the conditional reductions are B: 8, C: 11, and D: 4. Candidate C has the largest reduction, and \(11\geq10\), so add C. The resulting RSS is \(222-11=211\) squared response units.
After refitting with A and C, suppose B would reduce RSS by 9 and D by 3. Neither reduction meets the minimum of 10, so the search stops with A and C. If the rule had instead required a reduction of at least 8, B would qualify because \(9\geq8\), and it would enter. The changed threshold therefore changes the stopping point and the selected model.
These numbers describe reductions conditional on the current model, not improvements that can be added together across every possible order of entry. The example also uses an absolute RSS cutoff: rescaling the response would rescale RSS, so the same numerical cutoff might no longer represent the same practical standard.
Stopping at a Chosen Model Size
A different stopping rule specifies the number of predictors to retain. For example, a forward procedure may repeatedly add the predictor that gives the largest improvement and stop as soon as the model contains \(k\) predictors. This rule controls the size of the selected model directly. It does not require each selected predictor to pass a p-value threshold or a minimum improvement threshold.
A size limit can also serve as a cap within another procedure: stop when either the model has \(k\) predictors or no candidate meets the entry rule, whichever happens first. These are distinct stopping conditions. If the target is exactly \(k\), the search may continue to that size even when the latest addition has a small improvement. If \(k\) is only a cap, the search can stop earlier when no addition is eligible.
Worked Example: Stopping After Two Additions
Suppose a hypothetical forward search uses reductions in RSS to rank additions and is required to stop at exactly two predictors. The intercept-only model has RSS 300 squared response units. At the first step, the reductions are X: 16, Y: 13, and Z: 9. The greatest reduction is 16, so X enters and RSS becomes \(300-16=284\).
After refitting with X, suppose the remaining conditional reductions are Y: 12, Z: 14, and W: 5. Z gives the greatest reduction, so Z enters and RSS becomes \(284-14=270\). The model now has two predictors, so the stated stopping rule ends the search at X and Z.
Suppose that, after fitting X and Z, the largest reduction available from another addition would be 7. A rule requiring at least three predictors could continue, but the chosen-size rule stops at two. Conversely, if the rule had required a minimum reduction of 10 as well as a maximum size of two, the search would also stop at X and Z because both selected reductions, 16 and 14, meet that cutoff. The stopping rule must therefore be read together with the rule that chooses additions.
Stopping at two predictors does not establish that X and Z are the best two-predictor subset among all possible subsets. It says only that this particular search reached two predictors along its specified path.
Comparing the Rules
Each rule answers a different operational question. A p-value rule asks whether a conditional coefficient test meets a threshold. An RSS-improvement rule asks whether a specified amount of in-sample error reduction is achieved. A model-size rule asks whether the search has reached a chosen number of predictors. Their differences can be summarized as follows.
| Rule | Decision it controls | What to keep in mind |
|---|---|---|
| Fixed p-value entry or removal threshold | Whether a predictor is eligible to enter or leave, conditional on the current model | Conditional p-values can change after each refit; the threshold is a search rule, not a guarantee of model quality. |
| Fit-improvement threshold | Whether an addition or other change improves fit enough to continue | Raw RSS is nonincreasing as predictors are added; an improvement standard or another stopping rule is needed. |
| Chosen model size | How many predictors the search will retain before stopping | A fixed size does not show that this path found the best subset of that size. |
The rules may be combined, but their roles should remain clear. For example, a procedure may use p-values to decide whether a candidate is eligible, choose the eligible candidate with the largest RSS reduction, and stop at a maximum model size. Reporting only that “stepwise selection was used” leaves important decisions unspecified. A useful description states the direction of the search, the entry and removal rules, how competing candidates are ranked, and the stopping condition.
Common Mistakes and Practical Checks
- Treating a threshold as a universal law. A rule such as \(p<0.05\) is a chosen procedure setting; another threshold can produce a different path. State the threshold rather than implying it is the only valid choice.
- Assuming a predictor’s p-value stays fixed. The p-value is conditional on the other predictors in the current model. Refit and reassess after each addition or removal.
- Using raw RSS as if it penalized complexity. On the same observations, a larger least-squares model cannot have a larger minimum RSS. A rule based only on “any decrease” has no meaningful minimum-improvement stopping point.
- Confusing a size limit with a fit requirement. “Stop at two predictors” says how many predictors to retain; it does not say that the second one met a particular p-value or improvement standard.
- Calling the endpoint the best model without qualification. The endpoint is the result of the stated search path and rules. It need not be the best subset under a separate model-comparison criterion.
Before interpreting a selected model, verify that the response observations are the same for the models being compared, the entry and removal thresholds are explicit, the handling of ties is stated when relevant, and the stopping rule is unambiguous. A transparent rule makes the path reproducible and helps distinguish an algorithmic decision from a substantive conclusion.
Check Your Understanding
Answer these questions using the rules and distinctions developed above.
- In a bidirectional procedure, what does an entry threshold control, and why might a candidate’s p-value change after another predictor enters?
- A model has RSS 180. Adding one predictor reduces RSS to 174. What is the RSS improvement, and would that addition meet a minimum-reduction rule of 8?
- Why can a rule that adds any predictor that decreases least-squares RSS fail to provide a useful stopping point?
- A forward search is required to stop at exactly three predictors. Does this requirement establish that the final model is the best three-predictor subset? Explain.
- Give one example of how changing a removal threshold could change the final model, even if the starting model is the same.