Two Different Decisions in Model Selection
When several regression models are under consideration, two questions must be answered. Which models will be considered? is answered by a search algorithm. How will the considered models be ranked? is answered by a selection criterion. These are separate choices: changing the algorithm can change which models are reached, while changing the criterion can change how the same candidates are ranked.
This distinction matters because a search need not evaluate every possible subset of predictors. Forward selection and backward elimination move through a restricted sequence of models, whereas exhaustive best-subset search evaluates the full candidate set. Even if all three use the same criterion, the restricted searches can miss the model that scores best overall.
A Criterion for Comparing Candidate Models
For a concrete comparison, use the BIC-type score \( \operatorname{BIC}^* \) for Gaussian linear regression. Suppose all candidate models use the same response observations, include an intercept, and differ only in which predictors they contain. Let \(n\) be the common sample size, \(p(M)\) the number of predictors in model \(M\), and \(\operatorname{SSE}(M)\) its residual sum of squares. With independent normal errors of common variance within each candidate model, one convenient score, with terms common to every model omitted, is
The intercept and the error-variance parameter are not included in \(p(M)\); their contributions to the usual BIC are common across these candidate models and are omitted from this comparison score. The likelihood-based comparison requires that the models be fitted to the same response observations on the same response scale. A smaller \( \operatorname{BIC}^* \) is preferred. The score is not a p-value, and a difference between scores is not itself a significance test.
A criterion does not decide which candidate models a search will visit. For example, the BIC* formula can rank every subset in an exhaustive search, or it can rank only the models reached at each step of a forward search. Its definition stays the same; the set being compared changes.
Worked Example: Calculating BIC* Scores
Suppose a hypothetical data set has \(n=40\) observations and three candidate predictors, A, B, and C. The table gives the residual sums of squares for all eight subsets. Calculate the BIC* scores and identify the exhaustive best-subset result.
| Model | \(p(M)\) | \(\operatorname{SSE}(M)\) | \(\operatorname{BIC}^*(M)\) |
|---|---|---|---|
| None | 0 | 100 | 36.652 |
| A | 1 | 95 | 38.289 |
| B | 1 | 96 | 38.708 |
| C | 1 | 80 | 31.415 |
| AB | 2 | 60 | 23.596 |
| AC | 2 | 78 | 34.091 |
| BC | 2 | 77 | 33.575 |
| ABC | 3 | 58 | 25.929 |
For model C, the calculation is
For model AB, it is
The same formula gives 36.652 for the null model, 38.289 for A, 38.708 for B, 34.091 for AC, 33.575 for BC, and 25.929 for ABC. The smallest score in the complete table is 23.596, for AB. Therefore, exhaustive search using BIC* selects AB. The criterion supplies the ranking; exhaustive search supplies the guarantee that every listed subset was compared.
Following Different Searches with the Same Criterion
Now apply the same BIC* scores to forward selection and backward elimination. Assume forward selection starts at the null model, adds at most one predictor per step, and stops if no available addition lowers the score. Assume backward elimination starts at ABC, removes at most one predictor per step, and stops if no available removal lowers the score. These are greedy rules: each next move is chosen from the models one step away from the current model.
Worked Example: Forward and Backward Paths
Use the table above to trace both algorithms. Because the criterion and candidate scores are unchanged, any difference in their outcomes comes from the models each algorithm visits.
Forward selection. From the null model, the one-predictor candidates have scores 38.289 for A, 38.708 for B, and 31.415 for C. C has the smallest score, and its score is below the null model's 36.652, so the first move is to C. From C, the one-step additions are AC and BC. Their scores are 34.091 and 33.575, respectively. Both exceed C's score of 31.415. Since neither available addition improves the criterion, this forward procedure stops at C.
It would be incorrect to compare ABC as a one-step addition to C: reaching ABC from C requires adding two predictors. Forward selection does not evaluate that move in one step. Nor does the search go back and replace C with A after stopping.
Backward elimination. Starting at ABC, the models formed by removing one predictor are BC (33.575), AC (34.091), and AB (23.596). Removing C gives AB, which has the lowest score and improves on ABC's 25.929, so the procedure moves to AB. From AB, removing A gives B (38.708), while removing B gives A (38.289). Both scores exceed 23.596, so the procedure stops at AB.
The three results are therefore: forward selection returns C; backward elimination returns AB; exhaustive search returns AB. The searches use the same criterion, yet forward selection returns a different model because its path does not include AB. In this example backward elimination reaches the exhaustive best model, but there is no general guarantee that it will.
This example separates a search's path from a criterion's ranking. Forward selection does not reject AB on the basis of its BIC*: it never evaluates AB. Exhaustive search finds AB because it evaluates AB directly. The forward result is best only among the models that its stopping rule has reached and compared.
State which predictors and models are eligible for comparison, along with any restrictions that apply.
Decide whether models will be evaluated by an exhaustive search or by a restricted path such as forward selection or backward elimination.
Specify how candidate models are scored and when the search ends. Report the algorithm and criterion together so the selected model has an unambiguous meaning.
Why the Criterion Can Change a Search Move
An algorithm determines which move is available next, but the criterion determines whether that move is worthwhile under the chosen rule. For a nested pair of models fitted to the same \(n\) observations, suppose the larger model adds one predictor, changing the residual sum of squares from \(SSE_{\text{old}}\) to \(SSE_{\text{new}}\). The change in BIC* is
The addition improves BIC* precisely when this difference is negative. Dividing by \(n\) and exponentiating gives an equivalent condition:
Thus, the improvement in fit, represented by the reduced SSE, must be large enough to offset the criterion's penalty for an additional predictor. This condition concerns a particular nested model comparison; it does not ensure that a greedy search will find the best model in the full candidate set.
Worked Example: One Addition, Two Criteria
Consider a hypothetical comparison of two nested models fitted to the same 40 observations. The smaller model has \(\operatorname{SSE}=60\) and two predictors; the larger model adds one predictor and has \(\operatorname{SSE}=55\). Compare the change in BIC* with the change in AIC*, using the corresponding scores \(n\ln(\operatorname{SSE}/n)+p\ln(n)\) and \(n\ln(\operatorname{SSE}/n)+2p\).
For BIC*, the larger-minus-smaller score difference is
The difference is positive, so BIC* prefers the smaller model. For AIC*, the corresponding difference is
This difference is negative, so AIC* prefers the larger model. Both criteria use the same reduction in SSE, but their penalties differ; they therefore rank this pair differently. If a forward algorithm is at the smaller model and considers this addition, the choice to move depends on the criterion. The algorithm makes the addition available, and the criterion judges its score.
What a Selected Model Does—and Does Not—Tell You
A model returned by a search is the outcome of the specified search procedure, not automatically the globally best model under the criterion. The phrase “best model” should therefore be qualified: best among the models evaluated, best along a particular path, or best among every model in the candidate set are different claims.
The distinction also makes results easier to reproduce. A report that names BIC* but omits the search algorithm leaves an important detail unspecified: exhaustive search and forward selection can use the same score yet return different models. Conversely, naming forward selection without giving its criterion and stopping rule does not fully describe how it chose among available moves.
The comparison is meaningful only when the criterion is being applied to eligible, comparable models. In the BIC* examples, all models use the same response observations and response scale. The score should not be used to rank models fitted to different response scales or different sets of observations as though the formula made those comparisons automatically comparable.
Common Mistakes
- Treating the criterion as the search. Saying “BIC selected C” can conceal whether BIC was applied to every subset or only to models visited by forward selection. Name both the algorithm and the criterion.
- Calling a path result the exhaustive best. Forward selection stops at C in the worked table, but AB has a smaller BIC* score. C is the forward result, not the exhaustive best.
- Assuming every algorithm must agree. A shared criterion does not make the search paths identical. Each algorithm can evaluate a different set of models.
- Confusing a lower score with statistical significance. A lower BIC* is preferred under that comparison rule; it does not establish a significant effect, prove a model true, or show that predictors cause changes in the response.
- Ignoring the stopping rule. A search that stops when no one-step move improves the score may stop even though a better model exists farther away in the candidate set.
Check Your Understanding
Answer each question using the definitions and examples above.
- In one sentence, distinguish a search algorithm from a selection criterion.
- In the BIC* table, why does forward selection stop at C even though AB has the lowest score overall?
- Which models does backward elimination compare immediately after starting at ABC, and which does it choose?
- For a single-predictor addition, what does a negative change in BIC* mean for the criterion's ranking?
- In the final worked example, why does AIC* prefer the larger model while BIC* prefers the smaller one?