Why Begin With the Full Model?
Forward selection builds a model by adding predictors. Backward elimination takes the opposite direction: it starts with a model containing every eligible predictor, then considers removing predictors one at a time. This approach is useful when a reasonable starting set is available and fitting the full model is feasible.
The phrase “remove the least useful predictor” is not a complete rule. Use a specified criterion to compare the candidate models formed by deleting one predictor at a time. The criterion determines what “least damaging” means, and a separate stopping rule determines whether to remove that predictor or retain the current model.
Using an AIC Score as the Deletion Rule
For a concrete rule, consider least-squares regression with a quantitative response, an intercept, and \(q\) predictors. Suppose the errors are independent and normally distributed with mean zero and constant variance. For comparing models, they must be fitted to the same response observations. Write \(RSS\) for the residual sum of squares of a fitted model.
For this setting, the Akaike information criterion (AIC) can be compared using the score below. The intercept and error-variance parameters are included in the full AIC calculation, but their penalty terms are the same for every candidate model, so they can be omitted from this comparison score. Here \(q\) counts predictors, not the intercept.
Lower scores are preferred. When considering a deletion, compare the score of the reduced model with that of the current model. Under an AIC deletion rule, make the deletion only if it lowers the score; if every available deletion raises it, stop. AIC balances fit, represented by \(RSS\), against model size, represented by \(2q\). It is a model-selection criterion, not a test of whether a coefficient is statistically significant.
To see the result, subtract the current score from the reduced-model score:
The reduced model cannot fit the observed data better in the least-squares sense, so \(RSS_R\) is at least \(RSS_F\). The question is whether the increase in \(RSS\) is small enough to be offset by the two-point reduction in the complexity penalty. For example, if the RSS rises by a factor of \(1.03\), the change is \(n\log(1.03)-2\); its sign depends on \(n\).
Worked Example: Trace an AIC Backward-Elimination Path
Worked Example: Trace an AIC Backward-Elimination Path
Suppose a hypothetical data set has \(n=40\) observations and three eligible predictors, \(A\), \(B\), and \(C\). All models include an intercept. The following fitted RSS values are hypothetical; they are mutually consistent with centered, orthogonal predictors whose contributions to the reduction in RSS are 4 for \(A\), 10 for \(B\), and 3 for \(C\). Use the AIC score \(S=40\log(RSS/40)+2q\), and stop when no one-predictor deletion lowers the score.
Start with the full model. It has \(q=3\) predictors and \(RSS=100\):
Compare every one-predictor deletion. Refit each reduced model, then calculate its score. The RSS values and scores are:
| Model after deletion | RSS | AIC score calculation | Score |
|---|---|---|---|
| \(BC\) (delete \(A\)) | 104 | \(40\log(104/40)+4\) | 42.22 |
| \(AC\) (delete \(B\)) | 110 | \(40\log(110/40)+4\) | 44.46 |
| \(AB\) (delete \(C\)) | 103 | \(40\log(103/40)+4\) | 41.83 |
Deleting \(C\) gives the lowest candidate score, \(41.83\), which is below the current score, \(42.65\). Therefore the rule removes \(C\), and the new current model is \(AB\). The deletion is favored because the RSS increase from 100 to 103 is small enough to be outweighed by the reduction in the complexity penalty.
Refit and compare again. The current \(AB\) model has \(RSS=103\) and score \(41.83\). Deleting \(A\) leaves \(B\) alone with \(RSS=107\); deleting \(B\) leaves \(A\) alone with \(RSS=113\):
Deleting \(A\), and thus retaining \(B\), is the better of these two candidate deletions because \(41.36<43.54\). But it does not improve on the current score: \(41.36>41.83\). The stopping rule therefore retains \(AB\). The selected model on this search path is \(AB\), not the single-predictor model with the smallest score among these two attempted deletions.
This result depends on the stated rule. If the procedure were instead required to delete a predictor regardless of whether the score rose, it could continue, but that would be a different stopping rule. Always distinguish choosing the best candidate deletion from deciding whether to make that deletion.
Worked Example: Why the Stopping Rule Matters
Worked Example: Why the Stopping Rule Matters
Consider a second hypothetical data set with \(n=40\). The full \(ABC\) model has \(RSS=80\), and the one-deletion models \(BC\), \(AC\), and \(AB\) have RSS values 101, 88, and 92, respectively. Calculate the current score and the three deletion scores.
The full-model score is \(40\log(80/40)+6=40\log(2)+6\approx33.73\). The deletion scores are:
The least damaging deletion is to remove \(B\), giving \(AC\) with score \(35.54\). However, this score still exceeds the current score \(33.73\), as do the other deletion scores. Under the rule used here, backward elimination stops immediately and retains \(ABC\). “Choose the best deletion” does not mean “make a deletion even when every candidate is worse.”
Worked Example: The Score Change Depends on Sample Size
Worked Example: The Score Change Depends on Sample Size
Suppose a full model has two predictors and a candidate reduced model has one. In each of two hypothetical comparisons, the full-model ratio \(RSS/n\) is 2 and the reduced-model ratio is 2.1. Use the AIC score to decide whether deletion is favored in each case.
For \(n=20\), the full-model score is \(20\log(2)+4\approx17.86\). The reduced-model score is \(20\log(2.1)+2\approx16.84\). The reduced score is lower by about \(1.02\), so the deletion is favored.
For \(n=100\), the corresponding scores are \(100\log(2)+4\approx73.31\) for the full model and \(100\log(2.1)+2\approx76.19\) for the reduced model. Now the reduced score is higher by about \(2.88\), so the deletion is not favored. The RSS ratios are the same in both comparisons, but the fit penalty scales with \(n\); therefore the same proportional loss of fit need not lead to the same decision.
What Backward Elimination Can and Cannot Establish
Backward elimination is a greedy search: at each step, it chooses from the deletions available in the current model. It does not compare every possible subset in every possible order. Consequently, its final model is the result of the chosen path and stopping rule; it is not guaranteed to be the best model among all subsets under the criterion.
The eligible predictor set also matters. Backward elimination begins with what it is given, so predictors excluded before the search cannot be selected by it. The initial full model must be estimable: for example, exact linear dependence among predictors can make some coefficients non-identifiable. Keep the observations and response the same when comparing these AIC scores; comparing scores from models fitted to different responses or different observations is not the comparison described here.
The AIC calculations rank models according to a specified criterion. They do not establish that a retained predictor causes a change in the response, nor do they turn the selection path into a collection of ordinary significance tests. Interpret the resulting model with the limits of the selection procedure in mind.
Common Mistakes
- Removing the predictor with the largest one-variable association. Backward elimination compares refitted reduced models under its specified rule; a predictor’s simple association with the response is not the same as its contribution conditional on the other included predictors.
- Deleting the best candidate automatically. The best deletion may still make the criterion worse. Compare its score with the current model and apply the stopping rule.
- Using the old fit after a deletion. Each candidate model must be refitted. Its RSS and score are not obtained by simply carrying the full-model coefficients forward.
- Treating the selected model as the global optimum. A backward path searches only a sequence of candidate models and need not find the lowest-scoring subset overall.
- Calling AIC a significance test. AIC is a model-comparison criterion. A lower score does not by itself show that a coefficient differs from zero or that a predictor has a causal effect.
Check Your Understanding
Answer these questions using the AIC deletion rule described above.
- In backward elimination, what is the starting model, and what models are compared at each step?
- A one-predictor deletion changes \(RSS_F\) to \(RSS_R\). Write the formula for the change in AIC score.
- If every candidate deletion has a higher score than the current model, what does the stated stopping rule require?
- Why must a reduced model be refitted after a predictor is deleted?
- Does a backward-elimination result guarantee the best-scoring model among all possible subsets? Explain briefly.