Forward Selection

Building a Model One Predictor at a Time

When many predictors are available, searching every possible subset can require fitting a very large number of models. Forward selection offers a more limited search: begin with an intercept-only model, then consider adding one predictor at a time. At each step, compare the eligible additions using a chosen model-selection criterion and take the addition that gives the best result. The resulting sequence of models is called the forward-selection path.

The procedure is a search algorithm, not a definition of which model is scientifically correct. Its path depends on the candidate predictors and on the criterion being used. To make its choices meaningful, specify those before carrying out the search.

Definition: In forward selection, start with a model containing an intercept and no candidate predictors. At each step, fit a model for each eligible predictor added individually to the current model. Add the predictor whose addition gives the best value of the chosen criterion, provided the criterion improves on the current model. Stop when no eligible addition improves it, or when a separately specified stopping condition is met.

Model Setup and AIC Comparisons

Suppose there are $n$ observations on a quantitative response $Y$ and candidate predictors $X_1,\ldots,X_p$. For a particular selected set $S$, a least-squares linear regression model has the form

Y_i=\beta_0+\sum_{j\in S}\beta_jX_{ij}+\varepsilon_i,\qquad i=1,\ldots,n.

For the AIC calculations below, assume that the response is modeled with independent normal errors having mean zero and common variance $\sigma^2$, and that the conditional mean is represented by the stated linear model. The candidate models must use the same response and observations so that their likelihood-based AIC values can be compared directly. For simplicity, each candidate predictor in the examples is a single numerical variable and adds one regression coefficient.

AIC is a model-selection criterion defined by $\mathrm{AIC}=-2\ell(\hat{\theta})+2k$, where $\ell(\hat{\theta})$ is the maximized log-likelihood and $k$ is the number of estimated parameters. Smaller AIC is preferred. In Gaussian linear regression, $k$ includes the intercept, the slopes in the model, and the error variance. Since all models in a comparison have an intercept and an estimated error variance, adding one predictor increases $k$ by one.

Result: For two Gaussian linear regression models fit to the same $n$ observations, if the larger model adds one predictor, then $$ \mathrm{AIC}_{\mathrm{larger}}-\mathrm{AIC}_{\mathrm{smaller}} =n\ln\!\left(\frac{\mathrm{RSS}_{\mathrm{larger}}}{\mathrm{RSS}_{\mathrm{smaller}}}\right)+2. $$ A negative difference means the addition lowers AIC; a positive difference means it raises AIC.

Here $\mathrm{RSS}$ is the residual sum of squares. The result follows from the Gaussian likelihood. At its maximum, the estimated error variance is $\widehat{\sigma}^2=\mathrm{RSS}/n$, and the maximized log-likelihood is

\ell(\hat{\theta})=-\frac{n}{2}\left[\ln(2\pi)+1+\ln\!\left(\frac{\mathrm{RSS}}{n}\right)\right].

Thus AIC for a model with $k$ estimated parameters is $n\ln(\mathrm{RSS}/n)+n[\ln(2\pi)+1]+2k$. Subtract the smaller-model AIC from the larger-model AIC. The terms $n[\ln(2\pi)+1]$ cancel, as does the shared $n$ inside the logarithms, leaving $n\ln(\mathrm{RSS}_{\mathrm{larger}}/\mathrm{RSS}_{\mathrm{smaller}})+2(k_{\mathrm{larger}}-k_{\mathrm{smaller}})$. One added predictor gives $k_{\mathrm{larger}}-k_{\mathrm{smaller}}=1$, yielding the stated formula.

Worked Example: Choosing the First Predictor

Suppose a hypothetical data set contains $n=30$ observations. The intercept-only model has $\mathrm{RSS}=240$. Two eligible one-coefficient predictors, humidity and temperature, give residual sums of squares 210 and 225 when added separately. Which should enter first under AIC?

For humidity, the AIC change relative to the intercept-only model is

30\ln(210/240)+2 =30\ln(0.875)+2 \approx -2.006.

For temperature, it is

30\ln(225/240)+2 =30\ln(0.9375)+2 \approx 0.063844.

Humidity gives the smaller AIC change, and its change is negative, so it is added. Temperature alone would slightly raise AIC relative to the starting model. These are hypothetical regression results; they show how to compare additions, not that either predictor has a causal effect on the response.

Tracing a Forward-Selection Path

At a given step, the comparison is conditional on the predictors already selected. A predictor that performs poorly when added to the intercept-only model may improve the fit after another predictor has entered, and the reverse can also occur. The procedure therefore repeats the candidate comparisons after every addition rather than relying only on the first set of comparisons.

Worked Example: A Complete AIC Path

Consider a hypothetical response measured on $n=20$ observations and three eligible one-coefficient predictors, $X_1$, $X_2$, and $X_3$. The intercept-only model has $\mathrm{RSS}=120$. The following residual sums of squares come from fitting each candidate addition at the indicated step. Use AIC and stop when no remaining addition lowers AIC.

Step 0: Start with the intercept. The current model has no predictors and $\mathrm{RSS}=120$. Fit each eligible one-predictor model. Adding $X_1$ gives $\mathrm{RSS}=90$, adding $X_2$ gives 100, and adding $X_3$ gives 110. Their AIC changes from the current model are

\begin{aligned} \Delta\mathrm{AIC}(X_1)&=20\ln(90/120)+2 =20\ln(0.75)+2\approx -3.754,\\ \Delta\mathrm{AIC}(X_2)&=20\ln(100/120)+2 \approx -1.646,\\ \Delta\mathrm{AIC}(X_3)&=20\ln(110/120)+2 \approx 0.2598. \end{aligned}

The smallest change is for $X_1$, and it lowers AIC, so the first selected model contains the intercept and $X_1$. In particular, $20\ln(0.75)\approx-5.754$; adding the penalty of 2 gives an AIC change of about $-3.754$.

Step 1: Consider the remaining predictors given $X_1$. The current model has $\mathrm{RSS}=90$. Adding $X_2$ gives $\mathrm{RSS}=70$, while adding $X_3$ gives $\mathrm{RSS}=78$. The changes from the current model are

\begin{aligned} \Delta\mathrm{AIC}(X_2\mid X_1)&=20\ln(70/90)+2\approx -3.026,\\ \Delta\mathrm{AIC}(X_3\mid X_1)&=20\ln(78/90)+2\approx -0.8620. \end{aligned}

Both additions would lower AIC, but $X_2$ gives the greater improvement (the more negative change). Add $X_2$. The current model is now the intercept plus $X_1$ and $X_2$, with $\mathrm{RSS}=70$.

Step 2: Consider $X_3$ given $X_1$ and $X_2$. Adding $X_3$ gives $\mathrm{RSS}=68$. Its change in AIC is

\Delta\mathrm{AIC}(X_3\mid X_1,X_2) =20\ln(68/70)+2 \approx 1.420.

The positive change means adding $X_3$ would raise AIC, so the stated stopping rule leaves it out. The selected path is intercept only, then intercept plus $X_1$, then intercept plus $X_1$ and $X_2$. The final model is the best AIC model encountered along this particular path under this stopping rule; this does not establish that it is best among every possible subset.

The Criterion Affects the Path

Forward selection needs a chosen criterion: the search rule says how to move through candidate models, while the criterion says how to rank those models. AIC is not the only available choice. For example, BIC uses a larger complexity penalty for an added one-coefficient predictor when $n>e^2$: its change is $n\ln(\mathrm{RSS}_{\mathrm{larger}}/\mathrm{RSS}_{\mathrm{smaller}})+\ln(n)$. The two criteria can therefore prefer different actions at the same step.

Worked Example: AIC and BIC Can Disagree

Suppose a hypothetical current model has $\mathrm{RSS}=100$ on $n=20$ observations, and one eligible predictor reduces the RSS to 88. The AIC change for adding it is

20\ln(88/100)+2 =20\ln(0.88)+2 \approx -0.5567.

AIC decreases, so an AIC-based forward search would add the predictor at this step. For BIC, the change is

20\ln(0.88)+\ln(20) \approx -2.5567+2.9957 =0.4390.

BIC increases, so a BIC-based search would not make this addition under the same improvement rule. The candidate model and data are identical; the decision differs because the penalties differ. This is why a search should state its criterion rather than describing a predictor simply as “selected.”

What the Search Does—and Does Not—Guarantee

At each step, forward selection makes the best available one-predictor addition according to the stated criterion. It does not compare every subset at every step. A predictor may be valuable only in combination with another predictor, yet neither may improve the current model enough to enter first. Also, once a predictor has entered, ordinary forward selection does not remove it. Consequently, the final model need not be the best-scoring model over the full candidate set, even when that best model could be fit.

The starting model and candidate set matter too. The intercept-only model is appropriate only when an intercept-only baseline is intended; required predictors, excluded predictors, groups of variables, and model hierarchy constraints require explicit handling. In the simple procedure presented here, each candidate is considered as a one-coefficient addition and all models are compared on the same observations.

A common mistake is to treat the largest drop in RSS as the automatic choice. RSS cannot increase when a predictor is added by least squares, so that rule would ignore model complexity. AIC, by contrast, balances the improvement in fit against a penalty for the added parameter. Another mistake is to conclude that a selected predictor is statistically significant, important in every model, or causally effective. Forward selection makes a criterion-based choice within a specified search; it does not by itself establish any of those claims.

Specify the search.
State the response, observations, candidate predictors, and criterion. For the calculations here, assume comparable Gaussian linear models on the same data.

Initialize.
Fit the intercept-only model and record its criterion value.

Compare additions.
Fit each eligible one-predictor addition to the current model and choose the addition with the best criterion value.

Update or stop.
Add the predictor if it improves the chosen criterion; otherwise stop under the stated rule. Repeat the comparisons after each addition.

Key takeaway: Forward selection is a greedy, criterion-guided search that builds a model from an intercept-only starting point. Its sequence is determined by the eligible predictors and the criterion, and it does not guarantee the best model among all subsets.

Check Your Understanding

Answer these questions using the definitions and examples above.

In forward selection, what models are compared at a step when the current model already contains two predictors?
A candidate addition changes RSS from 150 to 135 on $n=25$ observations. Calculate its AIC change using $n\ln(\mathrm{RSS}_{\mathrm{larger}}/\mathrm{RSS}_{\mathrm{smaller}})+2$. Does it lower AIC?
Why does an AIC comparison between models with one additional predictor include a penalty of 2?
If every eligible addition raises AIC, what does the stopping rule used in the main example recommend?
Why can the model found by forward selection differ from the best-scoring model among all possible predictor subsets?

What You'll Learn