What Forward Selection Does
When there are many candidate predictors, fitting every possible subset can require a large number of models. Forward selection offers a sequential alternative: it starts with a simple model and adds one predictor at a time. At each step, the algorithm asks which currently unused predictor gives the best addition to the model under a specified comparison criterion.
For this tutorial, the comparison criterion is training residual sum of squares (RSS). This makes each step concrete: fit the model formed by adding each eligible predictor, compare the resulting RSS values, and append the predictor whose addition produces the smallest RSS. The algorithm evaluates additions to the current model; it does not start over at each step or remove a predictor that has already been selected.
All candidate fits at a particular step must use the same response observations and the same modeling conventions, including an intercept. A model formed by adding one predictor has the same number of predictors as every other candidate addition at that step. Comparing these same-size models by RSS therefore directly identifies which addition gives the best training fit under this criterion. The algorithm then uses the enlarged set as the starting point for the next step.
The algorithm describes a search path, not an exhaustive comparison of every possible subset. If a procedure specifies that two predictors are to be added, forward selection takes two successive decisions. It does not guarantee that its two-predictor model is the best among all two-predictor models; that would require comparing the relevant subsets directly. Here the focus is on how the sequential decisions are made, not on when to stop the search.
How Each Addition Is Evaluated
Suppose the current model contains predictors in \(S\). For each eligible predictor \(X_j\), fit the model using all predictors in \(S\) together with \(X_j\), re-estimating the coefficients for the whole model. Record the candidate model's RSS. Select the candidate with the smallest RSS and add \(j\) to \(S\). If there is only one eligible predictor, it is the only possible addition.
The coefficient estimates are refitted because least squares estimates depend on which predictors are in the model. Keeping earlier estimates fixed would generally not produce the least squares fit for the enlarged model. The algorithm selects a predictor based on the resulting fit, then proceeds with all selected predictors present.
Set \(S=\varnothing\). Fit the intercept-only model and record its RSS.
For each \(j\notin S\), fit the model with predictor set \(S\cup\{j\}\) and record its RSS.
Choose the eligible predictor whose addition has the smallest RSS, then replace \(S\) with \(S\cup\{j\}\).
Use the enlarged set as the current model and evaluate the remaining eligible predictors again.
Why an Addition Cannot Increase Training RSS
The least squares fit in a model minimizes the sum of squared residuals over the coefficient values allowed by that model. When one predictor is added, the enlarged model can reproduce the old fit by assigning coefficient zero to the new predictor. It can also choose other coefficient values, so its minimum RSS cannot be greater than the old model's minimum RSS.
This inequality applies to training RSS when both models use the same observations and the enlarged model contains all the terms in the original model. It does not say that every addition reduces RSS strictly; the reduction can be zero. Nor does it say that adding a predictor improves performance on new data. The forward algorithm's RSS comparison concerns the fitted training observations.
A Partial-Residual Derivation
The RSS reduction from adding one predictor can be understood by removing the part already explained by the current model. Assume ordinary least squares with an intercept, the same observations in both fits, and a full-rank design for the coefficients being estimated. Let \(r\) be the residual vector from the current model. Let \(z\) be the part of the candidate predictor left after projecting it onto the current model's columns, including the intercept. Thus \(z\) is orthogonal to the current model's fitted-value space.
Once the current model is included, the additional fitted component from the candidate is \(b z\) for some coefficient \(b\). Since \(r\) is orthogonal to the current model, the new residual vector is \(r-bz\). Its squared length is
To minimize this expression over \(b\), differentiate with respect to \(b\) and set the derivative to zero. If \(z^\mathsf{T}z>0\), the minimizing value is \(b=(r^\mathsf{T}z)/(z^\mathsf{T}z)\). Substituting that value gives the reduction in RSS:
This result explains why the candidate must be assessed conditional on the predictors already selected. The relevant candidate direction is \(z\), not necessarily the original predictor \(X_j\): any part of \(X_j\) explained by the current model has already been accounted for. The formula also shows that the reduction is nonnegative. If \(z\) is the zero vector, the candidate is entirely redundant with the current model and cannot supply a distinct direction; the full-rank condition for this addition fails.
Worked Example: Choosing the First Two Predictors
A hypothetical analyst is building a least squares model for a response \(Y\) from three candidate predictors \(X_1,X_2,X_3\). All fits use the same observations and an intercept. A constructed centered design has mutually orthogonal predictors, each with sum of squares 4. The response has the decomposition \(Y=2X_1+X_2+1.5X_3+e\), where \(e\) is orthogonal to the three predictors and has sum of squares 7. Use forward selection by training RSS to choose two predictors.
Method. Fit each eligible addition to the intercept-only model, select the one with lowest RSS, then compare additions to that selected model. Orthogonality makes the RSS calculations direct.
Start with the intercept. The response is centered, so the intercept-only fitted value is its mean, zero. The total sum of squares is the sum of the squared components:
First addition. Adding \(X_1\) removes \(2^2(4)=16\) from the RSS, giving \(36-16=20\). Adding \(X_2\) removes \(1^2(4)=4\), giving \(36-4=32\). Adding \(X_3\) removes \(1.5^2(4)=9\), giving \(36-9=27\). The lowest candidate RSS is 20, so append \(X_1\).
Second addition. The current RSS is 20. Adding \(X_2\) removes 4, giving \(20-4=16\). Adding \(X_3\) removes 9, giving \(20-9=11\). Since 11 is smaller than 16, append \(X_3\). The selected two-predictor model contains \(X_1\) and \(X_3\), and its RSS is 11.
Check. The remaining component is \(X_2+e\), with squared length \(4+7=11\), matching the calculated RSS. Common mistake avoided: choosing a predictor based on its coefficient alone. Forward selection compares the resulting candidate-model fits; here the larger reduction from \(X_3\) makes it the second addition.
Worked Example: Comparing Conditional Additions
A hypothetical risk analyst has three candidate predictors \(X_1,X_2,X_3\) and is to make two forward additions using training RSS. Each fit includes an intercept and uses the same observations. The least squares fits give an intercept-only RSS of 100; the RSS values after adding \(X_1,X_2,X_3\) individually are 80, 88, and 90, respectively. After selecting the first predictor, the RSS values for the two possible enlarged models are \( \operatorname{RSS}(\{X_1,X_2\})=71\) and \( \operatorname{RSS}(\{X_1,X_3\})=72\). Which predictors are selected?
Method. At each step, compare the RSS values for the eligible additions to the current set. These values are assumed to be the outputs of the specified least squares fits.
First addition. Relative to the intercept-only RSS of 100, the reductions are \(100-80=20\) for \(X_1\), \(100-88=12\) for \(X_2\), and \(100-90=10\) for \(X_3\). Thus \(X_1\) gives the smallest candidate RSS, 80, and is appended.
Second addition. The eligible predictors are now \(X_2\) and \(X_3\). Their candidate RSS values, with \(X_1\) retained, are 71 and 72. The reductions from the current RSS of 80 are \(80-71=9\) for \(X_2\) and \(80-72=8\) for \(X_3\). Append \(X_2\).
Check. The selected sequence is \(X_1\), then \(X_2\), and the two-predictor model has RSS 71. Common mistake avoided: reusing the single-predictor RSS values at the second step. Each candidate must be fitted with \(X_1\) already in the model, because the value of an additional predictor depends on what the current model explains.
Worked Example: Updating the Eligible Set
A hypothetical forecasting team has four candidate predictors \(A,B,C,D\). Using the same observations and an intercept in every least squares fit, the intercept-only RSS is 250. The single-predictor RSS values are 220 for \(A\), 230 for \(B\), 214 for \(C\), and 225 for \(D\). After the first addition, the two-predictor candidate RSS values formed by adding one remaining predictor are: \( \operatorname{RSS}(\{C,A\})=198\), \( \operatorname{RSS}(\{C,B\})=202\), and \( \operatorname{RSS}(\{C,D\})=205\). Find the sequence for two additions and state the RSS of the final model.
Method. Select the smallest RSS among eligible additions, append that predictor, and form the next candidate models by retaining it.
First addition. The four candidate RSS values are 220, 230, 214, and 225. Their reductions from 250 are \(30,20,36,\) and \(25\), respectively. Predictor \(C\) gives the largest reduction, so the first selected set is \(\{C\}\), with RSS 214.
Second addition. \(C\) is already selected and is therefore not eligible. The remaining candidates produce RSS values 198, 202, and 205. Their reductions from 214 are \(16,12,\) and \(9\), respectively. Append \(A\), giving the set \(\{C,A\}\) and RSS 198.
Check. The RSS decreased from 250 to 214 to 198, and the final set contains exactly the two appended predictors. Common mistake avoided: treating the second step as a fresh search among all four single-predictor fits. The current model retains \(C\), and only \(A,B,D\) are eligible additions.
What the Algorithm Does—and Does Not—Guarantee
Forward selection is a greedy procedure: at a given step, it chooses the best addition available to the current model under the specified criterion. Once a predictor has been appended, the current set changes, and the next comparison is conditional on that set. This makes the search more manageable than fitting all possible subsets, but it also means that the result depends on the sequence of local choices.
Several distinctions help keep the procedure straight:
- It compares additions, not isolated predictors after the first step. Later candidate fits contain every predictor already selected.
- It re-estimates the coefficients. Each candidate is a newly fitted least squares model, not an old fit with one term tacked on at a fixed coefficient.
- It appends one predictor per step. The eligible set is reduced by the selected predictor, which stays in the model during subsequent steps.
- It minimizes training RSS only when that is the chosen criterion. A training-fit comparison alone does not establish how well a model predicts new observations.
The RSS comparison is a computational rule and does not require normally distributed errors. If coefficient tests or confidence intervals are used for other purposes, their interpretation requires the assumptions appropriate to those inferential procedures. The forward-selection steps described here are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus, especially the distinction between evaluating a candidate addition and evaluating a predictor in isolation.
Check Your Understanding
For each problem, follow the specified forward-selection criterion and show which predictor is appended.
- A model has intercept-only RSS 90. The single-predictor RSS values for \(P,Q,R\) are 74, 78, and 69. Which predictor is selected first, and what is the reduction in RSS? (Answer: \(R\); reduction \(21\).)
- After \(R\) has been selected, the candidate RSS values for adding \(P\) and \(Q\) are 61 and 63. Which predictor is selected second, and what is the resulting RSS? (Answer: \(P\); RSS \(61\).)
- A current model has RSS 52 and includes \(M\). The RSS values after adding eligible predictors \(N,O\) are 49 and 45. Which predictor is appended, and by how much does RSS fall? (Answer: \(O\); reduction \(7\).)
- A current set is \(\{A,C\}\), with RSS 38. The candidate RSS values after adding \(B,D,E\) are 32, 35, and 30. Name the selected addition, state the resulting set, and calculate the RSS reduction. (Answer: \(E\); set \(\{A,C,E\}\); reduction \(8\).)
- An intercept-only model has RSS 120. The first-step candidate RSS values for \(U,V,W\) are 102, 111, and 105. After the selected predictor is retained, the RSS values for adding the other two predictors are 90 and 94. State the two additions and the final RSS. (Answer: \(U\), then the candidate with RSS \(90\); final RSS \(90\).)