Scoring Each Forward-Selection Step
Forward selection builds a regression model one predictor at a time. At a given step, the selected predictors stay in the model, and each eligible predictor is considered in turn as an addition. The practical task is to fit each resulting candidate model and compare its score with the current model’s score. A predictor’s score from an earlier step is not necessarily its score now: the current model changes what remains for that predictor to explain.
This tutorial uses residual sum of squares (RSS) as the main score. Smaller RSS means better fit to the observed response, so the candidate with the largest RSS reduction gives the best fit among additions at that step. A later example uses AIC, for which smaller scores are also preferred. The calculations below trace paths for specified steps; they do not determine when a search should stop.
Use the same response observations and the same treatment of the intercept for all candidate fits. Otherwise, differences in RSS can reflect different data rather than the added predictor. In the examples, models include an intercept. For centered variables, the intercept-only RSS is \(y'y\), and cross-products such as \(x_j'y\) refer to the centered response and predictor values.
Why the Candidate’s Conditional Contribution Matters
When predictors are correlated, adding a predictor after other variables have entered does not generally remove as much RSS as it did in a one-predictor model. Its new contribution is what remains after accounting for the predictors already selected. The following calculation gives that contribution directly.
Let \(P_S\) be the projection matrix onto the columns for the current model, including its intercept, and let \(M_S=I-P_S\). The current residual vector is \(r=M_Sy\). For candidate predictor \(x_j\), its part not explained by the current model is \(z=M_Sx_j\). Adding \(x_j\) fits a coefficient \(b\) to \(z\) in the residuals \(r\). The remaining squared error is \((r-bz)'(r-bz)\).
Differentiating with respect to \(b\) and setting the derivative to zero gives \(-2z'r+2b z'z=0\), so the least-squares coefficient is \(b=z'r/(z'z)\), provided \(z'z>0\). Substituting this value into the squared error shows that the RSS reduction is
If \(z'z=0\), the candidate has no variation left after accounting for the current model; it adds no independent direction and this formula is not defined. Such a predictor cannot be evaluated as an ordinary additional least-squares direction. The formula also explains why the marginal reduction from a predictor considered alone is generally not its reduction later: after variables enter, both its residualized response association and its residualized variation can change.
Worked Examples
Worked Example: Comparing RSS Reductions at Two Steps
A hypothetical analyst models a centered claim-cost response using predictors A, B, and C. There are six observations. The centered cross-products satisfy \(y'y=100\); the three predictor columns are mutually orthogonal, each with squared length 4; and \(x_A'y=12\), \(x_B'y=8\), and \(x_C'y=4\). Find the first two additions using RSS.
Method. At the first step, the current model contains only the intercept. For a centered predictor \(j\), its RSS reduction is \((x_j'y)^2/(x_j'x_j)\). Thus the reductions are:
- A: \(12^2/4=144/4=36\).
- B: \(8^2/4=64/4=16\).
- C: \(4^2/4=16/4=4\).
The intercept-only RSS is 100. The three candidate RSS values are therefore \(100-36=64\) for A, \(100-16=84\) for B, and \(100-4=96\) for C. The smallest is 64, so A enters first.
At the second step, A is already in the model. Since the predictors are mutually orthogonal, fitting A does not change either the cross-product or squared length for B or C after residualization. The reduction for B remains \(8^2/4=16\); its candidate RSS is \(64-16=48\). The reduction for C remains \(4^2/4=4\); its candidate RSS is \(64-4=60\). B is the next addition. After two steps the path is A, then B, and the RSS is 48.
Check. The two reductions total \(36+16=52\), and \(100-52=48\), agreeing with the final candidate RSS. The common mistake avoided here is choosing by predictor label or by an uncomputed impression of association rather than comparing every eligible candidate’s score.
Worked Example: Recalculating After a Correlated Predictor Enters
For a hypothetical centered response, \(y'y=40\). Predictors A and B each have squared length 4, their cross-product is \(x_A'x_B=2\), and \(x_A'y=6\), \(x_B'y=7\). Predictor C is orthogonal to both A and B, with \(x_C'x_C=4\) and \(x_C'y=4\). Find the first two additions by minimizing RSS.
First step. The marginal RSS reductions are A: \(6^2/4=9\), B: \(7^2/4=49/4=12.25\), and C: \(4^2/4=4\). The corresponding RSS values are 31, 27.75, and 36. B has the smallest candidate RSS, so B enters first.
Second step. C remains orthogonal to B, so its conditional reduction is \(4^2/4=4\). Its candidate RSS is \(27.75-4=23.75\). For A, first remove the part explained by B. The residualized cross-product with the response is \(6-(2/4)(7)=2.5\). The residualized squared length is \(4-2^2/4=3\). Therefore A’s conditional reduction is \(2.5^2/3=6.25/3=25/12\), or approximately 2.0833. Its candidate RSS is \(27.75-25/12=25.6667\), rounded to four decimal places. C has the smaller candidate RSS, so C enters second.
Check. After B and C enter, A’s residualized reduction is still \(25/12\), because C is orthogonal to A and B. Adding A would give RSS \(23.75-25/12=21.6667\), rounded to four decimal places. The path after two steps is B, then C.
Common mistake and correction. It is wrong to reuse A’s marginal reduction of 9 after B has entered: that would give the false candidate RSS \(27.75-9=18.75\), incorrectly making A appear better than C. The corrected conditional reduction is \(25/12\), not 9, so the actual next choice is C.
Worked Example: Comparing AIC Changes Along a Path
A hypothetical analyst has 12 observations and fits Gaussian linear regression models with an intercept. For a centered response and three mutually orthogonal centered predictors, \(y'y=50\); each predictor has squared length 4; and their response cross-products are \(x_A'y=8\), \(x_B'y=6\), and \(x_C'y=4\). Trace three additions using AIC, reporting each candidate’s change from the current AIC.
Score definition and first step. For independent Gaussian errors with unknown variance, a convenient AIC score, up to constants common to the candidate models, is \(12\ln(\operatorname{RSS}/12)+2q\), where \(q\) is the number of regression coefficients, including the intercept. Each addition increases \(q\) by one. Thus the AIC change for an addition that changes RSS from \(R\) to \(R_{\text{new}}\) is \(12\ln(R_{\text{new}}/R)+2\). The Gaussian model assumption matters for this AIC expression; it is not a general formula for every likelihood.
The first-step RSS reductions are A: \(8^2/4=16\), B: \(6^2/4=9\), and C: \(4^2/4=4\). The candidate RSS values are 34, 41, and 46. Their AIC changes from the intercept-only model are:
- A: \(12\ln(34/50)+2=12\ln(0.68)+2\approx-2.6279\).
- B: \(12\ln(41/50)+2=12\ln(0.82)+2\approx-0.3814\).
- C: \(12\ln(46/50)+2=12\ln(0.92)+2\approx0.9994\).
A has the smallest change and therefore the lowest candidate AIC. After A enters, orthogonality means B still reduces RSS by 9 and C by 4. From the current RSS of 34, adding B gives RSS 25 and AIC change \(12\ln(25/34)+2\approx-1.6898\); adding C gives RSS 30 and change \(12\ln(30/34)+2\approx0.4980\). B is the next addition.
After A and B enter, the current RSS is 25. Adding C reduces RSS by 4, giving RSS 21 and AIC change \(12\ln(21/25)+2\approx-0.0922\). The path for three additions is A, B, C. The displayed changes are rounded to four decimal places. The common mistake avoided is comparing raw RSS reductions as though they were AIC changes: AIC includes a parameter penalty, so its numerical change must be calculated using its own formula.
How to Keep a Hand Calculation Organized
At each step, write down the current model and its score before evaluating additions. Then list every eligible predictor, calculate the score of the model formed by adding it, and compare the candidate scores. This table structure prevents a common bookkeeping error: subtracting a reduction from the wrong current RSS or forgetting that a predictor already in the model is no longer eligible.
Write the selected predictor set and its score, such as the current RSS.
Fit or calculate the score for the current set plus that candidate. For RSS, use the candidate’s conditional contribution when predictors are correlated.
For RSS and AIC, a smaller candidate score is preferred. If scores tie, the score alone does not distinguish the candidates.
Use the chosen model as the new current model, then recompute or obtain every remaining candidate’s score for that step.
The calculations are only comparable when the response observations and scoring definition stay fixed across candidates. If a predictor has missing values and its candidate fit uses fewer observations, its RSS is not directly comparable to a fit on the original observations. Likewise, when a score includes a model-size penalty, use that full score rather than treating improved training fit alone as the objective.
A forward-selection path records local choices: at each step, the selected addition was preferred among the candidates then considered under the stated score. It does not, by itself, establish that the resulting model is best among all possible predictor subsets. The hand calculation is valuable because it makes the comparison and its dependence on the current model explicit.
Check Your Understanding
For each problem, identify the preferred addition or path using the score stated. Answers use the given values without intermediate rounding.
- A centered response has \(y'y=60\). Two orthogonal centered predictors each have squared length 5, with \(x_1'y=10\) and \(x_2'y=5\). Using RSS, find the first two additions and the RSS after both. (Answer: \(x_1\), then \(x_2\); RSS \(=35\).)
- The current RSS is 48. After residualizing against the selected predictors, eligible predictors D and E have \((x_D'M_Sy,x_D'M_Sx_D)=(5,4)\) and \((x_E'M_Sy,x_E'M_Sx_E)=(4,4)\). Which addition is preferred by RSS, and what is its candidate RSS? (Answer: D; reduction \(=25/4=6.25\), candidate RSS \(=41.75\).)
- With 20 observations, the current RSS is 100. Two additions reduce RSS by 15 and 8, respectively. Using the AIC change \(20\ln(R_{\text{new}}/R_{\text{current}})+2\), which candidate has the smaller AIC change? (Answer: the reduction of 15; its change is \(20\ln(0.85)+2\approx-1.2504\).)
- The current RSS is 80. Candidates A and B each reduce it by 12, while C reduces it by 5. After A is selected as the tie choice, the conditional reductions for B and C are 3 and 6. What is the preferred second addition and the RSS after two steps? (Answer: C; RSS \(=80-12-6=62\).)
- The current RSS is 25. Candidate P gives RSS 21, Q gives RSS 20, and R gives RSS 22. Which candidate is preferred by RSS, and by how much does it reduce RSS? (Answer: Q; reduction \(=5\).)