Backward Elimination Worked Example

What a Backward-Elimination Example Must Track

Backward elimination starts with a specified full regression model and considers smaller models formed by deleting one predictor at a time. At each stage, it compares the eligible deletions, chooses the one with the best score, and decides whether that score improves on the current model. The result is a sequence of models along a single search path, not a comparison of every possible subset.

To make each decision unambiguous, the score and stopping rule must be stated before the search begins. The examples below use a simple penalized residual sum of squares. The numerical data are hypothetical regression summaries created for illustration; they are not observations from an insurer or other organization.

Definition: Let $M$ denote a candidate linear regression model, let $\operatorname{RSS}(M)$ be its least-squares residual sum of squares, and let $p(M)$ be its number of predictors, not counting the intercept. For a prespecified penalty $\lambda>0$, define the score $$ Q(M)=\operatorname{RSS}(M)+\lambda p(M). $$ The search prefers smaller scores. From the current model, fit every candidate formed by deleting exactly one predictor, select the candidate with the smallest score, and move to it only if its score is strictly smaller than the current score. If no candidate has a strictly smaller score, stop.

For these calculations, every candidate uses the same response observations and includes an intercept; each is refitted by least squares. The score is an illustrative selection criterion, not an assertion that a particular information criterion has been calculated. The arithmetic does not require normally distributed errors, but the data and fitting procedure must be comparable across candidate models. If the observations or response differ, their residual sums of squares are not directly comparable in this way.

How to Calculate and Compare a Deletion

Suppose the current model has $p$ predictors. A candidate formed by deleting one predictor has $p-1$ predictors, so its score is its own refitted RSS plus $\lambda(p-1)$. The intercept is retained and is not charged the predictor penalty. Compare each candidate score with the other deletion scores first; then compare the winning candidate with the current model's score.

Q(M_{\text{candidate}})=\operatorname{RSS}(M_{\text{candidate}})+\lambda(p-1), \qquad Q(M_{\text{current}})=\operatorname{RSS}(M_{\text{current}})+\lambda p.

This expression follows directly from the definition of $Q$: deleting one predictor reduces the predictor count by one, while the RSS is recalculated for the reduced model. Refitting matters. The reduced model's coefficients are estimated anew; its RSS is not generally the RSS obtained by merely setting one current coefficient to zero and leaving every other coefficient unchanged.

Record the current model and score.
Count predictors, compute the current score, and keep the intercept convention fixed.

Fit each one-predictor deletion.
For every predictor in the current model, form the model without that predictor and use its refitted RSS.

Select the smallest candidate score.
Compare all eligible one-variable deletions, not just one chosen in advance.

Apply the stopping rule.
Move to the best candidate only if its score is strictly less than the current score; otherwise stop at the current model.

Worked Example: Correctly Tracing a Three-Predictor Search

A hypothetical least-squares analysis uses the same $30$ observations in every model. The full model contains predictors $U,V,W$, and the score is $Q=\operatorname{RSS}+7p$. The reported RSS values are:

Model	RSS	Predictors $p$
$UVW$	40	3
$VW$	43	2
$UW$	45	2
$UV$	42	2
$U$	50	1
$V$	48	1
Intercept only	60	0

Step 1: Score the full model. It has three predictors, so $Q(UVW)=40+7(3)=61$.

Step 2: Score every first deletion. Deleting $U$ leaves $VW$, with score $43+7(2)=57$. Deleting $V$ leaves $UW$, with score $45+7(2)=59$. Deleting $W$ leaves $UV$, with score $42+7(2)=56$. The smallest candidate score is $56$, so delete $W$ and move to $UV$. Since $56<61$, this move improves the score.

Step 3: Continue from $UV$. The current score is $42+7(2)=56$. Deleting $U$ leaves $V$, with score $48+7(1)=55$. Deleting $V$ leaves $U$, with score $50+7(1)=57$. The best candidate is $V$, with score $55$, so delete $U$. This is another strict improvement.

Step 4: Test the final deletion. The model $V$ has score $48+7(1)=55$. Deleting $V$ leaves the intercept-only model, with score $60+7(0)=60$. Since $60$ is not less than $55$, stop at $V$.

Result and check: The selected sequence is $UVW \rightarrow UV \rightarrow V$, with scores $61 \rightarrow 56 \rightarrow 55$. The chosen score decreases at each accepted step, and the only next deletion would increase it. A common mistake is to say that deleting $U$ leaves $UV$; in fact, it leaves $VW$. The correct first-stage comparison includes $UV$, whose score is $56$, the minimum of the three candidates.

A Second Search With More Than One Accepted Deletion

The same procedure applies when the penalty and predictor set differ. Keep a written record of the model associated with each RSS: a numerical score is useful only if it is attached to the correct candidate deletion.

Worked Example: Comparing Deletions at Two Stages

A hypothetical model fitted to a common sample of $40$ observations starts with predictors $A,B,C,D$. Use $Q=\operatorname{RSS}+4p$. The full model has RSS $30$; the first-stage reduced-model RSS values are given below.

Candidate after deletion	RSS	Score calculation	Score
$ABC$ (delete $D$)	32	$32+4(3)$	44
$ABD$ (delete $C$)	33	$33+4(3)$	45
$ACD$ (delete $B$)	31	$31+4(3)$	43
$BCD$ (delete $A$)	34	$34+4(3)$	46

The full model score is $30+4(4)=46$. The minimum first-stage candidate score is $43$, for $ACD$, so delete $B$. The move improves the score because $43<46$.

Now $ACD$ is the current model, with RSS $31$ and score $31+4(3)=43$. The next candidate scores are: $CD$, RSS $37$, score $37+4(2)=45$; $AD$, RSS $35$, score $35+4(2)=43$; and $AC$, RSS $34$, score $34+4(2)=42$. The best candidate is $AC$, so delete $D$. The score falls from $43$ to $42$.

For completeness, the one-predictor candidates from $AC$ have RSS $43$ for $A$ and RSS $46$ for $C$. Their scores are $43+4=47$ and $46+4=50$, respectively. Neither improves on the current score of $42$, so the search stops. The full sequence is $ABCD \rightarrow ACD \rightarrow AC$, with scores $46 \rightarrow 43 \rightarrow 42$. The method avoids the mistake of stopping after the first deletion: the stopping decision must be reassessed at each selected model.

When the Best Deletion Is Not Good Enough

At a stage, the best one-variable deletion is not necessarily an acceptable move. It may have the smallest score among all deletion candidates while still having a score greater than or equal to the current model's score. Under the strict-improvement rule used here, that is a stopping point: do not take a worsening move simply because it is the best available deletion.

Worked Example: The Minimum Candidate Score Can Still Lose

A hypothetical full model $PQR$, fitted to the same $25$ observations as its reductions, has RSS $24$. Use the prespecified score $Q=\operatorname{RSS}+5p$.

The full-model score is $24+5(3)=39$. Deleting $P$ leaves $QR$, with RSS $28$ and score $28+5(2)=38$. Deleting $Q$ leaves $PR$, with RSS $25$ and score $25+5(2)=35$. Deleting $R$ leaves $PQ$, with RSS $27$ and score $27+5(2)=37$. The minimum is $35$, so delete $Q$ and move to $PR$.

At $PR$, the current score is $25+5(2)=35$. Deleting $P$ leaves $R$, with RSS $34$ and score $34+5=39$. Deleting $R$ leaves $P$, with RSS $40$ and score $40+5=45$. The best candidate score is $39$, but $39>35$. Therefore, stop at $PR$; do not delete $P$.

The resulting path is $PQR \rightarrow PR$, with scores $39 \rightarrow 35$. The key check is the comparison of $39$ with the current score $35$, not merely the fact that $39$ is the smaller of the two deletion scores.

Common Scoring and Bookkeeping Traps

Mislabeling a deletion. If $U$ is deleted from $UVW$, the candidate is $VW$, not $UV$. Write the remaining predictors beside each deletion before calculating its score.
Using the wrong predictor count. The intercept does not count toward $p$ in the score defined here. A two-predictor model receives penalty $2\lambda$, not $3\lambda$.
Comparing only candidate scores. Finding the best deletion does not establish that the move is worthwhile. Also compare its score with the current model's score.
Using an RSS without refitting. Each candidate's RSS must come from fitting that reduced model, rather than carrying over coefficients from the larger model.
Changing the criterion mid-search. Keep the same $\lambda$, response, observations, intercept convention, and scoring rule at every stage.

Key takeaway: At every backward-elimination stage, identify each model produced by deleting one predictor, calculate its score with the stated rule, choose the minimum, and accept it only if it strictly improves on the current score. Record both the model and its score before continuing.

Check Your Understanding

For each problem, use the stated score and the strict-improvement stopping rule. Each set of RSS values refers to least-squares fits on the same observations with an intercept retained.

A current model $ABC$ has RSS $20$, and $Q=\operatorname{RSS}+3p$. The deletion candidates are $AB$, RSS $22$; $AC$, RSS $21$; and $BC$, RSS $24$. From the selected candidate, the one-predictor RSS values are $29$ for $A$ and $31$ for $C$. Find the full search path and stopping score. (Answer: $ABC$ has score $29$; delete $B$ to reach $AC$ with score $27$; then stop because the scores for $A$ and $C$ are $32$ and $34$, so the path is $ABC \rightarrow AC$, ending at $27$.)
A current model $WXYZ$ has RSS $35$, with $Q=\operatorname{RSS}+6p$. Deleting $W,X,Y,Z$, respectively, gives models $XYZ$ with RSS $39$, $WYZ$ with RSS $37$, $WXZ$ with RSS $41$, and $WXY$ with RSS $40$. From the winning candidate $WYZ$, deletion gives $WY$ with RSS $44$, $WZ$ with RSS $41$, and $YZ$ with RSS $46$. From the resulting best model, the one-predictor RSS values are $48$ for $W$ and $52$ for $Z$. Find the sequence and stopping score. (Answer: $WXYZ$ scores $59$; delete $X$ to reach $WYZ$ at $55$; delete $Y$ to reach $WZ$ at $53$; stop because $W$ and $Z$ score $54$ and $58$, respectively. The path is $WXYZ \rightarrow WYZ \rightarrow WZ$, ending at $53$.)
A model $ABC$ has RSS $14$, and $Q=\operatorname{RSS}+2p$. Deleting $A,B,C$, respectively, gives $BC$ with RSS $17$, $AC$ with RSS $15$, and $AB$ with RSS $15$. If tied deletion scores are resolved by deleting the alphabetically earliest tied predictor, which first move is selected? (Answer: The current score is $20$; $BC$ scores $21$, while $AC$ and $AB$ both score $19$. Delete $B$ by the tie rule, reaching $AC$ with score $19$.)
A model $ABCD$ has RSS $25$, and $Q=\operatorname{RSS}+3p$. Its deletion candidates $BCD,ACD,ABD,ABC$ have RSS values $27,26,28,27$, respectively. From the best candidate $ACD$, the RSS values for $CD,AD,AC$ are $32,29,30$, respectively. Should the search continue after this second-stage comparison? (Answer: The full score is $37$. The best first deletion is $ACD$, with score $35$. From $ACD$, the best deletion is $AD$, with score $35$, which does not strictly improve on the current score $35$. Stop at $ACD$; the path is $ABCD \rightarrow ACD$.)

Model	RSS	Predictors \(p\)
\(UVW\)	40	3
\(VW\)	43	2
\(UW\)	45	2
\(UV\)	42	2
\(U\)	50	1
\(V\)	48	1
Intercept only	60	0

Candidate after deletion	RSS	Score calculation	Score
\(ABC\) (delete \(D\))	32	\(32+4(3)\)	44
\(ABD\) (delete \(C\))	33	\(33+4(3)\)	45
\(ACD\) (delete \(B\))	31	\(31+4(3)\)	43
\(BCD\) (delete \(A\))	34	\(34+4(3)\)	46

What You'll Learn