Why Backward Elimination Starts at the Full Model
When many predictors are available, a selection procedure can search for a smaller model by removing predictors from an initial model. Backward elimination begins with the full candidate model: all predictors designated for consideration are included, subject to the model being estimable. At each step, it evaluates the effect of deleting each remaining predictor, removes the predictor selected by a specified rule, refits the model, and repeats.
The method is a sequence of comparisons, not a single test of the full model. A predictor that appears removable while other predictors are present may look different after one of them is deleted. Therefore, every round uses the current model and requires fresh comparisons. This procedure is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
A Partial F Test for Each Proposed Deletion
One common version of backward elimination uses partial F tests. Suppose the current model is an ordinary least-squares linear regression with an intercept and \(k\) predictors, fitted to \(n\) observations. Let its residual sum of squares be \(RSS_{\text{current}}\). For a proposed deletion of one predictor \(j\), let the reduced model's residual sum of squares be \(RSS_{-j}\). The reduced model is nested within the current model, and the deletion removes one coefficient.
For the usual exact partial F test, assume the errors are independent, normally distributed with mean zero and common variance \(\sigma^2\), and that the design matrix has full column rank. The null hypothesis for deleting \(j\) is that its coefficient is zero, conditional on the other predictors in the current model. The residual degrees of freedom in the current model are \(n-k-1\).
The numerator measures the increase in residual sum of squares caused by deleting \(j\), per coefficient removed. The denominator is the current model's estimate of the error variance, its mean squared error. Under the null hypothesis and the stated assumptions, \(F_j\) has an \(F\) distribution with 1 and \(n-k-1\) degrees of freedom. For a single coefficient, this partial F statistic equals the square of its usual t statistic.
Why the Partial F Formula Works
The reduced model imposes one additional restriction: the coefficient of predictor \(j\) must be zero. Because the reduced model is nested in the current model, fitting the current model cannot have a larger residual sum of squares than fitting the reduced model. The difference \(RSS_{-j}-RSS_{\text{current}}\) is the extra unexplained variation caused by imposing that one restriction.
Under the null hypothesis, this increase, divided by the number of restrictions, estimates \(\sigma^2\). The current model's residual sum of squares divided by its residual degrees of freedom also estimates \(\sigma^2\). Comparing these two estimates gives the partial F statistic. Under the null and the normal-error assumptions, the ratio has the stated F distribution. A large value indicates that deleting the predictor worsens fit substantially relative to the estimated error variation.
At a chosen significance level \(\alpha\), the common rule is to remove the predictor with the largest partial-test p-value if that p-value exceeds \(\alpha\). Equivalently, among the one-predictor deletion tests, remove the predictor with the smallest \(F_j\) if that statistic is below the upper-tail critical value \(F_{1,n-k-1;1-\alpha}\). If the smallest statistic is already at or above the critical value, stop. The degrees of freedom, and hence the critical value, can change after a deletion.
Worked Example: Comparing Deletions Through More Than One Round
A hypothetical insurer models a continuous claim-cost measure using predictors \(A\), \(B\), and \(C\). There are \(n=24\) observations. Use partial F tests at \(\alpha=0.05\), and assume the linear-model conditions for the tests hold. The full model has \(RSS=200\). For the initial model, deleting \(A\), \(B\), or \(C\) gives respective reduced-model RSS values of 209, 260, and 215. At the next rounds, the needed reduced-model RSS values are given below. Find the selected model.
Method. At each step, divide each proposed RSS increase by the current model's mean squared error. Remove the predictor with the smallest partial F statistic if it is below the applicable critical value; otherwise stop.
Initial model. It has three predictors and an intercept, so the residual degrees of freedom are \(24-3-1=20\), and \(MSE=200/20=10\). The deletion statistics are:
The critical value is approximately \(F_{1,20;0.95}=4.351\). The smallest statistic is \(F_A=0.900\), which is below the critical value, so delete \(A\). The refitted model containing \(B\) and \(C\) has \(RSS=209\).
Second round. The refitted model has two predictors, so its residual degrees of freedom are \(24-2-1=21\). Its mean squared error is \(209/21=9.952\) (rounded to three decimals). Deleting \(B\) gives \(RSS=260\); deleting \(C\) gives \(RSS=211\). Thus:
The critical value is approximately \(F_{1,21;0.95}=4.325\). Since \(F_C=0.201\) is the smaller statistic and is below the critical value, delete \(C\). The model containing only \(B\) has \(RSS=211\).
Final round. The model with only \(B\) has \(24-1-1=22\) residual degrees of freedom and \(MSE=211/22=9.591\) (rounded to three decimals). Deleting \(B\) gives the intercept-only model with \(RSS=260\), so:
The critical value is approximately \(F_{1,22;0.95}=4.301\). Since \(5.109>4.301\), do not delete \(B\). Selected model: the model containing \(B\) alone. Each round used the current model's error estimate and degrees of freedom.
Common mistake avoided: comparing RSS increases alone would not be the stated partial F rule. The increases must be scaled by the current model's mean squared error.
The Algorithm in Operational Form
List the candidate predictors and choose a rule, such as removing the largest p-value when it exceeds a fixed significance level \(\alpha\). Any required predictors must be handled separately and not treated as removable candidates.
Record its residual sum of squares, residual degrees of freedom, and error estimate. These quantities are the reference for every proposed deletion in this round.
For every predictor still present, compare the current model with the model omitting that predictor. Calculate its partial F statistic, or obtain the corresponding p-value from the same partial test.
Under the significance-based rule, delete the predictor with the largest p-value only if that p-value exceeds \(\alpha\). If none exceeds \(\alpha\), stop.
After a deletion, refit the reduced model and evaluate all remaining deletions anew. Do not carry forward the previous round's test statistics or p-values.
Reassessment Can Change the Decision
The next example shows why the algorithm needs a new comparison after each deletion. Predictor effects are assessed conditional on the other predictors currently in the model. Removing one predictor changes that conditioning set, the fitted coefficients, and the residual variation used in each test.
Worked Example: Recalculate Before Applying the Stopping Rule
A hypothetical portfolio model has predictors \(A\), \(B\), \(C\), and \(D\), with \(n=32\) observations. Use the partial F rule at \(\alpha=0.05\), and assume its test conditions hold. The full model has \(RSS=270\). Deleting \(A\), \(B\), \(C\), or \(D\) separately gives RSS values 271, 315, 312, and 330. After deleting \(A\), the current model has \(RSS=271\); deleting \(B\), \(C\), or \(D\) from that model gives RSS values 316, 314, and 332. Determine the outcome.
First fit. There are four predictors, so the residual degrees of freedom are \(32-4-1=27\), and \(MSE=270/27=10\). The four statistics are:
The critical value is approximately \(F_{1,27;0.95}=4.210\). The smallest statistic is \(F_A=0.100\), so delete \(A\). The fact that \(F_C=4.200\) is just below the initial critical value does not select \(C\): the rule selects the smallest statistic for deletion.
Refit after deleting \(A\). Three predictors remain, giving \(32-3-1=28\) residual degrees of freedom and \(MSE=271/28=9.679\) (rounded to three decimals). Recalculate the comparisons:
The new critical value is approximately \(F_{1,28;0.95}=4.196\). The smallest current statistic is \(F_C=4.443\), which is greater than the critical value. None of the remaining predictors meets the rule for deletion, so stop with \(B\), \(C\), and \(D\) in the model.
Common mistake avoided: using the initial \(F_C=4.200\) after deleting \(A\) would incorrectly suggest removing \(C\). The correct statistic after refitting is \(4.443\), and the correct stopping decision is to retain all three remaining predictors.
What the Rule Does—and Does Not—Establish
A partial F test asks whether deleting a predictor causes a sufficiently large increase in residual variation, conditional on the other predictors currently included. Backward elimination repeatedly applies this local comparison; it does not evaluate every possible predictor subset. The chosen result depends on the starting candidate set and the specified removal rule.
The procedure requires an estimable starting model. If the design matrix is rank deficient, some coefficients cannot be separately estimated, so the ordinary coefficient tests and deletion comparisons may not be defined as stated. Earlier material on collinearity and the dummy-variable trap helps identify such issues. Also, although the classical test has its stated distribution under its assumptions before selection, repeatedly testing and selecting predictors affects the interpretation of ordinary p-values. The stopping rule is a model-selection procedure, not a guarantee that the final coefficients have unadjusted inferential properties.
A frequent exam trap is to assume that the predictor with the largest individual p-value in the original full model must be the one removed at every later stage. It is considered for removal only in the round in which that p-value was calculated. After deletion, the model is different, so all remaining candidate deletions must be reassessed. Another trap is to keep deleting until only one predictor remains: under the stated rule, the procedure stops as soon as no remaining deletion satisfies the criterion.
Check Your Understanding
For each question, use the partial F rule where specified. Assume the required linear-model test conditions hold and use the stated approximate critical values.
- A model with \(n=20\) observations has predictors \(X_1\) and \(X_2\), with current \(RSS=90\). Deleting \(X_1\) gives \(RSS=96\), and deleting \(X_2\) gives \(RSS=91\). At \(\alpha=0.05\), use \(F_{1,17;0.95}=4.451\). Which predictor is removed? (Answer: Remove \(X_2\); its partial \(F\) is \(0.189\), less than \(4.451\).)
- A current model has \(n=25\), three predictors, and \(RSS=168\). Deleting \(A\), \(B\), or \(C\) separately gives RSS values 172, 208, and 171. At \(\alpha=0.05\), use \(F_{1,21;0.95}=4.325\). Which predictor is selected for deletion? (Answer: Remove \(C\); the partial \(F\) values are \(0.500\), \(5.000\), and \(0.375\), respectively.)
- A current model has \(n=16\), predictors \(P\) and \(Q\), and \(RSS=52\). Deleting \(P\) gives \(RSS=71\); deleting \(Q\) gives \(RSS=68\). At \(\alpha=0.05\), use \(F_{1,13;0.95}=4.667\). Which predictor is removed? (Answer: Remove \(Q\); \(F_P=4.750\) and \(F_Q=4.000\), so the smallest statistic is below the critical value.)
- After previous deletions, a model has \(n=30\), one remaining predictor, and \(RSS=140\). The intercept-only model has \(RSS=163\). At \(\alpha=0.05\), use \(F_{1,28;0.95}=4.196\). Should the remaining predictor be deleted? (Answer: No; the deletion statistic is \((163-140)/(140/28)=4.600\), which exceeds the critical value, so stop and retain the predictor.)