Why the Path Is Nested
Backward elimination is a sequential search through regression models. Its defining move is local: at each step, it considers deleting one predictor from the current model, then either makes one deletion or stops. Because predictors that have been deleted are not added back, the search follows a nested path rather than revisiting arbitrary subsets. This property describes the algorithm’s structure; it does not guarantee that the path contains every model of interest or that the final model is best by some separate measure.
For the usual linear-regression interpretation, take a common response vector \(y\), the same observations at every step, an intercept in every model, and the same predictor columns whenever a predictor is retained. Then the design matrix for a reduced model consists of columns from the larger model’s design matrix. The predictor sets are nested, and so are the corresponding spaces of fitted values. A change in sample, coding, or model terms would need separate consideration; it is not the one-predictor deletion operation defined here.
A common rule chooses the predictor with the largest partial-test p-value and deletes it if that p-value exceeds a chosen significance level \(\alpha\). The structural result below does not depend on that particular rule: another rule can select a different predictor, but it still makes a nested path if it deletes only one predictor at a time and never restores deleted ones.
The One-Deletion Result
Suppose the current predictor set is \(S_k\). A backward-elimination step selects a predictor \(j_k\) that is still in that set, so \(j_k\in S_k\), and defines the next set by removing it. The defining set operation immediately gives the result.
The inclusion is strict because \(j_k\) is present in \(S_k\) and absent from \(S_{k+1}\). The size falls by one because exactly one element is removed. Applying this argument at every step proves that the predictor sets form a chain:
This also explains the corresponding statement about model spaces. Write \(\mathcal{C}(S)\) for the set of all fitted vectors obtainable from model \(M(S)\) as its regression coefficients vary. A fitted vector from the smaller model can also be represented in the larger model: assign coefficient zero to the predictor that was removed and keep the other coefficients unchanged. Therefore,
This is a statement about which fitted values the models can represent, not about the numerical estimates or the elimination rule. If deletion continues until no predictors remain, a model beginning with \(p\) predictors has \(p+1\) models on its path, counting the full model and the intercept-only model. If a stopping rule ends the search earlier, the path contains fewer models. At no stage can its predictor count increase.
Tracing the Path Under a Deletion Rule
For classical partial \(t\)-tests in a linear model, one commonly assumes \(Y=X\beta+\varepsilon\), with fixed full-rank design matrix \(X\) and independent errors \(\varepsilon_i\sim N(0,\sigma^2)\). These assumptions support the usual exact finite-sample \(t\)-tests when their conditions hold. The nested-path proof itself does not require normal errors or p-values; those specify how a predictor may be selected, not why deletion produces nested sets.
Worked Example: Following a Partial-Test Rule
Method. Apply backward elimination with \(\alpha=0.05\), deleting the predictor with the largest supplied partial-test p-value whenever that value exceeds \(\alpha\). A hypothetical actuary models a claim-adjustment measure using predictors \(A\), \(B\), and \(C\). The p-values below are supplied regression-summary results for the current model at each step.
| Current predictors | Partial-test p-values |
|---|---|
| \(A,B,C\) | \(A: 0.012,\ B: 0.310,\ C: 0.080\) |
| \(A,C\) | \(A: 0.018,\ C: 0.110\) |
| \(A\) | \(A: 0.018\) |
Step 1. In \(M(\{A,B,C\})\), the largest p-value is \(0.310\), since \(0.310>0.080>0.012\). It belongs to \(B\), and \(0.310>0.05\), so delete \(B\). The new set is \(\{A,C\}\).
Step 2. In \(M(\{A,C\})\), the largest p-value is \(0.110\), since \(0.110>0.018\). It belongs to \(C\), and \(0.110>0.05\), so delete \(C\). The new set is \(\{A\}\).
Step 3. In \(M(\{A\})\), the only candidate p-value is \(0.018\). Because \(0.018\leq0.05\), the rule stops. The visited sets are \(\{A,B,C\}\supset\{A,C\}\supset\{A\}\); the predictor count falls from three to two to one. Check: each successive set removes exactly the selected predictor and retains all other predictors. Common mistake avoided: comparing a p-value from one model against a p-value from a different model; each deletion decision here uses the values supplied for the current model only.
Worked Example: A Candidate Model the Path Does Not Visit
Method. Trace the same largest-p-value rule, now with \(\alpha=0.05\). In a hypothetical model of a policy-level outcome, the candidate predictors are \(P,Q,R,S\), and the current-model p-values are supplied as follows.
| Current predictors | Partial-test p-values |
|---|---|
| \(P,Q,R,S\) | \(P: 0.040,\ Q: 0.200,\ R: 0.030,\ S: 0.160\) |
| \(P,R,S\) | \(P: 0.060,\ R: 0.020,\ S: 0.120\) |
| \(P,R\) | \(P: 0.030,\ R: 0.010\) |
| \(R\) | \(R: 0.010\) |
Step 1. In the full model, \(Q\) has the largest p-value: \(0.200>0.160>0.040>0.030\). Since \(0.200>0.05\), delete \(Q\), leaving \(\{P,R,S\}\).
Step 2. The largest p-value in \(\{P,R,S\}\) is \(0.120\), for \(S\), since \(0.120>0.060>0.020\). Delete \(S\), because \(0.120>0.05\), leaving \(\{P,R\}\).
Step 3. In \(\{P,R\}\), the largest p-value is \(0.030\), for \(P\). It is not greater than \(0.05\), so stop. The visited sets are \(\{P,Q,R,S\}\), \(\{P,R,S\}\), and \(\{P,R\}\); the last model retains both \(P\) and \(R\).
The two-predictor set \(\{P,S\}\) is not on this path. Once \(Q\) is deleted, it cannot return; then \(S\) is deleted before the path reaches two predictors. The algorithm did not compare every two-predictor model in this sequence. Check: the displayed steps each remove the predictor with the largest current p-value, and the stopping comparison is \(0.030\leq0.05\). Common mistake avoided: assuming that visiting a model of a given size means every subset of that size has been visited.
Worked Example: Ties and a Stopping Decision
Method. Use the largest-p-value rule with \(\alpha=0.10\), and stipulate that if the largest values tie, the predictor whose label comes first alphabetically is deleted. A hypothetical analyst models a reserve-related response using \(U,V,W\).
| Current predictors | Partial-test p-values |
|---|---|
| \(U,V,W\) | \(U: 0.240,\ V: 0.240,\ W: 0.070\) |
| \(V,W\) | \(V: 0.130,\ W: 0.060\) |
| \(W\) | \(W: 0.060\) |
Step 1. The maximum is \(0.240\), tied between \(U\) and \(V\). The stipulated tie rule selects \(U\). Since \(0.240>0.10\), delete \(U\), leaving \(\{V,W\}\).
Step 2. The maximum is \(0.130\), for \(V\), and \(0.130>0.10\). Delete \(V\), leaving \(\{W\}\).
Step 3. The only p-value is \(0.060\), and \(0.060\leq0.10\), so the procedure stops. The path is \(\{U,V,W\}\supset\{V,W\}\supset\{W\}\). The initial tie means the p-values alone do not specify a unique first deletion; the stated tie rule makes this particular path determinate. Check: the process stops at \(W\), rather than deleting it, because its p-value is below the threshold. Common mistake avoided: treating a tied maximum as though only one possible path exists without specifying how the tie is resolved.
What Nesting Does—and Does Not—Tell You
Nesting gives a useful way to check whether a sequence could have come from ordinary backward elimination. Every later predictor set must be contained in the previous one, and its size must fall by exactly one at each actual deletion. A sequence such as \(\{A,B,C\}\), \(\{A,B\}\), \(\{A,C\}\) cannot be a backward-elimination path: the last set restores \(C\), which was removed in the preceding step. Merely listing models in decreasing order of size is not enough to establish nesting.
Nesting is a constraint on the search path, not a claim about statistical quality. A model can be omitted from the path simply because the sequence of earlier deletion choices made it unreachable. Likewise, the fact that a model appears later on the path does not by itself show that it is preferable under a different assessment. The structural question answered here is narrower: given the current set and a one-predictor deletion step, what form must the next set take?
Check Your Understanding
For each question, use the stated deletion rule and report the visited predictor sets. Unless a question says otherwise, delete the largest supplied p-value if it exceeds \(\alpha\), and stop otherwise.
- With \(\alpha=0.05\), start with \(\{L,M,N\}\). The initial p-values are \(L:0.02,\ M:0.14,\ N:0.07\). In the resulting two-predictor model, the p-values are \(L:0.03,\ N:0.04\). What is the path and where does it stop? (Answer: \(\{L,M,N\}\supset\{L,N\}\); stop at \(\{L,N\}\), since both remaining p-values are at most \(0.05\).)
- With \(\alpha=0.10\), start with \(\{A,B,C,D\}\). The first largest p-value is for \(C\), at \(0.30\); after deleting \(C\), the largest is for \(A\), at \(0.18\); after deleting \(A\), the remaining p-values for \(B,D\) are \(0.08,0.04\). Give the visited sets and stopping model. (Answer: \(\{A,B,C,D\}\supset\{A,B,D\}\supset\{B,D\}\); stop at \(\{B,D\}\), since \(0.08\leq0.10\).)
- A search starts at \(\{J,K,L\}\), then visits \(\{J,L\}\). Could its next model be \(\{J,K\}\) under backward elimination? Explain using the one-deletion property. (Answer: No. \(K\) was removed in the first step and cannot be restored; \(\{J,K\}\) is not a subset of \(\{J,L\}\).)
- With \(\alpha=0.05\), predictors \(R,S,T\) have initial p-values \(R:0.22,\ S:0.22,\ T:0.03\). A stated tie rule deletes \(S\) before \(R\). In the resulting model \(\{R,T\}\), the p-values are \(R:0.04,\ T:0.02\). Give the path and stopping decision. (Answer: \(\{R,S,T\}\supset\{R,T\}\); stop at \(\{R,T\}\), because the largest remaining p-value, \(0.04\), is at most \(0.05\).)
- A full model contains six predictors and backward elimination makes four deletions before stopping. How many predictors remain, and how many models have been visited, counting the initial model? (Answer: Two predictors remain; five models have been visited.)