Why a Selected Subset Can Change
Subset selection returns a particular set of predictors, but that set can depend on small details of the data. If two candidate predictors explain nearly the same amount of variation, a modest response change may make one appear slightly better than the other. A small change to the candidate predictors can have a similar effect. The selected variable set changes, even though the competing models’ fits may remain close.
This sensitivity is called selection instability. It is especially relevant when predictors are strongly related or when competing models have nearly tied scores. Instability does not, by itself, show that the search algorithm was implemented incorrectly. It does mean that the selected set should not automatically be treated as uniquely established by the data.
To make the idea precise, consider ordinary least squares with an intercept. Let \(y\) be the response vector and \(x_j\) a candidate predictor vector. Center both vectors by subtracting their sample means. Comparisons below use the same observations and the same response, and compare models with one predictor unless stated otherwise. The residual sum of squares, abbreviated RSS, measures the squared difference between observed and fitted responses. For models with the same number of predictors, the model with smaller RSS fits the observed data better by this measure.
Why Small Changes Can Reverse a Choice
For centered vectors, fitting a one-predictor regression through the origin is equivalent to fitting the corresponding predictor with an intercept before centering. The estimated slope is \(x_j^\mathsf{T}y/(x_j^\mathsf{T}x_j)\). The next derivation gives the resulting RSS and shows why a small change in the inner product \(x_j^\mathsf{T}y\) can affect the ranking.
The last equality follows by substituting the least-squares slope into both terms: the two subtracted contributions combine to one copy of \((x_j^\mathsf{T}y)^2/(x_j^\mathsf{T}x_j)\). Thus the reduction in RSS from adding \(x_j\) to an intercept-only model is \((x_j^\mathsf{T}y)^2/(x_j^\mathsf{T}x_j)\). This formula assumes ordinary least squares with an intercept and uses centered vectors; it does not directly describe a model with a different loss function or a different set of observations.
When predictors are centered and mutually orthogonal, the reductions in RSS from including them together add. This gives a convenient way to see that an expanded model can fit almost exactly while the one-predictor model chosen by a selection rule changes.
Worked Example: A Small Response Change Reverses the Winner
Suppose two centered, orthogonal candidate predictors satisfy \(x_1^\mathsf{T}x_1=x_2^\mathsf{T}x_2=4\). Initially, let \(y=x_1+x_2+q\), where \(q\) is orthogonal to both predictors and \(q^\mathsf{T}q=0.0002\). Compare the one-predictor models before and after a small response perturbation that changes the response to \(y^*=0.995x_1+1.005x_2+q\). Choose the model with the smaller RSS.
Method. Use the centered-vector RSS formula for each one-predictor model. The orthogonal residual \(q\) contributes to total variation but has zero inner product with either predictor.
Before the perturbation. The centered total sum of squares is \(y^\mathsf{T}y=4+4+0.0002=8.0002\). The inner products are \(x_1^\mathsf{T}y=4\) and \(x_2^\mathsf{T}y=4\). Therefore,
The two one-predictor models are tied. The model containing both predictors has RSS \(q^\mathsf{T}q=0.0002\).
After the perturbation. The new inner products are \(x_1^\mathsf{T}y^*=0.995(4)=3.98\) and \(x_2^\mathsf{T}y^*=1.005(4)=4.02\). The new centered total sum of squares is \(4(0.995^2)+4(1.005^2)+0.0002=3.9601+4.0401+0.0002=8.0004\). Hence,
The selected one-predictor model switches from a tie to \(x_2\). In the two-predictor model, the explained sums of squares are \(3.9601\) and \(4.0401\), for a total of \(8.0002\). Its RSS is \(8.0004-8.0002=0.0002\), the same as before the perturbation. The response change reverses the one-predictor ranking, while the full model’s RSS remains very small.
Common mistake avoided. The full-model RSS is not the response total sum of squares and is not either one-predictor RSS. Subtract the sum of the two orthogonal predictors’ explained sums of squares from the centered total: \(8.0004-8.0002=0.0002\).
Changing the Candidate Predictors
Selection can also be sensitive to which predictors are offered to the procedure. In particular, a new candidate may be a close substitute for an existing predictor. The selected variable can then change even if the new candidate yields a similar fitted direction and a similar fit score. This connects with the earlier material on collinearity and predictor substitution: related predictors can carry overlapping information, making it difficult to distinguish their individual contributions.
Worked Example: A Similar Candidate Crosses a Selection Boundary
Let \(x_1\) and \(x_2\) be centered orthogonal predictors, each with squared length 4, and let \(y=x_1+0.10x_2\). A candidate \(z=x_1+cx_2\) is considered in a one-predictor model. Find the positive value of \(c\) at which \(z\) and \(x_1\) have equal RSS, and determine which model has lower RSS at \(c=0.05\).
Method. Compare the RSS reductions from \(z\) and \(x_1\). Since \(y^\mathsf{T}y=4+4(0.10)^2=4.04\), the model containing \(x_1\) has reduction \(4\) and RSS \(4.04-4=0.04\). For \(z\), \(z^\mathsf{T}y=4(1+0.10c)\) and \(z^\mathsf{T}z=4(1+c^2)\), so its reduction is \(4(1+0.10c)^2/(1+c^2)\).
Equality of RSS is equality of reductions. Therefore,
The roots are \(c=0\) and \(c=0.20/0.99=20/99\approx0.20202\). The positive crossing is \(20/99\), not \(0.20\).
At \(c=0.05\), \(z^\mathsf{T}y=4.02\) and \(z^\mathsf{T}z=4.01\). Its reduction is \(4.02^2/4.01=16.1604/4.01\approx4.030025\), so its RSS is approximately \(4.04-4.030025=0.009975\). The RSS for \(x_1\) is \(0.04\). Thus \(z\) wins at \(c=0.05\).
Check. The crossing equation has the nonzero root \(20/99\), and \(0.05<20/99\). For positive \(c\) below this crossing, the reduction for \(z\) exceeds 4, consistent with its lower RSS at \(c=0.05\).
Common mistake avoided. Do not round the equation to \(c=0.20\) before solving it. The coefficient of \(c^2\) on the left is \(0.01\), so moving terms gives \(0.20c-0.99c^2=0\), not \(0.20c-c^2=0\).
Worked Example: A Tiny Predictor Change Near a Tie
Continue with \(x_1,x_2,y\) from the preceding example. Compare the candidates \(z=x_1+0.20x_2\) and \(w=x_1+0.21x_2\) against \(x_1\). State the selected model at each value and explain why the switch is sensitive to the candidate set.
Method. Use the reduction formula \(4(1+0.10c)^2/(1+c^2)\), and compare each reduction with 4, the reduction for \(x_1\). The positive equality point is \(20/99\approx0.20202\).
At \(c=0.20\), the reduction for \(z\) is \(4(1.02)^2/1.04=4(1.0404)/1.04=4.001538\) to six decimal places. Its RSS is \(4.04-4.001538=0.038462\), which is less than the \(x_1\) RSS of \(0.04\). Thus \(z\) wins.
At \(c=0.21\), the reduction for \(w\) is \(4(1.021)^2/1.0441=4(1.042441)/1.0441\approx3.993644\). Its RSS is approximately \(4.04-3.993644=0.046356\), which is greater than \(0.04\). Thus \(x_1\) wins. The candidate change is only \(0.01x_2\), but it crosses the equality point and reverses the preferred one-predictor model.
Check. The ordering \(0.20<20/99<0.21\) predicts the two winners, and the calculated reductions confirm it. This is sensitivity to the candidate predictor’s definition, not a response-data change.
How to Interpret an Unstable Selection
A changed subset does not necessarily imply a comparably large change in fitted values or predictive performance. If two predictors are similar, one can partly substitute for the other. The model names and coefficient interpretations may change while fitted responses remain close. Conversely, two competing subsets can have nearly equal RSS but produce meaningfully different predictions for some observations. The RSS comparison alone does not settle that question.
When a selected subset is unstable, report the selection rule and the comparison that led to the choice. For close candidates, distinguish the variable-selection result from the strength of evidence for a unique predictor: “this procedure selected \(x_2\) under this data and candidate set” is more precise than claiming that \(x_2\) is definitively important while its alternative is not. A useful diagnostic is to examine whether small plausible changes to the response or predictor definitions alter the selected subset, and whether the competing models’ fit scores change substantially.
- Near-ties invite switches. When two candidates have almost equal fit scores, a small change can reverse their ranking.
- Substitutes complicate interpretation. Related predictors may offer similar fits, so selection of one need not show that the other contains no useful information.
- Separate model identity from fit. A different selected set is not, by itself, evidence of a large change in model performance.
- Keep comparisons like-for-like. Use the same response, observations, and comparison rule when attributing a changed result to a response perturbation or a candidate-predictor change.
A common error is to compare RSS values across models with different numbers of predictors and conclude that the smallest RSS identifies the best subset. Adding predictors cannot increase ordinary least-squares RSS on the same observations, so raw RSS alone rewards larger models. Use the relevant selection rule, and when comparing equal-size subsets, make clear that RSS is being used to rank their fit. The goal here is to recognize sensitivity, not to replace the selection rule with an unqualified RSS comparison.
Check Your Understanding
Use the stated centered-vector settings and show the key RSS calculations before checking each answer.
- Two centered orthogonal predictors each have squared length 4. A response has centered total sum of squares \(8.000275\), and the predictors’ explained sums of squares in the full model are \(3.980025\) and \(4.020025\). Find the full-model RSS. (Answer: \(8.000275-(3.980025+4.020025)=0.000225\).)
- For two centered orthogonal predictors with squared lengths 4, suppose a response update gives \(x_1^\mathsf{T}y^*=3.98\), \(x_2^\mathsf{T}y^*=4.02\), and \(y^{*\mathsf{T}}y^*=8.0004\). Find both one-predictor RSS values and select the better model. (Answer: \(\operatorname{RSS}(x_1)=8.0004-3.98^2/4=4.0403\), \(\operatorname{RSS}(x_2)=8.0004-4.02^2/4=3.9603\); select \(x_2\).)
- Let \(x_1,x_2\) be centered and orthogonal with squared lengths 4, \(y=x_1+0.10x_2\), and \(z=x_1+cx_2\). Find the positive \(c\) where \(z\) and \(x_1\) tie, then choose at \(c=0.05\). (Answer: \(c=20/99\approx0.20202\); at \(c=0.05\), select \(z\).)
- For centered orthogonal predictors with squared lengths 4, let the response total sum of squares be \(8.0002\), with inner products \(x_1^\mathsf{T}y=4.01\) and \(x_2^\mathsf{T}y=3.99\). Calculate the one-predictor RSS values and identify the winner. (Answer: \(\operatorname{RSS}(x_1)=8.0002-4.01^2/4=3.980175\), \(\operatorname{RSS}(x_2)=8.0002-3.99^2/4=4.020175\); select \(x_1\).)
- In the setting \(y=x_1+0.10x_2\) above, compare \(z=x_1+0.20x_2\) and \(w=x_1+0.21x_2\) with \(x_1\). Which wins at each value? (Answer: \(z\) wins at \(c=0.20\) with RSS approximately \(0.038462\); \(x_1\) wins at \(c=0.21\), since \(w\) has RSS approximately \(0.046356\) versus \(0.04\).)