What Does “Selected” Mean?
A subset-selection result is easy to overstate. If a procedure returns a model containing two predictors, that tells us which model the procedure chose from the options it considered, under its chosen comparison rule. It does not, by itself, show that those predictors are uniquely important, that no alternative would work as well, or that either predictor causes changes in the response.
To interpret the result, keep three ingredients in view: the candidate set of predictors, the search procedure that determines which models are considered, and the comparison rule used to rank those models. A change in any of these can change the selected subset.
For example, the candidate set might contain experience, a risk classification, and a regional indicator. A search procedure might examine every one-predictor model, or it might follow a stepwise path that examines only a fraction of all possible subsets. A comparison rule might rank models by a specified fit-and-size score. The phrase “the selected model” has a precise interpretation only when these choices are understood.
A Regression Setup for Interpreting Selection
Suppose the response \(Y_i\) is modeled using a subset \(M\) of candidate predictors. For a linear regression model with an intercept, write
Here, \(Y_i\) is the response for observation \(i\), \(x_{ij}\) is its value of predictor \(j\), \(\beta_0\) and the \(\beta_j\) are regression coefficients, and \(\varepsilon_i\) is the error. In the examples below, models are fitted by ordinary least squares on the same observations. We assume independent errors with mean zero and finite variance; normality is not needed to calculate or compare the residual sums of squares shown. These conditions do not turn a selected model into a causal explanation.
A comparison score must be interpreted according to its own definition. In some examples below, the rule is to choose the model with the smallest residual sum of squares (RSS), but only among models of a specified size. Elsewhere, the problem supplies a score directly and says lower is better. A model selected under one rule need not be selected under another.
Worked Examples
Worked Example: Adding a Candidate Changes the Selection
A hypothetical analyst fits one-predictor least-squares models to the same 60 observations. The candidate predictors are \(A\), \(B\), and \(C\). The table gives each model’s RSS. The rule is to select the one-predictor model with the smallest RSS. What is selected from candidate set \(\{A,B\}\), and what is selected when the set is expanded to \(\{A,B,C\}\)?
| One-predictor model | RSS |
|---|---|
| \(A\) | 84 |
| \(B\) | 79 |
| \(C\) | 73 |
Method. Hold the response observations, model size, and comparison rule fixed. Compare only the models allowed by each candidate set.
With candidates \(A\) and \(B\), the available scores are 84 and 79. Since \(79<84\), model \(B\) is selected. After \(C\) is admitted, the available scores are 84, 79, and 73. Since \(73<79<84\), model \(C\) is selected.
Check. Both answers minimize RSS within their respective candidate sets. The conclusion is not that \(B\) or \(C\) is universally best; each is best only among the available one-predictor models under this rule and on these observations.
Common mistake avoided: Reporting “\(B\) is the selected predictor” without saying which predictors were candidates. The expanded candidate set changes the answer even though the data, model size, and comparison rule are unchanged.
Worked Example: A Search Path Can Miss the Best Subset
A hypothetical analyst has three candidate predictors \(A\), \(B\), and \(C\), and must return a model containing exactly two predictors. All models are fitted on the same observations. The score is RSS, with lower preferred. The table shows the supplied RSS values for every subset.
| Subset | RSS | Subset | RSS |
|---|---|---|---|
| Empty model | 120 | \(A\) | 91 |
| \(B\) | 88 | \(C\) | 95 |
| \(AB\) | 82 | \(AC\) | 79 |
| \(BC\) | 81 | \(ABC\) | 76 |
First consider exhaustive comparison at the required size. The three eligible two-predictor scores are \(82\) for \(AB\), \(79\) for \(AC\), and \(81\) for \(BC\). Their minimum is 79, so exhaustive size-two selection returns \(AC\).
Now consider forward selection that begins with the empty model and adds the predictor giving the smallest RSS at each step, stopping after two predictors. At the first step, the single-predictor RSS values are 91 for \(A\), 88 for \(B\), and 95 for \(C\). Since \(88<91\) and \(88<95\), it first selects \(B\). At the second step, the two possible additions give \(AB\) with RSS 82 and \(BC\) with RSS 81. Since \(81<82\), the procedure returns \(BC\).
Check. Forward selection returns a score of 81, while exhaustive size-two comparison finds a score of 79 for \(AC\). Thus this particular forward path does not reach the best two-predictor model in the table.
Method and common mistake avoided. Follow the specified path for the greedy search, then separately compare all eligible subsets for the exhaustive result. It would be incorrect to call \(BC\) the best two-predictor model overall: it is best only along this search path.
Worked Example: The Comparison Rule Affects the Result
A hypothetical modeler compares two one-predictor models, \(P\) and \(Q\), fitted to the same response observations. Two different comparison rules are supplied, and lower scores are preferred under each rule. Which model is selected under each rule?
| Model | Rule 1 score | Rule 2 score |
|---|---|---|
| \(P\) | 40.2 | 13.4 |
| \(Q\) | 40.0 | 13.7 |
Under Rule 1, compare 40.2 for \(P\) with 40.0 for \(Q\). Since \(40.0<40.2\), Rule 1 selects \(Q\). Under Rule 2, compare 13.4 for \(P\) with 13.7 for \(Q\). Since \(13.4<13.7\), Rule 2 selects \(P\).
For additional context, suppose the fitted models give predictions of 90, 110, and 130 for three specified cases under \(P\), and 91, 109, and 130 under \(Q\). The absolute prediction differences are \(|90-91|=1\), \(|110-109|=1\), and \(|130-130|=0\). Their mean is \((1+1+0)/3=2/3\), or approximately 0.667 to three decimals. The models can therefore make close predictions for these cases while still being ranked differently by the two rules.
Check. Rule 1 selects \(Q\); Rule 2 selects \(P\). The prediction comparison is \(0.667\) on average for the three cases, calculated to three decimals.
Common mistake avoided: Treating “selected” as an intrinsic property of the data. The selection depends on the stated comparison rule, and the result alone does not establish that the chosen predictor has a unique scientific or causal role.
Why the Candidate Set and Search Matter
The dependence on the candidate set can be stated directly. Suppose the same score \(Q(M)\) is used throughout, lower is better, and the eligible collection changes from \(\mathcal{M}_1\) to a larger collection \(\mathcal{M}_2\), where \(\mathcal{M}_1\subseteq\mathcal{M}_2\). Then
This inequality holds because every model in \(\mathcal{M}_1\) is still available in \(\mathcal{M}_2\), and the larger collection may also contain a model with a lower score. The inequality concerns the best score; it does not guarantee that the identity of the selected model stays the same. In Example 1, the best score changed from 79 to 73 when \(C\) became eligible.
Likewise, a search procedure may evaluate only part of the full model collection. If it follows a particular path or stops at a chosen size, its returned subset reflects those restrictions. In Example 2, a forward path settled on \(BC\), although the exhaustive search found a lower-RSS size-two subset, \(AC\). The procedure’s output remains a valid description of what that search returned, but it is not evidence that the search examined or rejected every other subset.
List the predictors, or predictor blocks, that were eligible. A predictor not offered to the procedure cannot appear in its selected subset.
Determine whether all permitted subsets were compared or whether the procedure followed a restricted path and stopped at a specified point.
State which score was used, whether smaller or larger values are preferred, and how ties were handled.
Report the selected subset as the result under those candidates, search choices, and comparison rule—not as a universal ranking of predictors.
What Selection Does—and Does Not—Say About Importance
The word “important” can refer to different things: improving a model’s prediction under a specified rule, being associated with the response after adjustment for other predictors, or having a causal effect. A subset-selection result addresses only the comparison that was carried out. It is not, by itself, a test of a causal claim or proof that the included variables are uniquely important.
Predictors can also represent overlapping information. If two candidate predictors are substitutes, a selection procedure may include one and omit the other even when either could represent much of the same signal. Selection of one member of such a pair does not show that the omitted predictor contains no information. Nor does the selected subset alone establish that the included predictor would remain selected under a different candidate set, search, or rule.
A careful interpretation is therefore conditional: “Among the listed candidates, the specified procedure returned this subset under the stated comparison rule.” That wording is more informative than an unqualified claim that the model “found the important variables.” It also makes clear what additional evidence would be needed for a broader scientific or predictive conclusion.
Check Your Understanding
For each problem, identify what the selection result establishes and avoid conclusions beyond the stated candidates, search, and rule.
- A one-predictor search uses candidates \(R\) and \(S\), with scores 52 and 49, respectively; smaller is better. A third predictor was not offered to the search. Which predictor is selected, and what can be concluded about the unoffered predictor? (Answer: \(S\) is selected from \(R\) and \(S\); nothing about the unoffered predictor’s score or importance is established.)
- A two-predictor search must return exactly two of \(A,B,C\). The scores are \(AB=34\), \(AC=32\), and \(BC=33\), with smaller preferred. What does exhaustive comparison select? (Answer: \(AC\), with score 32.)
- A greedy procedure evaluates only \(AB\) and \(BC\), with scores 18 and 17. A separate unvisited model \(AC\) has score 15. What does the greedy procedure select, and which model has the lowest score among the three stated pairs? (Answer: the procedure selects \(BC\); \(AC\) has the lowest score among the three pairs.)
- With the same candidates and two rules that both prefer smaller scores, \(U\) and \(V\) have scores 7.8 and 8.1 under Rule A, and 12.4 and 12.0 under Rule B. Which model does each rule select, and does the result establish one universally best model? (Answer: Rule A selects \(U\), Rule B selects \(V\); no universal best model is established.)
- A procedure compares only the one-predictor models \(D\) and \(E\), selecting \(D\). A second procedure allows all subsets and selects \(EF\) under the same stated score, where \(EF\) has a lower score than \(D\). Give the careful interpretation of both results. (Answer: \(D\) is best among the one-predictor models \(D\) and \(E\) under the first procedure; \(EF\) is selected among the subsets allowed by the second. The first result does not show that \(D\) is best among all subsets.)