Defining Candidate Model Sets

Why Define the Candidate Set First?

A model-selection procedure can compare only the models it has been given permission to consider. If an important predictor is absent from that collection, no search can select it. If a categorical predictor is represented by separately selectable columns rather than as one term, the procedure may consider partial versions of its effect. The candidate set is therefore part of the analysis design, not a neutral detail that can be postponed until after selection.

To make a candidate set clear, specify the response, the eligible predictor terms, and the rules that determine which combinations are allowed. These choices establish the scope of what selection can discover. They do not, by themselves, determine which model is best or how a search will be carried out.

Defining the Candidate Model Set

Consider a multiple linear regression setting with \(n\) observations. Let \(Y\) be the response, and let the eligible predictor terms be indexed by \(j=1,\ldots,p\). A candidate model uses an allowed subset \(S\) of those terms. The intercept is included in every model and is not counted among the \(p\) optional terms.

Y_i=\beta_0+\sum_{j\in S}\beta_j x_{ij}+\varepsilon_i,\qquad i=1,\ldots,n

Here \(x_{ij}\) denotes the value or encoded contribution for term \(j\) on observation \(i\), and \(\beta_j\) is its coefficient or coefficient vector, as appropriate. The error is \(\varepsilon_i\). A model with \(S=\varnothing\) contains the intercept alone, sometimes called the intercept-only model.

Definition: A candidate model set is a specified collection of regression models eligible for consideration. It is defined by (1) the response and analysis observations, (2) the predictor terms eligible for inclusion, and (3) the permissible combinations of those terms. The models must use a clearly specified coding and model form.

For the usual least-squares linear model, the working assumptions are that the response is modeled as a linear combination of the included terms plus errors with conditional mean zero and constant variance, and that errors from different observations are independent. Normal errors are additionally assumed when exact small-sample normal-theory inference is required. A unique least-squares coefficient estimate also requires the included design-matrix columns to be linearly independent. Defining a candidate set does not establish these assumptions: they concern the data and model fit, and must be considered separately.

A predictor term is a scientifically defined contribution that may be eligible for a model. A term may use one coefficient, as with a single quantitative predictor, or several coefficients, as with a multi-level categorical predictor. It is important to say whether selection is allowed to include or omit a term as a whole or to choose among its encoded columns separately. That decision changes the collection of candidate models.

Counting Unrestricted Subsets

If there are \(p\) optional terms and every combination is permitted, each term has two choices: include it or omit it. The intercept is present regardless. The number of subsets, and hence the number of candidate models, is:

$$ 2^p $$

One way to justify this count is to represent each model by a string of \(p\) binary choices. For example, with three terms, 101 means include terms 1 and 3 and omit term 2. There are two choices at each of \(p\) positions, so the multiplication rule gives \(2\times 2\times\cdots\times 2=2^p\) distinct strings. Each string identifies exactly one subset.

This count treats each specified term as one inclusion unit, not necessarily one coefficient. If a factor is one selectable term represented by multiple columns, it contributes one binary choice to \(p\). If its columns are independently selectable, they contribute separate choices and define a different candidate set.

Worked Example: Four Optional Predictors

Suppose a hypothetical dataset contains 48 independent clinic visits. The response is systolic blood pressure in mmHg. The four eligible quantitative predictor terms are age in years, body mass in kg, weekly exercise hours, and daily sodium intake in mg. Assume the analysis uses the same 48 visits for every model, the specified predictor columns are full rank for each model in which they appear, and the least-squares error assumptions are suitable for the intended analysis. These assumptions are stipulated for the example; the model count alone cannot verify them.

If all combinations are allowed, then \(p=4\), and the number of candidate models is:

2^p=2^4=2\times2\times2\times2=16

The 16 models comprise the intercept-only model, four models with one predictor, six with two predictors, four with three predictors, and one with all four:

$$ 1+4+6+4+1=16 $$

The intercept-only model counts as one candidate. Counting only models with at least one predictor would instead give \(16-1=15\). Neither count is universally correct without a stated rule; the candidate-set definition determines whether the intercept-only model is eligible.

Permissible Combinations and Restricted Counts

A candidate set may deliberately allow only some subsets. For example, an analysis plan may limit the number of terms in a model, or a scientific requirement may rule out certain combinations. The permitted collection can be written as a family \(\mathcal{A}\) of subsets of the eligible terms. The candidate models are then exactly the models \(M_S\) for which \(S\in\mathcal{A}\).

If every subset of size at most \(k\) is allowed, the count is the sum of the numbers of subsets of each permitted size:

\sum_{r=0}^{k}\binom{p}{r}

Here \(\binom{p}{r}\) is the number of ways to choose \(r\) distinct terms from \(p\). The term for \(r=0\) includes the intercept-only model. This formula counts eligible models; it does not prescribe a search algorithm.

Worked Example: Limiting Model Size

Suppose a hypothetical study has 36 independent observations on a response measured in minutes and five eligible quantitative predictor terms. For a planned comparison, only models containing at most two optional terms are permitted. Assume the same observations are used in every model, each included design is full rank, and the error conditions for the intended least-squares analysis are reasonable. These conditions concern the model analysis, not the counting rule.

There is one model with zero optional terms, five choices with one term, and \(\binom{5}{2}=10\) choices with two terms. Thus:

\sum_{r=0}^{2}\binom{5}{r} =\binom{5}{0}+\binom{5}{1}+\binom{5}{2} =1+5+10=16

If no size restriction were imposed, the same five terms would yield \(2^5=32\) subsets. The restriction therefore reduces the candidate set from 32 to 16 models. It also means that no model with three, four, or five terms can be selected, regardless of how well such a model might describe the response.

Coding Choices Change What Can Be Discovered

Categorical predictors make the distinction between terms and coefficient columns especially important. A factor with three levels can be represented using two indicator columns alongside an intercept. If the whole factor is treated as one selectable term, it is either included as a two-coefficient contribution or omitted. If the two columns are eligible independently, the candidate set also includes models containing only one of those columns. Those are different model specifications, with different possible discoveries.

Some coding choices are merely alternative ways to represent the same model when all the relevant columns are included. But if the selection unit changes from a complete factor to individual columns, the set of permitted fitted relationships can change. The analyst should therefore specify the coding and the unit of selection before counting candidate models. A count based only on the number of coefficients can be wrong when terms have different numbers of coefficients.

Worked Example: A Three-Level Categorical Predictor

Suppose a hypothetical dataset has 60 independent houses. The response is monthly electricity use in kWh. The eligible predictors are neighborhood, with three levels, and two quantitative terms: floor area in square meters and outdoor temperature in degrees Celsius. Assume all three neighborhoods are represented, the encoded columns are full rank when included, and the usual least-squares error assumptions are appropriate for the intended analysis.

Whole-factor selection. With a reference-level coding, neighborhood uses two indicator columns plus the intercept. If the entire neighborhood factor is one selectable term, there are three optional terms altogether: neighborhood, floor area, and temperature. The candidate count is:

$$ 2^3=8 $$

Independent-column selection. If the two neighborhood indicators are instead eligible as separate terms, there are four optional terms: two indicators, floor area, and temperature. The count becomes:

$$ 2^4=16 $$

The second specification allows models with just one neighborhood indicator; the first does not. Thus 8 and 16 are both correct for their respective definitions, but they describe different candidate sets. A statement such as “there are three predictors” is not enough unless it clarifies whether the factor is selected as a single term.

Exclusions Define the Boundary of Selection

A predictor can be excluded for a reason established before searching: it may not be measured in the analysis sample, may be outside the scientific question, or may not be usable under the planned model form. Once excluded, it cannot enter any candidate model. Likewise, a transformation or a new predictor term that was never included among the eligible terms cannot be discovered by comparing the specified models.

This is why candidate-set design matters for interpretation. A selected model is selected from the permitted options, not from every model that could conceivably be written. A narrow set can make the search manageable and scientifically focused, but it can also omit useful alternatives. State the response scale, analysis observations, eligible terms, coding, and combination restrictions so readers can see what was and was not available.

Common Mistake: Treating a Search as Open-Ended

Wrong reasoning: “The procedure selected the best predictors, so any important predictor would have appeared.”
Correction: A procedure can select only from its candidate set. If a predictor was excluded or encoded in a way that prevents its intended effect from appearing as a permitted model, selection cannot recover it.
Wrong reasoning: “There are four coefficients, so there must be \(2^4\) candidate models.”
Correction: Count the selectable terms and permitted combinations. One categorical term may require multiple coefficients but still have one inclusion choice; independent selection of those coefficient columns defines a different set.
Wrong reasoning: “The unrestricted count is always \(2^p\).”
Correction: \(2^p\) applies only when all subsets of \(p\) optional terms are allowed. With restrictions, count only the permitted subsets.

Key takeaway: Before comparing regression models, define the response and analysis sample, identify the eligible predictor terms, specify their coding and selection units, and state which combinations are permitted. The resulting candidate set determines the scope of every selection result.

Check Your Understanding

Answer each question by referring to the specified candidate set.

There are six optional terms, and every subset is allowed. How many candidate models are there if the intercept-only model is included?
With five eligible terms, only models containing at most one optional term are allowed. Write the count as a sum of binomial coefficients and evaluate it.
A four-level factor is represented by three indicator columns. Explain why treating the factor as one selectable term differs from selecting its columns independently.
A relevant predictor was not included among the eligible terms. Can any search over the candidate set select it? Explain.
Why must a candidate-set description state whether the intercept-only model is permitted?

What You'll Learn