Why Constrain a Predictor Search?
A regression search does not have to consider every possible combination of every recorded variable. Some predictors must remain for scientific or design reasons; others should not be considered at all. A categorical factor can also need special treatment: its several indicator coefficients represent one predictor concept, so selecting only some of them can make the model depend on the arbitrary choice of reference category.
These rules define which models are eligible before a search algorithm or model-comparison criterion is applied. They do not replace the choice of algorithm or criterion. Instead, they make sure that the search operates on models that answer the intended question.
To make the rules precise, let \(\mathcal{M}\) be the set of candidate models and let \(Q(M)\) be the selection criterion for a model \(M\). The search compares only models in \(\mathcal{M}\). For example, if \(F\) is the set of forced predictors, \(E\) is the set of excluded predictors, and \(G\) is a group of terms, eligibility requires every model to include \(F\), include none of \(E\), and contain either all of \(G\) or none of \(G\). The criterion then ranks the eligible models; it does not decide which restrictions to impose.
Model and Assumptions
For the regression calculations below, suppose the outcome \(Y_i\) for observation \(i\) follows a linear model with independent, normally distributed errors of constant variance:
The conditional mean is assumed to have the stated linear form, and the error variance is the same for all observations. The exact partial F test also relies on the normal-error assumption. Candidate models must be fit to the same observations, and their included terms must not be perfectly linearly dependent. In practice, the model form and residual behavior should be checked; the calculations do not make these assumptions true.
For a categorical predictor with \(L\) levels, a standard reference-level coding uses \(L-1\) indicator variables. Together those indicators represent the factor's contribution relative to the reference level. Treating them as a group means either all \(L-1\) indicators are present or none are. Different valid codings can describe the same factor effect, but the group decision must apply to the full factor, not to selected individual indicators.
Worked Example: Excluding a Predictor and Forcing an Adjustment
Suppose a hypothetical set of 24 observations records an outcome measured after a laboratory task, a required baseline temperature \(T\), optional humidity \(H\), optional room size \(R\), and a post-task measurement \(P\). The analysis plan requires adjustment for baseline temperature and excludes \(P\), because it is measured after the outcome. Assume diagnostic checks for this hypothetical analysis do not show a serious conflict with the linear-model assumptions.
The candidate models must all include \(T\), must not include \(P\), and may include \(H\), \(R\), both, or neither. Suppose least-squares fits give residual sums of squares (RSS) of 120 for the model with \(T\), 90 for the model with \(T,H\), and 108 for the model with \(T,R\). For illustration, compare these three models using the Gaussian AIC score \(n\ln(\mathrm{RSS}/n)+2k\), omitting a common constant. Here \(k\) counts the regression coefficients, including the intercept, and the error variance.
The required-only model has an intercept and one slope, plus the variance parameter, so \(k=3\):
Adding humidity gives \(k=4\):
Adding room size instead also gives \(k=4\):
Of these three eligible models, the humidity model has the lowest AIC. This comparison does not consider a model without \(T\) or a model containing \(P\): those models are ineligible, regardless of their fit. A full search would also compare the eligible model containing both optional predictors; the three calculations here illustrate how the restrictions operate.
Grouped Terms and the Partial F Test
A partial F test compares a reduced model with a larger model that adds a specified set of terms. For a factor, the reduced model omits the entire factor and the larger model adds all its indicator variables. If the factor has \(L\) levels under reference-level coding, the number of added coefficients is \(q=L-1\).
The numerator is the reduction in residual sum of squares per added coefficient. The denominator is the full model's residual mean square, an estimate of the error variance. Thus the statistic compares the improvement from adding the group with the model's remaining unexplained variation. A large value is evidence against the joint null hypothesis; it is not a test of each coefficient separately.
Worked Example: Testing a Four-Level Factor as One Group
In a hypothetical analysis of \(n=36\) patients, an outcome is modeled using a required age adjustment and a four-level clinic factor. Assume the observations are independent, the linear mean and constant-variance assumptions are reasonable, and the normal-error assumption is adequate for the exact test. The reduced model includes age but not clinic; its RSS is 240. The full model adds all three clinic indicators and has RSS 198. The full model has an intercept, one age coefficient, and three clinic coefficients, so \(p_F=5\) and \(df=36-5=31\).
The joint null hypothesis is that all three clinic coefficients are zero. The test statistic is
Compared with an \(F\) distribution with 3 and 31 degrees of freedom, this gives a p-value of about 0.11. At significance level \(\alpha=0.05\), the test does not provide sufficient evidence to add the clinic factor under this entry rule. The decision is about the factor as a whole: the search does not keep one or two clinic indicators while dropping the rest. This result alone does not establish that clinic has no association with the outcome.
Implementing the Rules Together
A practical way to define the search is to list the forced terms, excluded terms, and optional groups before fitting candidate models. Then generate only combinations that respect those rules. For instance, if a required adjustment variable is \(Z\), a two-indicator factor is \(G\), and a numeric predictor is \(W\), eligible models have the form \(Z\), possibly together with \(W\), and either both indicators for \(G\) or neither. A prohibited variable \(V\) is absent from every one of these models.
The adjustment variable is retained even if its fitted coefficient is small or its individual p-value is large. Its role in the analysis is the reason for forcing it; its entry is not conditional on passing the search criterion. Likewise, a factor's indicators are assessed as a set. Selection rules should not quietly change the meaning of the candidate model halfway through the search.
Record which predictors are mandatory, which are prohibited, and which terms form indivisible groups. Give a substantive reason for each restriction.
Include every forced predictor, omit every excluded predictor, and include or omit each grouped set as a whole. Check that each model can be fit without perfect collinearity.
Use the specified algorithm and compare or test only eligible models. The restrictions do not, by themselves, select a criterion or guarantee a unique result.
Name forced and excluded predictors, describe grouped terms, state the criterion or test used, and identify the selected model in context.
Worked Example: Selecting a Factor Alongside a Required Adjustment
Consider a hypothetical analysis of 40 observations on an outcome, baseline age \(A\), a three-level service type, a numeric severity score \(S\), and a measurement taken after the outcome \(Q\). The plan forces age as an adjustment variable and excludes \(Q\). Service type is represented by two indicators and is treated as one group; severity is optional. Suppose diagnostic review finds no major departure from the assumed linear-model conditions. The following RSS values are hypothetical least-squares results for the four eligible models:
| Eligible model | RSS | Regression coefficients, including intercept |
|---|---|---|
| Age | 400 | 2 |
| Age and service type | 350 | 4 |
| Age and severity | 360 | 3 |
| Age, service type, and severity | 320 | 5 |
Use the same Gaussian AIC score as before, counting the error variance in \(k\). Thus \(k\) is one more than the number of regression coefficients in the table. For the age-only model, \(k=3\), giving
For age and service type, \(k=5\):
For age and severity, \(k=4\):
For the model containing both optional predictors, \(k=6\):
The model with age, the complete service-type group, and severity has the smallest AIC among the four eligible models. Age remains in that model because it was forced, not because AIC selected it. The comparison neither considered the excluded post-outcome measurement nor allowed a model with just one service-type indicator. In this hypothetical example, the selected model describes conditional associations; the selection itself does not establish causal effects.
Why the Restrictions Matter
Constraints can preserve the purpose of an analysis. A variable may be required because the analysis plan calls for adjustment, or excluded because it would not be available when predictions are made. Grouping protects the interpretation of a categorical factor from arbitrary coding choices. These decisions should be made and documented before interpreting the winning model: the selected model is best only among the models the search was allowed to consider.
Common mistake: A search finds that one indicator for a four-level factor improves the criterion, so the analyst keeps that indicator but drops the other indicators. This treats a multi-level factor as though one particular comparison had been selected in advance. Which level's indicator appears can depend on the reference category, and the resulting model no longer represents the factor as a single grouped predictor. Correction: define the factor as a group of all its coding terms and enter or remove that group together. Similarly, do not remove a forced adjustment just because its individual test is not significant, and do not restore an excluded predictor merely because it improves fit.
Check Your Understanding
Answer each question by applying the restrictions to the candidate models, not by assuming that the best-fitting model is automatically eligible.
- A required adjustment variable has a large p-value in every candidate model. Under a forced-predictor rule, may it be removed? Explain.
- A categorical factor has five levels and is represented by four indicators. What does it mean to group the factor during selection?
- A predictor measured after the outcome is excluded. Can it be considered if adding it substantially lowers RSS?
- In a partial F test for adding a three-indicator factor, what is the numerator degrees of freedom, and what joint null hypothesis is tested?
- Why should a report describe forced, excluded, and grouped predictors as well as the selected model?