Why Constrain the Search?
Automated selection can compare many candidate regression models, but not every possible combination of terms is equally interpretable. For example, a model may include an interaction between two predictors while omitting both predictors’ individual terms. Or it may include a squared predictor without its linear term. These models can be fitted mathematically, but their meaning can depend awkwardly on how the predictors are measured or centered.
Hierarchy is a rule for which combinations of terms are admissible in a model search. It requires selected higher-order terms to be accompanied by specified lower-order terms. The rule does not decide which admissible model fits best; it changes the candidate set that an algorithm is allowed to search.
Terms and Hierarchy Rules
Consider a regression model with quantitative predictors \(X\) and \(Z\). The main effects are the terms \(X\) and \(Z\); the two-way interaction is \(XZ\). In the model \(Y=\beta_0+\beta_X X+\beta_Z Z+\beta_{XZ}XZ+\varepsilon\), the interaction allows the association between \(X\) and the mean response to vary with \(Z\), and vice versa. If \(Z\) is held fixed, the slope with respect to \(X\) is \(\beta_X+\beta_{XZ}Z\).
For example, the model containing \(X\), \(Z\), and \(XZ\) satisfies both rules. The model containing \(X\) and \(XZ\), but not \(Z\), satisfies weak hierarchy only. A model containing \(XZ\) alone satisfies neither. A model with just the two main effects satisfies both, because no interaction is present.
The same principle applies to polynomial terms. A squared term \(X^2\) is a higher-order term built from \(X\), so a hierarchical polynomial model that includes \(X^2\) also includes the linear term \(X\). For a cubic term, a common hierarchy rule requires the lower powers \(X\) and \(X^2\) as well. With a single underlying predictor, there is only one immediate lower-order parent for \(X^2\), so the strong-versus-weak distinction used for two-predictor interactions does not create two different choices.
For higher-order interactions, conventions about weak hierarchy can differ between procedures. One practical convention is to regard the immediate lower-order terms formed by deleting one factor as the parent terms: strong hierarchy requires all such parents, and weak hierarchy requires at least one. A search must state the convention it uses. The examples here focus on two-way interactions and a squared term.
How Constraints Change the Candidate Set
A candidate model is a specified set of terms. A selection procedure, such as forward selection or backward elimination, moves among candidate models using a rule for adding or removing terms. Hierarchy filters those moves: a term may be statistically eligible under the search rule but still be unavailable because adding or removing it would violate the chosen hierarchy constraint.
For \(XZ\), the parents are \(X\) and \(Z\). For \(X^2\), the lower-order parent is \(X\).
For a two-way interaction, decide whether both parents (strong) or at least one parent (weak) must accompany it.
Allow only additions and removals that leave the resulting model within the permitted candidate set.
Among the admissible models or moves, use the specified criterion or entry and removal rule to make the search decision.
This is a constraint on the search, not a new selection criterion. If the search uses an information criterion, hierarchy restricts which models can be compared. If it uses an entry rule, hierarchy restricts which terms can enter at a given step. A term can therefore be unavailable even when its addition would improve the chosen fit statistic.
The regression framework is \(Y_i=\beta_0+\sum_j \beta_j T_{ij}+\varepsilon_i\), where each \(T_{ij}\) is a candidate term such as \(X\), \(Z\), or \(XZ\). For standard regression inference, the usual working assumptions include a correctly specified conditional mean, independent errors with mean zero and constant variance; normal errors are additionally used for exact small-sample \(t\) and \(F\) procedures. Hierarchy itself is a restriction on the terms, not an additional error-distribution assumption. Automated selection can be carried out without claiming that hierarchy proves the model is true.
Worked Example: Strong and Weak Hierarchy With an Interaction
Suppose a hypothetical data set contains \(n=12\) observations. A candidate search considers two quantitative predictors \(X\) and \(Z\), their interaction \(XZ\), and an intercept. For illustration, compare models using the score \(S=n\ln(RSS/n)+2k\), where \(RSS\) is residual sum of squares and \(k\) is the number of regression coefficients, including the intercept. Lower scores are preferred. All models use the same observations.
The hypothetical model-fit summaries are: the additive model \(\{X,Z\}\) has \(RSS=20\) and \(k=3\); the model \(\{X,XZ\}\) has \(RSS=17\) and \(k=3\); and the model \(\{X,Z,XZ\}\) has \(RSS=15\) and \(k=4\). The term braces list the included predictors, with the intercept counted separately in \(k\).
Under strong hierarchy, \(\{X,XZ\}\) is inadmissible because it includes \(XZ\) but omits \(Z\). The additive model is admissible, as is the model with both main effects and the interaction. Their scores are
The second score is lower, so among these strong-hierarchy candidates it is preferred by the stated score. Under weak hierarchy, \(\{X,XZ\}\) is also admissible. Its score is
Among the three listed models, that weak-hierarchy candidate has the lowest score. This comparison illustrates how changing the constraint changes the candidate set and can change the selected model. It does not show that weak hierarchy is always preferable: the rule should be chosen before interpreting a search result, and the score only compares the models presented.
Hierarchy During a Stepwise Search
In a forward search, a predictor or term is added to the current model at each step. Hierarchy affects the eligible additions. Under strong hierarchy, an interaction cannot enter until both of its main effects are already present. Under weak hierarchy, it can enter once either main effect is present. Under either rule, adding an interaction does not remove the need to respect the rule in the resulting model.
Backward elimination has a corresponding restriction. If a model contains \(X\), \(Z\), and \(XZ\), removing \(X\) while retaining \(XZ\) would violate strong hierarchy. A strong-hierarchy procedure must first remove the interaction or otherwise make a move that keeps the resulting model admissible. Under weak hierarchy, removal of one main effect may be allowed if the other main effect remains with the interaction. In all cases, the algorithm’s entry and removal rules operate only among moves that preserve the constraint.
Worked Example: Eligible Moves in Forward Selection
A hypothetical forward search begins with \(X\) already in the model, perhaps because it was required by the analysis plan. It may consider adding \(Z\), \(W\), or the interaction \(XZ\). Suppose fitting the candidate models to the same 30 observations gives these residual sums of squares: \(RSS(X)=70\), \(RSS(X,Z)=50\), \(RSS(X,W)=60\), \(RSS(X,XZ)=40\), \(RSS(X,Z,W)=45\), and \(RSS(X,Z,XZ)=35\). At each step, this illustration selects the eligible addition giving the smallest resulting RSS.
With strong hierarchy, \(XZ\) is not eligible at the starting model \(\{X\}\), because \(Z\) is absent. The eligible additions are \(Z\) and \(W\). Since \(50<60\), the search adds \(Z\), giving \(\{X,Z\}\) with \(RSS=50\).
At \(\{X,Z\}\), both main effects are present, so \(XZ\) becomes eligible under strong hierarchy. The search can compare adding \(W\), which gives \(RSS=45\), with adding \(XZ\), which gives \(RSS=35\). Since \(35<45\), it adds \(XZ\). The resulting model \(\{X,Z,XZ\}\) satisfies strong hierarchy.
With weak hierarchy, \(XZ\) is already eligible at the starting model because \(X\) is present. The three possible resulting RSS values are \(50\) for adding \(Z\), \(60\) for adding \(W\), and \(40\) for adding \(XZ\). Thus this illustrative rule adds \(XZ\) first. This path reaches an interaction without first adding \(Z\), a move unavailable under strong hierarchy. The numerical comparisons illustrate eligibility and path changes; an actual procedure may use a different entry rule or stopping criterion.
Why Lower-Order Terms Matter
A useful justification for including lower-order terms is that it can make an interaction model stable in meaning when predictors are recentered. Recenter \(X\) and \(Z\) by defining \(X'=X-a\) and \(Z'=Z-b\). Then \(X=X'+a\) and \(Z=Z'+b\). A model containing both main effects and their interaction can be rewritten in terms of \(X'\) and \(Z'\) without changing its fitted values:
The recentered expression still has an intercept, both main effects, and the interaction. If a model contains \(XZ\) but excludes both \(X\) and \(Z\), its form is \(\beta_0+\beta_{XZ}XZ\). Replacing \(X\) and \(Z\) by \(X'+a\) and \(Z'+b\) produces terms proportional to \(X'\) and \(Z'\), which that restricted model cannot generally represent. Thus the fitted relationship can depend on the chosen zero points. This algebra motivates hierarchy as a modeling convention; it does not make nonhierarchical models impossible to fit.
Worked Example: A Quadratic Term After Centering
Suppose a hypothetical fitted mean response is \(\hat{Y}=10+2X+0.5X^2\), with \(X\) measured in a chosen unit. Let \(t=X-4\), so \(X=t+4\). Substituting and expanding gives
The same fitted curve, expressed using the centered predictor \(t\), still has a linear term and a squared term. Now consider the nonhierarchical form \(\hat{Y}=10+0.5X^2\). After the same substitution,
A model containing only an intercept and \(t^2\) cannot represent this expression in general because it has no linear \(t\) term. Including \(X\) alongside \(X^2\) therefore allows the quadratic model to retain its form when the predictor’s zero point changes. The coefficients themselves change with centering; the fitted curve does not.
Common Mistakes and Practical Choices
- Confusing a search constraint with evidence. Requiring main effects alongside an interaction does not prove those effects are individually important. It defines which models the search considers.
- Assuming every model must obey hierarchy. Hierarchy is a commonly useful modeling convention, not a mathematical condition required for least-squares estimation. If a nonhierarchical model is used, its interpretation and dependence on coding or centering need care.
- Applying the rule only when adding terms. A removal can also create a violation. Check the resulting model after each addition or deletion.
- Forgetting polynomial parents. A model with \(X^2\) but without \(X\) is nonhierarchical under the usual polynomial rule, even though there is no second predictor involved.
- Treating strong and weak hierarchy as identical. For an \(XZ\) interaction, \(\{X,XZ\}\) is allowed by weak hierarchy but not strong hierarchy.
Before running a constrained automated search, specify the candidate terms, identify their lower-order parents, state the hierarchy convention, and decide how forced or excluded terms interact with that rule. For instance, forcing \(XZ\) into a strong-hierarchy search also requires both main effects to be available and included; excluding a required parent may make that interaction inadmissible. These choices should be recorded because they define the model set from which the algorithm selects.
Check Your Understanding
Answer each question using the hierarchy rules defined above.
- Does the model \(\{X,XZ\}\) satisfy strong hierarchy, weak hierarchy, both, or neither? Explain.
- Under strong hierarchy, which main effects must be present before an algorithm can add \(XZ\)?
- A forward search currently includes \(Z\) but not \(X\). Is \(XZ\) eligible under weak hierarchy? Is it eligible under strong hierarchy?
- Why can a model with \(X^2\) but no \(X\) behave differently when the predictor is centered?
- In backward elimination from \(\{X,Z,XZ\}\), which single main-effect deletion would violate strong hierarchy if the interaction were left in the model?