Model Hierarchy and Complexity in GLM Selection

Why Hierarchy Matters in GLM Selection

A GLM can improve its fit by adding interactions or higher powers of predictors. But a term should not be judged only by whether it reduces deviance: the model's structure and the number of parameters added matter too. Hierarchy provides a disciplined way to build interpretable candidate models, while complexity accounting makes comparisons fair.

This tutorial focuses on two common cases: an interaction such as \(xz\), and a polynomial term such as \(x^2\). The same principles apply whether a GLM uses an identity, log, or logit link. They do not dictate which model must be selected; they help ensure that the candidate models and their comparisons are sensible.

For categorical predictors, “main effect” and “interaction” refer to sets of coefficients, not necessarily single coefficients. If factor A has \(a\) levels and factor B has \(b\) levels, their main effects usually require \(a-1\) and \(b-1\) coefficients, and their interaction requires \((a-1)(b-1)\) coefficients under standard reference-level coding. The hierarchy principle retains both main-effect sets when the interaction set is included.

Why Lower-Order Terms Belong with Interactions

Consider the linear predictor

The interaction says that the association between \(x\) and the linear predictor can change with \(z\): its slope with respect to \(x\) is \(\beta_1+\beta_3z\). The main effects describe the corresponding lower-order contributions. Omitting them can make the interaction depend unnecessarily on the particular zero points used for the predictors.

The reason can be shown directly. Let \(z'=z-c\), so \(z=z'+c\). Substituting this into the complete model gives

After shifting the origin of \(z\), the intercept and the main-effect coefficient of \(x\) change, while the interaction remains. Now consider a model containing only an intercept and \(\beta_3xz\). Replacing \(z\) by \(z'+c\) produces \(\beta_3xz'+\beta_3cx\): a main effect of \(x\) appears solely because the origin changed. The model without lower-order terms is therefore not generally invariant to a harmless change of reference point. This is one reason strong hierarchy is a useful default.

With a complete polynomial, shifting the origin also changes the coefficients of lower powers. For example, \((x+c)^2=x^2+2cx+c^2\), so a quadratic effect expressed around a shifted origin includes a linear term and a changed intercept. Including the lower-order terms allows the same curve to be represented after recentering.

Hierarchy and Parameter Counts

A candidate model's complexity is determined by the number of estimated parameters, not by the number of terms as written in a formula. A term can represent several coefficients. For example, an interaction between two factors with three and four levels adds six coefficients, not one. The intercept and any estimated dispersion parameter also count when calculating an information criterion such as AIC.

For models fit to the same observations with maximized log-likelihood \(\ell\), AIC is

where \(k\) is the number of estimated parameters under the convention being used. Lower AIC is preferred among the candidates being compared. The likelihood improvement rewards fit; the \(2k\) term penalizes complexity. AIC does not make an incoherent or non-hierarchical candidate automatically appropriate.

For nested models fit to the same data, the likelihood-ratio statistic is \(2(\ell_{\text{larger}}-\ell_{\text{smaller}})\). When a term adds \(q\) estimable coefficients, the usual large-sample reference comparison uses \(q\) degrees of freedom, subject to the model and regularity conditions for that test. A term that adds six coefficients should not be assessed as if it added only one.

Polynomial Terms and Hierarchical Comparisons

For a polynomial of degree \(r\), strong hierarchy retains each power below \(r\). Thus a cubic model contains \(x\), \(x^2\), and \(x^3\). This gives the polynomial a coherent interpretation: the curve can express both slope and curvature, and the terms remain available when the predictor is centered at another value.

A common comparison is between a linear model and its quadratic extension. These are nested if they use the same data, response, link, and other terms; the quadratic model adds one coefficient, \(x^2\). In contrast, comparing a model with \(x^2\) alone against one with \(x\) is not the usual hierarchical test of whether curvature improves a linear trend. The two specifications answer different questions and may not be nested.

Factor Interactions: Count the Coefficients, Not the Labels

For a factor interaction, the increase in complexity can be substantial. Suppose factor A has three levels and factor B has four levels. With an intercept, standard reference-level coding gives two coefficients for A, three for B, and six for the interaction. The full model therefore has \(1+2+3+6=12\) regression coefficients. The hierarchical main-effects model has \(1+2+3=6\). In a Gaussian GLM with an estimated variance, the corresponding total parameter counts are 13 and 7.

The interaction represents departures from a model in which the factor effects combine additively on the linear-predictor scale. It is not a single extra parameter merely because it appears as one formula term. Sparse or empty combinations of factor levels can also make some interaction coefficients unestimable; in that case, the nominal count may not equal the number of estimable parameters. Check the fitted model and the data structure before interpreting a comparison.

Using Hierarchy in a Selection Decision

Hierarchy is a structural guide, not a substitute for the comparison objective. Earlier material on GLM comparison distinguishes likelihood-based fit, information criteria, and predictive performance. In each case, the candidate set should be built so that the terms have meaningful interpretations and the complexity is counted correctly.

A lower deviance for a larger model is expected when it is a nested extension fit by maximum likelihood to the same data. That fact alone does not establish that every added term is worthwhile. Conversely, a small AIC difference does not prove that one model is substantively correct. Report the candidate structure, the number of added coefficients, the comparison criterion, and any important limitations.

Common Mistakes and Exam Traps

• Keeping an interaction but dropping both main effects. This violates strong hierarchy and can make the model's meaning depend on arbitrary predictor origins. Retain both main effects by default.
• Keeping \(x^3\) but omitting \(x\) or \(x^2\). A cubic hierarchical model contains every lower power. Compare the complete polynomial extension with its lower-degree counterpart.
• Counting formula terms instead of coefficients. A factor-by-factor interaction can add several degrees of freedom. Count the level contrasts and their products.
• Ignoring parameters outside the linear predictor. For Gaussian likelihood-based criteria, an estimated variance is a parameter too. Use a consistent parameter-count convention for all models.
• Selecting only by the smallest deviance. Fit usually improves as terms are added. Consider the complexity penalty or the stated predictive objective rather than equating better in-sample fit with better selection.
• Assuming hierarchy is an absolute law. It is a widely useful modeling principle, not a theorem that forbids every non-hierarchical specification. If an exception is used, justify the scientific meaning and interpretability.