Why Nesting Is About Model Structure
Regression analysis often compares a simpler specification with one that includes extra terms. For example, a model might predict an outcome from study time, while another includes both study time and class attendance. Before asking what the comparison shows, we need to know whether the models are nested: can every mean pattern allowed by the simpler model also be represented by the larger one?
Nesting is a relationship between model specifications, not a conclusion about which model is more useful or which predictors matter. It is determined by the terms in the models and the restrictions on their coefficients. A model with more terms is not automatically a full model for every other model; the required relationship is that the reduced model can be obtained from it by imposing restrictions.
For the standard comparisons in this unit, the models describe the same outcome and use the same observations. Their specifications may differ in which predictors or terms they include. Holding the outcome and sample fixed makes the comparison about the model structure rather than about changes in the data being analyzed.
To keep the definition precise, consider a regression model for an outcome \(Y\) whose conditional mean, given predictor values, is written as a linear combination of specified terms. “Linear” here means linear in the unknown coefficients; a term such as \(x^2\) can be included even though it is a nonlinear function of \(x\). The possible mean functions are all functions that result as the coefficients vary over their allowed values.
Coefficient Restrictions and Model Inclusion
Suppose the full model contains terms \(x\) and \(z\), along with an intercept:
If the reduced model contains only an intercept and \(x\), it is
The reduced model is represented inside the full model by the restriction \(\beta_2=0\). Under that restriction, the full model becomes \(\beta_0+\beta_1x+0z=\beta_0+\beta_1x\), exactly the reduced model. Thus every mean function available under the reduced specification is available under the full specification, which is the defining inclusion.
For several added terms, nesting may require several coefficient restrictions. If the full model adds two predictors \(z_1\) and \(z_2\), the reduced model that omits both is obtained by setting both associated coefficients to zero. The restrictions must be considered together: setting just one coefficient to zero removes only one term, not the entire added block.
A useful way to see the same idea is through the design matrix. Each model’s design matrix has one column for each included term, such as an intercept column, an \(x\) column, or a \(z\) column. The model’s possible fitted mean vectors are linear combinations of its columns. If the reduced model’s columns are among the full model’s columns, then each reduced-model mean vector can also be expressed using the full model. More generally, nesting holds when the reduced model’s column space is contained in the full model’s column space, even if the columns are represented differently.
Worked Examples
Worked Example: Adding a Second Quantitative Predictor
Suppose a hypothetical data set records quiz score \(Y\), hours of study \(x\), and hours of class attendance \(z\) for the same students. Compare the study-only model with a model that includes both predictors. The scores and predictor values are illustrative, not results from a real study.
| Student | Quiz score \(Y\) | Study hours \(x\) | Attendance hours \(z\) |
|---|---|---|---|
| A | 72 | 2 | 3 |
| B | 81 | 4 | 4 |
| C | 75 | 3 | 2 |
The reduced model is \(E(Y\mid x)=\beta_0+\beta_1x\). The full model is \(E(Y\mid x,z)=\beta_0+\beta_1x+\beta_2z\). The restriction is \(\beta_2=0\); substituting it into the full-model expression gives \(\beta_0+\beta_1x+0z=\beta_0+\beta_1x\). Therefore, the study-only model is nested in the model with study time and attendance.
For a concrete check, use \(x=2\) and \(z=3\). If \(\beta_0=50\), \(\beta_1=5\), and \(\beta_2=0\), the full-model mean at these predictor values is \(50+5(2)+0(3)=60\). The reduced model gives \(50+5(2)=60\), the same value. These coefficients illustrate the restriction; they are not estimates calculated from the three listed students.
The observation rows do not determine whether the specifications are nested. The equations do: the reduced model is exactly the full model with the attendance coefficient set to zero.
Worked Example: A Linear Model Nested in a Quadratic Model
A hypothetical study records a response \(Y\) at predictor values \(x=1,2,3\). Consider a straight-line model and a quadratic model. A quadratic model is still a regression model linear in its coefficients, because the unknown coefficients are multiplied by the known terms \(1\), \(x\), and \(x^2\).
The reduced model is \(E(Y\mid x)=\beta_0+\beta_1x\). The full model is \(E(Y\mid x)=\beta_0+\beta_1x+\beta_2x^2\). Setting \(\beta_2=0\) gives \(\beta_0+\beta_1x+0x^2=\beta_0+\beta_1x\). Therefore the straight-line model is nested within the quadratic model.
For example, take \(x=2\) and illustrative coefficients \(\beta_0=10\), \(\beta_1=3\), and \(\beta_2=0\). The quadratic expression gives \(10+3(2)+0(2^2)=10+6+0=16\), exactly the linear expression \(10+3(2)=16\). If \(\beta_2\) were not zero, the quadratic term could contribute to the mean; nesting requires that the full model allow the particular case in which that contribution is zero.
The reduced model must retain the intercept and the linear term for this direct restriction to work. A model containing only an intercept and \(x^2\) does not generally include every straight line in \(x\), because setting a coefficient to zero cannot create the missing \(x\) term.
Worked Example: Removing a Categorical Predictor Block
Suppose a hypothetical data set records a response \(Y\) and a three-level treatment-group label: A, B, or C. With A as the reference group, define \(I_B=1\) for group B and \(0\) otherwise, and define \(I_C=1\) for group C and \(0\) otherwise. Compare a model with no group terms to one that includes both indicators.
The reduced model is \(E(Y)=\beta_0\). The full model is \(E(Y\mid I_B,I_C)=\beta_0+\beta_BI_B+\beta_CI_C\). The reduced model is obtained by imposing the two restrictions \(\beta_B=0\) and \(\beta_C=0\). Both restrictions are needed to remove the group block.
Check the predictions using illustrative values \(\beta_0=70\), \(\beta_B=4\), and \(\beta_C=-2\). For group B, \(I_B=1\) and \(I_C=0\), so the full-model mean is \(70+4(1)-2(0)=74\). Under the reduced-model restrictions, it is \(70+0(1)+0(0)=70\), which is the reduced model’s mean for every group. For group C, \(I_B=0\) and \(I_C=1\), and the restricted full model gives \(70+0(0)+0(1)=70\) as well.
Thus the group-free model is nested in the model with the two group indicators. This example also shows why a categorical predictor can require a block of coefficients rather than a single coefficient.
A Short Justification Using Mean Vectors
The coefficient-restriction definition and the mean-function definition express the same relationship for the ordinary linear models considered here. Write the full model’s possible mean vector for the observed predictor values as \(X_F\boldsymbol{\beta}\), where \(X_F\) is its design matrix and \(\boldsymbol{\beta}\) is its coefficient vector. Suppose the reduced specification is obtained by imposing linear restrictions on \(\boldsymbol{\beta}\). Every coefficient vector that satisfies those restrictions is still an allowed coefficient vector for the full model. Consequently, every reduced-model mean vector is also a full-model mean vector.
For example, with full-model columns for an intercept, \(x\), and \(z\), the mean vector is \(\beta_0\mathbf{1}+\beta_1\mathbf{x}+\beta_2\mathbf{z}\). Under the restriction \(\beta_2=0\), it becomes \(\beta_0\mathbf{1}+\beta_1\mathbf{x}\). That is a mean vector available to the full model and is precisely the reduced-model form. This establishes the inclusion directly; it does not depend on particular observed outcome values.
The argument concerns the mean structure, not the distribution of errors. To use standard regression inference about model comparisons, the relevant regression assumptions and conditions must also be considered. They are not part of the definition of nesting: two specifications can be nested or not nested as a structural matter even before an inferential procedure is chosen.
When Two Models Are Not Nested
Having different numbers of coefficients does not by itself establish nesting. Consider a model with \(x\) alone and another with \(z\) alone. Neither generally contains the other: a mean function that changes with \(x\) cannot necessarily be written as a function of \(z\), and vice versa. Unless a special relationship among the predictor columns makes one model’s mean functions representable by the other, the specifications are not nested.
Likewise, a model that includes one predictor instead of another is not automatically a larger version of either model. The key question is not “Which model has more terms?” but “Can every mean pattern allowed by the proposed reduced model be expressed by the proposed full model?” In simple specifications this can be checked by setting coefficients to zero. With different encodings or overlapping columns, compare the model spaces or possible mean functions.
Common Mistakes and Why the Distinction Matters
- Counting terms instead of checking restrictions. More terms do not guarantee that one model contains another. Confirm that the reduced specification can be recovered from the full one.
- Forgetting a coefficient in a block. Omitting a categorical predictor with several indicator terms requires restricting all of its coefficients, not just one.
- Calling the coefficient restriction a result. A restriction describes how the models relate; by itself it does not establish that an added predictor has an association with the outcome, or that it is useful.
- Confusing nesting with a test outcome. Nesting is a property of model specifications. Whether data provide evidence against particular restrictions is a separate inferential question.
- Assuming every model with added terms is a sensible specification. A model can be nested yet still be poorly motivated or inappropriate for the scientific question. Structural inclusion alone does not establish practical value or causation.
Nesting matters because it identifies when a simpler model is a restricted case of a more flexible one. That relationship is the structural foundation for later comparisons of model fit and tests of coefficient restrictions. At this stage, the essential task is to state the models clearly and identify the restrictions that connect them.
Check Your Understanding
For each question, focus on the model terms and the restrictions linking the specifications.
- A reduced model is \(E(Y\mid x)=\beta_0+\beta_1x\), and a full model is \(E(Y\mid x,z)=\beta_0+\beta_1x+\beta_2z\). State the restriction that recovers the reduced model.
- A model includes an intercept and \(x\); another includes an intercept, \(x\), and \(x^2\). Which is reduced, and what restriction establishes nesting?
- A three-level categorical predictor is represented by two indicator terms. What restrictions are needed to remove the predictor from the model?
- Explain why a model with \(x\) alone and a model with \(z\) alone are not necessarily nested, even if each has one predictor.
- Does establishing that two specifications are nested show that the added terms are important? Explain the distinction.