Why Polynomial Terms Come in a Hierarchy
A polynomial regression model represents a conditional mean using powers of a predictor. If the mean curve bends, a squared term may be useful; if its shape needs further flexibility, a cubed term may be considered. The terms are related: changing the predictor’s origin mixes powers together. Consequently, a model that includes a higher power but omits a lower one can have an interpretation that depends strongly on where the predictor is measured from.
The hierarchy principle is a practical rule for keeping polynomial models interpretable and stable to changes of origin. In a model of degree \(d\), the usual hierarchical specification includes every power from one through \(d\), along with an intercept. This rule concerns which terms are included; it does not say that every coefficient must be large, precisely estimated, or statistically significant.
For example, the hierarchical quadratic and cubic mean functions have the forms
and
Here \(E(Y\mid x)\) is the mean response at predictor value \(x\), and the \(\beta\) values are model coefficients. The response for an individual observation is not generally exactly equal to that mean; it also includes an error.
Model and Assumptions
To discuss least-squares fitting and model comparisons, suppose the observations follow a polynomial regression model
The error \(\varepsilon_i\) describes the difference between observation \(i\) and the model’s conditional mean. For the usual least-squares interpretation, assume the errors have mean zero and a common variance, and are independent across observations. The predictor values are treated as given. These assumptions make the specified polynomial the mean function being fitted; they do not guarantee that the polynomial describes every feature of the data well. Normal errors are additionally used for some exact small-sample inference, but no such inference is needed for the hierarchy arguments here.
The hierarchy principle applies to the powers in the chosen mean function. A model containing \(x^3\) but no \(x^2\), for instance, is mathematically possible. The principle recommends against that specification as a default, because omission of a lower-order term can make the model depend on the arbitrary choice of origin and can constrain the curve in ways that are not apparent from the coefficient list.
Worked Example: Retaining the Linear Term in a Quadratic Model
Suppose a hypothetical analysis relates a test score \(Y\), measured in points, to study time centered at 8 hours. Let \(z=x-8\), where \(x\) is study time in hours. The fitted quadratic mean function is
Because the model includes \(z^2\), hierarchy calls for retaining \(z\) as well as the intercept. At \(z=0\), corresponding to 8 hours of study, the predicted score is \(62\) points. At \(z=1\), corresponding to 9 hours, it is
At \(z=2\), or 10 hours, the prediction is
The linear coefficient \(4.5\) describes the contribution of the linear term at the chosen origin; it is not, by itself, a constant increase in score for each additional study hour throughout the curve. The quadratic term makes the relationship vary with \(z\). Even if a separate analysis found little evidence that the linear coefficient differs from zero, the hierarchy principle would ordinarily keep that term while \(z^2\) remains. Its presence allows the curvature term to be interpreted within the full quadratic model rather than under an extra constraint.
Why the Predictor’s Origin Matters
Centering replaces \(x\) by \(z=x-c\), where \(c\) is a chosen constant. This changes the numerical values of the coefficients, but a full quadratic model can represent exactly the same fitted curves in either coordinate. To see why hierarchy matters, expand a centered quadratic:
The expansion contains a constant, a linear term, and a squared term. In general, the coefficient of \(x\) after changing origins is \(\beta_1-2c\beta_2\), not simply \(\beta_1\). A complete quadratic model therefore remains a complete quadratic model after centering. This is one reason not to treat the linear coefficient as an optional nuisance when a square is included.
Worked Example: What Omitting a Lower-Order Term Does
Suppose \(x\) is a hypothetical temperature measurement in degrees Celsius, and define \(z=x-10\), so \(z=0\) represents 10°C. Consider the hierarchical centered model
Substitute \(z=x-10\) and expand:
The same curve has a linear term in the uncentered coordinate too. The coefficients differ because the origin changed, not because the fitted curve changed.
Now omit the centered linear term and fit \(\widehat{Y}=40-0.20z^2\). In terms of \(x\), this restricted model is
This equation does contain a linear term in \(x\), but its coefficient is forced to be \(4\); it cannot vary freely. Thus “no linear term in the centered coordinate” is a particular restriction on the curve, not a general statement that the relationship has no linear component. The unrestricted hierarchical quadratic allows the linear and squared contributions to be estimated together.
Hierarchy and Model Comparison
A model with more polynomial terms includes all the fitted curves available to a corresponding lower-degree model. For example, the linear model is obtained from the hierarchical quadratic by setting \(\beta_2=0\). The quadratic model is obtained from the hierarchical cubic by setting \(\beta_3=0\). Such models are called nested: the smaller model is contained within the larger one.
This nesting gives an important least-squares result. The sum of squared residuals (SSE), the sum of squared differences between observed and fitted responses, cannot increase when a term is added. The larger model can always reproduce the smaller model by setting the added coefficient to zero; its best-fitting SSE must therefore be no greater. This establishes a property of fit, not a guarantee that the more complex model is preferable.
Hierarchy also clarifies what a model comparison asks. Comparing a quadratic with a linear model concerns whether allowing a squared term improves the fit beyond the linear model, with the intercept and linear term retained in both. Comparing a cubic with a linear model concerns adding both the squared and cubed terms together. These are different model changes. A comparison based only on whether the highest-power coefficient appears important does not, by itself, answer every question about whether a higher-degree model is useful.
Worked Example: Reading Nested-Model Summaries
A hypothetical data set contains \(n=18\) independent measurements of an outcome in milligrams and a predictor \(x\) in days. Suppose least-squares output reports the following residual sums of squares for hierarchical models fitted to these same observations:
| Model | Terms beyond intercept | SSE (mg²) | Residual degrees of freedom |
|---|---|---|---|
| Linear | \(x\) | 216 | 16 |
| Quadratic | \(x, x^2\) | 144 | 15 |
| Cubic | \(x, x^2, x^3\) | 132 | 14 |
The linear model estimates an intercept and one slope, so its residual degrees of freedom are \(18-2=16\). The quadratic estimates three coefficients, giving \(18-3=15\); the cubic estimates four, giving \(18-4=14\). Each model retains all the lower-order powers required by hierarchy.
From linear to quadratic, the SSE decreases by
From quadratic to cubic, it decreases by
The decreases are consistent with the nesting result: adding terms cannot increase least-squares SSE. But the cubic model’s smaller SSE alone does not establish that it is the better choice. It uses more coefficients, and the decrease must be considered alongside the purpose of the analysis, the adequacy of the mean-function assumptions, and appropriate measures of model performance. The comparison of cubic to linear changes two terms at once, \(x^2\) and \(x^3\); it is not the same question as comparing quadratic to linear or cubic to quadratic.
Common Mistakes and Practical Consequences
- Dropping a lower-order term because its estimate is small. A small estimated linear coefficient does not make a quadratic model hierarchical if the squared term is retained. Keep the lower-order term unless there is a clear reason to impose that restriction.
- Assuming “no linear term” means the same thing at every origin. The expansion of a centered polynomial shows that changing the origin changes the linear coefficient. State the predictor’s origin and units when interpreting coefficients.
- Choosing a more complex model just because SSE is lower. SSE cannot increase when terms are added, even when the added flexibility is not useful. Fit improvement alone is not a complete model-selection argument.
- Comparing models that answer different questions as though only one term changed. A linear-to-cubic comparison adds two powers; a linear-to-quadratic comparison adds one. Identify exactly which terms differ before describing the comparison.
- Treating hierarchy as an absolute law. A nonhierarchical model can be fitted when a specific scientific or design constraint calls for it. It should be recognized as a restriction, with its dependence on the predictor’s origin made clear.
The practical value of hierarchy is therefore broader than a tidy list of terms. It keeps polynomial coefficients connected to a coherent curve, makes centered and uncentered forms easier to relate, and makes nested comparisons explicit. It does not remove the need to judge whether the model is sensible for the data or whether its predictions are useful.
Check Your Understanding
Answer each question using the definitions and examples above.
- Which terms should ordinarily appear in a hierarchical quartic model?
- A quadratic model includes \(x^2\) but omits \(x\). Why can changing the origin change the restriction represented by this model?
- In a centered quadratic model, what happens to the fitted curve when the predictor is recentered but all terms are retained?
- Why can the SSE of a hierarchical cubic model not be larger than that of its nested hierarchical quadratic model?
- A comparison changes a linear model directly to a cubic model. Which powers are added, and why is that different from comparing a quadratic to a linear model?