Orthogonal Polynomials for Correlated Power Terms

Why Use Orthogonal Polynomial Terms?

A curved relationship between a response and a numerical predictor can be represented by adding powers of that predictor to a regression model. For example, a quadratic model uses a constant, a linear term, and a squared term. But these columns of the design matrix can be strongly correlated. When they are, it can be difficult to distinguish how the terms contribute to the fitted curve, and coefficient estimates can be sensitive to the data.

Orthogonal polynomials provide a different set of columns spanning the same polynomial model. Their values are constructed to have zero cross-products in the observed data. This changes the coefficient coordinates, not the set of curves the model can fit. For interpretation or prediction, it is therefore essential to know how to move between the orthogonal basis and ordinary powers of the original predictor.

Here, “orthogonal” refers to the observed design: the vectors of term values across the \(n\) observations. It does not mean that the predictor itself has been transformed into a random variable independent of its square. Unless stated otherwise, the construction below is for ordinary least squares with equal weight on each observation. Weighted least squares instead requires orthogonality under the corresponding weighted inner product.

Raw Powers and a Quadratic Orthogonal Basis

For a quadratic curve, the raw-power model is \(y_i=\beta_0+\beta_1x_i+\beta_2x_i^2+\varepsilon_i\). If the predictor values are unevenly distributed, the columns for \(x\) and \(x^2\) may be highly correlated. One way to create an orthogonal quadratic column is to regress the column of squared values on the constant and linear columns, then use the residuals as the quadratic column. By construction, those residuals have zero inner product with both columns.

The next example carries out this construction. Its small design also makes the dependence between the raw powers directly visible.

Why Orthogonality Reduces Coefficient Dependence

Consider a linear model written using \(p\) orthogonal design columns, collected in the matrix \(Z\): \(y=Z\gamma+\varepsilon\). Suppose the columns include the intercept, each has positive sum of squares, and the errors have mean zero and common variance \(\sigma^2\), with zero covariance between distinct errors. Ordinary least squares estimates \(\gamma\) by \((Z^\mathsf{T}Z)^{-1}Z^\mathsf{T}y\). Orthogonality makes the cross-products in \(Z^\mathsf{T}Z\) zero, so this calculation separates into one projection per column.

The result follows directly from the least-squares equations. Because each distinct pair of columns has zero inner product, the \(j\)th diagonal entry of \(Z^\mathsf{T}Z\) is \(z_j^\mathsf{T}z_j\), and every off-diagonal entry is zero. Its inverse is diagonal with entries \(1/(z_j^\mathsf{T}z_j)\). Multiplying that inverse by \(Z^\mathsf{T}y\) gives the projection formula above. The usual covariance formula for least squares is \(\sigma^2(Z^\mathsf{T}Z)^{-1}\), which is diagonal here. This is why orthogonal columns remove covariance between the coefficient estimates under the stated error assumptions.

Orthogonality does not guarantee small standard errors: a column with a small sum of squares can still have an imprecise coefficient. Nor does it remove dependence from the raw-power coefficient estimates after conversion back to raw powers. It simplifies coefficient estimation and covariance in the orthogonal coordinates.

Computing Fitted Values and Transforming Coefficients

For any observation, a fitted value can be calculated directly from the orthogonal-basis model by evaluating each basis function at that observation’s predictor value. To obtain raw-power coefficients, expand each basis polynomial and collect like powers. These are two coordinate descriptions of the same fitted polynomial when the bases span the same polynomial space.

Interpretation and Common Traps

An orthogonal polynomial coefficient describes the contribution of one constructed basis column while the other orthogonal columns are present. It is generally not the slope or curvature in the raw predictor scale. For example, in \(\hat{y}=\hat{\gamma}_0+\hat{\gamma}_1u+\hat{\gamma}_2q\), the coefficient \(\hat{\gamma}_1\) multiplies \(u\), not \(x\) alone; the raw linear coefficient also receives a contribution from the quadratic basis after expansion. The raw quadratic coefficient depends on the chosen scaling of the highest-degree basis term.

Orthogonalizing the columns leaves the model’s fitted values unchanged when the transformation is invertible and the original and transformed columns span the same space. It does not make a misspecified curve correct, establish a causal relationship, or eliminate uncertainty in predictions. The basis is also tied to its construction. If the predictor values change, an orthogonal basis built for the new data may have different coefficients and different polynomial expressions.

• Trap: assuming zero correlation means the predictor and its square are unrelated. The zero cross-products concern constructed columns on the observed design, not a general probabilistic independence claim.
• Trap: interpreting an orthogonal coefficient as a raw-power coefficient. Expand the basis to recover the coefficient of \(x\) or \(x^2\); do not simply relabel \(\hat{\gamma}_1\) as the linear slope.
• Trap: rebuilding the basis inconsistently for prediction. Use the exact basis definition from fitting. Applying a different centering or residualization changes the calculation.
• Trap: thinking orthogonality changes the fitted curve. With the same polynomial span and a full-rank change of basis, fitted values are unchanged; what changes are the coefficient coordinates and their covariance structure.