Collinearity in Polynomial and Interaction Terms

Why Constructed Terms Can Be Collinear

Regression models can represent curved relationships and interactions by adding constructed predictors, such as \(x^2\) or \(xw\), alongside their component predictors. These terms can be useful, but they are often correlated with those components. For example, when \(x\) is mostly positive, larger values of \(x\) will usually also produce larger values of \(x^2\). In an interaction model, the product \(xw\) can track \(x\) when \(w\) is usually positive and does not vary much.

Such correlations are not automatically evidence that a model is wrong. They do matter for interpreting coefficients: when predictors overlap in the information they carry, an individual coefficient may be less precisely estimated. Centering—a change of origin for a predictor—can make some of these correlations smaller and can give coefficients more useful reference points. It does not create new information or guarantee that every correlation will disappear.

Definition: A constructed term is a predictor formed from one or more existing predictors. A polynomial term such as \(x^2\) is formed from \(x\); an interaction term such as \(xw\) is formed from predictors \(x\) and \(w\). Centering a predictor \(x\) at a constant \(c\) means replacing it by \(z=x-c\). If \(c=\bar{x}\), then the centered sample predictor has mean zero.

To discuss these terms in a regression setting, consider the usual linear model for the conditional mean of a quantitative response:

Y_i=\beta_0+\beta_1x_i+\beta_2x_i^2+\varepsilon_i

Here \(Y_i\) is the response for observation \(i\), and \(\varepsilon_i\) is its error. For standard least-squares inference, the usual assumptions are that the specified mean function is appropriate, the errors have mean zero, have constant variance, and are independent; normal errors are additionally used for exact small-sample \(t\)- and \(F\)-inference. The correlations discussed below are properties of the observed predictors themselves, so they can be examined without assuming that the errors are normal. Correlation among predictors describes overlap, not a causal relationship.

Polynomial Terms and Centering

When \(x\) is far from zero or has a narrow range around a positive value, \(x\) and \(x^2\) can be strongly correlated. That correlation depends on the observed values; it is not a mathematical certainty for every data set. Centering changes the polynomial basis from \(x,x^2\) to \(z,z^2\), where \(z=x-c\). This may reduce the correlation, particularly when the values of \(x\) are roughly balanced around \(c\).

Worked Example: Correlation Between a Predictor and Its Square

Suppose a hypothetical data set has predictor values \(x=1,2,3,4\). Compare the sample correlation of \(x\) with \(x^2\) to the correlation of \(z=x-2.5\) with \(z^2\).

The values of \(x^2\) are \(1,4,9,16\). Their means are \(\bar{x}=2.5\) and \(\overline{x^2}=7.5\). The sum of cross-products of deviations is \(1.5(6.5)+0.5(3.5)+0.5(1.5)+1.5(8.5)=25\). The sums of squared deviations are \(\sum(x_i-\bar{x})^2=5\) and \(\sum(x_i^2-\overline{x^2})^2=129\). Therefore,

r_{x,x^2}=\frac{25}{\sqrt{5(129)}}=\frac{25}{\sqrt{645}}\approx 0.9844.

This is a strong positive sample correlation. Now center at \(2.5\). The \(z\) values are \(-1.5,-0.5,0.5,1.5\), and the \(z^2\) values are \(2.25,0.25,0.25,2.25\). Their means are \(0\) and \(1.25\). The sum of cross-products of deviations is \((-1.5)(1)+(-0.5)(-1)+(0.5)(-1)+(1.5)(1)=0\). The correlation of \(z\) with \(z^2\) is therefore \(0\). This balanced example shows that centering can substantially reduce this particular correlation; it does not establish that centering will always do so.

What Centering Changes—and What It Preserves

The effect of centering on the coefficient meanings follows directly from substitution. If \(z=x-c\), then \(x=z+c\). In the quadratic model, replacing \(x\) by \(z+c\) gives

\begin{aligned} \beta_0+\beta_1x+\beta_2x^2 &=\beta_0+\beta_1(z+c)+\beta_2(z+c)^2\\ &=(\beta_0+\beta_1c+\beta_2c^2) +(\beta_1+2\beta_2c)z+\beta_2z^2. \end{aligned}

Thus, with an intercept and both polynomial terms included, the centered model describes the same set of possible fitted curves. Its intercept is the fitted mean at \(x=c\), because \(z=0\) there. Its coefficient of \(z\) is the local slope at \(x=c\), not necessarily the slope at \(x=0\). The quadratic coefficient is unchanged.

Worked Example: Interpreting a Centered Quadratic Model

Suppose a hypothetical fitted mean function, with \(x\) measured in hours, is \(\widehat{Y}=10+3x+2x^2\), where \(Y\) is a test score in points. Re-express the function using \(z=x-2.5\), and interpret the coefficients.

Substitute \(x=z+2.5\):

\begin{aligned} \widehat{Y} &=10+3(z+2.5)+2(z+2.5)^2\\ &=10+3z+7.5+2(z^2+5z+6.25)\\ &=30+13z+2z^2. \end{aligned}

At \(x=2.5\) hours, \(z=0\), so the fitted score is 30 points. The coefficient 13 means the fitted score’s instantaneous rate of change at \(x=2.5\) hours is 13 points per hour. The curve’s quadratic coefficient remains 2 points per squared hour. The coefficient is a local slope at the centering value, not a constant change in score for every one-hour increase.

A useful qualification is that centering at the sample mean does not automatically make \(z\) and \(z^2\) uncorrelated. Since the centered values have mean zero, their sample covariance is proportional to \(\sum z_i^3\). That sum is zero for data symmetric around the centering point, as in the first example, but need not be zero otherwise. Centering can reduce correlation, leave it similar, or—depending on the data—fail to help.

Interaction Terms and Their Component Predictors

For two predictors, an interaction model commonly includes both main effects and their product:

Y_i=\beta_0+\beta_1x_i+\beta_2w_i+\beta_3x_iw_i+\varepsilon_i.

The interaction coefficient allows the association between \(x\) and the fitted mean of \(Y\) to depend on \(w\), and vice versa. The product \(xw\) can be correlated with \(x\) or \(w\), especially when one predictor has a nonzero mean. Centering both predictors gives \(u=x-a\) and \(v=w-b\). Expanding the product shows why the model can be rewritten without changing its fitted values, provided it contains an intercept, both main effects, and the interaction:

$$ xw=(u+a)(v+b)=uv+b u+a v+ab. $$

The centered model uses \(u\), \(v\), and \(uv\) in place of \(x\), \(w\), and \(xw\). The coefficients change, but the model’s possible fitted values do not. The interaction coefficient is unchanged; the two main-effect coefficients now describe slopes at the chosen reference value of the other predictor.

Worked Example: Centering an Interaction

Suppose \(x\) is a hypothetical measurement in minutes, \(w\) is a hypothetical measurement in units, and a fitted response \(Y\) is measured in points. The model is \(\widehat{Y}=40+0.8x+1.5w+0.06xw\). Center at \(a=50\) minutes and \(b=10\) units, so \(u=x-50\) and \(v=w-10\). Find the centered coefficients.

Use \(x=u+50\) and \(w=v+10\). The intercept becomes \(40+0.8(50)+1.5(10)+0.06(50)(10)=125\) points. The coefficient of \(u\) is \(0.8+0.06(10)=1.4\) points per minute, and the coefficient of \(v\) is \(1.5+0.06(50)=4.5\) points per unit. The product coefficient remains \(0.06\) points per minute per unit. Thus,

\widehat{Y}=125+1.4u+4.5v+0.06uv.

At \(w=10\) units (\(v=0\)), the fitted change in response per additional minute of \(x\) is 1.4 points. At \(x=50\) minutes (\(u=0\)), the fitted change per additional unit of \(w\) is 4.5 points. These are reference-specific slopes; the interaction means that each slope varies with the other predictor.

Checking an Interaction Correlation

A small constructed example makes clear how centering can change correlations among the terms. Suppose a hypothetical data set contains all four combinations of \(x=0,2\) and \(w=10,14\). The product values \(xw\) are \(0,0,20,28\). The means are \(\bar{x}=1\) and \(\overline{xw}=12\). The sum of cross-products of deviations for \(x\) and \(xw\) is \((-1)(-12)+(-1)(-12)+(1)(8)+(1)(16)=48\). The sums of squared deviations are 4 for \(x\) and 608 for \(xw\), so

r_{x,xw}=\frac{48}{\sqrt{4(608)}}\approx 0.9733.

Now center at the respective means: \(u=x-1\) and \(v=w-12\). In the order \((0,10),(0,14),(2,10),(2,14)\), the values of \(uv\) are \(2,-2,-2,2\), while the values of \(u\) are \(-1,-1,1,1\). Their cross-products sum to \(-2+2-2+2=0\), so their sample correlation is zero. The balanced combinations also make \(uv\) uncorrelated with \(v\). This result follows from the particular data pattern; other data patterns can yield nonzero correlations after centering.

Practical Interpretation and Common Pitfalls

When diagnosing polynomial or interaction terms, inspect correlations as an initial screen, then consider how the terms relate together in the full predictor set. A small pairwise correlation does not rule out more complex redundancy. For a specific centering choice, say what value was subtracted and why: the sample mean is common, but a meaningful scientific or practical reference value may make coefficients easier to explain.

Centering is a re-expression, not a remedy for every estimation problem. It can change the pairwise correlations among columns of the design matrix and the meanings of individual coefficients. With the complete polynomial or interaction terms and an intercept included, it does not change fitted values or add information. In particular, a centered model can still have strong collinearity, and centering does not make an interaction disappear.

Mistake: “The product is correlated with a main effect, so the interaction should be removed.” The correlation alone does not show that the interaction is unnecessary. The model’s purpose and the evidence for the terms matter.
Mistake: “Centering always makes the terms uncorrelated.” Centering changes the reference point; the remaining correlation depends on the predictor values and their joint pattern.
Mistake: “The centered linear coefficient is the overall slope.” In a quadratic model it is the slope at the centering value. In an interaction model, a main-effect coefficient is the slope when the other predictor equals its centering value.
Mistake: “Centering changes the fitted relationship.” With an intercept and all corresponding terms retained, centering changes the parameterization, not the set of fitted values.

Key takeaway: Polynomial and interaction terms can overlap strongly with their component predictors. Centering can make coefficients easier to interpret and can reduce some correlations, but its effects depend on the data. Always identify the reference point and distinguish a change of parameterization from a change in the fitted model.

Check Your Understanding

Use the ideas and calculations in this tutorial to answer the following questions.

Why can \(x\) and \(x^2\) be strongly correlated when the observed values of \(x\) are mostly positive?
In a quadratic model centered at \(c\), what does the coefficient of \(x-c\) describe?
Does centering guarantee that a predictor and its square are uncorrelated? Explain briefly.
In a model with \(x\), \(w\), and \(xw\), what does the coefficient of centered \(x\) describe when \(w\) is centered at \(b\)?
Under what model terms does centering preserve the set of possible fitted values?