Polynomial Features for Curved Predictor Effects

Why Add Polynomial Features?

A straight-line predictor effect changes by the same amount for each one-unit increase in a predictor. Some relationships instead flatten, steepen, or reverse direction over the predictor’s range. Adding powers of a continuous predictor gives a model a way to represent such curvature without dividing the predictor into categories.

For example, a model can include both $x$ and $x^2$. The squared feature gives the model a curved shape, while the coefficients are still combined linearly. Polynomial features are predictors created from the existing predictor; they do not, by themselves, establish that the real relationship is polynomial or that extrapolation beyond the observed range is reliable.

Definition: Given a continuous predictor $x$, a degree-$p$ polynomial feature set consists of the powers $x, x^2, \ldots, x^p$. A polynomial predictor model combines these features with coefficients. For a quadratic model, the conditional mean is $E[Y\mid x]=\beta_0+\beta_1x+\beta_2x^2$; for a cubic model it is $E[Y\mid x]=\beta_0+\beta_1x+\beta_2x^2+\beta_3x^3$.

These equations specify a model for the conditional mean of $Y$ given $x$. They assume that the stated polynomial form is an appropriate description of that conditional mean over the range being considered. If the model is fitted as ordinary least squares, the fitted conditional mean is computed from the estimated coefficients in place of the $\beta$ coefficients. Standard errors and inferential claims require additional assumptions, such as independent errors with mean zero and an appropriate variance model; adding powers does not automatically satisfy those assumptions.

The curvature in a quadratic model comes from the $x^2$ term. Its rate of change with $x$ is $\beta_1+2\beta_2x$, which depends on $x$ when $\beta_2\ne 0$. In a cubic model the corresponding rate of change is $\beta_1+2\beta_2x+3\beta_3x^2$. Thus, a polynomial model permits a changing slope; it does not assign one constant effect to a one-unit increase in $x$.

Worked Example: Building and Reading a Quadratic Feature

A hypothetical analyst models expected annual service cost, in thousands of dollars, using a continuous risk score $x$. The observed scores run from 1 to 5, and the fitted conditional-mean model is $E[Y\mid x]=2+0.6x-0.05x^2$. Construct the squared feature, calculate the fitted means at each observed score, and find the fitted change from score 2 to score 3.

Method. Create $x^2$ by squaring each score, then substitute the score and its square into the fitted model. The model coefficients are treated as given estimates; the calculation does not assert that the hypothetical data are real.

For $x=1,2,3,4,5$, the squared-feature values are $1,4,9,16,25$. The predictor ranges are therefore $1$ to $5$, and the squared predictor ranges from $1$ to $25$.

At $x=1$, the fitted mean is $2+0.6(1)-0.05(1)=2+0.6-0.05=2.55$ thousand dollars. At $x=2$, it is $2+0.6(2)-0.05(4)=2+1.2-0.2=3.00$. At $x=3$, it is $2+1.8-0.45=3.35$. At $x=4$, it is $2+2.4-0.8=3.60$. At $x=5$, it is $2+3.0-1.25=3.75$.

The fitted change from $x=2$ to $x=3$ is $3.35-3.00=0.35$ thousand dollars, or $350. As a check, the formula for the one-unit change in this quadratic model is $\beta_1+\beta_2(2x+1)$. For $x=2$, this gives $0.6-0.05(5)=0.35$ thousand dollars, agreeing with the difference between the two fitted means.

Common mistake avoided: The change is not always $\beta_1=0.6$. That coefficient alone is not the total one-unit change when the model also contains $x^2$; the contribution of the squared term depends on the starting score.

Why the Polynomial Formula Gives a Changing Effect

The finite change in fitted mean is useful when the predictor is measured in whole units. It can be derived directly by subtracting the model value at $x$ from the value at $x+1$. For a quadratic conditional mean, assume $E[Y\mid x]=\beta_0+\beta_1x+\beta_2x^2$. Then:

\begin{aligned} E[Y\mid x+1]-E[Y\mid x] &=\beta_1\big((x+1)-x\big)+\beta_2\big((x+1)^2-x^2\big)\\ &=\beta_1+\beta_2\big(x^2+2x+1-x^2\big)\\ &=\beta_1+\beta_2(2x+1). \end{aligned}

The intercept cancels because it is unchanged between the two predictor values. The linear term contributes $\beta_1$, while the squared term contributes $\beta_2(2x+1)$. This explains why a coefficient on a polynomial feature should not be interpreted as the entire effect of increasing the original predictor. For a continuous predictor, differentiating the same mean function gives the local slope $\beta_1+2\beta_2x$; the finite one-unit change and local slope are related but are not identical.

A quadratic term can produce a U-shaped or an inverted-U-shaped mean curve, depending on the sign of $\beta_2$. A cubic term allows additional changes in curvature. These shapes should be judged within the observed predictor range and in the application’s context. A higher degree is not automatically better: it adds flexibility but can also produce unstable or extreme fitted values near the boundaries and beyond them.

Centering and Polynomial Features

A polynomial can be expressed using the original predictor $x$, or using a shifted predictor $z=x-c$, where $c$ is a chosen constant. The shift changes the numerical values of the features and their coefficients. It does not change the set of curves available to a complete polynomial model of a given degree: the binomial expansions convert one basis into the other.

For example, with $z=x-c$, $x=c+z$, so $x^2=c^2+2cz+z^2$. For a cubic model, $x^3=c^3+3c^2z+3cz^2+z^3$. Consequently, expressing a quadratic or cubic model using centered powers generally changes its intercept and lower-order coefficients as well as the feature values. Centering is a change of coordinates, not a reason to leave out the constant or lower-order terms in a usual polynomial model.

Worked Example: Rewriting a Cubic Model Around a Center

A hypothetical fitted mean function over scores from 8 to 14 is $f(x)=10+0.5x-0.02x^2+0.001x^3$. Rewrite it in terms of $z=x-10$, compare predictions at $x=12$, and inspect the raw and centered feature ranges.

Method. Substitute $x=10+z$, expand each power, and collect coefficients of like powers. This shows exactly how centering changes the coefficients without changing fitted values.

The expansions are $x^2=(10+z)^2=100+20z+z^2$ and $x^3=(10+z)^3=1000+300z+30z^2+z^3$. Substitution gives:

\begin{aligned} f(x) &=10+0.5(10+z)-0.02(100+20z+z^2)\\ &\quad+0.001(1000+300z+30z^2+z^3)\\ &=14+0.4z+0.01z^2+0.001z^3. \end{aligned}

For $x=12$, $z=12-10=2$. Using the original expression gives $10+0.5(12)-0.02(144)+0.001(1728)=10+6-2.88+1.728=14.848$. Using the centered expression gives $14+0.4(2)+0.01(4)+0.001(8)=14+0.8+0.04+0.008=14.848$. The matching results check the algebra.

Over the observed interval, $x$ ranges from 8 to 14 and $z$ from $-2$ to 4. The raw powers range from 64 to 196 for $x^2$, and from 512 to 2,744 for $x^3$. The centered square $z^2$ ranges from 0 to 16, and the centered cube $z^3$ from $-8$ to 64. In particular, $z^2$ does not have its minimum at an endpoint: it is zero at $z=0$, which corresponds to $x=10$.

Common mistake avoided: Centering does not mean replacing $x^2$ by $x^2-10^2$. The centered square is $(x-10)^2$, which includes the cross-product term when expanded. The complete expansion is $x^2-20x+100$.

Creating Features and Checking Their Ranges in R

For raw polynomial powers in R, make the arithmetic explicit. If a data frame is called d, one can create features with d$x2 <- d$x^2 and d$x3 <- d$x^3. In a formula, use I(x^2) to indicate that the square is arithmetic on the predictor: for example, lm(y ~ x + I(x^2), data = d). The I() wrapper protects the arithmetic expression from the special formula interpretation of operators.

A formula such as y ~ x^2 does not safely specify an arithmetic square in R’s formula language. The caret has a formula-expansion meaning there; it is not the same as typing the square operation in ordinary R arithmetic. Use I(x^2) for a raw square. R’s poly(x, 3) uses orthogonal polynomial columns by default, not the raw columns $x, x^2, x^3$. If raw powers are specifically required, poly(x, 3, raw = TRUE) requests them.

Inspect ranges after constructing the features, rather than assuming their ranges from the original predictor’s endpoints. For a predictor containing negative values, a square can have its minimum at an interior value near zero; a cube preserves the ordering of the original values. A brief R check is:

\texttt{range(d\$x)} \qquad \texttt{range(d\$x2)} \qquad \texttt{range(d\$x3)}

In applied work, also inspect the actual feature values and their scale, especially if the predictor covers a wide numerical range. For negative and positive predictor values, checking only the smallest and largest original values may miss an interior minimum of the square. Ranges describe the observed features; they do not guarantee that a polynomial model behaves reasonably outside that range.

Worked Example: Constructing a Centered Cubic Design in R

A hypothetical analyst has predictor values $x=2,4,6,8,10$, and chooses $c=6$. The proposed fitted mean is $f(x)=20+0.5z-0.1z^2+0.01z^3$, where $z=x-6$. List the centered features, calculate the prediction at $x=8$, and give suitable R formula code.

Method. Calculate $z$ first, then its powers. The center is a fixed value for this calculation, and each feature is calculated from the same original $x$.

The centered values are $z=-4,-2,0,2,4$. Squaring gives $z^2=16,4,0,4,16$, so the observed range of $z^2$ is 0 to 16. Cubing gives $z^3=-64,-8,0,8,64$, so the range of $z^3$ is $-64$ to 64. The centered predictor itself ranges from $-4$ to 4.

At $x=8$, $z=8-6=2$, $z^2=4$, and $z^3=8$. The fitted mean is $20+0.5(2)-0.1(4)+0.01(8)=20+1-0.4+0.08=20.68$. As a check, the feature values satisfy $z^3=z\cdot z^2=2\cdot4=8$.

One clear R implementation is d$z <- d$x - 6, followed by lm(y ~ z + I(z^2) + I(z^3), data = d). The formula includes the linear term and both higher powers, matching the stated cubic model.

Common mistake avoided: Do not cube the raw predictor and call it the centered cubic feature. At $x=8$, the raw cube is $8^3=512$, whereas the centered cube is $(8-6)^3=8$. They are different columns with different coefficients.

Practical Checks and Common Traps

Polynomial features are most useful when a curved effect is plausible and the predictor has a meaningful continuous scale. Before interpreting a fitted model, check that the requested powers were actually constructed, that their ranges are consistent with the original values, and that the prediction being discussed falls within a range where the model is supported by data.

Reading a coefficient as a constant effect. In a model with powers, a one-unit change in $x$ generally changes the fitted mean by an amount that depends on $x$.
Confusing centered and raw powers. The centered square is $(x-c)^2$, not $x^2-c^2$. Expand it if needed to verify the relationship.
Checking only endpoints. For $x$ values that straddle zero, the minimum of $x^2$ can occur inside the range. Inspect the transformed values or use a suitable range check.
Assuming more powers are always better. Higher-degree terms can create rapid changes in the fitted curve, particularly near the edges. A mathematical fit over the observed values is not evidence that extrapolation is dependable.
Using a formula operator as arithmetic. In R, use I(x^2) for a raw squared feature in a model formula, and distinguish that from the default orthogonal columns returned by poly().

Key takeaway: Polynomial features let a model represent curvature by combining $x$ with powers such as $x^2$ and $x^3$. Centering changes the feature values and coefficients but, when the full polynomial basis is retained, preserves the same fitted curve. Verify the construction and observed ranges before interpreting the result.

Check Your Understanding

Work through each calculation and check that the feature values and predictions agree.

A predictor takes values 3, 5, and 7. Construct its squared feature and report the ranges of the original predictor and its square. (Answer: $x^2=9,25,49$; ranges are 3 to 7 and 9 to 49.)
A fitted mean is $f(x)=4+2x-0.25x^2$. Calculate $f(2)$, $f(6)$, and the change $f(6)-f(2)$. (Answer: $f(2)=7$, $f(6)=7$, change $=0$.)
For $x=8$ and center $c=5$, calculate $z$, $z^2$, and verify $x^2=c^2+2cz+z^2$. (Answer: $z=3$, $z^2=9$, and $64=25+30+9$.)
An R model needs the raw quadratic predictor effect. Which formula is appropriate, and why is the unwrapped caret not the arithmetic square? (Answer: use `y ~ x + I(x^2)`; the caret has a formula-expansion meaning, so `I()` specifies arithmetic squaring.)
Values of $x$ run from 1 to 5 and $z=x-3$. Give the range of $z$, the range of $z^2$, and the fitted value at $x=5$ for $f(x)=3-0.2z+0.05z^2$. (Answer: $z$ ranges from $-2$ to 2, $z^2$ from 0 to 4, and $f(5)=3-0.4+0.2=2.8$.)

What You'll Learn