Polynomial Regressions and Functional-Form Error

Why Allow a Regression to Bend?

A straight-line regression imposes a constant fitted change in the outcome for a one-unit increase in the regressor. That restriction can be too strong: an outcome might rise quickly at first and then level off, or the fitted relationship might change direction. Polynomial regressions relax the straight-line shape by adding powers of a regressor, while remaining linear in the coefficients estimated by ordinary least squares (OLS).

A polynomial is still a particular functional form, not a guarantee that the conditional mean has been captured correctly. If the underlying relationship curves but the fitted regression is linear, the resulting approximation error can affect both fitted values and the OLS coefficients. The size and direction of that effect depend partly on which regressor values are common in the population.

For observations \((Y_i,X_i)\), the degree-\(K\) OLS regression minimizes the sum of squared residuals over coefficients:

This is an ordinary linear regression whose columns include powers of \(X\). The design matrix must have full column rank for the coefficients to be uniquely determined. In practice, the observed \(X_i\) values must provide enough distinct variation to distinguish the included powers. A polynomial can be fitted without assuming it is the true conditional mean; in that case it gives a least-squares projection onto the chosen set of powers.

Reading a Fitted Polynomial’s Shape

The coefficients of a polynomial are not, in general, constant marginal effects. The fitted change in \(Y\) for a small change in \(x\) is described by the derivative of the fitted polynomial. For a quadratic, the derivative varies linearly with \(x\); for a cubic, it is quadratic in \(x\). These derivatives describe the fitted curve, not automatically a causal effect.

For a cubic, the derivative can have up to two roots, so the fitted curve can change direction more than once. Its second derivative describes how the slope changes; a point where the second derivative changes sign is an inflection point.

Omitted Curvature Changes a Linear Projection

A linear fit to a curved conditional mean is not generally the same line one would obtain by retaining the linear part of the underlying equation. OLS chooses the line that minimizes squared approximation error over the regressor values represented in the population or sample. Consequently, the linear coefficient can absorb some of the omitted curvature.

To see this precisely, suppose the population relationship is \(Y=a+bX+cX^2+u\), where \(E[u\mid X]=0\), \(0<\operatorname{Var}(X)<\infty\), and the moments needed below exist. Fit a population linear projection with an intercept, \(Y=\gamma_0+\gamma_1X+e\). The normal equations imply that the projection slope is \(\operatorname{Cov}(X,Y)/\operatorname{Var}(X)\). Conditional mean zero gives \(\operatorname{Cov}(X,u)=0\), so direct substitution yields:

The intercept is \(\gamma_0=E[Y]-\gamma_1E[X]\). Unless \(c=0\) or \(\operatorname{Cov}(X,X^2)=0\), the linear projection slope differs from \(b\). The residual from the linear projection includes a systematic approximation component, even though the disturbance \(u\) has conditional mean zero. This is functional-form error: the chosen regression basis does not represent the conditional mean exactly.

Polynomial Degree, Approximation, and Interpretation

Adding powers expands the set of curves available to the regression. A linear model can fit a straight line, a quadratic can represent one bend, and a cubic can represent changing curvature. If the higher-degree regression contains all the lower-degree regressors and its design matrix has full rank, its least-squares objective cannot be worse in the sample: it can always reproduce the lower-degree fit by setting the added coefficients to zero. This fact alone does not establish that the higher-degree curve better represents the population relationship.

For an approximate polynomial, the population projection minimizes average squared error over the distribution of \(X\). Values of \(X\) that occur more often therefore have more influence on the fitted coefficients. Even when the same underlying function is being approximated, changing the distribution of \(X\) can change the projection coefficients. This is why coefficients from different samples or populations need not have a simple structural interpretation when the polynomial is only an approximation.

The coefficient on \(x\) in a quadratic or cubic is the fitted slope at \(x=0\), not a universal marginal effect. For a quadratic, \(\hat\beta_1+2\hat\beta_2x\) is the fitted slope at \(x\); for a cubic, it is \(\hat\beta_1+2\hat\beta_2x+3\hat\beta_3x^2\). If zero is outside the relevant range or has no useful meaning, focus on fitted values and slopes at relevant observed values instead.

Common Mistakes

• Reading a polynomial coefficient as a constant slope. The coefficient on \(x\) is not the slope everywhere once powers of \(x\) are included. Differentiate the fitted polynomial to obtain its slope at a specified \(x\).
• Assuming a linear fit recovers the linear term of a curved relationship. The projection example shows that omitted curvature can change the slope as well as the intercept. The fitted coefficient is determined by the projection, not by selecting one term from the underlying equation.
• Calling a polynomial regression nonlinear in its parameters. Powers such as \(x^2\) and \(x^3\) are transformed regressors; the unknown coefficients still enter linearly and can be estimated by OLS.
• Extrapolating a fitted polynomial without caution. A polynomial’s behavior outside the observed range may be very different from its behavior where data were observed. Shape interpretations should be tied to that range.