Why Polynomial Terms Still Belong to Linear Regression
A straight-line mean model can miss systematic curvature. One way to represent curvature is to add powers of a predictor, such as \(x^2\) or \(x^3\), to the regression. The response mean may then be a curved function of \(x\), but the model can still be estimated by the ordinary least-squares methods used for linear regression.
The key distinction is between being linear in the predictor and being linear in the coefficients. A polynomial model is generally not a straight line as \(x\) changes. It is nevertheless a linear regression model when each unknown coefficient enters as a first-power multiplier of a known predictor column.
For example, \(\beta_0+\beta_1x+\beta_2x^2\) is linear in the coefficients, even though it is usually curved in \(x\). By contrast, a model such as \(\beta_0+\exp(\beta_1x)\) is not linear in its coefficients as written: the unknown \(\beta_1\) is inside an exponential function. This distinction determines whether the usual linear-model least-squares calculation applies directly.
The Polynomial Design Matrix
For observations \((x_i,y_i)\), a quadratic model uses the row \((1,x_i,x_i^2)\) for observation \(i\); a cubic model adds \(x_i^3\). In matrix notation, the model is \(Y=X\beta+\varepsilon\), where \(Y\) is the response vector, \(X\) is the design matrix, \(\beta\) is the coefficient vector, and \(\varepsilon\) is the error vector. The entries of \(X\) are constructed from the observed predictor values, so they are known when estimating the coefficients.
The ordinary least-squares estimate selects the coefficient vector that minimizes the sum of squared residuals. The standard assumptions for the basic linear model are that the conditional mean is correctly specified, the predictor values are treated as fixed, and the errors have mean zero and constant variance \(\sigma^2\), with no correlation between distinct observations. Normal errors are additionally assumed when exact small-sample normal-theory inference is required; they are not necessary to define the least-squares estimate.
For a polynomial of degree \(d\), \(d+1\) distinct predictor values guarantee full column rank. To see why, suppose a linear combination of its columns were zero at every observed predictor value. That combination would be a polynomial of degree at most \(d\) with at least \(d+1\) distinct roots. A nonzero polynomial of degree at most \(d\) cannot have more than \(d\) distinct roots, so the combination must be the zero polynomial and all its coefficients must be zero. This guarantees a unique coefficient estimate. It does not guarantee that the polynomial is a good description of the mean or that predictions will be reliable outside the observed range.
Worked Example: Checking a Quadratic Least-Squares Fit
A hypothetical insurer studies a response \(Y\), measured in thousands of dollars, against a numerical rating \(x\). The four observations are \((-2,2),(-1,-0.5),(1,7.5),(2,8)\). Consider the quadratic model \(Y_i=\beta_0+\beta_1x_i+\beta_2x_i^2+\varepsilon_i\). Assume the errors have mean zero, common variance \(\sigma^2\), and are uncorrelated; normality is not needed for the least-squares check below.
Method. Construct the quadratic design, then verify that the proposed coefficient estimates satisfy the least-squares normal equations. For \(\hat{\beta}=(3,2,0.5)^\mathsf{T}\), the fitted value is \(3+2x+0.5x^2\).
| \(x_i\) | \(y_i\) | \(1\) | \(x_i\) | \(x_i^2\) | \(\hat{y}_i\) | \(e_i=y_i-\hat{y}_i\) |
|---|---|---|---|---|---|---|
| -2 | 2 | 1 | -2 | 4 | 1 | 1 |
| -1 | -0.5 | 1 | -1 | 1 | 1.5 | -2 |
| 1 | 7.5 | 1 | 1 | 1 | 5.5 | 2 |
| 2 | 8 | 1 | 2 | 4 | 9 | -1 |
The fitted values are calculated row by row: at \(x=-2\), \(3-4+2=1\); at \(x=-1\), \(3-2+0.5=1.5\); at \(x=1\), \(3+2+0.5=5.5\); and at \(x=2\), \(3+4+2=9\). Subtracting these fitted values from the observations gives residuals \(1,-2,2,-1\). Thus, the polynomial gives the fitted values; it does not describe the observed responses exactly.
To verify the least-squares fit, check that each design column has zero inner product with the residual vector. For the intercept column, \(1-2+2-1=0\). For the \(x\) column, \((-2)(1)+(-1)(-2)+(1)(2)+(2)(-1)=0\). For the \(x^2\) column, \(4(1)+1(-2)+1(2)+4(-1)=0\). Therefore \(X^\mathsf{T}e=0\), the normal-equation condition, so these coefficients are the least-squares estimates. The residual sum of squares is \(1^2+(-2)^2+2^2+(-1)^2=10\). There are four distinct predictor values, more than the three needed for full rank in a quadratic design.
Check and common mistake. The residuals are not all zero, so it would be incorrect to say the observations lie exactly on the fitted polynomial. Their zero inner products with all three design columns verify least squares, not a perfect fit. The nonzero residual sum of squares, \(10\), confirms that distinction.
Why the Ordinary Least-Squares Formula Applies
The formula for \(\hat{\beta}\) follows from minimizing squared error, not from the predictor being a straight line. Once the polynomial terms are placed in the columns of \(X\), the fitted vector is \(X\beta\), just as in any other linear regression. The minimization is with respect to the coefficients.
For a candidate coefficient vector \(b\), the residual vector is \(Y-Xb\), and the residual sum of squares is \(Q(b)=(Y-Xb)^\mathsf{T}(Y-Xb)\).
Expanding gives \(Q(b)=Y^\mathsf{T}Y-2b^\mathsf{T}X^\mathsf{T}Y+b^\mathsf{T}X^\mathsf{T}Xb\). Its gradient is \(-2X^\mathsf{T}Y+2X^\mathsf{T}Xb\).
A least-squares solution satisfies \(X^\mathsf{T}Xb=X^\mathsf{T}Y\), the normal equations. If \(X\) has full column rank, \(X^\mathsf{T}X\) is invertible, yielding \(\hat{\beta}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}Y\).
The derivation uses only the design matrix and the squared-error criterion. It therefore applies when the columns are \(x,x^2,x^3\), or other known predictor functions, provided the model remains linear in its coefficients and the design has full column rank. The assumptions about error means, variances, and dependence concern the statistical properties of the estimates and residuals; they do not change this algebraic minimization.
Worked Example: Building a Cubic Design and Verifying Its Fit
For a hypothetical response \(Y\), consider five predictor values \(x=-2,-1,0,1,2\) and the cubic model \(Y_i=\beta_0+\beta_1x_i+\beta_2x_i^2+\beta_3x_i^3+\varepsilon_i\). The observed responses are \(6.5,3.25,7,1.75,6.5\), respectively. Assume errors have mean zero and common variance and are uncorrelated. The proposed coefficient estimates are \((4,-1,0.5,0.25)\).
Construct the design. For each \(x\), calculate the four entries \(1,x,x^2,x^3\). The rows are \((1,-2,4,-8)\), \((1,-1,1,-1)\), \((1,0,0,0)\), \((1,1,1,1)\), and \((1,2,4,8)\). This design has five distinct \(x\) values, so its four columns have full rank.
Calculate fitted values and residuals. At \(x=-2\), the fitted value is \(4+2+2-2=6\), giving residual \(6.5-6=0.5\). At \(x=-1\), it is \(4+1+0.5-0.25=5.25\), giving residual \(3.25-5.25=-2\). At \(x=0\), it is \(4\), giving residual \(7-4=3\). At \(x=1\), it is \(4-1+0.5+0.25=3.75\), giving residual \(1.75-3.75=-2\). At \(x=2\), it is \(4-2+2+2=6\), giving residual \(6.5-6=0.5\).
Check least squares. The residual vector is \((0.5,-2,3,-2,0.5)\). Its products summed against the intercept column give \(0.5-2+3-2+0.5=0\). Against \(x\), the sum is \(-1+2+0-2+1=0\). Against \(x^2\), it is \(2-2+0-2+2=0\). Against \(x^3\), it is \(-4+2+0-2+4=0\). Thus \(X^\mathsf{T}e=0\), confirming the proposed coefficients are the least-squares estimates. The residual sum of squares is \(0.25+4+9+4+0.25=17.5\).
Check and common mistake. There are five distinct predictor values, not four, and all four polynomial columns are linearly independent. Do not count columns as distinct predictor values: the rank conclusion follows here from the five distinct \(x\) values and the degree-three design. The nonzero residuals also show that the cubic fitted values are not the observed responses themselves.
Using a Polynomial Design for a Fitted Prediction
After estimating coefficients, prediction uses the same design-row construction. For a new predictor value \(x_*\), the row is \((1,x_*,x_*^2,\ldots,x_*^d)\); multiplying this row by the coefficient estimates gives the fitted mean at \(x_*\). This calculation does not change the estimation method. The numerical example below focuses on constructing that row and evaluating the fitted polynomial.
Worked Example: Constructing a Quadratic Prediction
A hypothetical insurer models average claim-processing expense in thousands of dollars using a rating \(x\) and a quadratic regression. Suppose a least-squares fit has produced \(\hat{\beta}_0=12\), \(\hat{\beta}_1=3\), and \(\hat{\beta}_2=-0.5\). Assume the underlying regression errors have mean zero and constant variance; the coefficients here are supplied as fitted estimates, so no refitting is required.
Method. At \(x_*=4\), the quadratic design row is \((1,4,4^2)=(1,4,16)\). Multiply it by the estimated coefficient vector: \(12(1)+3(4)+(-0.5)(16)=12+12-8=16\). The fitted mean is \(16\) thousand dollars, or \(\$16{,}000\).
As a check on the row construction, at \(x_*=2\) the row is \((1,2,4)\), giving \(12+6-2=16\) thousand dollars. In both calculations the coefficient of \(x^2\) multiplies the squared predictor value. A common mistake is to use \(x\) in that column—for \(x=4\), that would give \(12+12-2=22\) thousand dollars, which is not the specified quadratic model's fitted mean.
What Polynomial Terms Do—and Do Not—Change
Adding polynomial columns changes the set of possible mean patterns and increases the number of coefficients to estimate. It does not, by itself, require a nonlinear optimization method. The estimation remains ordinary least squares as long as the coefficients enter linearly and the design matrix has full column rank.
A polynomial model also does not transform the response. The response remains \(Y\); it is the predictor that contributes additional columns. This differs from a log-response model, for example, where the response scale is changed. Nor does adding powers automatically establish that a higher-degree model is appropriate: adequacy, uncertainty, and the usefulness of predictions remain matters for statistical assessment.
| Model form | Linear in coefficients? | Reason |
|---|---|---|
| \(\beta_0+\beta_1x+\beta_2x^2\) | Yes | Each unknown coefficient enters as a multiplier of a known column. |
| \(\beta_0+\beta_1x+\beta_2x^3\) | Yes | The cubic column is known once \(x\) is observed. |
| \(\beta_0+\exp(\beta_1x)\) | No, as written | The unknown coefficient is inside an exponential function. |
| \(\beta_0+\beta_1^2x\) | No, as written | The coefficient itself enters as a square rather than linearly. |
- Count parameters correctly. A quadratic model with an intercept has three coefficients; a cubic model with an intercept has four.
- Build powers from the observed predictor. The quadratic column contains \(x_i^2\), and the cubic column contains \(x_i^3\).
- Check the rank condition. At least \(d+1\) distinct predictor values are needed for a unique degree-\(d\) polynomial fit with an intercept.
- Keep fitted values and observations separate. Residuals are observed responses minus fitted values; a nonzero residual means the model did not fit that observation exactly.
The central skill is to identify what is known when the regression is fitted: the observed predictor values and their powers form fixed design columns, while the coefficients are unknown. This distinction is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus and is essential whenever a regression includes polynomial terms.
Check Your Understanding
For each problem, construct or use the polynomial design carefully before checking the answer.
- A quadratic regression includes an intercept. What design row corresponds to \(x=3\), and how many coefficients does the model contain? (Answer: The row is \((1,3,9)\), and there are 3 coefficients.)
- A cubic model uses \(x=-1\). Write its design row in the order \(1,x,x^2,x^3\). (Answer: \((1,-1,1,-1)\).)
- A fitted quadratic model has coefficients \((1,-0.5,0.25)\). Calculate its fitted value at \(x=2\). (Answer: \(1+(-0.5)(2)+0.25(4)=1\).)
- A quadratic design uses predictor values \(-1,0,1\). Does it have full column rank, and how many distinct predictor values are present? (Answer: Yes; it has 3 distinct values, which is sufficient for the 3 columns.)
- A quadratic model has four observations at \(x=0,1,1,2\). How many distinct predictor values are present, and is the design full rank? (Answer: There are 3 distinct values, so the quadratic design has full column rank.)