Comparing Polynomial Fits with Design Matrices

From Polynomial Models to Design Matrices

Suppose measurements are recorded as pairs \((x_i,y_i)\), where \(x_i\) is an input and \(y_i\) is its observed response. A polynomial model predicts the response using powers of the input. The design matrix organises those powers into columns, so fitting the model becomes a least-squares problem in linear algebra. Comparing polynomial degrees then means comparing the column spaces of different design matrices—not simply counting observations or comparing the number of coefficients.

Throughout, assume the inputs and responses are real. There are \(m\) observations, with input vector \(\mathbf{x}=(x_1,\dots,x_m)^\top\) and response vector \(\mathbf{y}=(y_1,\dots,y_m)^\top\). A polynomial of degree at most \(d\) has the form \(p(x)=\beta_0+\beta_1x+\cdots+\beta_dx^d\). The fitted values at the observed inputs are \(p(x_1),\dots,p(x_m)\). The degree is an upper bound: the coefficient \(\beta_d\) may be zero, so the resulting polynomial may have lower degree.

The number of observations is the number of rows, \(m\). The number of coefficient positions is the number of columns, \(d+1\). These counts answer different questions. For instance, four observations and a quadratic model give a \(4\times3\) design matrix: there are four data points but three coefficients.

Computing and Checking a Fit

For any real matrix \(\mathbf{A}\), its column space \(\operatorname{Col}\mathbf{A}\) is the set of all vectors \(\mathbf{A}\boldsymbol{\beta}\). Least squares seeks the vector in this column space closest to \(\mathbf{y}\). The projection property of least squares says that the residual at a fit is orthogonal to every column of the design matrix. Consequently, a least-squares coefficient vector satisfies the normal equations \(\mathbf{A}^{\top}\mathbf{A}\boldsymbol{\beta}=\mathbf{A}^{\top}\mathbf{y}\). If the columns are dependent, the coefficient vector need not be unique; the fitted values are still unique.

Why Increasing the Degree Cannot Worsen the Fit

For degrees \(d_1\le d_2\), every degree-\(d_1\) polynomial is also a degree-\(d_2\) polynomial: its additional coefficients can be set to zero. The design-matrix version of this fact is that the first \(d_1+1\) columns of \(\mathbf{A}_{d_2}\) are exactly the columns of \(\mathbf{A}_{d_1}\). Thus \(\operatorname{Col}\mathbf{A}_{d_1}\subseteq\operatorname{Col}\mathbf{A}_{d_2}\). The following result makes the consequence for residuals precise.

Proof. A least-squares fitted-value vector is the orthogonal projection of \(\mathbf{y}\) onto its design matrix's column space. Hence \(\mathbf{y}-\widehat{\mathbf{y}}_2\) is orthogonal to every vector in \(S_2\). Both fitted vectors belong to \(S_2\), because \(S_1\subseteq S_2\); therefore \(\widehat{\mathbf{y}}_2-\widehat{\mathbf{y}}_1\in S_2\). Write \(\mathbf{y}-\widehat{\mathbf{y}}_1=(\mathbf{y}-\widehat{\mathbf{y}}_2)+(\widehat{\mathbf{y}}_2-\widehat{\mathbf{y}}_1)\). The two terms on the right are orthogonal, so expanding the squared norm gives \(\|\mathbf{y}-\widehat{\mathbf{y}}_1\|^2=\|\mathbf{y}-\widehat{\mathbf{y}}_2\|^2+\|\widehat{\mathbf{y}}_2-\widehat{\mathbf{y}}_1\|^2\). The last term is nonnegative, proving the inequality. It equals zero exactly when the two fitted-value vectors are equal, which proves the equality condition as well. \(\square\)

Polynomial design matrices have nested column spaces as degree increases, so this theorem applies directly to their least-squares fits. A larger degree cannot increase the sum of squared residuals on the same observations. It may leave that sum unchanged: this happens precisely when the added columns do not change the fitted values for the response being considered. This is a statement about the observed data and fitted values, not a guarantee that a higher-degree model is preferable for other purposes.

Coefficient Uniqueness and the Number of Observations

A fitted-value vector can be unique even when several coefficient vectors produce it. The distinction is governed by the null space \(\operatorname{Nul}\mathbf{A}\), the set of coefficient vectors \(\mathbf{z}\) satisfying \(\mathbf{A}\mathbf{z}=\mathbf{0}\).

Proof. Least squares has a fitted-value vector \(\widehat{\mathbf{y}}\) in \(\operatorname{Col}\mathbf{A}\), the projection of \(\mathbf{y}\) onto that finite-dimensional column space. If \(\operatorname{Nul}\mathbf{A}=\{\mathbf{0}\}\), any two minimizing coefficient vectors \(\boldsymbol{\beta}\) and \(\boldsymbol{\gamma}\) must give this same fitted vector, so \(\mathbf{A}\boldsymbol{\beta}=\mathbf{A}\boldsymbol{\gamma}\). Thus \(\mathbf{A}(\boldsymbol{\beta}-\boldsymbol{\gamma})=\mathbf{0}\); the null-space assumption implies \(\boldsymbol{\beta}-\boldsymbol{\gamma}=\mathbf{0}\), so the coefficients are unique. Conversely, if \(\mathbf{z}\ne\mathbf{0}\) and \(\mathbf{A}\mathbf{z}=\mathbf{0}\), then for any minimizing coefficient vector \(\boldsymbol{\beta}\), the distinct vector \(\boldsymbol{\beta}+\mathbf{z}\) gives \(\mathbf{A}(\boldsymbol{\beta}+\mathbf{z})=\mathbf{A}\boldsymbol{\beta}\). It therefore gives the same fitted values and the same least-squares error, so the minimizing coefficients are not unique. \(\square\)

A degree-\(d\) polynomial model has \(d+1\) coefficient positions, regardless of the number of observations. If there are fewer than \(d+1\) observations, the matrix has more columns than rows, so its columns cannot be independent; a nonzero null vector exists, and the coefficients cannot be unique. Even with at least \(d+1\) observations, repeated inputs can make columns dependent. The next example shows why a high-degree fit can have exact fitted values without identifying all its coefficients.

Repeated Inputs and an Unchanged Fit

Common Pitfall: Confusing Degree with Data Count

A common mistake is to say that a quadratic model has two coefficients because its degree is two, or that three observations automatically mean a quadratic model. Both statements confuse different counts. A polynomial of degree at most \(d\) has \(d+1\) coefficient positions, including the constant term. The number of observations determines the number of rows, not the degree.

For example, with three observations, the degree-three design matrix has four columns, not three. It cannot have four independent columns, and as the cubic example shows, its coefficients may not be unique even when the observations are fit exactly. Conversely, four observations can be fitted with a degree-one model, which still has just two coefficient positions. Before solving, write down the powers included in the model and count the columns. When comparing degrees, keep the response vector and observations fixed, and compare the fitted values or residuals rather than assuming the higher-degree coefficients must all be identifiable.

Polynomial design matrices are useful because they translate curved trends into linear combinations of columns: the unknown coefficients enter linearly even though the inputs appear as powers. The projection viewpoint then gives a reliable comparison. Higher degree enlarges the available column space, so the best residual cannot become worse; whether it becomes smaller depends on whether the enlarged space changes the projection of the observed response.