A Worked Best-Fit Line From Data

From Data to a Best-Fit Line

Given data points \((x_i,y_i)\), a line will not generally pass through every point. A best-fit line instead chooses its intercept and slope to make the total squared vertical error as small as possible. The calculation can be organized as a least-squares problem: the unknown coefficients form a vector, and the columns of a design matrix describe the constant and \(x\)-dependent parts of the line.

Here we work over \(\mathbb{R}\), with \(n\) data points and not all \(x_i\) equal. Write a candidate line as \(\widehat y=a+bx\), where \(a\) is the intercept and \(b\) is the slope. Its residual at data point \(i\) is \(r_i=y_i-(a+bx_i)\). The least-squares line minimizes the sum of squared residuals, \(\sum_{i=1}^n r_i^2\). In matrix form, the coefficient vector is \(\mathbf{c}=\begin{bmatrix}a\\b\end{bmatrix}\), and each row of the design matrix records the constant term and the corresponding \(x_i\).

The first column of \(\mathbf{X}\) represents the intercept; its second column represents the slope contribution. Thus \(\mathbf{X}\mathbf{c}\) contains the fitted values \(a+bx_i\). When the \(x_i\) are not all equal, these two columns are linearly independent: if \(\alpha+\beta x_i=0\) for every \(i\), subtracting the equations for two distinct \(x_i\) gives \(\beta=0\), and then \(\alpha=0\). This condition makes the least-squares coefficients unique.

Why the Normal Equations Identify the Best Fit

For a candidate coefficient vector \(\mathbf{c}\), the residual is \(\mathbf{r}=\mathbf{y}-\mathbf{X}\mathbf{c}\). The normal equations are \(\mathbf{X}^{\top}\mathbf{X}\mathbf{c}=\mathbf{X}^{\top}\mathbf{y}\), equivalently \(\mathbf{X}^{\top}\mathbf{r}=\mathbf{0}\). For a line, this says that the residual is orthogonal to each of the two design-matrix columns: the constant column and the \(x\)-values column.

Proof. For any change \(\mathbf{h}\in\mathbb{R}^p\), the residual from the coefficients \(\mathbf{c}+\mathbf{h}\) is \(\mathbf{r}-\mathbf{X}\mathbf{h}\). Expanding its squared norm gives

If \(\mathbf{X}^{\top}\mathbf{r}=\mathbf{0}\), the middle term vanishes, and the last term is nonnegative. Therefore no change \(\mathbf{h}\) can reduce the squared residual, so \(\mathbf{c}\) is a minimizer. Conversely, suppose \(\mathbf{c}\) is a minimizer. For any \(\mathbf{h}\), changing the coefficients to \(\mathbf{c}+t\mathbf{h}\), where \(t\in\mathbb{R}\), changes the squared residual by \(-2t\mathbf{h}^{\top}\mathbf{X}^{\top}\mathbf{r}+t^2\|\mathbf{X}\mathbf{h}\|^2\). If \(\mathbf{h}^{\top}\mathbf{X}^{\top}\mathbf{r}\ne0\), choosing \(t\) with the same sign as this quantity and sufficiently small magnitude makes this change negative: the linear term is negative and dominates the quadratic term. That contradicts minimality. Thus \(\mathbf{h}^{\top}\mathbf{X}^{\top}\mathbf{r}=0\) for every \(\mathbf{h}\); taking \(\mathbf{h}\) to be each standard basis vector gives \(\mathbf{X}^{\top}\mathbf{r}=\mathbf{0}\). Finally, if the columns are independent, then \(\mathbf{X}\mathbf{h}=\mathbf{0}\) implies \(\mathbf{h}=\mathbf{0}\). For a minimizer, the displayed expansion reduces to \(\|\mathbf{r}\|^2+\|\mathbf{X}\mathbf{h}\|^2\), which is strictly larger whenever \(\mathbf{h}\ne\mathbf{0}\). Hence the minimizer is unique. \(\square\)

Worked Example: Solve the Normal Equations

The Closed-Form Coefficients

The same calculation can be written using averages and centered data. Define \(\bar x=\frac1n\sum_{i=1}^n x_i\) and \(\bar y=\frac1n\sum_{i=1}^n y_i\). Provided the \(x_i\) are not all equal, the best-fit line has slope and intercept

These formulas follow by writing the two normal equations as \(na+b\sum x_i=\sum y_i\) and \(a\sum x_i+b\sum x_i^2=\sum x_i y_i\). The first gives \(a=\bar y-b\bar x\). Substituting this into the second gives \(b\bigl(\sum x_i^2-(\sum x_i)^2/n\bigr)=\sum x_i y_i-(\sum x_i)(\sum y_i)/n\). The parenthesized expression equals \(\sum(x_i-\bar x)^2\), and the right side equals \(\sum(x_i-\bar x)(y_i-\bar y)\), yielding the stated slope. The denominator is a sum of squares; it is zero only if every \(x_i=\bar x\).

Worked Example: A Second Dataset

Common Pitfall: Omitting the Intercept

A frequent setup error is using the single-column design matrix \(\begin{bmatrix}0&1&2&3\end{bmatrix}^{\top}\) for the first dataset. That matrix fits lines of the form \(\widehat y=bx\), which are forced through the origin; it does not fit the general line \(a+bx\). For that restricted model the least-squares slope is \(\sum x_i y_i/\sum x_i^2=18/14=9/7\), not \(9/10\). Its residuals are \(1,5/7,-4/7,1/7\), whose sum is \(1+5/7-4/7+1/7=9/7\ne0\). This fails the constant-column orthogonality condition because the constant column was omitted. The correction is to include a column of ones whenever the model includes an intercept.

For a correctly formed line-fitting design matrix, the two orthogonality checks have a direct interpretation: residuals sum to zero, and their \(x\)-weighted sum is zero. They are not separate rules to memorize; together they say \(\mathbf{X}^{\top}\mathbf{r}=\mathbf{0}\), which certifies the least-squares minimum. When the \(x\)-values vary, the fit is unique, and either the normal equations or the centered-sum formulas produces the same coefficients.