From a Best-Fit Line to Two Equations
Suppose we have \(n\) data points \((x_1,y_1),\dots,(x_n,y_n)\), where each \(x_i\) and \(y_i\) is real. We want a line of the form \(y=a+bx\), with intercept \(a\) and slope \(b\), that fits the observed values as closely as possible in the least-squares sense. That means choosing \(a\) and \(b\) to minimize the sum of squared residuals, where a residual is the observed value minus its fitted value.
In matrix form, the design matrix has a column of ones for the intercept and a column of input values for the slope. The least-squares normal equations state that the residual vector is orthogonal to both columns. Here this gives two scalar equations. We will solve them and express the result using the sample means and centered data. The key condition is that the inputs cannot all be equal if the slope and intercept are to be determined uniquely.
Write the design matrix as \(\mathbf{A}=\begin{bmatrix}1&x_1\\ \vdots&\vdots\\1&x_n\end{bmatrix}\), the coefficient vector as \(\begin{bmatrix}a\\b\end{bmatrix}\), and the data vector as \(\begin{bmatrix}y_1\\ \vdots\\y_n\end{bmatrix}\). The two columns of \(\mathbf{A}\) are the vector of ones and the input vector. The normal equations say that \(\mathbf{A}^{\top}\mathbf{r}=\mathbf{0}\). In terms of the residuals, this is \(\sum_i r_i=0\) and \(\sum_i x_i r_i=0\).
The first equation expresses that the residuals sum to zero; the second expresses that the residuals have zero dot product with the input values. Both are needed: solving only the first equation chooses an intercept for a fixed slope, but does not by itself determine the least-squares slope.
Why the Normal Equations Characterize a Least-Squares Fit
The normal equations are useful because they characterize minimizers, not merely possible coefficients. The following argument also explains why checking residual orthogonality at the end of a calculation is a meaningful check.
Proof. First suppose \(\mathbf{A}^{\top}\mathbf{r}=\mathbf{0}\). For any other coefficient vector \(\mathbf{d}\), write \(\mathbf{h}=\mathbf{d}-\mathbf{c}\). Its residual is \(\mathbf{y}-\mathbf{A}\mathbf{d}=\mathbf{r}-\mathbf{A}\mathbf{h}\). Expanding the squared norm gives
because \(\mathbf{r}^{\top}\mathbf{A}\mathbf{h}=(\mathbf{A}^{\top}\mathbf{r})^{\top}\mathbf{h}=0\). Thus \(\mathbf{c}\) minimizes the squared residual norm. Conversely, suppose \(\mathbf{c}\) is a minimizer. If there were an \(\mathbf{h}\) with \(z=\mathbf{r}^{\top}\mathbf{A}\mathbf{h}\ne0\), then \(\mathbf{A}\mathbf{h}\ne\mathbf{0}\). For any real \(t\), the coefficient vector \(\mathbf{c}+t\mathbf{h}\) has squared residual norm \(\|\mathbf{r}\|^2-2tz+t^2\|\mathbf{A}\mathbf{h}\|^2\). Choosing \(t=z/\|\mathbf{A}\mathbf{h}\|^2\) makes this \(\|\mathbf{r}\|^2-z^2/\|\mathbf{A}\mathbf{h}\|^2\), which is strictly smaller than \(\|\mathbf{r}\|^2\). This contradicts minimality. Therefore \(\mathbf{r}^{\top}\mathbf{A}\mathbf{h}=0\) for every \(\mathbf{h}\in\mathbb{R}^2\), which means \(\mathbf{A}^{\top}\mathbf{r}=\mathbf{0}\). \(\square\)
Centering the Data Reveals the Slope
To solve the two normal equations, first rewrite their right sides using the means. The first equation gives \(a=\bar{y}-b\bar{x}\). Substitute this expression for \(a\) into the second equation. The resulting equation isolates the slope and shows why deviations from the mean, rather than the raw inputs alone, govern its value.
Since \(\sum_i x_i=n\bar{x}\), the quantity multiplying \(b\) is \(\sum_i(x_i-\bar{x})^2\). The remaining difference on the right is \(\sum_i(x_i-\bar{x})(y_i-\bar{y})\). These identities follow by expanding the products and using \(\sum_i(x_i-\bar{x})=0\) and \(\sum_i(y_i-\bar{y})=0\).
Proof. Set \(S_{xx}=\sum_i(x_i-\bar{x})^2\) and \(S_{xy}=\sum_i(x_i-\bar{x})(y_i-\bar{y})\). The second normal equation minus \(\bar{x}\) times the first is \(bS_{xx}=S_{xy}\), by the identities above. Each square in \(S_{xx}\) is nonnegative, and \(S_{xx}=0\) exactly when every \(x_i=\bar{x}\), which is exactly when all inputs are equal. If they are not all equal, \(S_{xx}>0\), so \(b=S_{xy}/S_{xx}\) is forced, and the first normal equation then forces \(a=\bar{y}-b\bar{x}\). These coefficients satisfy both equations, so the preceding theorem shows they give a least-squares fit. The equations force both coefficients, proving uniqueness. If instead every \(x_i=c\), then every fitted value is the same number \(q=a+bc\). For any such \(q\), direct expansion gives \(\sum_i(y_i-q)^2=\sum_i(y_i-\bar{y})^2+n(q-\bar{y})^2\), because \(\sum_i(y_i-\bar{y})=0\). This is minimized exactly when \(q=\bar{y}\). Thus the minimizing coefficient pairs are precisely those with \(a+bc=\bar{y}\); there are multiple such pairs, and the slope is not uniquely determined. \(\square\)
The numerator in the slope formula is a centered cross-product sum: it compares each input’s deviation from its mean with the corresponding output’s deviation from its mean. The denominator measures the spread of the inputs about their mean. The intercept then places the fitted line through the point \((\bar{x},\bar{y})\), since \(a+b\bar{x}=\bar{y}\). This does not mean the line passes through every data point; it identifies a particular point the least-squares line must pass through when the coefficients are unique.
Worked Calculations
Worked Example: Solving the Normal Equations Directly
Find the least-squares line for the observations \((0,1),(1,2),(2,2)\). First compute the sums needed in the normal equations: \(n=3\), \(\sum x_i=0+1+2=3\), \(\sum y_i=1+2+2=5\), \(\sum x_i^2=0+1+4=5\), and \(\sum x_i y_i=0+2+4=6\). Thus the equations are
Subtract the first equation from the second to get \(2b=1\), so \(b=\frac12\). Substitute into the first: \(3a+\frac32=5\), hence \(3a=\frac72\) and \(a=\frac76\). Therefore the line is \(y=\frac76+\frac12x\).
Check: The coefficients satisfy the two equations: \(3\left(\frac76\right)+3\left(\frac12\right)=\frac72+\frac32=5\), and \(3\left(\frac76\right)+5\left(\frac12\right)=\frac72+\frac52=6\). The residuals are \(-\frac16,\frac13,-\frac16\). Their sum is \(0\), and their input-weighted sum is \(0(-\frac16)+1(\frac13)+2(-\frac16)=0\), as required.
Worked Example: Using Centered Data
Consider \((-2,1),(-1,2),(1,5),(2,4)\). The input mean is \(\bar{x}=(-2-1+1+2)/4=0\), and the output mean is \(\bar{y}=(1+2+5+4)/4=3\). The centered input deviations are \(-2,-1,1,2\), and the centered output deviations are \(-2,-1,2,1\). Therefore
It follows that \(b=\frac9{10}\) and \(a=3-\frac9{10}\cdot0=3\). The least-squares line is \(y=3+\frac9{10}x\).
Check: Here \(\sum x_i=0\), \(\sum y_i=12\), \(\sum x_i^2=10\), and \(\sum x_i y_i=-2-2+5+8=9\). The normal equations become \(4a=12\) and \(10b=9\), which the stated coefficients satisfy. The residuals are \(-\frac15,-\frac1{10},\frac{11}{10},-\frac45\); their sum is \(0\), and their input-weighted sum is \(\frac25+\frac1{10}+\frac{11}{10}-\frac85=0\).
Worked Example: Equal Inputs Do Not Determine a Slope
Take the observations \((3,1),(3,2),(3,4)\). The means are \(\bar{x}=3\) and \(\bar{y}=\frac73\). Since every input equals \(3\), the centered denominator is \(S_{xx}=0\), so the slope formula with a denominator cannot be used. The normal-equation sums are \(\sum x_i=9\), \(\sum y_i=7\), \(\sum x_i^2=27\), and \(\sum x_i y_i=21\). Hence
The second equation is three times the first, so the equations determine only \(a+3b=\frac73\), not a unique pair. For instance, \(a=\frac73,b=0\) gives fitted values \(\frac73,\frac73,\frac73\). Any other pair satisfying \(a+3b=\frac73\) gives the same fitted values for these three inputs.
Check: For the displayed choice, the residuals are \(-\frac43,-\frac13,\frac53\). They sum to \(0\), and their input-weighted sum is \(3(-\frac43-\frac13+\frac53)=0\). The fitted value is the sample mean, as the theorem predicts.
A Common Mistake: Using Raw Inputs in the Slope
When a line includes an intercept, the slope is not generally \(\sum_i x_i y_i/\sum_i x_i^2\). That uncentered ratio omits the effect of the intercept. In the first worked example, it would give \(b=6/5\), since \(\sum_i x_i y_i=6\) and \(\sum_i x_i^2=5\), rather than the correct slope \(1/2\). If we then use the mean-based intercept formula, we obtain \(a=5/3-(6/5)(1)=7/15\). The residuals for this incorrect line are \(\frac8{15},\frac5{15},-\frac{13}{15}\). They sum to zero, but their input-weighted sum is \(\frac5{15}-\frac{26}{15}=-\frac75\ne0\). Thus the second normal equation fails: fitting the intercept correctly for a chosen slope does not make that slope least-squares.
Use the centered formula when the model includes an intercept. The denominator is positive precisely when the inputs vary, and the numerator uses deviations of both variables from their sample means. If the inputs do not vary, report that the slope and intercept are not individually determined; the fitted value at the observed input is determined, but infinitely many coefficient pairs can represent it.
Check Your Understanding
Use the normal equations and centered formulas where applicable. For each proposed fit, check the residual sums.
- For \((0,0),(1,1),(2,3)\), calculate \(\bar{x}\), \(\bar{y}\), \(S_{xx}\), and \(S_{xy}\), then find \(a\) and \(b\).
- Explain why the slope formula cannot be applied when every input is the same value.
- For inputs \(1,2,4\) and outputs \(3,2,5\), write the two normal equations before solving them.
- State the two residual orthogonality equations that a least-squares line must satisfy.
- For data with \(\bar{x}=0\) and \(\bar{y}=4\), what does the intercept formula imply about \(a\)?
