The Fit and the Residual as Vectors
For a system \(\mathbf{A}\mathbf{x}=\mathbf{b}\), an exact solution would make the residual \(\mathbf{b}-\mathbf{A}\mathbf{x}\) equal to zero. When there is no exact solution, least squares chooses a vector \(\mathbf{x}\) that makes the residual as short as possible. The geometry is easiest to read by separating the fitted vector \(\mathbf{A}\mathbf{x}\) from the residual: the fitted vector lies in the column space of \(\mathbf{A}\), while the residual of a best fit points perpendicular to that space.
Let \(\mathbf{A}\in\mathbb{R}^{m\times n}\) and \(\mathbf{b}\in\mathbb{R}^m\). The column space \(\operatorname{Col}\mathbf{A}\) is the set of all linear combinations of the columns of \(\mathbf{A}\). By the definition of matrix multiplication, it is also the set of all vectors \(\mathbf{A}\mathbf{x}\) as \(\mathbf{x}\) ranges over \(\mathbb{R}^n\). A candidate coefficient vector \(\mathbf{x}\) therefore gives a fitted vector \(\mathbf{y}=\mathbf{A}\mathbf{x}\in\operatorname{Col}\mathbf{A}\) and a residual \(\mathbf{r}=\mathbf{b}-\mathbf{y}\). A least-squares solution is a choice of \(\mathbf{x}\) that minimizes \(\|\mathbf{b}-\mathbf{A}\mathbf{x}\|\), or equivalently minimizes its square.
The earlier results on orthogonal projections give an orthogonal decomposition of any \(\mathbf{b}\in\mathbb{R}^m\) relative to a subspace \(S\): \(\mathbf{b}=\mathbf{b}_S+\mathbf{b}_{S^\perp}\), where \(\mathbf{b}_S\in S\) and \(\mathbf{b}_{S^\perp}\in S^\perp\). These two components are unique. Applied to \(S=\operatorname{Col}\mathbf{A}\), the component \(\mathbf{b}_S\) is the part available to be fitted, and \(\mathbf{b}_{S^\perp}\) is the part that lies outside the model space.
Why the Best-Fit Residual Is Perpendicular
The following characterization expresses least squares entirely in terms of vectors in the column space. It applies to any subspace \(S\) of \(\mathbb{R}^m\), so it does not require the columns of \(\mathbf{A}\) to be independent. For a least-squares problem, take \(S=\operatorname{Col}\mathbf{A}\). The statement is conditional: it characterizes a best fit whenever one is under consideration; existence is a separate question.
Proof. Set \(\mathbf{e}=\mathbf{b}-\mathbf{y}_*\). First suppose \(\mathbf{e}\in S^\perp\). For any \(\mathbf{y}\in S\), the difference \(\mathbf{y}_*-\mathbf{y}\) belongs to \(S\), so it is perpendicular to \(\mathbf{e}\). Since \(\mathbf{b}-\mathbf{y}=\mathbf{e}+(\mathbf{y}_*-\mathbf{y})\), expansion of the squared norm gives \(\|\mathbf{b}-\mathbf{y}\|^2=\|\mathbf{e}\|^2+\|\mathbf{y}_*-\mathbf{y}\|^2\geq\|\mathbf{e}\|^2=\|\mathbf{b}-\mathbf{y}_*\|^2\). Thus \(\mathbf{y}_*\) minimizes the distance.
Conversely, suppose \(\mathbf{y}_*\) minimizes the distance. Take any nonzero \(\mathbf{u}\in S\). For every real \(t\), the vector \(\mathbf{y}_*+t\mathbf{u}\) belongs to \(S\). Minimality implies \(\|\mathbf{e}-t\mathbf{u}\|^2\geq\|\mathbf{e}\|^2\) for every \(t\in\mathbb{R}\). Expanding the left side gives \(-2t(\mathbf{e}\cdot\mathbf{u})+t^2\|\mathbf{u}\|^2\geq0\). If \(\mathbf{e}\cdot\mathbf{u}\ne0\), choose \(t=(\mathbf{e}\cdot\mathbf{u})/\|\mathbf{u}\|^2\). The left side then equals \(-(\mathbf{e}\cdot\mathbf{u})^2/\|\mathbf{u}\|^2<0\), a contradiction. Hence \(\mathbf{e}\cdot\mathbf{u}=0\) for every nonzero \(\mathbf{u}\in S\); the same equality holds for \(\mathbf{u}=\mathbf{0}\). Therefore \(\mathbf{e}\in S^\perp\). \(\square\)
With \(S=\operatorname{Col}\mathbf{A}\), this says that any best-fit vector \(\mathbf{y}_*\) has residual \(\mathbf{r}_*=\mathbf{b}-\mathbf{y}_*\) perpendicular to every column combination. The residual is not an extra direction that the model could still use: if it had a nonzero component inside the column space, moving the fit in that direction would shorten the error.
What a Fit Can and Cannot Capture
Write the orthogonal decomposition of \(\mathbf{b}\) relative to \(S=\operatorname{Col}\mathbf{A}\) as \(\mathbf{b}=\mathbf{b}_S+\mathbf{b}_{S^\perp}\). The first component can be represented as \(\mathbf{A}\mathbf{x}\) for at least one coefficient vector, because it belongs to the column space. The second is perpendicular to every possible fitted vector. Thus a best fit captures \(\mathbf{b}_S\), and its residual is \(\mathbf{b}_{S^\perp}\). The decomposition also separates the error for any candidate fit.
This identity follows because \(\mathbf{b}-\mathbf{y}=\mathbf{b}_{S^\perp}+(\mathbf{b}_S-\mathbf{y})\), and the two terms are perpendicular: the first lies in \(S^\perp\), while the second lies in \(S\). It gives a useful interpretation of the smallest possible error. No choice of coefficients can remove \(\mathbf{b}_{S^\perp}\), because every fitted vector remains in \(S\).
Worked Example: A Best Fit in a Plane
Let \(\mathbf{A}=\begin{bmatrix}1&0\\0&1\\1&1\end{bmatrix}\) and \(\mathbf{b}=\begin{bmatrix}1\\2\\4\end{bmatrix}\). Find a fitted vector and residual, and check that the residual is perpendicular to the column space.
The columns of \(\mathbf{A}\) are \(\mathbf{a}_1=(1,0,1)^\top\) and \(\mathbf{a}_2=(0,1,1)^\top\). The vector \(\mathbf{n}=(-1,-1,1)^\top\) is perpendicular to both, since \(\mathbf{n}\cdot\mathbf{a}_1=-1+1=0\) and \(\mathbf{n}\cdot\mathbf{a}_2=-1+1=0\). The component of \(\mathbf{b}\) along \(\mathbf{n}\) is \(\frac{\mathbf{b}\cdot\mathbf{n}}{\mathbf{n}\cdot\mathbf{n}}\mathbf{n} =\frac{-1-2+4}{1+1+1}\mathbf{n} =\frac13\mathbf{n}\). Thus take \(\mathbf{r}=\frac13\mathbf{n}=(-\frac13,-\frac13,\frac13)^\top\) and \(\mathbf{y}=\mathbf{b}-\mathbf{r}=(\frac43,\frac73,\frac{11}3)^\top\).
To verify that this fitted vector is available from the matrix, use \(\mathbf{x}=(\frac43,\frac73)^\top\): \(\mathbf{A}\mathbf{x}=(\frac43,\frac73,\frac43+\frac73)^\top=(\frac43,\frac73,\frac{11}3)^\top=\mathbf{y}\). The residual check is \(\mathbf{b}-\mathbf{A}\mathbf{x}=(1-\frac43,2-\frac73,4-\frac{11}3)^\top=(-\frac13,-\frac13,\frac13)^\top=\mathbf{r}\). Also \(\mathbf{r}\cdot\mathbf{a}_1=-\frac13+\frac13=0\) and \(\mathbf{r}\cdot\mathbf{a}_2=-\frac13+\frac13=0\). By the theorem, this is a best-fit vector. Its squared error is \(\|\mathbf{r}\|^2=\frac19+\frac19+\frac19=\frac13\).
Worked Example: Orthogonal Columns Make the Geometry Visible
Let \(\mathbf{A}=\begin{bmatrix}1&0\\1&0\\0&1\\0&1\end{bmatrix}\) and \(\mathbf{b}=(2,0,3,1)^\top\). The columns \(\mathbf{a}_1=(1,1,0,0)^\top\) and \(\mathbf{a}_2=(0,0,1,1)^\top\) are perpendicular. Their squared norms are both \(2\). The projection coefficients onto these perpendicular directions are \(\frac{\mathbf{b}\cdot\mathbf{a}_1}{\|\mathbf{a}_1\|^2}=\frac{2}{2}=1\) and \(\frac{\mathbf{b}\cdot\mathbf{a}_2}{\|\mathbf{a}_2\|^2}=\frac{4}{2}=2\). So choose \(\mathbf{x}=(1,2)^\top\). Then \(\mathbf{A}\mathbf{x}=(1,1,2,2)^\top=\mathbf{y}\) and \(\mathbf{r}=\mathbf{b}-\mathbf{y}=(1,-1,1,-1)^\top\).
Indeed, \(\mathbf{r}\cdot\mathbf{a}_1=1-1=0\) and \(\mathbf{r}\cdot\mathbf{a}_2=1-1=0\), so the residual is perpendicular to the whole column space. The fitted vector is verified by multiplying the matrix by the stated coefficients, and the residual by subtracting it from \(\mathbf{b}\). Its squared length is \(1+1+1+1=4\); that is the part of the data vector these two column directions cannot capture.
Worked Example: An Uncapturable Component Along a Line
Let \(\mathbf{A}=(1,1,1)^\top\) and \(\mathbf{b}=(1,0,2)^\top\). Every fitted vector has the form \(\mathbf{y}=t(1,1,1)^\top\) for some \(t\in\mathbb{R}\). The projection coefficient is \(t=\frac{\mathbf{b}\cdot(1,1,1)}{(1,1,1)\cdot(1,1,1)}=\frac{1+0+2}{3}=1\). This gives \(\mathbf{y}_*=(1,1,1)^\top=\mathbf{A}(1)\) and \(\mathbf{r}_*=\mathbf{b}-\mathbf{y}_*=(0,-1,1)^\top\). Since \(\mathbf{r}_*\cdot(1,1,1)=0-1+1=0\), the residual is perpendicular to the line and the fit is best.
For any other coefficient \(t\), the residual is \(\mathbf{b}-t(1,1,1)^\top=\mathbf{r}_*+(1-t)(1,1,1)^\top\). The two terms on the right are perpendicular, so \(\|\mathbf{b}-t(1,1,1)^\top\|^2=\|\mathbf{r}_*\|^2+(1-t)^2\|(1,1,1)\|^2=2+3(1-t)^2\geq2\). Equality holds at \(t=1\). This check shows both what the line can capture and why the perpendicular component of squared length \(2\) remains in every error.
A Common Mistake: Perpendicular to the Wrong Vector
A residual of a best fit is perpendicular to the column space, not necessarily to the data vector \(\mathbf{b}\), the coefficient vector \(\mathbf{x}\), or each residual component considered separately. For the plane example, the correct residual is \(\mathbf{r}=(-\frac13,-\frac13,\frac13)^\top\). A tempting but wrong check is to claim that \(\mathbf{r}\cdot\mathbf{b}=0\). In fact, \(\mathbf{r}\cdot\mathbf{b}=-\frac13- \frac23+\frac43=\frac13\ne0\). The fit is still best: the relevant checks are \(\mathbf{r}\cdot\mathbf{a}_1=0\) and \(\mathbf{r}\cdot\mathbf{a}_2=0\), which imply perpendicularity to every linear combination of the columns.
There is also an important distinction between a fitted vector and its coefficients. The geometry takes place in \(\mathbb{R}^m\), where \(\mathbf{b}\), \(\mathbf{A}\mathbf{x}\), and the residual live. The coefficient vector \(\mathbf{x}\) lives in \(\mathbb{R}^n\). It is the image \(\mathbf{A}\mathbf{x}\), not \(\mathbf{x}\) itself, that is compared with \(\mathbf{b}\) and belongs to \(\operatorname{Col}\mathbf{A}\).
| Vector | Where it lies | Geometric role |
|---|---|---|
| \(\mathbf{b}\) | \(\mathbb{R}^m\) | Data vector being approximated |
| \(\mathbf{A}\mathbf{x}\) | \(\operatorname{Col}\mathbf{A}\) | Candidate fitted vector |
| \(\mathbf{b}-\mathbf{A}\mathbf{x}\) | \(\mathbb{R}^m\) | Residual; perpendicular to the column space at a best fit |
| \(\mathbf{x}\) | \(\mathbb{R}^n\) | Coefficients used to produce a fitted vector |
Check Your Understanding
Use the column-space geometry in each question, and check computations by substitution or dot products.
- For the plane example, verify directly that \(\mathbf{b}=\mathbf{y}+\mathbf{r}\) and that \(\mathbf{y}\cdot\mathbf{r}=0\).
- In the orthogonal-columns example, compute the dot product of the residual with each column and state why those two checks suffice.
- For a subspace \(S\), explain why \(\mathbf{b}_{S^\perp}\) cannot be changed by choosing a different fitted vector in \(S\).
- In the line example, evaluate \(2+3(1-t)^2\) at \(t=0\) and compare it with the squared error at \(t=1\).
- Give the correct geometric condition on the residual of a best fit, and explain why perpendicularity to \(\mathbf{b}\) is not required.
