What a Residual Check Can Tell You
A computed least-squares answer is a candidate until it is checked against the data in the matrix equation. The residual provides a direct test: if the candidate is a minimizer, its residual must be orthogonal to every column of the matrix. This test checks optimality without requiring you to repeat the method used to find the candidate. It can also expose whether an incorrect answer came from arithmetic or from using the wrong data in the setup.
Work over \(\mathbb{R}\). Let \(\mathbf{A}\in\mathbb{R}^{m\times n}\) have columns \(\mathbf{a}_1,\dots,\mathbf{a}_n\), let \(\mathbf{b}\in\mathbb{R}^m\), and let \(\mathbf{x}\in\mathbb{R}^n\) be a proposed coefficient vector. The corresponding fitted vector is \(\mathbf{A}\mathbf{x}\). Its residual is the difference between the target and the fit.
The residual has \(m\) entries, one for each equation, while each column also has \(m\) entries. Thus the test consists of \(n\) dot products. In matrix form, the same test is \(\mathbf{A}^{\top}\mathbf{r}=\mathbf{0}\): the \(j\)th entry of \(\mathbf{A}^{\top}\mathbf{r}\) is exactly \(\mathbf{a}_j^{\top}\mathbf{r}\). In particular, the test is not that \(\mathbf{r}=\mathbf{0}\). A nonzero residual is allowed; least squares asks for the smallest possible residual norm.
The Exact Test for a Proposed Fit
Proof. First suppose \(\mathbf{A}^{\top}\mathbf{r}=\mathbf{0}\). For any other coefficient vector \(\mathbf{y}\in\mathbb{R}^n\), write \(\mathbf{y}=\mathbf{x}+\mathbf{h}\), where \(\mathbf{h}=\mathbf{y}-\mathbf{x}\). The residual for \(\mathbf{y}\) is \(\mathbf{b}-\mathbf{A}\mathbf{y}=\mathbf{r}-\mathbf{A}\mathbf{h}\). Expanding its squared norm gives
The middle equality holds because \(\mathbf{r}^{\top}\mathbf{A}\mathbf{h}=(\mathbf{A}^{\top}\mathbf{r})^{\top}\mathbf{h}=0\). Therefore no \(\mathbf{y}\) has a smaller residual norm than \(\mathbf{x}\), so \(\mathbf{x}\) is a least-squares solution.
Conversely, suppose \(\mathbf{x}\) is a least-squares solution, but \(\mathbf{A}^{\top}\mathbf{r}\ne\mathbf{0}\). Then some entry \(j\) of \(\mathbf{A}^{\top}\mathbf{r}\) is nonzero, so \(\mathbf{a}_j^{\top}\mathbf{r}\ne0\). In particular, \(\mathbf{a}_j\ne\mathbf{0}\), since the zero column has dot product zero with every vector. Change only coefficient \(j\) by setting \(\mathbf{y}=\mathbf{x}+t\mathbf{e}_j\), where \(t=(\mathbf{a}_j^{\top}\mathbf{r})/\|\mathbf{a}_j\|^2\). The new residual is \(\mathbf{r}-t\mathbf{a}_j\), and
This contradicts the assumption that \(\mathbf{x}\) minimizes the residual norm. Hence \(\mathbf{A}^{\top}\mathbf{r}=\mathbf{0}\). Both implications hold, proving the test. \(\square\)
The condition is also the normal equations for a candidate: substituting \(\mathbf{r}=\mathbf{b}-\mathbf{A}\mathbf{x}\) into \(\mathbf{A}^{\top}\mathbf{r}=\mathbf{0}\) gives \(\mathbf{A}^{\top}\mathbf{A}\mathbf{x}=\mathbf{A}^{\top}\mathbf{b}\). For checking a proposed answer, it is often more transparent to compute the residual first and test its dot products with the original columns.
Checking a Candidate and Correcting an Arithmetic Error
Worked Example: Verify a Proposed Fit
Let \(\mathbf{A}=\begin{bmatrix}1&0\\1&1\\1&2\end{bmatrix}\), \(\mathbf{b}=\begin{bmatrix}1\\2\\2\end{bmatrix}\), and suppose the proposed answer is \(\mathbf{x}=\begin{bmatrix}7/6\\1/2\end{bmatrix}\). Check whether it is a least-squares solution.
First compute the fitted vector by multiplying each row of \(\mathbf{A}\) by \(\mathbf{x}\):
Subtract this from \(\mathbf{b}\) to find the residual, then take its dot product with each column \(\mathbf{a}_1=(1,1,1)^{\top}\) and \(\mathbf{a}_2=(0,1,2)^{\top}\):
Both column dot products are zero, so the residual test proves that the proposed \(\mathbf{x}\) is a least-squares solution. It is not an exact solution of \(\mathbf{A}\mathbf{x}=\mathbf{b}\), since its residual is nonzero; exact fit and least-squares fit are different conditions.
Worked Example: Catch an Arithmetic Error
Keep the same \(\mathbf{A}\) and \(\mathbf{b}\), but suppose an arithmetic slip produced \(\widetilde{\mathbf{x}}=\begin{bmatrix}1\\1/2\end{bmatrix}\). Use residual orthogonality to check the answer and see whether a nearby coefficient change improves the fit.
The proposed fit and residual are
The dot products with the columns are \(\mathbf{a}_1^{\top}\widetilde{\mathbf{r}}=1/2\) and \(\mathbf{a}_2^{\top}\widetilde{\mathbf{r}}=1/2\). Neither is zero, so this candidate is not a least-squares solution. The theorem’s perturbation argument gives a specific improvement: change the first coefficient by \(t=(\mathbf{a}_1^{\top}\widetilde{\mathbf{r}})/\|\mathbf{a}_1\|^2=(1/2)/3=1/6\). The new coefficient vector is \(\begin{bmatrix}7/6\\1/2\end{bmatrix}\), and its residual is
The original squared residual norm was \(0^2+(1/2)^2+0^2=1/4\). The new squared norm is \(1/36+1/9+1/36=1/6\), which is smaller. Checking the corrected residual against both columns gives zero dot products, as in the previous example. The calculation both rejects the arithmetic result and confirms the corrected fit.
Separating a Setup Error from a Valid Fit
Residual orthogonality checks a candidate against the particular \(\mathbf{A}\) and \(\mathbf{b}\) supplied. If the data were put into the wrong order or an entry was copied incorrectly, the candidate may solve the problem that was accidentally entered rather than the intended problem. It is therefore important to form the residual using the original, correctly ordered data—not a modified vector used during an earlier calculation.
Worked Example: Detect a Data-Order Error
Let \(\mathbf{A}=\begin{bmatrix}1&0\\0&1\\1&1\end{bmatrix}\) and \(\mathbf{b}=\begin{bmatrix}1\\2\\4\end{bmatrix}\). Suppose the first two entries of \(\mathbf{b}\) were accidentally swapped while calculating, producing the candidate \(\widetilde{\mathbf{x}}=\begin{bmatrix}7/3\\4/3\end{bmatrix}\). Test it using the stated \(\mathbf{b}\).
For the swapped vector \(\widetilde{\mathbf{b}}=\begin{bmatrix}2\\1\\4\end{bmatrix}\), the normal equations are \(\mathbf{A}^{\top}\mathbf{A}\widetilde{\mathbf{x}}=\mathbf{A}^{\top}\widetilde{\mathbf{b}}\), with \(\mathbf{A}^{\top}\mathbf{A}=\begin{bmatrix}2&1\\1&2\end{bmatrix}\) and \(\mathbf{A}^{\top}\widetilde{\mathbf{b}}=\begin{bmatrix}6\\5\end{bmatrix}\). Indeed, \(2(7/3)+4/3=6\) and \(7/3+2(4/3)=5\). But the residual for the intended data is
The columns are \(\mathbf{a}_1=(1,0,1)^{\top}\) and \(\mathbf{a}_2=(0,1,1)^{\top}\). Their dot products with the residual are \(\mathbf{a}_1^{\top}\mathbf{r}=-4/3+1/3=-1\) and \(\mathbf{a}_2^{\top}\mathbf{r}=2/3+1/3=1\). Thus the candidate fails the test for the stated data.
For comparison, the normal equations using the correctly ordered \(\mathbf{b}\) have right-hand side \(\mathbf{A}^{\top}\mathbf{b}=\begin{bmatrix}5\\6\end{bmatrix}\). Solving \(2x_1+x_2=5\) and \(x_1+2x_2=6\) gives \(x_1=4/3\) and \(x_2=7/3\). Substitution yields \(\mathbf{A}\mathbf{x}=\begin{bmatrix}4/3\\7/3\\11/3\end{bmatrix}\) and residual \(\begin{bmatrix}-1/3\\-1/3\\1/3\end{bmatrix}\). Its dot products with the columns are \(-1/3+1/3=0\) and \(-1/3+1/3=0\). The corrected candidate passes the test for the intended data.
What the Test Says When Coefficients Are Not Unique
The orthogonality condition decides whether a candidate is a minimizer; it does not promise that the coefficients are unique. If some columns are dependent, different coefficient vectors can produce the same fitted vector. The test remains valid: each minimizer’s residual must be orthogonal to every column, including dependent or zero columns. Testing every column is important, since it checks the stated matrix directly rather than relying on a selected subset.
Worked Example: Several Coefficient Vectors Pass
Let \(\mathbf{A}=\begin{bmatrix}1&2\\0&0\\1&2\end{bmatrix}\) and \(\mathbf{b}=\begin{bmatrix}1\\1\\3\end{bmatrix}\). Check the coefficient vectors \(\mathbf{x}_1=\begin{bmatrix}2\\0\end{bmatrix}\) and \(\mathbf{x}_2=\begin{bmatrix}0\\1\end{bmatrix}\).
The two columns are \(\mathbf{a}_1=(1,0,1)^{\top}\) and \(\mathbf{a}_2=(2,0,2)^{\top}=2\mathbf{a}_1\). For the first coefficient vector, \(\mathbf{A}\mathbf{x}_1=(2,0,2)^{\top}\), so \(\mathbf{r}_1=(-1,1,1)^{\top}\). Its dot products are \(\mathbf{a}_1^{\top}\mathbf{r}_1=-1+1=0\) and \(\mathbf{a}_2^{\top}\mathbf{r}_1=-2+2=0\).
For the second vector, \(\mathbf{A}\mathbf{x}_2=(2,0,2)^{\top}\) as well. It therefore has the same residual \((-1,1,1)^{\top}\), and the same two zero dot products. Both vectors are least-squares solutions. The fit is the same because \(x_1+2x_2=2\) for each candidate; the residual test verifies optimality without requiring a unique coefficient vector.
A Reliable Checking Routine
When auditing a proposed answer, keep the order of operations explicit. This reduces the risk of testing the wrong residual or overlooking a column.
Check that \(\mathbf{A}\) has the intended rows and columns, that \(\mathbf{b}\) has the corresponding entries in the intended order, and that the proposed \(\mathbf{x}\) has \(n\) entries.
Multiply \(\mathbf{A}\mathbf{x}\), then subtract it from \(\mathbf{b}\) in the order \(\mathbf{r}=\mathbf{b}-\mathbf{A}\mathbf{x}\).
Compute \(\mathbf{a}_j^{\top}\mathbf{r}\) for every \(j=1,\dots,n\), or compute \(\mathbf{A}^{\top}\mathbf{r}\) once.
If every dot product is zero, the candidate is a least-squares solution. If any is nonzero, it is not; recheck the arithmetic and the setup before solving again.
A common mistake is to stop after seeing that the residual entries sum to zero. That is not the test unless the relevant column is specifically the all-ones vector. For example, with \(\mathbf{a}=(1,2)^{\top}\) and \(\mathbf{r}=(1,-1)^{\top}\), the residual entries sum to \(1+(-1)=0\), but \(\mathbf{a}^{\top}\mathbf{r}=1-2=-1\ne0\). The correction is to use the actual columns of \(\mathbf{A}\) in the dot products, not a general summary of the residual entries.
Likewise, a zero residual is sufficient for a least-squares fit, but it is not necessary. The meaningful requirement is orthogonality to the columns: it says that moving the fitted vector in any direction available from the columns cannot reduce the error. That is why the same check applies to exact fits, nonzero residuals, and problems with dependent columns.
Check Your Understanding
For each proposed fit, form the residual before deciding whether it is a least-squares solution.
- For \(\mathbf{A}=\begin{bmatrix}1\\1\end{bmatrix}\), \(\mathbf{b}=\begin{bmatrix}1\\3\end{bmatrix}\), and \(x=2\), compute the residual and test its dot product with the column of \(\mathbf{A}\).
- Explain why \(\mathbf{A}^{\top}\mathbf{r}=\mathbf{0}\) is the same condition as orthogonality to every column of \(\mathbf{A}\).
- For \(\mathbf{A}=\begin{bmatrix}1&0\\0&1\\1&1\end{bmatrix}\), \(\mathbf{b}=\begin{bmatrix}1\\2\\4\end{bmatrix}\), test \(\mathbf{x}=\begin{bmatrix}1\\2\end{bmatrix}\).
- If a residual is not orthogonal to column \(\mathbf{a}_j\), what coefficient change from the proof gives a strictly smaller residual norm?
- Can two different coefficient vectors both pass the residual test? Explain how dependent columns make this possible.
