Why an Inverse Is Not Enough
For a square invertible matrix \(\mathbf{A}\), the equation \(\mathbf{A}\mathbf{x}=\mathbf{b}\) has the unique solution \(\mathbf{x}=\mathbf{A}^{-1}\mathbf{b}\). This relies on two features: \(\mathbf{A}\) is square, and the equation has a unique exact solution for every \(\mathbf{b}\). Rectangular systems may have no exact solution or many exact solutions. A square matrix that is singular can also have no solution for some right-hand sides and many solutions for others. In these cases, the inverse either does not exist or cannot answer the question we need to ask.
Two choices make a useful replacement. When no exact solution exists, choose a vector \(\mathbf{x}\) for which \(\mathbf{A}\mathbf{x}\) is as close as possible to \(\mathbf{b}\). When many such choices are possible, select the one with the smallest norm. The pseudoinverse combines these choices into one linear map.
The norm measures the size of a vector, and the column space \(\operatorname{Col}\mathbf{A}\) is the set of all vectors \(\mathbf{A}\mathbf{x}\). Thus least squares asks for a vector in \(\operatorname{Col}\mathbf{A}\) closest to \(\mathbf{b}\), then chooses an input that produces it. These are two distinct tasks: the closest output need not equal \(\mathbf{b}\), and more than one input may produce the same output.
When the System Has No Exact Solution
Worked Example: A Rectangular Inconsistent System
Consider the \(3\times1\) matrix and right-hand side
The equation \(\mathbf{A}x=\mathbf{b}\) would require \(x=1\), \(x=0\), and \(0=1\) simultaneously, so it has no exact solution. For a candidate scalar \(x\), the squared error is
The squared error is smallest at \(x=\frac12\). Therefore the least-squares solution is \(\frac12\), and the fitted output and residual are
Check: the residual is orthogonal to the column of \(\mathbf{A}\), since \(\begin{bmatrix}1&1&0\end{bmatrix} \begin{bmatrix}\frac12\\-\frac12\\1\end{bmatrix}=0\). Also, the squared error at \(x=\frac12\) is \(\frac32\), as the completed-square expression gives. Here there is one least-squares solution, but no exact one.
The least-squares answer is not an exact solution in disguise: it minimizes an error that remains nonzero. In geometric terms, it chooses the closest point in \(\operatorname{Col}\mathbf{A}\) to \(\mathbf{b}\). This description also applies when the system has infinitely many exact solutions, but then the input producing that closest point may not be unique.
When There Are Many Exact Solutions
Worked Example: An Underdetermined System
Let \(\mathbf{A}=\begin{bmatrix}1&1\end{bmatrix}\) and \(b=2\). The equation \(\mathbf{A}\mathbf{x}=b\) is \(x_1+x_2=2\), so it has infinitely many exact solutions. Write them as \(\mathbf{x}=\begin{bmatrix}t\\2-t\end{bmatrix}\), where \(t\in\mathbb{R}\). Their squared norms are
This is smallest at \(t=1\). The minimum-norm solution is \(\begin{bmatrix}1\\1\end{bmatrix}\).
Check: \(\begin{bmatrix}1&1\end{bmatrix}\begin{bmatrix}1\\1\end{bmatrix}=2\), so it solves the original equation exactly. Its squared norm is \(2\), and the completed-square expression shows that every other solution has squared norm at least \(2\), with equality only at \(t=1\).
Here least squares alone does not distinguish among solutions: every exact solution has error zero. Minimizing the norm supplies a second criterion, selecting one solution from the entire line of exact solutions. In general, this selection matters because changing an input by a vector in the null space \(\operatorname{Nul}\mathbf{A}\) does not change its output.
Existence and Uniqueness of the Selected Solution
The preceding examples illustrate a general fact. Suppose \(\mathbf{A}\in\mathbb{R}^{m\times n}\). The finite-dimensional subspace \(\operatorname{Col}\mathbf{A}\) has an orthonormal basis \(\mathbf{u}_1,\ldots,\mathbf{u}_r\); if it is the zero subspace, take \(r=0\). Define the projection of \(\mathbf{b}\) onto this column space by \(\mathbf{p}=\sum_{i=1}^r(\mathbf{u}_i^{\top}\mathbf{b})\mathbf{u}_i\), with the empty sum interpreted as \(\mathbf{0}\). The difference \(\mathbf{b}-\mathbf{p}\) is orthogonal to every vector in \(\operatorname{Col}\mathbf{A}\). The next result explains why this projection and a minimum-norm input solve the two tasks.
Proof. First, \(\mathbf{p}\in\operatorname{Col}\mathbf{A}\), so for every \(\mathbf{z}\in\operatorname{Col}\mathbf{A}\), the vector \(\mathbf{p}-\mathbf{z}\) belongs to that same subspace. Since \(\mathbf{b}-\mathbf{p}\) is orthogonal to the subspace, it is orthogonal to \(\mathbf{p}-\mathbf{z}\). Expanding the squared norm therefore gives \(\|\mathbf{b}-\mathbf{z}\|^2=\|\mathbf{b}-\mathbf{p}\|^2+\|\mathbf{p}-\mathbf{z}\|^2\). This is minimized uniquely when \(\mathbf{z}=\mathbf{p}\). Every output \(\mathbf{A}\mathbf{x}\) lies in \(\operatorname{Col}\mathbf{A}\), and \(\mathbf{p}\) lies there too, so some \(\mathbf{x}_0\) satisfies \(\mathbf{A}\mathbf{x}_0=\mathbf{p}\). The least-squares solutions are exactly the inputs with this output.
To select the smallest-norm input, decompose \(\mathbf{x}_0=\mathbf{w}+\mathbf{n}_0\), where \(\mathbf{n}_0\in\operatorname{Nul}\mathbf{A}\) and \(\mathbf{w}\perp\operatorname{Nul}\mathbf{A}\). Such an orthogonal decomposition exists for every vector and subspace in a finite-dimensional inner-product space. Since \(\mathbf{A}\mathbf{n}_0=\mathbf{0}\), we have \(\mathbf{A}\mathbf{w}=\mathbf{p}\). Every other input with output \(\mathbf{p}\) is \(\mathbf{w}+\mathbf{n}\) for some \(\mathbf{n}\in\operatorname{Nul}\mathbf{A}\): subtracting \(\mathbf{w}\) from that input gives a vector annihilated by \(\mathbf{A}\). Orthogonality gives \(\|\mathbf{w}+\mathbf{n}\|^2=\|\mathbf{w}\|^2+\|\mathbf{n}\|^2\). This is minimized uniquely when \(\mathbf{n}=\mathbf{0}\). Thus \(\mathbf{w}\) is the unique minimum-norm least-squares solution and has the stated characterization. \(\square\)
This existence result also gives a single linear rule for finding the selected input. The projection \(\mathbf{b}\mapsto\mathbf{p}\) is linear, as its orthonormal-basis expression is a sum of scalar multiples of \(\mathbf{u}_i\), with each scalar linear in \(\mathbf{b}\). The map \(\mathbf{A}\), restricted to vectors orthogonal to its null space, sends those vectors one-to-one and onto \(\operatorname{Col}\mathbf{A}\): uniqueness and existence of the selected input establish these two properties. Its inverse on that subspace is linear, since sums and scalar multiples of inputs map to the corresponding sums and scalar multiples of outputs, and the input is unique.
The definition describes what the pseudoinverse does, rather than a recipe for computing it. It also clarifies the role of ordinary inversion. If \(\mathbf{A}\) is square and invertible, there is exactly one solution to \(\mathbf{A}\mathbf{x}=\mathbf{b}\), namely \(\mathbf{A}^{-1}\mathbf{b}\). That solution has zero error and is therefore the unique least-squares solution; since it is the only one, it is also the minimum-norm least-squares solution. In this case, \(\mathbf{A}^{+}\mathbf{b}=\mathbf{A}^{-1}\mathbf{b}\) for every \(\mathbf{b}\).
A Singular Square System
Worked Example: Inconsistent Data and a Minimum-Norm Choice
Consider the singular square matrix \(\mathbf{A}=\begin{bmatrix}1&1\\2&2\end{bmatrix}\) and \(\mathbf{b}=\begin{bmatrix}1\\0\end{bmatrix}\). An exact solution would require \(x_1+x_2=1\) and \(2x_1+2x_2=0\), which cannot both hold. Set \(s=x_1+x_2\). The squared error is
It is minimized when \(s=\frac15\), so all least-squares solutions satisfy \(x_1+x_2=\frac15\). Write these as \(\mathbf{x}=\begin{bmatrix}t\\\frac15-t\end{bmatrix}\). Their squared norms are \(t^2+(\frac15-t)^2=2(t-\frac1{10})^2+\frac1{50}\), which is minimized at \(t=\frac1{10}\). Hence the minimum-norm least-squares solution is \(\begin{bmatrix}\frac1{10}\\\frac1{10}\end{bmatrix}\).
Check: multiplying back gives \(\mathbf{A}\begin{bmatrix}\frac1{10}\\\frac1{10}\end{bmatrix} =\begin{bmatrix}\frac15\\\frac25\end{bmatrix}\). The residual \(\mathbf{b}-\mathbf{A}\mathbf{x}=\begin{bmatrix}\frac45\\-\frac25\end{bmatrix}\) is orthogonal to the column \(\begin{bmatrix}1\\2\end{bmatrix}\), since \(\frac45-2\cdot\frac25=0\). Its squared norm is \(\frac{16}{25}+\frac4{25}=\frac45\), matching the minimum in the completed-square expression. The input has squared norm \(\frac1{50}\), the minimum among inputs with coordinate sum \(\frac15\).
Common Pitfall: Applying an Inverse Formula Without Checking
A familiar-looking inverse calculation can fail when the matrix is rectangular or rank-deficient. For the underdetermined example \(\mathbf{A}=\begin{bmatrix}1&1\end{bmatrix}\), a tempting approach is to try to invert \(\mathbf{A}^{\top}\mathbf{A}\) to solve the system. But the calculation gives
The nonzero vector in the second calculation shows that \(\mathbf{A}^{\top}\mathbf{A}\) has no inverse. Treating its inverse as available is an invalid step, not a way to resolve the infinitely many solutions. The minimum-norm answer found directly is \(\begin{bmatrix}1\\1\end{bmatrix}\), and multiplying by \(\mathbf{A}\) verifies that it gives \(2\).
The key distinction is between the task and the tool. An inverse answers an exact, unique-solution problem for an invertible square matrix. A pseudoinverse instead selects a least-squares solution and, among those, the one with minimum norm. The selection is meaningful for rectangular matrices and singular square matrices, whether or not an exact solution exists.
Check Your Understanding
For each question, distinguish the output that best fits the data from the input that is selected among possible solutions.
- For \(\mathbf{A}=\begin{bmatrix}1\\2\end{bmatrix}\) and \(\mathbf{b}=\begin{bmatrix}1\\0\end{bmatrix}\), find the least-squares scalar \(x\) by minimizing \(\|\mathbf{A}x-\mathbf{b}\|^2\).
- For \(x_1+x_2=4\), find the solution with minimum norm and verify it satisfies the equation.
- Explain why a least-squares solution to an inconsistent system need not be an exact solution.
- For a matrix \(\mathbf{A}\), describe the set of inputs that produce a fixed output \(\mathbf{p}\in\operatorname{Col}\mathbf{A}\), if one such input \(\mathbf{x}_0\) is known.
- Why is \(\mathbf{A}^{\top}\mathbf{A}\) not invertible for \(\mathbf{A}=\begin{bmatrix}1&1\end{bmatrix}\)?
