Pseudoinverses as Orthogonal Projections

Two Products, Two Spaces

A matrix \(\mathbf{A}\in\mathbb{R}^{m\times n}\) maps vectors in \(\mathbb{R}^n\) to vectors in \(\mathbb{R}^m\). Its column space is a subspace of \(\mathbb{R}^m\), while its row space is a subspace of \(\mathbb{R}^n\). The pseudoinverse \(\mathbf{A}^{+}\in\mathbb{R}^{n\times m}\) maps in the opposite direction. Consequently, the two products \(\mathbf{A}\mathbf{A}^{+}\) and \(\mathbf{A}^{+}\mathbf{A}\) act on different spaces: the first on \(\mathbb{R}^m\), the second on \(\mathbb{R}^n\). Each product keeps the component lying in the corresponding space and removes the orthogonal component.

Throughout this tutorial, matrices and vectors are real, and orthogonality uses the usual dot product. For a subspace \(W\), the orthogonal projection of \(\mathbf{x}\) onto \(W\) is the vector \(\mathbf{p}\in W\) for which \(\mathbf{x}-\mathbf{p}\) is orthogonal to every vector in \(W\). The SVD gives a direct way to see why the pseudoinverse products perform precisely these projections.

Symmetry expresses that the matrix does not distinguish the two arguments of the dot product, while idempotence means that once a vector has been projected, applying the same matrix again makes no change. The following fact explains why these algebraic conditions characterize orthogonal projection.

Proof. For any \(\mathbf{x}\), the vector \(\mathbf{P}\mathbf{x}\) belongs to \(\operatorname{im}\mathbf{P}\) by the definition of the range. Also, \(\mathbf{x}-\mathbf{P}\mathbf{x}\) is orthogonal to every \(\mathbf{y}\in\operatorname{im}\mathbf{P}\). Indeed, write \(\mathbf{y}=\mathbf{P}\mathbf{z}\). Then symmetry and idempotence give \(\langle\mathbf{x}-\mathbf{P}\mathbf{x},\mathbf{y}\rangle =(\mathbf{x}-\mathbf{P}\mathbf{x})^{\top}\mathbf{P}\mathbf{z} =\mathbf{x}^{\top}\mathbf{P}\mathbf{z}-\mathbf{x}^{\top}\mathbf{P}^{\top}\mathbf{P}\mathbf{z} =\mathbf{x}^{\top}\mathbf{P}\mathbf{z}-\mathbf{x}^{\top}\mathbf{P}^{2}\mathbf{z}=0\). Thus \(\mathbf{P}\mathbf{x}\) lies in the range and the residual is orthogonal to the range, which is the defining property of the orthogonal projection. \(\square\)

The Projection Theorem for the Pseudoinverse

Let \(\mathbf{A}\) have rank \(r\), and use a full SVD \(\mathbf{A}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\), where \(\mathbf{U}\) and \(\mathbf{V}\) are orthogonal and \(\boldsymbol{\Sigma}\) is an \(m\times n\) matrix whose first \(r\) diagonal entries \(\sigma_1,\dots,\sigma_r\) are positive and whose remaining entries are zero. The SVD construction of the pseudoinverse replaces each positive singular value by its reciprocal, transposes the rectangular diagonal matrix, and swaps the orthogonal factors: \(\mathbf{A}^{+}=\mathbf{V}\boldsymbol{\Sigma}^{+}\mathbf{U}^{\top}\). Here \(\boldsymbol{\Sigma}^{+}\) is \(n\times m\), with diagonal entries \(1/\sigma_i\) for \(1\le i\le r\) and zeros elsewhere. When \(r=0\), both \(\boldsymbol{\Sigma}\) and \(\boldsymbol{\Sigma}^{+}\) are zero.

Proof. Substituting the SVD expressions and using \(\mathbf{U}^{\top}\mathbf{U}=\mathbf{I}_m\) and \(\mathbf{V}^{\top}\mathbf{V}=\mathbf{I}_n\), we obtain \(\mathbf{A}\mathbf{A}^{+}=\mathbf{U}(\boldsymbol{\Sigma}\boldsymbol{\Sigma}^{+})\mathbf{U}^{\top}\) and \(\mathbf{A}^{+}\mathbf{A}=\mathbf{V}(\boldsymbol{\Sigma}^{+}\boldsymbol{\Sigma})\mathbf{V}^{\top}\). The product \(\boldsymbol{\Sigma}\boldsymbol{\Sigma}^{+}\) is the \(m\times m\) diagonal matrix with \(1\) in its first \(r\) diagonal positions and \(0\) in the others. Likewise, \(\boldsymbol{\Sigma}^{+}\boldsymbol{\Sigma}\) is the \(n\times n\) diagonal matrix with \(1\) in its first \(r\) positions and \(0\) in the others. These statements also hold when \(r=0\), in which case both products are zero.

Both displayed products are symmetric and idempotent: for example, \(\mathbf{A}\mathbf{A}^{+}=\mathbf{U}\mathbf{D}_m\mathbf{U}^{\top}\), where \(\mathbf{D}_m\) is diagonal with entries \(0\) or \(1\), so \((\mathbf{A}\mathbf{A}^{+})^{\top}=\mathbf{A}\mathbf{A}^{+}\) and \((\mathbf{A}\mathbf{A}^{+})^2=\mathbf{U}\mathbf{D}_m^2\mathbf{U}^{\top}=\mathbf{U}\mathbf{D}_m\mathbf{U}^{\top}\). The same calculation applies to \(\mathbf{A}^{+}\mathbf{A}\).

It remains to identify their ranges. From the SVD, the columns of \(\mathbf{A}\) lie in \(\operatorname{span}\{\mathbf{u}_1,\dots,\mathbf{u}_r\}\), where \(\mathbf{u}_i\) are the first \(r\) columns of \(\mathbf{U}\). Conversely, each \(\mathbf{u}_i\) for \(1\le i\le r\) is in \(\operatorname{Col}\mathbf{A}\), since \(\mathbf{A}\mathbf{v}_i=\sigma_i\mathbf{u}_i\) and \(\sigma_i\ne0\). Thus \(\operatorname{Col}\mathbf{A}=\operatorname{span}\{\mathbf{u}_1,\dots,\mathbf{u}_r\}\). Applying the same argument to \(\mathbf{A}^{\top}=\mathbf{V}\boldsymbol{\Sigma}^{\top}\mathbf{U}^{\top}\) gives \(\operatorname{Row}\mathbf{A}=\operatorname{Col}\mathbf{A}^{\top}=\operatorname{span}\{\mathbf{v}_1,\dots,\mathbf{v}_r\}\). The ranges of the two products are therefore exactly the stated spaces. By the proposition, each product is the orthogonal projector onto its range. \(\square\)

A Rank-One Example: Projection onto a Line

A Rank-Two Example: Projection onto a Plane

Rank Deficiency and What Gets Removed

Common Pitfall: Interchanging the Products

The products are both projectors, but they generally have different sizes and project onto different spaces. For the \(3\times2\) matrix in the first example, \(\mathbf{A}\mathbf{A}^{+}\) is \(3\times3\), so it acts on vectors in \(\mathbb{R}^3\). The product \(\mathbf{A}^{+}\mathbf{A}\) is \(2\times2\), so it acts on vectors in \(\mathbb{R}^2\).

Wrong calculation: applying \(\mathbf{A}^{+}\mathbf{A}\) to \(\mathbf{b}=(2,0,2)^{\top}\) as if it were a column-space vector would require multiplying a \(2\times2\) matrix by a \(3\times1\) vector. The inner dimensions do not match, so that product is undefined. Correction: to project this \(\mathbf{b}\in\mathbb{R}^3\) onto \(\operatorname{Col}\mathbf{A}\), use \(\mathbf{A}\mathbf{A}^{+}\mathbf{b}\). Use \(\mathbf{A}^{+}\mathbf{A}\) only for inputs in \(\mathbb{R}^2\), projecting onto \(\operatorname{Row}\mathbf{A}\).

One practical interpretation is that \(\mathbf{A}^{+}\mathbf{b}\) first finds a coefficient vector in the domain, while \(\mathbf{A}\mathbf{A}^{+}\mathbf{b}\) returns the corresponding projected vector in the codomain. The theorem identifies the target of that projection without requiring the input \(\mathbf{b}\) to lie in the column space already. In particular, the residual \(\mathbf{b}-\mathbf{A}\mathbf{A}^{+}\mathbf{b}\) is orthogonal to every column of \(\mathbf{A}\). In the domain, \(\mathbf{A}^{+}\mathbf{A}\) keeps precisely the component lying in the row space and removes components in directions that the matrix cannot use.