The Pseudoinverse in Singular-Vector Coordinates

Reading the Pseudoinverse Through an SVD

A singular value decomposition (SVD) describes a matrix using orthonormal coordinate directions in its domain and codomain. In those coordinates, the matrix sends each right singular vector to a scaled left singular vector. The pseudoinverse reverses that action for every nonzero singular value: it sends the matching left singular vector back to the right singular vector, scaled by the reciprocal singular value. Any component in a left singular direction associated with a zero singular value is instead sent to zero.

Let \(\mathbf{A}\in\mathbb{F}^{m\times n}\), where \(\mathbb{F}\) is \(\mathbb{R}\) or \(\mathbb{C}\), and let \(r=\operatorname{rank}\mathbf{A}\). Use a full SVD \(\mathbf{A}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{*}\), where \(\mathbf{U}\) and \(\mathbf{V}\) are unitary (orthogonal when \(\mathbb{F}=\mathbb{R}\)). Write the columns of \(\mathbf{U}\) as \(\mathbf{u}_1,\dots,\mathbf{u}_m\) and the columns of \(\mathbf{V}\) as \(\mathbf{v}_1,\dots,\mathbf{v}_n\). The positive singular values are \(\sigma_1,\dots,\sigma_r\); the remaining diagonal entries of \(\boldsymbol{\Sigma}\) are zero. Thus, for \(1\le i\le r\), \(\mathbf{A}\mathbf{v}_i=\sigma_i\mathbf{u}_i\), and \(\mathbf{A}\mathbf{v}_j=\mathbf{0}\) for the right singular vectors with zero singular value.

Here \(\mathbf{u}_i^{*}\mathbf{y}\) is the \(i\)th coordinate of \(\mathbf{y}\) in the orthonormal left singular-vector coordinates. Over \(\mathbb{R}\), the conjugate transpose \(\mathbf{u}_i^{*}\) is just \(\mathbf{u}_i^{\top}\). The sum for \(\mathbf{A}^{+}\) contains only the positive singular values, so it is valid for square or rectangular matrices, and for matrices of any rank, including rank zero.

Why the Formula Gives the Pseudoinverse

The formula is not just a convenient rule for calculating. It satisfies the four Moore-Penrose equations, which characterize the pseudoinverse; uniqueness of a matrix satisfying those equations is the standard result established in the earlier material. To see why the SVD formula meets the equations, define \(\mathbf{P}_{U}=\sum_{i=1}^{r}\mathbf{u}_i\mathbf{u}_i^{*}\) and \(\mathbf{P}_{V}=\sum_{i=1}^{r}\mathbf{v}_i\mathbf{v}_i^{*}\). Orthonormality gives \(\mathbf{P}_{U}^{*}=\mathbf{P}_{U}\), \(\mathbf{P}_{V}^{*}=\mathbf{P}_{V}\), and \(\mathbf{P}_{U}^{2}=\mathbf{P}_{U}\), \(\mathbf{P}_{V}^{2}=\mathbf{P}_{V}\).

Proof. Since the singular vectors are orthonormal, \(\mathbf{u}_i^{*}\mathbf{u}_j=0\) for \(i\ne j\) and equals \(1\) for \(i=j\); the same statements hold for the \(\mathbf{v}_i\). Multiplying the sums therefore gives \(\mathbf{A}\mathbf{B}=\sum_{i=1}^{r}\mathbf{u}_i\mathbf{u}_i^{*}=\mathbf{P}_{U}\) and \(\mathbf{B}\mathbf{A}=\sum_{i=1}^{r}\mathbf{v}_i\mathbf{v}_i^{*}=\mathbf{P}_{V}\). Consequently, \(\mathbf{A}\mathbf{B}\mathbf{A}=\mathbf{P}_{U}\mathbf{A} =\sum_{i=1}^{r}\sigma_i\mathbf{u}_i\mathbf{v}_i^{*}=\mathbf{A}\), because \(\mathbf{P}_{U}\mathbf{u}_i=\mathbf{u}_i\) for each term in \(\mathbf{A}\). Likewise, \(\mathbf{B}\mathbf{A}\mathbf{B}=\mathbf{P}_{V}\mathbf{B} =\sum_{i=1}^{r}\frac{1}{\sigma_i}\mathbf{v}_i\mathbf{u}_i^{*}=\mathbf{B}\). The displayed sums for \(\mathbf{A}\mathbf{B}\) and \(\mathbf{B}\mathbf{A}\) are self-adjoint, so the other two equations hold. Thus \(\mathbf{B}\) satisfies all four equations, and uniqueness identifies it as \(\mathbf{A}^{+}\). \(\square\)

The Action on Singular-Vector Coordinates

Proof. Substitute the expansion of \(\mathbf{y}\) into the SVD formula for \(\mathbf{A}^{+}\): \(\mathbf{A}^{+}\mathbf{y}=\sum_{i=1}^{r}\frac{1}{\sigma_i}\mathbf{v}_i\mathbf{u}_i^{*}\sum_{j=1}^{m}c_j\mathbf{u}_j\). Distributing the products gives terms containing \(\mathbf{u}_i^{*}\mathbf{u}_j\). Orthonormality makes that scalar zero when \(i\ne j\) and one when \(i=j\), leaving exactly \(\sum_{i=1}^{r}\frac{c_i}{\sigma_i}\mathbf{v}_i\). Taking \(\mathbf{y}=\mathbf{u}_k\) gives \(c_k=1\) and all other coordinates zero. If \(k\le r\), the result is \(\mathbf{v}_k/\sigma_k\); if \(k>r\), the sum has no \(k\)th term and the result is \(\mathbf{0}\). \(\square\)

This coordinate formula separates two effects. A component \(c_i\mathbf{u}_i\) in a left singular direction with positive singular value becomes \((c_i/\sigma_i)\mathbf{v}_i\). A component in any of the remaining left singular directions contributes nothing to \(\mathbf{A}^{+}\mathbf{y}\).

Proof. The SVD gives \(\mathbf{A}^{*}\mathbf{y}=\sum_{i=1}^{r}\sigma_i\mathbf{v}_i(\mathbf{u}_i^{*}\mathbf{y})\). Since the \(\mathbf{v}_i\) are linearly independent and every \(\sigma_i\) in this sum is nonzero, this expression equals \(\mathbf{0}\) exactly when \(\mathbf{u}_i^{*}\mathbf{y}=0\) for every \(i\le r\). Expand \(\mathbf{y}\) in the complete orthonormal basis \(\mathbf{u}_1,\dots,\mathbf{u}_m\). Those coordinate conditions say that only \(\mathbf{u}_{r+1},\dots,\mathbf{u}_m\) can have nonzero coefficients. Hence the null space is the stated span. The coordinate-action theorem shows that \(\mathbf{A}^{+}\) annihilates every vector in that span. If \(r=m\), the span has no listed vectors and means \(\{\mathbf{0}\}\); the conclusion still holds. \(\square\)

Worked Coordinate Calculations

A Common Mistake and a Useful Interpretation

Do not multiply by the singular value when applying the pseudoinverse. For the matrix in the first example, the incorrect calculation \(\mathbf{A}^{+}\mathbf{u}_1=3\mathbf{v}_1\) uses \(\sigma_1\) instead of its reciprocal. The correction is \(\mathbf{A}^{+}\mathbf{u}_1=(1/3)\mathbf{v}_1\), since \(\mathbf{A}\mathbf{v}_1=3\mathbf{u}_1\) and the pseudoinverse reverses this scaling. A second error is to assume every left singular-vector component contributes: in that example, \(\mathbf{u}_3\) has zero singular value, so \(\mathbf{A}^{+}\mathbf{u}_3=\mathbf{0}\), not a scaled right singular vector.

The coordinate formula is useful because it separates the input into directions that the matrix can reverse and directions it discards. For each positive singular value, the matching pair \((\mathbf{v}_i,\mathbf{u}_i)\) tells exactly how the pseudoinverse acts. The reciprocal also explains why directions with small positive singular values can be strongly rescaled. None of this requires \(\mathbf{A}\) to be square or invertible; the dimensions of \(\mathbf{A}^{+}\) are always \(n\times m\).