Computing Pseudoinverses of Small Matrices

From a Supplied SVD to a Pseudoinverse

A singular value decomposition turns a matrix into orthogonal changes of coordinates and a diagonal scaling. The pseudoinverse reverses each scaling that is nonzero and reverses the direction of the map: it sends left singular vectors back toward their corresponding right singular vectors. The computation is compact, but dimensions and zero singular values need careful attention, especially when the matrix is rectangular or rank deficient.

All matrices in this tutorial have real entries. Let \(\mathbf{A}\in\mathbb{R}^{m\times n}\). In a full singular value decomposition, \(\mathbf{A}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\), where \(\mathbf{U}\in\mathbb{R}^{m\times m}\) and \(\mathbf{V}\in\mathbb{R}^{n\times n}\) are orthogonal, and \(\boldsymbol{\Sigma}\in\mathbb{R}^{m\times n}\) is rectangular diagonal. Its diagonal entries are the singular values, conventionally listed in nonincreasing order. Suppose there are exactly \(r\) positive singular values \(\sigma_1,\dots,\sigma_r\); any remaining diagonal entries are zero.

For computations, it is often convenient to use the compact SVD, which keeps only the singular-vector columns corresponding to positive singular values. Write \(\mathbf{A}=\mathbf{U}_r\boldsymbol{\Sigma}_r\mathbf{V}_r^{\top}\), where \(\mathbf{U}_r\in\mathbb{R}^{m\times r}\) and \(\mathbf{V}_r\in\mathbb{R}^{n\times r}\) have orthonormal columns, and \(\boldsymbol{\Sigma}_r=\operatorname{diag}(\sigma_1,\dots,\sigma_r)\in\mathbb{R}^{r\times r}\). The same computation becomes \(\mathbf{A}^{+}=\mathbf{V}_r\boldsymbol{\Sigma}_r^{-1}\mathbf{U}_r^{\top}\). If \(r=0\), then \(\mathbf{A}=\mathbf{O}\) and \(\mathbf{A}^{+}=\mathbf{O}\) of size \(n\times m\); there are no positive singular values to reciprocate.

Why the Computation Has These Dimensions

The rectangular diagonal factor changes shape when it is pseudoinverted. If \(\boldsymbol{\Sigma}\) is \(m\times n\), then \(\boldsymbol{\Sigma}^{+}\) is \(n\times m\). Consequently, the product \(\mathbf{V}\boldsymbol{\Sigma}^{+}\mathbf{U}^{\top}\) has dimensions \((n\times n)(n\times m)(m\times m)=n\times m\), as required for a map that reverses the input and output dimensions of \(\mathbf{A}\).

Proof. Let \(\mathbf{u}_i\) and \(\mathbf{v}_i\) be the \(i\)th columns of \(\mathbf{U}\) and \(\mathbf{V}\). The SVD writes \(\mathbf{A}=\sum_{i=1}^{r}\sigma_i\mathbf{u}_i\mathbf{v}_i^{\top}\); zero singular values contribute zero terms. Reversing each nonzero rank-one term gives \(\mathbf{A}^{+}=\sum_{i=1}^{r}(1/\sigma_i)\mathbf{v}_i\mathbf{u}_i^{\top}\). In matrix form this is exactly \(\mathbf{V}\boldsymbol{\Sigma}^{+}\mathbf{U}^{\top}\): the reciprocal \(1/\sigma_i\) occupies the \(i\)th diagonal position, while zero singular values have no term to reverse. Each outer product \(\mathbf{v}_i\mathbf{u}_i^{\top}\) has \(n\) rows and \(m\) columns, so their sum has size \(n\times m\). This also covers \(r=0\), when the sum is empty and equals the zero matrix. \(\square\)

There is a direct multiplication check behind this rule. Orthonormality gives \(\mathbf{u}_i^{\top}\mathbf{u}_j=0\) for \(i\ne j\), \(\mathbf{u}_i^{\top}\mathbf{u}_i=1\), and the corresponding identities for the \(\mathbf{v}_i\). Therefore

For example, in the first product, multiplying the sums produces terms \((1/\sigma_i)\sigma_j\mathbf{v}_i(\mathbf{u}_i^{\top}\mathbf{u}_j)\mathbf{v}_j^{\top}\). Every term with \(i\ne j\) is zero; when \(i=j\), the scalar factor is \(1\). The second identity follows by the same multiplication using orthonormality of the \(\mathbf{v}_i\). These identities are useful checks: their sizes are \(n\times n\) and \(m\times m\), respectively.

Full-Column-Rank and Full-Row-Rank Cases

Rank Deficiency: Leave Zero Singular Values at Zero

Common Mistakes and a Reliable Checklist

Mistake: reciprocating every diagonal entry. In the rank-one example, a tempting but invalid step is to write \(1/0\) for the second singular value. Division by zero is undefined. The correct factor is \(\boldsymbol{\Sigma}^{+}=\begin{bmatrix}\tfrac16&0\\0&0\\0&0\end{bmatrix}\): transpose the rectangular shape, reciprocate the positive value \(6\), and leave the zero positions zero.

Mistake: keeping the original matrix dimensions. If \(\mathbf{A}\) is \(m\times n\), then \(\mathbf{A}^{+}\) is \(n\times m\), not \(m\times n\) unless \(m=n\). In the full-row-rank example, \(\mathbf{A}\) is \(2\times3\), but its pseudoinverse is \(3\times2\). The factor order \(\mathbf{V}\boldsymbol{\Sigma}^{+}\mathbf{U}^{\top}\) makes the dimension reversal explicit.

For a supplied SVD, a dependable computation follows these checks:

• Record \(m\), \(n\), and the dimensions of \(\mathbf{U}\), \(\boldsymbol{\Sigma}\), and \(\mathbf{V}\).
• Count the positive singular values; do not infer full rank merely from the matrix being rectangular.
• Transpose the rectangular diagonal factor and replace each positive \(\sigma_i\) by \(1/\sigma_i\).
• Keep zero entries zero, then multiply in the order \(\mathbf{V}\boldsymbol{\Sigma}^{+}\mathbf{U}^{\top}\).
• Check that the result is \(n\times m\) and, when useful, verify both products using the supplied singular vectors.

This procedure applies whether the matrix has full column rank, full row rank, or rank smaller than both dimensions. In every case, the singular vectors identify the directions to pair, the reciprocal singular values undo the nonzero scalings, and the zero singular values remain excluded from the computation.