The SVD Recipe
A rectangular matrix need not have an inverse: its input and output spaces may have different dimensions, or its rank may be smaller than either dimension. The singular value decomposition gives a systematic way to build a matrix that reverses the action of the original matrix on the directions it does not erase. The construction has two steps: replace each positive singular value by its reciprocal, and transpose the rectangular diagonal factor so its dimensions reverse.
Work over \(\mathbb{R}\). Let \(\mathbf{A}\in\mathbb{R}^{m\times n}\) have rank \(r\), and write a full singular value decomposition as \(\mathbf{A}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\). Here \(\mathbf{U}\in\mathbb{R}^{m\times m}\) and \(\mathbf{V}\in\mathbb{R}^{n\times n}\) are orthogonal, so \(\mathbf{U}^{\top}\mathbf{U}=\mathbf{I}_m\) and \(\mathbf{V}^{\top}\mathbf{V}=\mathbf{I}_n\). The matrix \(\boldsymbol{\Sigma}\in\mathbb{R}^{m\times n}\) is rectangular diagonal: its diagonal entries are the singular values \(\sigma_1\ge\cdots\ge\sigma_r>0\), followed by zeros. In particular, a zero singular value is not inverted.
The dimensions explain the order of the factors: \(\mathbf{V}\) is \(n\times n\), \(\boldsymbol{\Sigma}^{+}\) is \(n\times m\), and \(\mathbf{U}^{\top}\) is \(m\times m\). Thus \(\mathbf{A}^{+}\) is \(n\times m\), reversing the input and output dimensions of \(\mathbf{A}\). For a tall matrix, its pseudoinverse is wide; for a wide matrix, its pseudoinverse is tall.
In full SVD coordinates, \(\boldsymbol{\Sigma}^{+}\) has \(1/\sigma_i\) in position \((i,i)\) for \(1\le i\le r\), and zero in every other position. If \(r=0\), then \(\mathbf{A}=\mathbf{O}\) and \(\boldsymbol{\Sigma}^{+}\) is the \(n\times m\) zero matrix. The recipe therefore covers the zero matrix as well as matrices with positive singular values.
What the Construction Reverses
The columns \(\mathbf{v}_i\) of \(\mathbf{V}\) are right singular vectors, and the columns \(\mathbf{u}_i\) of \(\mathbf{U}\) are left singular vectors. For \(1\le i\le r\), the SVD gives \(\mathbf{A}\mathbf{v}_i=\sigma_i\mathbf{u}_i\). Applying the constructed matrix in the reverse direction gives \(\mathbf{A}^{+}\mathbf{u}_i=(1/\sigma_i)\mathbf{v}_i\). The reciprocal undoes the scaling by \(\sigma_i\). Directions associated with zero singular values are treated differently: no reciprocal is taken, so the construction sends the corresponding left singular vectors to zero.
Proof. The first identity is the column-by-column form of \(\mathbf{A}\mathbf{V}=\mathbf{U}\boldsymbol{\Sigma}\). For \(1\le i\le r\), orthogonality gives \(\mathbf{U}^{\top}\mathbf{u}_i=\mathbf{e}_i\), where \(\mathbf{e}_i\) is the \(i\)th standard basis vector. By construction, \(\boldsymbol{\Sigma}^{+}\mathbf{e}_i=(1/\sigma_i)\mathbf{e}_i\), now regarded as a vector in \(\mathbb{R}^n\). Hence \(\mathbf{A}^{+}\mathbf{u}_i=\mathbf{V}\boldsymbol{\Sigma}^{+}\mathbf{U}^{\top}\mathbf{u}_i=(1/\sigma_i)\mathbf{V}\mathbf{e}_i=(1/\sigma_i)\mathbf{v}_i\). If \(j>r\), the \(j\)th column of \(\boldsymbol{\Sigma}^{+}\) is zero, so \(\boldsymbol{\Sigma}^{+}\mathbf{e}_j=\mathbf{0}\) and \(\mathbf{A}^{+}\mathbf{u}_j=\mathbf{0}\). If \(r=0\), there are no indices \(i\le r\), and the same zero-column argument applies to every \(\mathbf{u}_j\). \(\square\)
This action also shows why the rank-deficient case needs no special division rule beyond leaving zero entries at zero. The reciprocal is used precisely on the directions where \(\mathbf{A}\) has a nonzero scale factor. The four-equation characterization and uniqueness of the Moore–Penrose pseudoinverse are standard results treated in the later material on those subjects; here the focus is the SVD construction and its direct consequences.
Worked Constructions
Worked Example: A Tall Matrix
Consider the tall \(3\times2\) matrix $$ \mathbf{A}=\begin{bmatrix}2&0\\0&3\\0&0\end{bmatrix}. $$ It already has an SVD with \(\mathbf{U}=\mathbf{I}_3\), \(\mathbf{V}=\mathbf{I}_2\), and \(\boldsymbol{\Sigma}=\mathbf{A}\). Its positive singular values are \(3\) and \(2\); they are conventionally listed in decreasing order, so for this displayed diagonal order we may instead use \(\mathbf{U}\) that swaps the first two standard basis vectors. Equivalently, the diagonal entries can be reordered by choosing the singular-vector columns accordingly. To keep the factors explicit, take $$ \mathbf{U}=\begin{bmatrix}0&1&0\\1&0&0\\0&0&1\end{bmatrix},\quad \boldsymbol{\Sigma}=\begin{bmatrix}3&0\\0&2\\0&0\end{bmatrix},\quad \mathbf{V}=\begin{bmatrix}0&1\\1&0\end{bmatrix}. $$ Then \(\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}=\mathbf{A}\), and the positive singular values appear in descending order.
Reciprocating the two positive diagonal entries and transposing the \(3\times2\) factor gives $$ \boldsymbol{\Sigma}^{+}=\begin{bmatrix}1/3&0&0\\0&1/2&0\end{bmatrix},\qquad \mathbf{A}^{+}=\mathbf{V}\boldsymbol{\Sigma}^{+}\mathbf{U}^{\top} =\begin{bmatrix}1/2&0&0\\0&1/3&0\end{bmatrix}. $$ Check both products: $$ \mathbf{A}^{+}\mathbf{A}=\begin{bmatrix}1&0\\0&1\end{bmatrix},\qquad \mathbf{A}\mathbf{A}^{+}=\begin{bmatrix}1&0&0\\0&1&0\\0&0&0\end{bmatrix}. $$ In particular, multiplying the second product by \(\mathbf{A}\) gives \((\mathbf{A}\mathbf{A}^{+})\mathbf{A}=\mathbf{A}\). The zero row in the original matrix corresponds to a direction in the output space that the matrix never reaches.
Worked Example: A Wide Matrix
Let $$ \mathbf{A}=\begin{bmatrix}2&0&0\\0&4&0\end{bmatrix}. $$ This is a wide \(2\times3\) matrix. Its singular values in decreasing order are \(4,2\). Set $$ \mathbf{U}=\begin{bmatrix}0&1\\1&0\end{bmatrix},\quad \boldsymbol{\Sigma}=\begin{bmatrix}4&0&0\\0&2&0\end{bmatrix},\quad \mathbf{V}=\begin{bmatrix}0&1&0\\1&0&0\\0&0&1\end{bmatrix}. $$ Direct multiplication gives \(\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}=\mathbf{A}\). The reciprocal factor and pseudoinverse are $$ \boldsymbol{\Sigma}^{+}=\begin{bmatrix}1/4&0\\0&1/2\\0&0\end{bmatrix},\qquad \mathbf{A}^{+}=\mathbf{V}\boldsymbol{\Sigma}^{+}\mathbf{U}^{\top} =\begin{bmatrix}1/2&0\\0&1/4\\0&0\end{bmatrix}. $$ The dimensions are now reversed: \(\mathbf{A}^{+}\) is \(3\times2\). The products check the construction: $$ \mathbf{A}\mathbf{A}^{+}=\begin{bmatrix}1&0\\0&1\end{bmatrix},\qquad \mathbf{A}^{+}\mathbf{A}=\begin{bmatrix}1&0&0\\0&1&0\\0&0&0\end{bmatrix}. $$ Thus \((\mathbf{A}\mathbf{A}^{+})\mathbf{A}=\mathbf{A}\), and \((\mathbf{A}^{+}\mathbf{A})\mathbf{A}^{+}=\mathbf{A}^{+}\), as direct multiplication confirms.
Worked Example: A Rank-Deficient Matrix
Consider $$ \mathbf{A}=\begin{bmatrix}1&1&0\\1&1&0\end{bmatrix}. $$ Its rank is one. Let $$ \mathbf{u}_1=\frac{1}{\sqrt{2}}\begin{bmatrix}1\\1\end{bmatrix},\qquad \mathbf{v}_1=\frac{1}{\sqrt{2}}\begin{bmatrix}1\\1\\0\end{bmatrix}. $$ Then \(\mathbf{A}\mathbf{v}_1=2\mathbf{u}_1\). Complete these to orthonormal bases with \(\mathbf{u}_2=(1,-1)^{\top}/\sqrt{2}\) and \(\mathbf{v}_2=(1,-1,0)^{\top}/\sqrt{2}\), \(\mathbf{v}_3=(0,0,1)^{\top}\). The corresponding singular values are \(2,0\), with one extra zero column in the \(2\times3\) rectangular factor. Hence $$ \mathbf{A}^{+}=\frac{1}{2}\mathbf{v}_1\mathbf{u}_1^{\top} =\begin{bmatrix}1/4&1/4\\1/4&1/4\\0&0\end{bmatrix}. $$ For a direct check, $$ \mathbf{A}\mathbf{A}^{+} =\begin{bmatrix}1/2&1/2\\1/2&1/2\end{bmatrix},\qquad \mathbf{A}^{+}\mathbf{A} =\begin{bmatrix}1/2&1/2&0\\1/2&1/2&0\\0&0&0\end{bmatrix}. $$ Multiplying gives \((\mathbf{A}\mathbf{A}^{+})\mathbf{A}=\begin{bmatrix}1&1&0\\1&1&0\end{bmatrix}=\mathbf{A}\). Also \((\mathbf{A}^{+}\mathbf{A})\mathbf{A}^{+}=\begin{bmatrix}1/4&1/4\\1/4&1/4\\0&0\end{bmatrix}=\mathbf{A}^{+}\). The zero singular value was left at zero; only the singular value \(2\) was reciprocated.
Why the Rank Controls the Recipe
Proof. In the full SVD, \(\mathbf{V}\) and \(\mathbf{U}^{\top}\) are square orthogonal matrices, so multiplication by them does not change rank. The rectangular diagonal matrix \(\boldsymbol{\Sigma}^{+}\) has exactly \(r\) nonzero diagonal entries, namely \(1/\sigma_1,\dots,1/\sigma_r\), and therefore has rank \(r\). Thus \(\operatorname{rank}\mathbf{A}^{+}=r\). Its dimensions are \(n\times m\) because the product \(\mathbf{V}\boldsymbol{\Sigma}^{+}\mathbf{U}^{\top}\) has those dimensions. If \(r=0\), \(\boldsymbol{\Sigma}\) and \(\boldsymbol{\Sigma}^{+}\) are zero, so the product defining \(\mathbf{A}^{+}\) is zero as well. \(\square\)
This proposition is a practical check on a proposed result: the pseudoinverse must have the reversed shape and the same rank as the original matrix. The number of reciprocal entries is determined by rank, not by the smaller of \(m\) and \(n\). A full-row-rank wide matrix has \(m\) positive singular values, while a rank-deficient wide matrix has fewer; the same distinction applies to tall matrices.
Common Pitfall: Trying to Invert Zero
A common incorrect step is to replace every diagonal entry by its reciprocal, including a zero singular value. For example, the rank-deficient matrix in the previous example has singular values \(2,0\). The calculation \(1/0\) is undefined, so the proposed reciprocal diagonal factor does not exist. The correction is to reciprocate only the positive singular values and retain zero entries wherever the original singular value is zero. Here the reciprocal factor has its only nonzero entry equal to \(1/2\), giving the displayed \(\mathbf{A}^{+}\).
A second dimensional check prevents another frequent error. For \(\mathbf{A}\in\mathbb{R}^{m\times n}\), the factor \(\boldsymbol{\Sigma}\) is \(m\times n\), while \(\boldsymbol{\Sigma}^{+}\) must be \(n\times m\). Simply taking reciprocals without transposing the rectangular layout can produce a matrix that cannot be multiplied in the order \(\mathbf{V}\boldsymbol{\Sigma}^{+}\mathbf{U}^{\top}\). The safe procedure is to write down dimensions before multiplying, then check the final matrix has \(n\) rows and \(m\) columns.
Check Your Understanding
For each question, use the SVD construction and check dimensions before multiplying.
- If \(\mathbf{A}\in\mathbb{R}^{4\times2}\) has rank \(2\), what are the dimensions of \(\boldsymbol{\Sigma}\), \(\boldsymbol{\Sigma}^{+}\), and \(\mathbf{A}^{+}\)?
- For singular values \(5,2,0\) in a \(3\times3\) diagonal factor, write the corresponding reciprocal diagonal factor.
- Construct the pseudoinverse of \(\begin{bmatrix}3&0\\0&0\\0&0\end{bmatrix}\) using its diagonal SVD.
- For \(\mathbf{A}=\begin{bmatrix}1&1\\1&1\end{bmatrix}\), identify its positive singular value and construct \(\mathbf{A}^{+}\).
- Explain why the reciprocal of a zero singular value is not part of the SVD construction.
