SVDs Of Rank-Deficient Matrices

What Rank Deficiency Changes in an SVD

When a matrix has full possible rank, every diagonal position in its singular-value matrix is positive. If the matrix is rank-deficient, at least one of those positions is zero. That zero is not merely a missing scale factor: it signals a direction that the matrix sends to the zero vector. A full SVD must also account for the other directions needed to make its left and right singular vectors orthonormal bases.

Let \(\mathbf{A}\in\mathbb{R}^{m\times n}\), and put \(p=\min(m,n)\). The matrix \(\mathbf{A}^{\top}\mathbf{A}\) is an \(n\times n\) symmetric matrix with nonnegative eigenvalues. The convention for rectangular matrices is important: the singular values are the square roots of the largest \(p\) eigenvalues, counting multiplicity, listed in nonincreasing order. The full SVD has the shape \(\mathbf{A}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\), where \(\mathbf{U}\) is \(m\times m\) orthogonal, \(\mathbf{V}\) is \(n\times n\) orthogonal, and \(\boldsymbol{\Sigma}\) is \(m\times n\), with the \(p\) singular values on its diagonal positions and zeros elsewhere.

For a wide matrix, \(m<n\), the Gram matrix has \(n\) eigenvalues but the singular-value list has only \(p=m\) entries. The additional \(n-m\) zero eigenvalues forced by the dimensions are not extra diagonal positions in \(\boldsymbol{\Sigma}\). A rank-deficient wide matrix can have further zero eigenvalues among the \(p\) eigenvalues used to define its singular values. Keeping these two kinds of zeros distinct prevents a common shape error.

Why a Zero Eigenvalue Gives a Null-Space Direction

The null space of \(\mathbf{A}\) is the set of vectors \(\mathbf{x}\in\mathbb{R}^n\) for which \(\mathbf{A}\mathbf{x}=\mathbf{0}\). The following fact identifies these vectors directly from the Gram matrix. It is especially useful when constructing the zero-singular-value columns of \(\mathbf{V}\).

Proof. If \(\mathbf{A}\mathbf{x}=\mathbf{0}\), then \(\mathbf{A}^{\top}\mathbf{A}\mathbf{x}=\mathbf{A}^{\top}\mathbf{0}=\mathbf{0}\). Conversely, suppose \(\mathbf{A}^{\top}\mathbf{A}\mathbf{x}=\mathbf{0}\). Taking the dot product with \(\mathbf{x}\) gives \(\mathbf{x}^{\top}\mathbf{A}^{\top}\mathbf{A}\mathbf{x}=0\). The left side equals \((\mathbf{A}\mathbf{x})^{\top}(\mathbf{A}\mathbf{x})=\|\mathbf{A}\mathbf{x}\|^2\). A squared Euclidean norm is zero only for the zero vector, so \(\mathbf{A}\mathbf{x}=\mathbf{0}\). Thus the two null spaces are equal. \(\square\)

This also explains why a zero-eigenvalue right singular vector is a null-space direction. For a positive eigenvalue \(\lambda_i=\sigma_i^2\) with unit eigenvector \(\mathbf{v}_i\), the corresponding left singular vector is \(\mathbf{u}_i=\mathbf{A}\mathbf{v}_i/\sigma_i\). For a zero eigenvalue, this division is not defined; instead, the theorem says \(\mathbf{A}\mathbf{v}_i=\mathbf{0}\), and \(\mathbf{v}_i\) is kept as a right singular vector while the left basis is completed separately.

Completing the Bases in a Full SVD

The positive singular values determine matched left and right singular vectors. The zero positions require care: right null-space directions are eigenvectors of \(\mathbf{A}^{\top}\mathbf{A}\), but they do not determine left singular vectors by dividing \(\mathbf{A}\mathbf{v}_i\) by zero. Instead, complete the left vectors already constructed to an orthonormal basis. The finite-dimensional Gram–Schmidt process supplies such completions.

Proof. Let \(r\) be the number of positive eigenvalues of \(\mathbf{A}^{\top}\mathbf{A}\), counting multiplicity. The real Spectral Theorem gives the stated orthonormal eigenbasis. For \(1\le i,j\le r\),

Thus the constructed left vectors are orthonormal. For each \(i>r\), the eigenvalue is zero, so the null-space theorem gives \(\mathbf{A}\mathbf{v}_i=\mathbf{0}\). The left vectors \(\mathbf{u}_1,\dots,\mathbf{u}_r\) can be completed to an orthonormal basis of \(\mathbb{R}^m\). If \(m\ge n\), there are \(n-r\) zero diagonal positions and \(m-r\) vectors in the left completion; pair \(n-r\) of those left vectors with the \(n-r\) zero-eigenvalue right vectors. The remaining \(m-n\) left vectors occupy rows with no diagonal position. If \(m<n\), there are \(m-r\) zero diagonal positions; pair that many vectors from the right zero-eigenvalue list with the \(m-r\) left-completion vectors. The remaining \(n-m\) right vectors occupy columns with no diagonal position.

It remains to verify the factorization. Every \(\mathbf{x}\in\mathbb{R}^n\) has an expansion \(\mathbf{x}=\sum_{i=1}^n c_i\mathbf{v}_i\). Therefore \(\mathbf{A}\mathbf{x}=\sum_{i=1}^r c_i\mathbf{A}\mathbf{v}_i =\sum_{i=1}^r c_i\sigma_i\mathbf{u}_i\), because the terms for \(i>r\) vanish. The product \(\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\mathbf{x}\) has exactly this value: \(\mathbf{V}^{\top}\mathbf{x}\) has entries \(c_i\), and \(\boldsymbol{\Sigma}\) retains the first \(r\) entries scaled by \(\sigma_i\), with all zero positions contributing zero. The two matrices act identically on every \(\mathbf{x}\), so they are equal. If \(r=0\), all eigenvalues are zero, every \(\mathbf{A}\mathbf{v}_i=\mathbf{0}\), and the same argument gives \(\mathbf{A}=\mathbf{O}\). \(\square\)

A Wide Matrix and the Extra Zero Eigenvalue

A Square Example with Two Positive Singular Values

Common Mistake: Counting Every Gram Eigenvalue as a Singular Value

For the wide matrix \(\mathbf{B}\) above, a tempting but incorrect statement is: “\(\mathbf{B}^{\top}\mathbf{B}\) has three eigenvalues, so the singular values are \(\sqrt{2},0,0\), and \(\boldsymbol{\Sigma}\) must have three diagonal entries.” This mixes the \(n=3\) eigenvalues of the Gram matrix with the \(p=2\) singular values in the rectangular SVD convention. The full \(\boldsymbol{\Sigma}\) is \(2\times3\), so it has only two diagonal positions. The correction is to list the singular values as \(\sqrt{2},0\), place them in those two positions, and recognize the third Gram eigenvalue as an extra zero associated with the wide shape.

A second pitfall is to try to form a left singular vector for a zero singular value by using \(\mathbf{A}\mathbf{v}/\sigma\). When \(\sigma=0\), this expression is undefined. The correct steps are to keep \(\mathbf{v}\) as a right null-space direction and complete the already constructed left vectors to an orthonormal basis.