Completing Singular Vector Bases At Zero

Why Zero Singular Values Need Their Own Vectors

For a nonzero singular value, a right singular vector \(\mathbf{v}\) determines its matching left singular vector: if \(\|\mathbf{v}\|=1\), \( \mathbf{A}^{\top}\mathbf{A}\mathbf{v}=\sigma^2\mathbf{v}\), and \(\sigma>0\), then \(\mathbf{u}=\mathbf{A}\mathbf{v}/\sigma\). This formula cannot be used when \(\sigma=0\), because division by zero is undefined. Instead, the zero singular vectors come from null spaces: on the right from \(\operatorname{Nul}\mathbf{A}\), and on the left from \(\operatorname{Nul}\mathbf{A}^{\top}\).

Let \(\mathbf{A}\in\mathbb{R}^{m\times n}\). Its right vectors lie in \(\mathbb{R}^n\), while its left vectors lie in \(\mathbb{R}^m\). These spaces can have different dimensions, so completing one basis does not automatically complete the other. The central task is to identify the relevant orthogonal complements and determine how many vectors each requires.

Zero Right Singular Vectors

The zero eigenvalue of \(\mathbf{A}^{\top}\mathbf{A}\) describes exactly the right null space. This follows directly from the dot product: for every \(\mathbf{x}\in\mathbb{R}^n\), \(\mathbf{x}^{\top}\mathbf{A}^{\top}\mathbf{A}\mathbf{x}=\|\mathbf{A}\mathbf{x}\|^2\).

Proof. Suppose first that \(\mathbf{A}^{\top}\mathbf{A}\mathbf{x}=\mathbf{0}\). Multiply on the left by \(\mathbf{x}^{\top}\). Then \(\|\mathbf{A}\mathbf{x}\|^2=\mathbf{x}^{\top}\mathbf{A}^{\top}\mathbf{A}\mathbf{x}=0\). A vector has squared norm zero only when it is the zero vector, so \(\mathbf{A}\mathbf{x}=\mathbf{0}\). Conversely, if \(\mathbf{A}\mathbf{x}=\mathbf{0}\), then \(\mathbf{A}^{\top}\mathbf{A}\mathbf{x}=\mathbf{A}^{\top}\mathbf{0}=\mathbf{0}\). The two sets of vectors are therefore equal. \(\square\)

In particular, an orthonormal basis for the zero eigenspace gives all the right zero singular vectors without dividing by a singular value. In an SVD construction, an orthonormal eigenbasis of the real symmetric matrix \(\mathbf{A}^{\top}\mathbf{A}\) supplies the right singular vectors; the zero-eigenvalue part of that basis is precisely the null-space basis described here. The existence of an orthonormal eigenbasis is the standard spectral theorem for real symmetric matrices.

Zero Left Singular Vectors

The left-side construction is governed by the column space. Recall that \(\operatorname{Col}\mathbf{A}\) is the set of vectors \(\mathbf{A}\mathbf{x}\) as \(\mathbf{x}\) ranges over \(\mathbb{R}^n\). A vector \(\mathbf{u}\in\mathbb{R}^m\) is perpendicular to every column of \(\mathbf{A}\) exactly when \(\mathbf{A}^{\top}\mathbf{u}=\mathbf{0}\). Consequently, the left zero singular vectors fill the orthogonal complement of the column space.

Proof. Let \(\mathbf{u}\in\mathbb{R}^m\). If \(\mathbf{u}\in(\operatorname{Col}\mathbf{A})^{\perp}\), then \(\mathbf{u}\) is perpendicular to every column \(\mathbf{a}_j\) of \(\mathbf{A}\). The entries of \(\mathbf{A}^{\top}\mathbf{u}\) are \(\mathbf{a}_j^{\top}\mathbf{u}\), so every entry is zero and \(\mathbf{A}^{\top}\mathbf{u}=\mathbf{0}\). Conversely, if \(\mathbf{A}^{\top}\mathbf{u}=\mathbf{0}\), then \(\mathbf{a}_j^{\top}\mathbf{u}=0\) for every column. Every vector in \(\operatorname{Col}\mathbf{A}\) is a linear combination of these columns, so it too is perpendicular to \(\mathbf{u}\). Hence \(\mathbf{u}\in(\operatorname{Col}\mathbf{A})^{\perp}\). \(\square\)

This theorem describes where to find the left zero vectors, but it also explains how to complete the left basis. The positive left singular vectors span \(\operatorname{Col}\mathbf{A}\): each is a scalar multiple of \(\mathbf{A}\mathbf{v}\), so belongs to the column space; and, when the right singular vectors form an orthonormal eigenbasis, expanding any input in those vectors shows that \(\mathbf{A}\mathbf{x}\) is a combination of the positive left vectors. An orthonormal basis of \((\operatorname{Col}\mathbf{A})^\perp\) then completes that positive family to a basis of \(\mathbb{R}^m\).

How Many Zero Vectors Are Needed?

Suppose there are \(r\) positive singular values, with corresponding orthonormal right vectors \(\mathbf{v}_1,\ldots,\mathbf{v}_r\) and left vectors \(\mathbf{u}_1,\ldots,\mathbf{u}_r\). In the full construction, the right zero vectors complete the basis in \(\mathbb{R}^n\), while the left zero vectors complete it in \(\mathbb{R}^m\). Therefore there are \(n-r\) right zero vectors and \(m-r\) left zero vectors. These counts need not agree.

More precisely, when the right vectors are an orthonormal eigenbasis of \(\mathbf{A}^{\top}\mathbf{A}\), the zero-eigenvalue vectors form an orthonormal basis of \(\operatorname{Nul}\mathbf{A}\), by the first theorem. The positive left vectors span \(\operatorname{Col}\mathbf{A}\), and the remaining left basis vectors form an orthonormal basis of \(\operatorname{Nul}\mathbf{A}^{\top}\), by the second theorem. The standard finite-dimensional fact used here is that an orthonormal basis of a subspace can be extended to an orthonormal basis of the whole space; one may apply Gram–Schmidt to additional vectors until the space is spanned.

The full matrix \(\mathbf{U}\) has \(m\) columns, and \(\mathbf{V}\) has \(n\) columns. The rectangular diagonal matrix \(\boldsymbol{\Sigma}\) has \(m\) rows and \(n\) columns. Thus a tall matrix can require more left than right zero vectors, and a wide matrix can require more right than left zero vectors. The extra vectors complete the orthonormal bases; they do not create additional nonzero singular values.

When the Left Null Space Is Not the Right Null Space

A common mistake is to use \(\operatorname{Nul}\mathbf{A}\) for both sides. The two null spaces belong to different ambient spaces: \(\operatorname{Nul}\mathbf{A}\subseteq\mathbb{R}^n\), whereas \(\operatorname{Nul}\mathbf{A}^{\top}\subseteq\mathbb{R}^m\). They describe different conditions, \(\mathbf{A}\mathbf{v}=\mathbf{0}\) and \(\mathbf{A}^{\top}\mathbf{u}=\mathbf{0}\), respectively.

For the tall matrix \(\mathbf{A}=\begin{bmatrix}1&0\\0&0\\0&0\end{bmatrix}\), the incorrect calculation would be to say that the right null space is \(\operatorname{span}\{(0,1)^{\top}\}\), and therefore the left null space is the same one-dimensional space. That conclusion cannot be right: a left vector must have three entries, not two. The correction is to calculate in the correct ambient space: \(\operatorname{Nul}\mathbf{A}^{\top}=\operatorname{span}\{(0,1,0)^{\top},(0,0,1)^{\top}\}\). This gives two left zero vectors, while \(\operatorname{Nul}\mathbf{A}\) gives one right zero vector.

At zero singular values there is no formula dividing \(\mathbf{A}\mathbf{v}\) by \(\sigma\). Use the right null space for the right basis, and the orthogonal complement of the column space—or, equivalently, \(\operatorname{Nul}\mathbf{A}^{\top}\)—for the left basis. The two completions are made separately, in their own dimensions.