Right Singular Vectors And Their Images

Right Singular Vectors Describe Input Directions

A matrix takes vectors in a domain to vectors in a codomain. Some unit input directions are especially informative: their images have a simple, predictable length, and their direction can be identified by normalizing the image. These inputs are the right singular vectors. The word “right” refers to their role as vectors in the domain of the matrix, not to a claim that their images point in the same direction as the inputs.

Throughout, let \(\mathbf{A}\in\mathbb{R}^{m\times n}\). Its domain is \(\mathbb{R}^n\), and its outputs lie in \(\mathbb{R}^m\). The matrix \(\mathbf{A}^{\top}\mathbf{A}\) is an \(n\times n\) matrix, so its eigenvectors are domain vectors. As established in the earlier material on singular values, its eigenvalues are nonnegative. If \(\lambda\) is such an eigenvalue, its associated singular value is \(\sigma=\sqrt{\lambda}\), using the nonnegative square root.

The eigenvector equation takes place in \(\mathbb{R}^n\), the domain. But \(\mathbf{A}\mathbf{v}\) lies in \(\mathbb{R}^m\), the codomain. To relate the equation to the image, take the dot product with \(\mathbf{v}\): the result measures the squared length of \(\mathbf{A}\mathbf{v}\). When \(\sigma>0\), this length lets us normalize the image and obtain a unit vector \(\mathbf{u}\) in the codomain.

From the Eigenvector Equation to the Image

Proof. First compute the squared norm of the image. Because \(\mathbf{v}\) is a unit vector and satisfies the given eigenvector equation,

Since \(\sigma>0\), the definition \(\mathbf{u}=\frac{1}{\sigma}\mathbf{A}\mathbf{v}\) is valid. Its squared norm is \(\|\mathbf{u}\|^2=\frac{1}{\sigma^2}\|\mathbf{A}\mathbf{v}\|^2=\frac{\sigma^2}{\sigma^2}=1\), so \(\mathbf{u}\) is a unit vector in \(\mathbb{R}^m\). Finally, multiplying its definition by \(\sigma\) gives \(\sigma\mathbf{u}=\mathbf{A}\mathbf{v}\), as required. \(\square\)

This result gives the image relation in the form

The domain vector \(\mathbf{v}\) is unit length. Its image has length \(\sigma\), since \(\|\mathbf{A}\mathbf{v}\|=\sigma\), and \(\mathbf{u}\) records the unit direction of that image. The positivity condition matters: when \(\sigma=0\), division by \(\sigma\) is not defined, and the image is instead the zero vector.

Worked Images in Small Matrices

What a Zero Singular Value Means

Proof. Suppose first that \(\mathbf{A}^{\top}\mathbf{A}\mathbf{v}=\mathbf{0}\). Then \(\|\mathbf{A}\mathbf{v}\|^2=\mathbf{v}^{\top}\mathbf{A}^{\top}\mathbf{A}\mathbf{v}=\mathbf{v}^{\top}\mathbf{0}=0\). A vector in \(\mathbb{R}^m\) has squared norm equal to the sum of the squares of its entries, so squared norm zero forces every entry to be zero. Hence \(\mathbf{A}\mathbf{v}=\mathbf{0}\). Conversely, if \(\mathbf{A}\mathbf{v}=\mathbf{0}\), then \(\mathbf{A}^{\top}\mathbf{A}\mathbf{v}=\mathbf{A}^{\top}\mathbf{0}=\mathbf{0}\). Both implications hold, proving the equivalence. \(\square\)

In particular, if \(\mathbf{v}\) is a right singular vector with singular value zero, then its eigenvector equation is \(\mathbf{A}^{\top}\mathbf{A}\mathbf{v}=0\mathbf{v}=\mathbf{0}\), and the theorem gives \(\mathbf{A}\mathbf{v}=\mathbf{0}\). Such a domain direction is not stretched to a nonzero output; it is sent to zero.

Reading the Relation as a Domain Statement

The defining eigenvector equation is a statement about a direction before it is acted on by \(\mathbf{A}\). For a unit right singular vector \(\mathbf{v}_i\) with positive singular value \(\sigma_i\), the image relation says that an input exactly in that direction produces the output \(\sigma_i\mathbf{u}_i\), where \(\mathbf{u}_i\) is the unit direction of the image. More generally, for any real scalar \(c\), linearity gives \(\mathbf{A}(c\mathbf{v}_i)=c\sigma_i\mathbf{u}_i\). If \(c\) is negative, the image points in the opposite direction from \(\mathbf{u}_i\); its length is \(|c|\sigma_i\).

This is why right singular vectors are useful when thinking about how a matrix acts on its domain. They identify particular unit inputs for which the amount of stretching is known exactly. The singular value gives that amount, while the normalized image records where the output points. If the singular value is zero, the corresponding input direction is erased by the matrix. These statements concern individual directions; they do not say that an arbitrary input vector is stretched by one singular value.

Common pitfall: replacing \(\sigma\) with \(\sigma^2\) in the image relation. The eigenvector equation contains \(\sigma^2\), but it is an equation for \(\mathbf{A}^{\top}\mathbf{A}\mathbf{v}\), not for \(\mathbf{A}\mathbf{v}\). For example, with \(\mathbf{A}=\begin{bmatrix}3&0\\0&2\\0&0\end{bmatrix}\) and \(\mathbf{v}_1=\begin{bmatrix}1\\0\end{bmatrix}\), one has \(\mathbf{A}^{\top}\mathbf{A}\mathbf{v}_1=\begin{bmatrix}9\\0\end{bmatrix}=3^2\mathbf{v}_1\), but \(\mathbf{A}\mathbf{v}_1=\begin{bmatrix}3\\0\\0\end{bmatrix}\), not \(9\mathbf{v}_1\). In this rectangular example the mistaken expression even has the wrong dimension: \(\mathbf{A}\mathbf{v}_1\in\mathbb{R}^3\), while \(\mathbf{v}_1\in\mathbb{R}^2\). The correction is to normalize the image: \(\mathbf{u}_1=\frac{1}{3}\mathbf{A}\mathbf{v}_1=\begin{bmatrix}1\\0\\0\end{bmatrix}\), giving \(\mathbf{A}\mathbf{v}_1=3\mathbf{u}_1\).