Repeated Singular Values And Nonunique Vectors

Why Singular Vectors Need Not Be Unique

A singular value decomposition can describe a matrix in more than one way. The source of this nonuniqueness is especially direct when \(A^{\top}A\) has a repeated positive eigenvalue: the corresponding eigenspace has more than one orthonormal basis, and each such basis gives valid right singular vectors. The left singular vectors change too, in a coordinated way, so the matrix represented by the decomposition stays fixed.

Throughout, let \(\mathbf{A}\in\mathbb{R}^{m\times n}\). A right singular vector \(\mathbf{v}\) with positive singular value \(\sigma\) is a unit eigenvector of \(\mathbf{A}^{\top}\mathbf{A}\) with eigenvalue \(\sigma^2\); its paired left singular vector is \(\mathbf{u}=\mathbf{A}\mathbf{v}/\sigma\). Thus, a repeated positive eigenvalue \(\lambda\) corresponds to the repeated singular value \(\sigma=\sqrt{\lambda}\). The eigenspace \(E_{\lambda}\) consists of all vectors \(\mathbf{v}\) satisfying \(\mathbf{A}^{\top}\mathbf{A}\mathbf{v}=\lambda\mathbf{v}\).

In what follows, we use the compact SVD, which lists the positive singular values: \(\mathbf{A}=\mathbf{U}_{r}\boldsymbol{\Sigma}_{r}\mathbf{V}_{r}^{\top}\). Here \(r\) is the number of positive singular values, the columns of \(\mathbf{U}_{r}\in\mathbb{R}^{m\times r}\) and \(\mathbf{V}_{r}\in\mathbb{R}^{n\times r}\) are orthonormal, and \(\boldsymbol{\Sigma}_{r}\) is the \(r\times r\) diagonal matrix of positive singular values. We focus on a block of \(k\) equal positive singular values \(\sigma\), corresponding to an eigenvalue \(\lambda=\sigma^2>0\) of \(\mathbf{A}^{\top}\mathbf{A}\).

Changing the Basis Inside a Repeated Block

Suppose \(\mathbf{v}_1,\dots,\mathbf{v}_k\) are orthonormal vectors in \(E_{\lambda}\), and let \(\mathbf{V}_{\lambda}\) be the matrix with these vectors as columns. The associated left vectors are \(\mathbf{u}_j=\mathbf{A}\mathbf{v}_j/\sigma\), where \(\sigma=\sqrt{\lambda}\). For another orthonormal basis of the same eigenspace, there is a \(k\times k\) orthogonal matrix \(\mathbf{Q}\), meaning \(\mathbf{Q}^{\top}\mathbf{Q}=\mathbf{I}_k\), such that the new basis matrix is \(\mathbf{V}_{\lambda}\mathbf{Q}\). The matching left-vector matrix is then \(\mathbf{U}_{\lambda}\mathbf{Q}\).

Theorem (Changing basis within a repeated positive singular-value block). Let \(\mathbf{A}\in\mathbb{R}^{m\times n}\) have a compact SVD, and suppose \(\sigma>0\) occurs \(k\) times in that SVD. Replacing the \(k\) right singular vectors in that block by any orthonormal basis of the same eigenspace \(E_{\sigma^2}\), and replacing their paired left singular vectors by \(\mathbf{A}\mathbf{v}/\sigma\), gives another valid compact SVD of \(\mathbf{A}\). The repeated singular value remains \(\sigma\).

Proof. Write \(\mathbf{V}_{\lambda}\) and \(\mathbf{U}_{\lambda}\) for the old right- and left-vector blocks. By the definition of the paired left vectors, \(\mathbf{A}\mathbf{V}_{\lambda}=\sigma\mathbf{U}_{\lambda}\). Let \(\mathbf{Q}\) be an orthogonal \(k\times k\) matrix and set \(\mathbf{V}'_{\lambda}=\mathbf{V}_{\lambda}\mathbf{Q}\). Its columns are an orthonormal basis of the same eigenspace: they lie in that space because it is closed under linear combinations, and their inner-product matrix is \((\mathbf{V}_{\lambda}\mathbf{Q})^{\top}(\mathbf{V}_{\lambda}\mathbf{Q})=\mathbf{Q}^{\top}\mathbf{Q}=\mathbf{I}_k\). The new paired left block is \(\mathbf{A}\mathbf{V}'_{\lambda}/\sigma=\mathbf{A}\mathbf{V}_{\lambda}\mathbf{Q}/\sigma=\mathbf{U}_{\lambda}\mathbf{Q}\), whose columns are also orthonormal because \((\mathbf{U}_{\lambda}\mathbf{Q})^{\top}(\mathbf{U}_{\lambda}\mathbf{Q})=\mathbf{I}_k\). In the compact SVD, the diagonal block for these singular values is \(\sigma\mathbf{I}_k\). Its contribution after the change is \((\mathbf{U}_{\lambda}\mathbf{Q})(\sigma\mathbf{I}_k)(\mathbf{V}_{\lambda}\mathbf{Q})^{\top}=\mathbf{U}_{\lambda}\mathbf{Q}(\sigma\mathbf{I}_k)\mathbf{Q}^{\top}\mathbf{V}_{\lambda}^{\top}=\mathbf{U}_{\lambda}(\sigma\mathbf{I}_k)\mathbf{V}_{\lambda}^{\top}\), since \(\mathbf{Q}\mathbf{Q}^{\top}=\mathbf{I}_k\). All other blocks are unchanged, so the represented matrix is still \(\mathbf{A}\), as required. \(\square\)

This result explains why the changes must be paired. Rotating the right vectors within the repeated block changes their images under \(\mathbf{A}\); the left vectors must rotate by the same orthogonal matrix. The equal diagonal entries \(\sigma\) are what allow the two changes to cancel in the product.

The Left Vectors Change with the Right Vectors

The formula \(\mathbf{u}=\mathbf{A}\mathbf{v}/\sigma\) does more than define the matching left vector: it guarantees orthonormality when the right vectors are an orthonormal basis of one positive eigenspace. If \(\mathbf{v}_i,\mathbf{v}_j\in E_{\sigma^2}\), then \(\langle \mathbf{A}\mathbf{v}_i,\mathbf{A}\mathbf{v}_j\rangle=\mathbf{v}_i^{\top}\mathbf{A}^{\top}\mathbf{A}\mathbf{v}_j=\sigma^2\mathbf{v}_i^{\top}\mathbf{v}_j\). Dividing by \(\sigma^2\) shows that the paired left vectors have dot product \(1\) when \(i=j\), and \(0\) when \(i\ne j\).

There is also a precise sense in which all orthonormal choices inside a fixed eigenspace are related this way. If two matrices \(\mathbf{V}_{\lambda}\) and \(\mathbf{W}_{\lambda}\) have orthonormal columns that are both bases of \(E_{\lambda}\), then \(\mathbf{Q}=\mathbf{V}_{\lambda}^{\top}\mathbf{W}_{\lambda}\) is orthogonal and \(\mathbf{W}_{\lambda}=\mathbf{V}_{\lambda}\mathbf{Q}\). To check this, each column of \(\mathbf{W}_{\lambda}\) lies in the column space of \(\mathbf{V}_{\lambda}\), and its coordinates in that orthonormal basis are given by multiplication by \(\mathbf{V}_{\lambda}^{\top}\). Hence \(\mathbf{W}_{\lambda}=\mathbf{V}_{\lambda}\mathbf{Q}\). Then \(\mathbf{Q}^{\top}\mathbf{Q}=\mathbf{W}_{\lambda}^{\top}\mathbf{V}_{\lambda}\mathbf{V}_{\lambda}^{\top}\mathbf{W}_{\lambda}=\mathbf{W}_{\lambda}^{\top}\mathbf{W}_{\lambda}=\mathbf{I}_k\), because \(\mathbf{V}_{\lambda}\mathbf{V}_{\lambda}^{\top}\) acts as the identity on vectors in that eigenspace. Thus the freedom is exactly an orthogonal change of coordinates within the eigenspace.

Why Distinct Singular Values Cannot Be Mixed

The freedom is limited to one eigenspace at a time. If different singular values are mixed, the resulting vectors need not be eigenvectors of \(\mathbf{A}^{\top}\mathbf{A}\), so they cannot be right singular vectors. Repeated eigenvalues matter because every vector in their eigenspace remains an eigenvector with that same eigenvalue; a combination of eigenvectors with different eigenvalues generally does not have this property.

Common mistake. A calculation such as “the columns of \(\mathbf{Q}\) are orthonormal, so rotating the right singular vectors always gives another SVD” omits the condition that \(\mathbf{Q}\) act only within a block of equal singular values. In the example above, \(\mathbf{Q}\) is orthogonal, but it mixes eigenvalues \(9\) and \(1\), and the off-diagonal entries \(-4\) in \(\mathbf{Q}^{\top}\mathbf{A}^{\top}\mathbf{A}\mathbf{Q}\) show the failure. The correction is to preserve the eigenspace blocks: orthogonal changes are allowed among vectors with the same positive singular value, with the paired left vectors changed by the same matrix.

What Changes at Singular Value Zero?

The positive-singular-value result uses division by \(\sigma\), so it does not apply when \(\sigma=0\). A right vector in the zero-eigenvalue eigenspace satisfies \(\mathbf{A}^{\top}\mathbf{A}\mathbf{v}=\mathbf{0}\), and its image under \(\mathbf{A}\) is zero. There is no paired left vector obtained by \(\mathbf{A}\mathbf{v}/0\). In a full SVD, zero-singular-value directions are completed using orthonormal vectors on the left and right; the positive-block pairing rule does not determine those left vectors.

For a repeated positive singular value, by contrast, the rule is fully determined: choose any orthonormal basis of its right eigenspace, apply \(\mathbf{A}/\sigma\) to obtain the matching orthonormal left vectors, and use the same orthogonal change in both blocks. The matrix stays fixed because the repeated diagonal block is a scalar multiple of the identity. This is nonuniqueness of the vectors, not nonuniqueness of the singular value or of the matrix being decomposed.