Singular Values From A Transpose A

From a Matrix to Its Singular Values

Eigenvalues describe how a square matrix acts on its eigenvectors. Singular values provide a related measurement for any real matrix, including rectangular ones. To define them, begin with the square matrix \(\mathbf{A}^{\top}\mathbf{A}\), where \(\mathbf{A}\in\mathbb{R}^{m\times n}\). This matrix is called the Gram matrix of \(\mathbf{A}\); it has \(n\) rows and \(n\) columns, so its eigenvalues correspond to vectors in the input space \(\mathbb{R}^n\).

The Gram matrix is symmetric, since \((\mathbf{A}^{\top}\mathbf{A})^{\top}=\mathbf{A}^{\top}\mathbf{A}\). The Spectral Theorem for real symmetric matrices says that it has real eigenvalues and an orthonormal basis of eigenvectors. We also use the standard result that the eigenvalues of \(\mathbf{A}^{\top}\mathbf{A}\) are nonnegative; its proof comes from examining why Gram matrices have nonnegative eigenvalues. These facts ensure that the square roots in the definition below are real.

There are \(n\) singular values, not necessarily \(\min(m,n)\) of them: the definition counts the \(n\) eigenvalues of the \(n\times n\) matrix \(\mathbf{A}^{\top}\mathbf{A}\), including zero eigenvalues. Some conventions list only the positive singular values, but here the full list is used. Repeated eigenvalues are repeated in the list, so repeated singular values are repeated as well. The right singular vectors are chosen from the corresponding eigenspaces; they need not be uniquely determined when an eigenspace has dimension greater than one.

Computing the Values and Vectors

The calculation has a consistent order: form \(\mathbf{A}^{\top}\mathbf{A}\), find its eigenvalues with multiplicity, take their nonnegative square roots, and find unit eigenvectors for those eigenvalues. The vector associated with a singular value \(\sigma\) is an eigenvector for \(\sigma^2\), not for \(\sigma\). The transpose in \(\mathbf{A}^{\top}\mathbf{A}\) also matters: the resulting eigenvectors are in \(\mathbb{R}^n\), the space of inputs to \(\mathbf{A}\).

Multiplicity and Choice of Right Singular Vectors

When an eigenvalue of \(\mathbf{A}^{\top}\mathbf{A}\) is repeated, its square root is repeated in the singular-value list. The corresponding right singular vectors belong to the same eigenspace. If that eigenspace has dimension greater than one, there is no unique choice of unit eigenvectors: any orthonormal basis of the eigenspace gives valid right singular vectors for that repeated singular value. The multiplicity records how many vectors are needed in an orthonormal eigenbasis, not a unique set of directions.

Proof. Let \(\mathbf{x}\in\mathbb{R}^n\). If \(\mathbf{x}\) is in the zero eigenspace, then \(\mathbf{A}^{\top}\mathbf{A}\mathbf{x}=\mathbf{0}\). Multiplying on the left by \(\mathbf{x}^{\top}\) gives \(\mathbf{x}^{\top}\mathbf{A}^{\top}\mathbf{A}\mathbf{x}=0\). Since \(\mathbf{x}^{\top}\mathbf{A}^{\top}\mathbf{A}\mathbf{x}=(\mathbf{A}\mathbf{x})^{\top}(\mathbf{A}\mathbf{x})\), this is the sum of the squares of the entries of \(\mathbf{A}\mathbf{x}\). A sum of real squares is zero only if every entry is zero, so \(\mathbf{A}\mathbf{x}=\mathbf{0}\), which means \(\mathbf{x}\in\operatorname{Nul}\mathbf{A}\). Conversely, if \(\mathbf{A}\mathbf{x}=\mathbf{0}\), then \(\mathbf{A}^{\top}\mathbf{A}\mathbf{x}=\mathbf{A}^{\top}\mathbf{0}=\mathbf{0}\), so \(\mathbf{x}\) belongs to the zero eigenspace. The two spaces are equal. Because \(\mathbf{A}^{\top}\mathbf{A}\) is real symmetric, the Spectral Theorem gives equality of the algebraic and geometric multiplicities of each eigenvalue. Thus the multiplicity of eigenvalue \(0\), and hence the number of zero singular values, is the dimension of that eigenspace, namely \(\dim(\operatorname{Nul}\mathbf{A})\). \(\square\)

This result explains why zero singular values must be retained in the full list. They correspond precisely to directions in the input space that the matrix sends to zero. In particular, a rectangular matrix can have a nontrivial null space even though its Gram matrix is square. If the Rank-Nullity Theorem is used, the number of zero singular values can also be written as \(n-\operatorname{rank}\mathbf{A}\).

Common Mistakes

Using the eigenvalues themselves as singular values. For the first example, a wrong calculation would stop at the eigenvalues \(9\) and \(4\) of \(\mathbf{A}^{\top}\mathbf{A}\) and report singular values \(9\) and \(4\). The correction is to take nonnegative square roots: \(\sqrt{9}=3\) and \(\sqrt{4}=2\). Substituting back confirms the relationship: \(3^2=9\) and \(2^2=4\).

Dropping the zero values or their vectors. In the rectangular example, the Gram matrix has three eigenvalues, including \(0\), so the full list has three singular values, \(2,1,0\). The right singular vector for \(0\) is not an exception to the definition: it is a unit eigenvector with eigenvalue \(0^2\).

Assuming a repeated value determines one unique vector. A repeated singular value specifies an eigenspace, not a single right singular vector. Check that a proposed vector is a unit eigenvector of \(\mathbf{A}^{\top}\mathbf{A}\) for the squared value; for a repeated value, different orthonormal choices may pass the same check.

For any real \(m\times n\) matrix, the defining data can be summarised as follows.