SVDs Of Orthogonal, Projection, And Scaled Matrices

Structure Can Make an SVD Immediate

For a general matrix, finding singular values usually begins with the eigenvalues of its Gram matrix. Orthogonal matrices and orthogonal projections have enough structure that their singular values can often be read off without solving a characteristic equation. The key is to use the defining equations of these matrices to identify the action on suitable orthonormal vectors.

We work over \(\mathbb{R}\). For a square matrix \(\mathbf{A}\in\mathbb{R}^{n\times n}\), a singular value decomposition (SVD) is a factorization \(\mathbf{A}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\), where \(\mathbf{U}\) and \(\mathbf{V}\) are orthogonal matrices and \(\boldsymbol{\Sigma}\) is diagonal with nonnegative entries. The columns of \(\mathbf{V}\) are right singular vectors; the corresponding columns of \(\mathbf{U}\) are left singular vectors. A paired set of columns \(\mathbf{v}_i,\mathbf{u}_i\) with diagonal entry \(\sigma_i\) satisfies \(\mathbf{A}\mathbf{v}_i=\sigma_i\mathbf{u}_i\).

Orthogonal Matrices

An orthogonal matrix preserves lengths: for every \(\mathbf{x}\in\mathbb{R}^n\), \(\|\mathbf{Q}\mathbf{x}\|^2=(\mathbf{Q}\mathbf{x})^{\top}(\mathbf{Q}\mathbf{x})=\mathbf{x}^{\top}\mathbf{Q}^{\top}\mathbf{Q}\mathbf{x}=\mathbf{x}^{\top}\mathbf{x}=\|\mathbf{x}\|^2\). For singular values, an even more direct fact is that \(\mathbf{Q}^{\top}\mathbf{Q}\) is the identity. Therefore every singular value is \(1\).

Proof. In the proposed factorization, \(\mathbf{U}=\mathbf{Q}\) is orthogonal by hypothesis, \(\mathbf{V}=\mathbf{I}\) is orthogonal, and \(\boldsymbol{\Sigma}=\mathbf{I}\) is diagonal with nonnegative entries. Multiplication gives \(\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}=\mathbf{Q}\mathbf{I}\mathbf{I}^{\top}=\mathbf{Q}\). Thus this is an SVD with every diagonal entry equal to \(1\). Equivalently, \(\mathbf{Q}^{\top}\mathbf{Q}=\mathbf{I}\), whose eigenvalues are all \(1\); their nonnegative square roots, the singular values, are all \(1\). \(\square\)

Orthogonal Projections

A projection acts differently on two complementary parts of the space. It leaves every vector in its image unchanged and sends every vector in its kernel to zero. For an orthogonal projection, these parts are perpendicular. This gives an orthonormal basis in which the projection has only \(1\)s and \(0\)s on its diagonal.

Proof. First, if \(\mathbf{x}\in\operatorname{im}\mathbf{P}\), then \(\mathbf{x}=\mathbf{P}\mathbf{y}\) for some \(\mathbf{y}\), and \(\mathbf{P}\mathbf{x}=\mathbf{P}^2\mathbf{y}=\mathbf{P}\mathbf{y}=\mathbf{x}\). If \(\mathbf{z}\in\ker\mathbf{P}\), then \(\mathbf{P}\mathbf{z}=\mathbf{0}\). For any such \(\mathbf{x},\mathbf{z}\), symmetry gives \(\mathbf{x}^{\top}\mathbf{z}=(\mathbf{P}\mathbf{y})^{\top}\mathbf{z}=\mathbf{y}^{\top}\mathbf{P}^{\top}\mathbf{z}=\mathbf{y}^{\top}\mathbf{P}\mathbf{z}=0\). So the image and kernel are orthogonal. Every \(\mathbf{w}\in\mathbb{R}^n\) decomposes as \(\mathbf{w}=\mathbf{P}\mathbf{w}+(\mathbf{w}-\mathbf{P}\mathbf{w})\). The first term lies in the image, and the second lies in the kernel because \(\mathbf{P}(\mathbf{w}-\mathbf{P}\mathbf{w})=\mathbf{P}\mathbf{w}-\mathbf{P}^2\mathbf{w}=\mathbf{0}\). Choose orthonormal bases for these two subspaces and combine them; their perpendicularity makes the combined list an orthonormal basis of \(\mathbb{R}^n\). Write its vectors as \(\mathbf{v}_1,\dots,\mathbf{v}_n\), with the first \(r\) in the image. Then \(\mathbf{P}\mathbf{v}_i=\mathbf{v}_i\) for \(i\le r\), and \(\mathbf{P}\mathbf{v}_i=\mathbf{0}\) for \(i>r\). Let \(\mathbf{V}\) have these basis vectors as columns and set \(\boldsymbol{\Sigma}=\operatorname{diag}(1,\dots,1,0,\dots,0)\), with \(r\) entries equal to \(1\). The equations on the basis vectors give \(\mathbf{P}\mathbf{V}=\mathbf{V}\boldsymbol{\Sigma}\). Since \(\mathbf{V}\) is orthogonal, multiplication on the right by \(\mathbf{V}^{\top}\) gives \(\mathbf{P}=\mathbf{V}\boldsymbol{\Sigma}\mathbf{V}^{\top}\). Taking \(\mathbf{U}=\mathbf{V}\) gives an SVD, with the stated singular values. \(\square\)

Scalar Multiples

Multiplying a matrix by a real scalar changes the lengths of its outputs by the absolute value of that scalar. A negative scalar also reverses their directions. In an SVD, the singular values must stay nonnegative, so the sign belongs in one of the orthogonal factors, not in the diagonal matrix.

Proof. For an orthogonal \(\mathbf{Q}\) and \(c\ne0\), the matrix \(\operatorname{sgn}(c)\mathbf{Q}\) is orthogonal because its transpose times itself is \(\operatorname{sgn}(c)^2\mathbf{Q}^{\top}\mathbf{Q}=\mathbf{I}\). The proposed product is \(\operatorname{sgn}(c)\mathbf{Q}|c|\mathbf{I}\mathbf{I}^{\top}=c\mathbf{Q}\), and its diagonal entries are nonnegative. For the projection, the earlier factorization is \(\mathbf{P}=\mathbf{V}\boldsymbol{\Sigma}_0\mathbf{V}^{\top}\), where \(\boldsymbol{\Sigma}_0\) has \(r\) ones followed by zeros. The proposed left factor \(\operatorname{sgn}(c)\mathbf{V}\) is orthogonal, and \((\operatorname{sgn}(c)\mathbf{V})(|c|\boldsymbol{\Sigma}_0)\mathbf{V}^{\top}=c\mathbf{P}\); again the diagonal entries are nonnegative. If \(c=0\), then \(c\mathbf{Q}\) and \(c\mathbf{P}\) are zero matrices, and \(\mathbf{O}=\mathbf{I}\mathbf{O}\mathbf{I}^{\top}\) is an SVD with every singular value zero. \(\square\)

Common Pitfall: Singular Values Cannot Be Negative

A tempting but incorrect factorization of \(-2\mathbf{P}\) is to keep \(\mathbf{U}=\mathbf{V}\) and write a diagonal matrix with entries \(-2,-2,0\). That is not an SVD, because its diagonal entries are negative. The correction is to use nonnegative singular values \(2,2,0\) and put the sign into \(\mathbf{U}\), as in the worked example. More generally, for any nonzero real \(c\), the singular values of \(c\mathbf{Q}\) are \(|c|\), not \(c\); those of \(c\mathbf{P}\) are \(|c|\) on the image directions and \(0\) on the kernel directions.

These constructions are useful because they identify not only the singular values but also the directions on which a matrix acts. An orthogonal matrix preserves every length, so all its singular values are \(1\). A projection preserves lengths on its image and annihilates its perpendicular kernel, so its singular values are \(1\) or \(0\). Scaling multiplies the nonzero lengths by \(|c|\). In each case, checking the orthogonal factors and multiplying \(\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\) back gives a direct verification of the proposed SVD.