A Workflow for SVD Problems
An SVD problem often asks for more than a list of singular values. Starting from a matrix, we may need to construct the singular vectors, verify the factorization, and then interpret the result as information about the matrix’s action. A reliable solution keeps these tasks connected: right singular vectors describe directions in the input space, left singular vectors describe their output directions, and singular values give the amount of stretching.
Throughout, matrices and vectors are real. For \(\mathbf{A}\in\mathbb{R}^{m\times n}\), a full singular value decomposition is \(\mathbf{A}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\), where \(\mathbf{U}\in\mathbb{R}^{m\times m}\) and \(\mathbf{V}\in\mathbb{R}^{n\times n}\) are orthogonal, and \(\boldsymbol{\Sigma}\in\mathbb{R}^{m\times n}\) is diagonal in the rectangular sense, with nonnegative diagonal entries in decreasing order. These entries are the singular values \(\sigma_1\ge\cdots\ge\sigma_{\min(m,n)}\ge0\). Orthogonal means that the columns form an orthonormal basis, so \(\mathbf{U}^{\top}\mathbf{U}=\mathbf{I}_m\) and \(\mathbf{V}^{\top}\mathbf{V}=\mathbf{I}_n\).
In a computation, the standard starting point is \(\mathbf{A}^{\top}\mathbf{A}\), an \(n\times n\) symmetric matrix. Its eigenvalues are nonnegative, and their square roots are singular values. If \(\mathbf{v}_i\) is a unit eigenvector with positive eigenvalue \(\sigma_i^2\), then \(\mathbf{u}_i=\mathbf{A}\mathbf{v}_i/\sigma_i\) is the corresponding left singular vector. The following result explains why this construction works.
Proof. First, every eigenvalue is nonnegative: if \(\mathbf{A}^{\top}\mathbf{A}\mathbf{v}_i=\lambda_i\mathbf{v}_i\) and \(\|\mathbf{v}_i\|=1\), then \(\lambda_i=\mathbf{v}_i^{\top}\mathbf{A}^{\top}\mathbf{A}\mathbf{v}_i=\|\mathbf{A}\mathbf{v}_i\|^2\ge0\). For positive eigenvalues, compute \(\mathbf{u}_i^{\top}\mathbf{u}_j=\mathbf{v}_i^{\top}\mathbf{A}^{\top}\mathbf{A}\mathbf{v}_j/(\sigma_i\sigma_j) =\lambda_j\mathbf{v}_i^{\top}\mathbf{v}_j/(\sigma_i\sigma_j)\). This equals \(1\) when \(i=j\) and \(0\) when \(i\ne j\), so the constructed left vectors are orthonormal. If \(\lambda_i=0\), then \(\|\mathbf{A}\mathbf{v}_i\|^2=0\), hence \(\mathbf{A}\mathbf{v}_i=\mathbf{0}\). Every vector \(\mathbf{x}\in\mathbb{R}^n\) has an expansion \(\mathbf{x}=\sum_{i=1}^n c_i\mathbf{v}_i\), so \(\mathbf{A}\mathbf{x}=\sum_{\lambda_i>0}c_i\sigma_i\mathbf{u}_i\). This is exactly the action represented by the stated compact factorization. Completing the left vectors to an orthonormal basis of \(\mathbb{R}^m\) supplies the full SVD. If there are no positive eigenvalues, the same calculation gives \(\mathbf{A}\mathbf{v}_i=\mathbf{0}\) for every basis vector, so \(\mathbf{A}=\mathbf{O}\), covered by the zero-SVD case. \(\square\)
This construction uses the standard Spectral Theorem for the real symmetric matrix \(\mathbf{A}^{\top}\mathbf{A}\). In an exact problem, the central arithmetic checks are the eigenvector equations, the normalization of each \(\mathbf{u}_i\), the dimensions of the factors, and the product \(\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\).
From Entries to a Square SVD
Worked Example: A Symmetric Matrix with a Sign Change
Find an SVD of \(\mathbf{A}=\begin{bmatrix}1&2\\2&1\end{bmatrix}\), then determine its rank, operator norm, and condition number.
Calculate the matrix that controls the right singular vectors:
The vectors \(\mathbf{v}_1=(1,1)^{\top}/\sqrt{2}\) and \(\mathbf{v}_2=(1,-1)^{\top}/\sqrt{2}\) are orthonormal. Direct multiplication gives \(\mathbf{A}^{\top}\mathbf{A}\mathbf{v}_1=9\mathbf{v}_1\) and \(\mathbf{A}^{\top}\mathbf{A}\mathbf{v}_2=\mathbf{v}_2\), so the singular values are \(3\) and \(1\). Now apply \(\mathbf{A}\): \(\mathbf{A}\mathbf{v}_1=3\mathbf{v}_1\), while \(\mathbf{A}\mathbf{v}_2=-\mathbf{v}_2\). The second output has the opposite direction, so its left singular vector is \(\mathbf{u}_2=-\mathbf{v}_2\), not \(\mathbf{v}_2\). Thus
Both \(\mathbf{U}\) and \(\mathbf{V}\) have orthonormal columns. To check the factorization, multiplying \(\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\) gives entries \((3-1)/2=1\), \((3+1)/2=2\), \((3+1)/2=2\), and \((3-1)/2=1\), in row order. This recovers \(\mathbf{A}\). Both singular values are positive, so \(\operatorname{rank}\mathbf{A}=2\). Its operator norm is \(3\), and its condition number is \(3/1=3\).
The sign in this example is important: singular values are nonnegative, even when \(\mathbf{A}\mathbf{v}_i\) points opposite to \(\mathbf{v}_i\). The sign is carried by the corresponding left singular vector. More generally, the left and right singular vectors need not be equal, even when the matrix is square.
Reading Geometry and Dimensions from an SVD
Proof. Write any \(\mathbf{x}\in\mathbb{R}^n\) in a full orthonormal right-singular-vector basis as \(\mathbf{x}=\sum_{i=1}^n c_i\mathbf{v}_i\). The SVD gives \(\mathbf{A}\mathbf{x}=\sum_{i=1}^r \sigma_i c_i\mathbf{u}_i\). Therefore every output lies in the span of the first \(r\) left singular vectors, and each such vector is attained because \(\mathbf{A}\mathbf{v}_i=\sigma_i\mathbf{u}_i\) with \(\sigma_i>0\). This proves the image statement and shows that the image has dimension \(r\), hence the rank is \(r\). The output is zero exactly when every \(c_i=0\) for \(1\le i\le r\), since the \(\mathbf{u}_i\) are independent and \(\sigma_i>0\). The remaining coefficients are unrestricted, proving the null-space statement. Finally, orthonormality gives \(\|\mathbf{x}\|^2=\sum_{i=1}^n c_i^2\) and \(\|\mathbf{A}\mathbf{x}\|^2=\sum_{i=1}^r\sigma_i^2c_i^2\le\sigma_1^2\sum_{i=1}^n c_i^2\). For a unit vector \(\mathbf{x}=\mathbf{v}_1\), equality holds, so the maximum is \(\sigma_1\). \(\square\)
This result turns a factorization into geometric information. Right singular vectors identify orthogonal input directions; positive singular values tell how much each direction is stretched; and the corresponding left singular vectors give the output directions. Zero singular values identify some directions sent to zero; for a wide matrix, additional right-null-space directions can occur even when all \(\min(m,n)\) singular values are positive.
Worked Example: A Tall Matrix and an Orthonormal Completion
Consider the \(3\times2\) matrix whose columns are \(3\mathbf{u}_1\) and \(2\mathbf{u}_2\), where \(\mathbf{u}_1=(1,-1,2)^{\top}/\sqrt{6}\) and \(\mathbf{u}_2=(1,1,0)^{\top}/\sqrt{2}\). In entries,
Check the left vectors before forming the decomposition. Their squared lengths are \((1+1+4)/6=1\) and \((1+1)/2=1\); their dot product is \((1-1+0)/\sqrt{12}=0\). A third unit vector orthogonal to both is \(\mathbf{u}_3=(-1,1,1)^{\top}/\sqrt{3}\). Indeed, the two dot-product numerators are \(-1-1+2=0\) and \(-1+1+0=0\), and its squared length is \(3/3=1\). Therefore the full left matrix is orthogonal. With \(\mathbf{V}=\mathbf{I}_2\), a full SVD is
The displayed first matrix is intentionally not needed for the factorization; the second matrix has columns \(\mathbf{u}_1,\mathbf{u}_2,\mathbf{u}_3\) and is the required \(\mathbf{U}\). Its columns have just been checked to be orthonormal. Multiplying \(\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\) returns columns \(3\mathbf{u}_1\) and \(2\mathbf{u}_2\), which are exactly the two columns of \(\mathbf{A}\). The singular values are \(3,2\), the rank is \(2\), and \(\operatorname{Nul}\mathbf{A}=\{\mathbf{0}\}\subseteq\mathbb{R}^2\). Its image is \(\operatorname{span}\{\mathbf{u}_1,\mathbf{u}_2\}\subseteq\mathbb{R}^3\). The operator norm is \(3\); since the matrix has full column rank, its condition number is \(3/2\).
A Wide, Rank-Deficient Example
Worked Example: Rank, Null Space, Image, and Norm
For \(\mathbf{A}=\begin{bmatrix}1&1&0\\2&2&0\end{bmatrix}\), find a full SVD and read off its rank, null space, image, operator norm, and condition number.
The rows are dependent, but the SVD gives a systematic account of the input and output directions. Choose \(\mathbf{u}_1=(1,2)^{\top}/\sqrt{5}\), \(\mathbf{u}_2=(-2,1)^{\top}/\sqrt{5}\), \(\mathbf{v}_1=(1,1,0)^{\top}/\sqrt{2}\), \(\mathbf{v}_2=(1,-1,0)^{\top}/\sqrt{2}\), and \(\mathbf{v}_3=(0,0,1)^{\top}\). The two left vectors are orthogonal unit vectors, as their dot product is \((-2+2)/5=0\) and both squared lengths equal \(1\). The three right vectors are also orthonormal: the first two have dot product \((1-1)/2=0\), each has squared length \(1\), and both are perpendicular to \(\mathbf{v}_3\).
Direct multiplication verifies the singular-vector equations:
Thus a full SVD is \(\mathbf{A}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\), with \(\mathbf{U}=[\mathbf{u}_1\ \mathbf{u}_2]\in\mathbb{R}^{2\times2}\), \(\mathbf{V}=[\mathbf{v}_1\ \mathbf{v}_2\ \mathbf{v}_3]\in\mathbb{R}^{3\times3}\), and
To check the product, the only nonzero rank-one term is \(\sqrt{10}\mathbf{u}_1\mathbf{v}_1^{\top}\). Its scalar coefficient on the outer product is \(\sqrt{10}/\sqrt{5}\sqrt{2}=1\), so it equals \(\begin{bmatrix}1&1&0\\2&2&0\end{bmatrix}\), as required. There is one positive singular value, hence \(\operatorname{rank}\mathbf{A}=1\). The null space is \(\operatorname{span}\{\mathbf{v}_2,\mathbf{v}_3\}\), equivalently the vectors with \(x_1+x_2=0\), with \(x_3\) unrestricted. The image is \(\operatorname{span}\{\mathbf{u}_1\}=\operatorname{span}\{(1,2)^{\top}\}\subseteq\mathbb{R}^2\). The operator norm is \(\sqrt{10}\). Because a singular value is zero, the matrix is rank-deficient and its usual condition number is infinite.
Common Mistakes and Final Checks
Do not confuse eigenvalues of \(\mathbf{A}^{\top}\mathbf{A}\) with singular values. In the first example, the eigenvalues are \(9\) and \(1\), while the singular values are their nonnegative square roots, \(3\) and \(1\). Calling \(9\) a singular value would give the wrong scale and the wrong operator norm.
Keep the dimensions visible. For an \(m\times n\) matrix, \(\mathbf{A}^{\top}\mathbf{A}\) is \(n\times n\), a right singular vector lies in \(\mathbb{R}^n\), and a left singular vector lies in \(\mathbb{R}^m\). The rectangular \(\boldsymbol{\Sigma}\) must be \(m\times n\). In the wide example, the nonzero right singular vectors describe the row-space directions, while the remaining right singular vectors span the null space.
Do not infer full rank from the number of displayed rows or columns. In the rank-deficient example, \(\mathbf{A}\) has two rows but only one positive singular value. Rank counts positive singular values, not the number of rows, columns, or vectors included in a full SVD. When a matrix is rank-deficient, the zero singular values matter: they identify lost input directions and make the usual condition number infinite.
A final audit of an SVD solution should answer these questions: do the factors have the correct dimensions; are the columns of \(\mathbf{U}\) and \(\mathbf{V}\) orthonormal; are the singular values nonnegative and ordered; do the singular-vector equations hold; and does multiplying the factors recover the original matrix? Once those checks pass, rank, null space, image, and norm follow from the same singular-vector information rather than from separate guesses.
Check Your Understanding
Use exact arithmetic, state the dimensions of the factors, and verify each proposed conclusion from the singular-vector equations.
- For \(\begin{bmatrix}2&0\\0&-1\end{bmatrix}\), find a full SVD and explain where the negative sign appears.
- For a \(4\times2\) matrix with singular values \(5\) and \(0\), state its rank and the dimension of its null space.
- Explain why \(\mathbf{A}\mathbf{v}=\mathbf{0}\) for a unit vector \(\mathbf{v}\) implies that \(0\) is an eigenvalue of \(\mathbf{A}^{\top}\mathbf{A}\). When does it also give a zero singular value in the list of \(\min(m,n)\) singular values?
- If a full-column-rank matrix has singular values \(6,3,2\), find its operator norm and condition number.
- For the wide matrix in the last worked example, verify directly that \(\mathbf{A}(1,-1,0)^{\top}=\mathbf{0}\).
