Why Wide Matrices Need a Different SVD Shape
A wide matrix has more columns than rows. If \(\mathbf{A}\in\mathbb{R}^{m\times n}\) with \(m<n\), then its right singular vectors lie in the larger space \(\mathbb{R}^n\), while its left singular vectors lie in \(\mathbb{R}^m\). For a full-row-rank matrix, there are \(m\) positive singular values. The remaining \(n-m\) right singular vectors correspond to zero eigenvalues of \(\mathbf{A}^{\top}\mathbf{A}\) and are needed to complete a full orthonormal basis of \(\mathbb{R}^n\).
Here, full row rank means \(\operatorname{rank}\mathbf{A}=m\). The Gram matrix \(\mathbf{A}^{\top}\mathbf{A}\) is \(n\times n\), so it provides an orthonormal basis of right singular vectors. Each positive-eigenvalue vector produces a left singular vector by applying \(\mathbf{A}\) and dividing by the associated singular value.
The distinction is important: in this full-row-rank setting, the \(m\) left singular vectors obtained from the positive eigenvalues already form an orthonormal basis of \(\mathbb{R}^m\). No extra left vectors are needed. To form the full SVD, complete the right singular vectors by including an orthonormal basis for the zero-eigenvalue eigenspace.
Why the Gram Matrix Has Exactly \(m\) Positive Eigenvalues
We use the Spectral Theorem for real symmetric matrices: a real symmetric matrix has an orthonormal basis of eigenvectors. We also use the Rank-Nullity Theorem. Both apply here because \(\mathbf{A}^{\top}\mathbf{A}\) is real symmetric and \(\mathbf{A}\) is a linear map from the finite-dimensional space \(\mathbb{R}^n\).
Proof. For every \(\mathbf{x}\in\mathbb{R}^n\),
This quantity is zero exactly when \(\mathbf{A}\mathbf{x}=\mathbf{0}\). Thus \(\operatorname{Nul}(\mathbf{A}^{\top}\mathbf{A})=\operatorname{Nul}\mathbf{A}\): one inclusion follows because \(\mathbf{A}\mathbf{x}=\mathbf{0}\) implies \(\mathbf{A}^{\top}\mathbf{A}\mathbf{x}=\mathbf{0}\); for the other, if \(\mathbf{A}^{\top}\mathbf{A}\mathbf{x}=\mathbf{0}\), the displayed identity gives \(\|\mathbf{A}\mathbf{x}\|^2=0\), hence \(\mathbf{A}\mathbf{x}=\mathbf{0}\). Since \(\operatorname{rank}\mathbf{A}=m\), rank-nullity gives \(\dim(\operatorname{Nul}\mathbf{A})=n-m\). Therefore the zero-eigenvalue eigenspace of \(\mathbf{A}^{\top}\mathbf{A}\) has dimension \(n-m\). Also, if \(\mathbf{x}\ne\mathbf{0}\) is an eigenvector with eigenvalue \(\lambda\), then \(\lambda\|\mathbf{x}\|^2=\|\mathbf{A}\mathbf{x}\|^2\ge0\), so \(\lambda\ge0\). By the Spectral Theorem, the \(n\) eigenvectors in an orthonormal eigenbasis account for all \(n\) eigenvalues with multiplicity. Exactly \(n-m\) are zero, and the remaining \(m\) are positive. \(\square\)
From Eigenvectors to Thin and Full SVDs
Order an orthonormal eigenbasis of \(\mathbf{A}^{\top}\mathbf{A}\) so that the positive eigenvalues come first. Write the eigenpairs as \((\lambda_i,\mathbf{v}_i)\), with \(\lambda_1\ge\cdots\ge\lambda_m>0\), followed by \(\lambda_{m+1}=\cdots=\lambda_n=0\). Define \(\sigma_i=\sqrt{\lambda_i}\) for \(1\le i\le m\). The vectors \(\mathbf{v}_i\) are the right singular vectors.
Find an orthonormal eigenbasis of \(\mathbf{A}^{\top}\mathbf{A}\), placing the \(m\) positive eigenvalues first.
For each positive eigenvalue \(\lambda_i\), set \(\sigma_i=\sqrt{\lambda_i}\).
Set \(\mathbf{u}_i=\mathbf{A}\mathbf{v}_i/\sigma_i\) for \(1\le i\le m\).
Use the first \(m\) right vectors for the thin SVD; include the remaining zero-eigenvalue vectors for the full SVD.
Proof. First, for \(1\le i,j\le m\), use \(\mathbf{A}^{\top}\mathbf{A}\mathbf{v}_j=\lambda_j\mathbf{v}_j\) and orthonormality of the \(\mathbf{v}_i\):
Thus the \(m\) vectors \(\mathbf{u}_1,\dots,\mathbf{u}_m\) are orthonormal in \(\mathbb{R}^m\), so they form an orthonormal basis of that space. For \(i\le m\), the definition gives \(\mathbf{A}\mathbf{v}_i=\sigma_i\mathbf{u}_i\). For \(i>m\), the eigenvalue is zero, and \(\|\mathbf{A}\mathbf{v}_i\|^2=\mathbf{v}_i^{\top}\mathbf{A}^{\top}\mathbf{A}\mathbf{v}_i=0\), so \(\mathbf{A}\mathbf{v}_i=\mathbf{0}\).
Let \(\mathbf{x}\in\mathbb{R}^n\). Expand it in the orthonormal basis \(\mathbf{v}_1,\dots,\mathbf{v}_n\), apply \(\mathbf{A}\), and use these image formulas:
Hence \(\mathbf{A}=\sum_{i=1}^{m}\sigma_i\mathbf{u}_i\mathbf{v}_i^{\top}\), which is exactly the thin factorization \(\mathbf{A}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\) with the stated thin shapes. Adding \(\mathbf{v}_{m+1},\dots,\mathbf{v}_n\) to \(\mathbf{V}\) and placing \(n-m\) zero columns after the \(m\) singular-value columns in the \(m\times n\) matrix \(\boldsymbol{\Sigma}\) does not change the sum. The resulting \(\mathbf{V}\) is orthogonal by construction, giving the full factorization as well. \(\square\)
Worked Calculations
Worked Example: A Simple \(2\times3\) Full-Row-Rank Matrix
Find the thin and full SVDs of \(\mathbf{A}=\begin{bmatrix}2&0&0\\0&1&0\end{bmatrix}\).
Compute the Gram matrix:
The eigenvalues in decreasing order are \(4,1,0\), with orthonormal eigenvectors \(\mathbf{v}_1=\mathbf{e}_1\), \(\mathbf{v}_2=\mathbf{e}_2\), and \(\mathbf{v}_3=\mathbf{e}_3\). Therefore \(\sigma_1=2\), \(\sigma_2=1\), and \(\mathbf{u}_1=\mathbf{A}\mathbf{e}_1/2=(1,0)^{\top}\), \(\mathbf{u}_2=\mathbf{A}\mathbf{e}_2=(0,1)^{\top}\). Thus \(\mathbf{U}=\mathbf{I}_2\).
The thin factors and their product are
For the full factorization, include \(\mathbf{v}_3\) as the third column of \(\mathbf{V}=\mathbf{I}_3\), and use \(\boldsymbol{\Sigma}=\begin{bmatrix}2&0&0\\0&1&0\end{bmatrix}\). Then \(\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}=\mathbf{A}\mathbf{I}_3=\mathbf{A}\), verifying the full factorization too.
Worked Example: A Mixed \(2\times3\) Matrix
Find both SVD shapes for \(\mathbf{A}=\begin{bmatrix}1&0&1\\0&1&1\end{bmatrix}\).
First,
Direct multiplication gives eigenvectors \((1,1,2)^{\top}\), \((1,-1,0)^{\top}\), and \((1,1,-1)^{\top}\), with eigenvalues \(3,1,0\), respectively. For example, the Gram matrix sends these vectors to \((3,3,6)^{\top}\), \((1,-1,0)^{\top}\), and \((0,0,0)^{\top}\). Their normalized versions are mutually orthogonal, so take
The singular values are \(\sqrt{3}\) and \(1\). Applying \(\mathbf{A}\) gives \(\mathbf{A}\mathbf{v}_1=(3,3)^{\top}/\sqrt{6}\), hence \(\mathbf{u}_1=(1,1)^{\top}/\sqrt{2}\); also \(\mathbf{A}\mathbf{v}_2=(1,-1)^{\top}/\sqrt{2}\), hence \(\mathbf{u}_2=(1,-1)^{\top}/\sqrt{2}\). The thin product is
For instance, the first row of the product is \(\tfrac12(1,1,2)+\tfrac12(1,-1,0)=(1,0,1)\), and the second is \(\tfrac12(1,1,2)-\tfrac12(1,-1,0)=(0,1,1)\). For the full SVD, append \(\mathbf{v}_3\) as the final column of \(\mathbf{V}\) and append a zero column to the thin diagonal matrix. Since \(\mathbf{A}\mathbf{v}_3=\mathbf{0}\), this extra column contributes zero, so the full product is the same \(\mathbf{A}\).
Worked Example: A \(3\times4\) Matrix with Three Positive Singular Values
Let \(\mathbf{A}=\begin{bmatrix}1&1&0&0\\1&-1&1&1\\0&0&2&-2\end{bmatrix}\). Its three rows are nonzero and mutually orthogonal, so they are independent and \(\mathbf{A}\) has full row rank. Compute \(\mathbf{A}^{\top}\mathbf{A}\):
Multiplication shows that \((0,0,1,-1)^{\top}\), \((1,-1,1,1)^{\top}\), \((1,1,0,0)^{\top}\), and \((1,-1,-1,-1)^{\top}\) have eigenvalues \(8,4,2,0\), respectively. Their squared lengths are \(2,4,2,4\), and their pairwise dot products are zero. The normalized eigenvectors are therefore
The singular values are \(2\sqrt{2},2,\sqrt{2}\). Applying \(\mathbf{A}\) to the first three normalized vectors gives \(\mathbf{A}\mathbf{v}_1=2\sqrt{2}\mathbf{e}_3\), \(\mathbf{A}\mathbf{v}_2=2\mathbf{e}_2\), and \(\mathbf{A}\mathbf{v}_3=\sqrt{2}\mathbf{e}_1\), where the \(\mathbf{e}_i\) here are standard basis vectors in \(\mathbb{R}^3\). Hence \(\mathbf{U}=[\mathbf{e}_3\ \mathbf{e}_2\ \mathbf{e}_1]\). The thin product checks directly:
The first product row is \(\sqrt{2}\mathbf{v}_3^{\top}=(1,1,0,0)\); the second is \(2\mathbf{v}_2^{\top}=(1,-1,1,1)\); the third is \(2\sqrt{2}\mathbf{v}_1^{\top}=(0,0,2,-2)\). This verifies every row. For the full factorization, append \(\mathbf{v}_4\) to the right-vector matrix and append a zero column to the \(3\times3\) thin diagonal matrix. The added term is zero because its singular value is zero, so the full product is again \(\mathbf{A}\).
A Common Mistake: Dividing by a Zero Singular Value
A tempting but invalid step is to apply \(\mathbf{u}=\mathbf{A}\mathbf{v}/\sigma\) to a zero-eigenvalue vector. In the \(2\times3\) example above, \(\mathbf{v}_3=(1,1,-1)^{\top}/\sqrt{3}\) satisfies \(\mathbf{A}\mathbf{v}_3=\mathbf{0}\) and has singular value \(0\). The calculation \(\mathbf{u}_3=\mathbf{0}/0\) is undefined; it does not produce a left singular vector. The correction is to construct left vectors only from positive singular values. The zero-eigenvalue vectors complete the right basis, and their images under \(\mathbf{A}\) are zero.
Another shape check prevents a related error: for an \(m\times n\) wide matrix, the full left factor is \(m\times m\), not \(n\times n\). The right factor is the one that must have \(n\) orthonormal columns. In the full-row-rank case, the \(m\) constructed left singular vectors already fill \(\mathbb{R}^m\); the \(n-m\) additional vectors belong on the right.
Check Your Understanding
Use the Gram matrix and check the shapes as well as the numerical factors.
- For a full-row-rank \(2\times5\) real matrix, how many positive and zero eigenvalues does \(\mathbf{A}^{\top}\mathbf{A}\) have, counted with multiplicity?
- For \(\mathbf{A}=\begin{bmatrix}1&0&1\\0&1&1\end{bmatrix}\), verify directly that \((1,1,-1)^{\top}\) is a zero-eigenvalue vector of \(\mathbf{A}^{\top}\mathbf{A}\).
- Explain why the \(m\) left singular vectors constructed from positive eigenvalues form a basis of \(\mathbb{R}^m\) in the full-row-rank case.
- State the dimensions of \(\mathbf{U}\), \(\boldsymbol{\Sigma}\), and \(\mathbf{V}\) in the thin SVD of an \(m\times n\) full-row-rank matrix with \(m<n\).
- For \(\mathbf{A}=\begin{bmatrix}2&0&0\\0&1&0\end{bmatrix}\), write its full \(\mathbf{V}\) and full \(\boldsymbol{\Sigma}\), and verify that their product with \(\mathbf{U}\) reconstructs \(\mathbf{A}\).
