Full, Thin, And Compact SVD Shapes

Why One SVD Has Several Shapes

The singular value decomposition of a matrix can be written with different numbers of columns in its orthogonal factors. These forms encode the same matrix, but they do not retain the same information about directions associated with zero singular values. To choose or interpret an SVD correctly, track three quantities: the row count \(m\), the column count \(n\), and the rank \(r\).

Throughout, let \(\mathbf{A}\in\mathbb{R}^{m\times n}\), with \(m,n\geq 1\), and put \(k=\min(m,n)\). An orthogonal matrix is a square real matrix \(\mathbf{Q}\) satisfying \(\mathbf{Q}^{\top}\mathbf{Q}=\mathbf{I}\). We use the real singular value decomposition established earlier in the course: there are orthogonal matrices \(\mathbf{U}\in\mathbb{R}^{m\times m}\) and \(\mathbf{V}\in\mathbb{R}^{n\times n}\) such that \(\mathbf{A}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\). The diagonal entries of the rectangular matrix \(\boldsymbol{\Sigma}\in\mathbb{R}^{m\times n}\) are the singular values \(\sigma_1\geq\cdots\geq\sigma_k\geq0\), with other entries zero.

Let \(r=\operatorname{rank}\mathbf{A}\), where rank is the dimension of the column space. The shape conventions below specify precisely how many columns of the orthogonal factors are kept. Terminology can vary across texts, so the dimensions—not just the word “thin”—are the reliable guide.

The Three Shape Conventions

For the full SVD, \(\boldsymbol{\Sigma}\) is rectangular and has \(\sigma_i\) in position \((i,i)\), for \(1\leq i\leq k\). In the thin and compact forms, the middle factor is square and diagonal: \(\mathbf{D}_k=\operatorname{diag}(\sigma_1,\ldots,\sigma_k)\) and \(\mathbf{D}_r=\operatorname{diag}(\sigma_1,\ldots,\sigma_r)\). The columns of \(\mathbf{U}_k,\mathbf{V}_k\) are the first \(k\) columns of the full factors; those of \(\mathbf{U}_r,\mathbf{V}_r\) are the first \(r\).

The dimensions make each product valid: for example, \((m\times k)(k\times k)(k\times n)\) gives an \(m\times n\) matrix. If \(r<k\), the compact form is smaller than the thin form because it omits the zero singular-value directions. If \(r=k\), thin and compact have identical dimensions. If \(r=0\), the compact factors have zero columns and \(\mathbf{D}_r\) is \(0\times0\); their product is the \(m\times n\) zero matrix.

Why Truncating the Full SVD Works

Proof. Write the columns of the full factors as \(\mathbf{u}_1,\ldots,\mathbf{u}_m\) and \(\mathbf{v}_1,\ldots,\mathbf{v}_n\). From the diagonal structure of \(\boldsymbol{\Sigma}\), multiplication gives \(\mathbf{A}=\sum_{i=1}^{k}\sigma_i\mathbf{u}_i\mathbf{v}_i^{\top}\). Let \(s\) be the number of positive singular values. Since the values are ordered, \(\sigma_i>0\) for \(1\leq i\leq s\), and \(\sigma_i=0\) for \(s<i\leq k\). For any \(\mathbf{x}\in\mathbb{R}^n\), this sum shows that \(\mathbf{A}\mathbf{x}=\sum_{i=1}^{s}\sigma_i(\mathbf{v}_i^{\top}\mathbf{x})\mathbf{u}_i\), so the column space is contained in \(\operatorname{span}\{\mathbf{u}_1,\ldots,\mathbf{u}_s\}\). Also \(\mathbf{A}\mathbf{v}_j=\sigma_j\mathbf{u}_j\) for \(1\leq j\leq s\), because the \(\mathbf{v}_i\) are orthonormal. Dividing by the nonzero \(\sigma_j\) puts each \(\mathbf{u}_j\) in the column space. Thus the column space equals that span. Its \(s\) vectors are orthonormal, hence linearly independent, so the rank is \(s\). The displayed sum is unchanged if it is written using the first \(k\) terms, including zero terms, or only the first \(s=r\) terms. Those two groupings are exactly the thin and compact products, respectively. \(\square\)

This argument also explains the roles of the columns. The thin SVD retains a complete list of \(k\) singular-vector pairs, even when some pairs are multiplied by zero. The compact SVD retains only pairs that contribute nonzero terms to the reconstruction. A zero singular value does not contribute to \(\mathbf{A}\), but its corresponding vectors can still be part of the full orthogonal bases.

Worked Shapes for a Wide Rank-Deficient Matrix

When Thin and Compact Have the Same Size

A Rank-Deficient Tall Example

Common Mistake: Moving a Singular Value Off the Diagonal

A singular-value matrix is diagonal-shaped: its only possibly nonzero entries are in positions \((i,i)\). It is not enough for a proposed middle matrix to multiply with two orthogonal matrices and reconstruct \(\mathbf{A}\); its entries must also be the ordered singular values in diagonal positions.

For the first example, a tempting but incorrect choice is \(\mathbf{U}=\mathbf{I}_2\), \(\mathbf{V}=\mathbf{I}_3\), and \(\boldsymbol{\Sigma}=\begin{bmatrix}0&3&0\\0&0&0\end{bmatrix}\). The product does equal \(\mathbf{A}\), but the diagonal entries are \(0,0\), while the \(3\) is off the diagonal. Therefore this is not an SVD. The corrected full factors are \(\mathbf{U}=\mathbf{I}_2\), \(\mathbf{V}=\begin{bmatrix}0&1&0\\1&0&0\\0&0&1\end{bmatrix}\), and \(\boldsymbol{\Sigma}=\begin{bmatrix}3&0&0\\0&0&0\end{bmatrix}\). The value \(3\) is now on the diagonal, and the reordered right singular vectors make the product reconstruct \(\mathbf{A}\).

Another useful check is to write the dimensions beside every factor before multiplying. A rectangular full \(\boldsymbol{\Sigma}\) is \(m\times n\); the middle matrix in the thin or compact convention is square. These choices are not interchangeable without also changing the dimensions and columns of the other factors.

Choosing a Form

The full SVD is useful when a complete orthogonal basis on both sides is needed. The thin SVD removes columns that are unnecessary solely because one dimension of the matrix is larger, but it still keeps zero singular-value directions among the first \(k\). The compact SVD is the smallest of the three: it retains exactly the directions that contribute to the matrix.

Before using a displayed factorization, check that the singular values are ordered, that the middle factor has the required diagonal shape, and that adjacent dimensions match. Then multiply the factors, or use the rank-one expansion \(\mathbf{A}=\sum_{i=1}^{r}\sigma_i\mathbf{u}_i\mathbf{v}_i^{\top}\) to check which terms actually contribute. This distinguishes a valid SVD from a product that happens to equal the matrix but has incorrectly arranged factors.