Constructing the Truncated SVD

Keeping the Leading Singular Components

A singular value decomposition expresses a matrix as a sum of simple, structured pieces. The truncated SVD keeps only the pieces corresponding to the largest singular values. This gives a specific rank-limited matrix and a residual whose two common measures of size can be read directly from the discarded singular values. Here the goal is to construct that matrix and calculate its errors; deciding whether another rank-\(k\) matrix could do better is a separate question.

Work over \(\mathbb{R}\). Let \(\mathbf{A}\in\mathbb{R}^{m\times n}\) have rank \(r\). A compact singular value decomposition writes \(\mathbf{A}\) as

Here \(\mathbf{u}_1,\dots,\mathbf{u}_r\) are orthonormal columns in \(\mathbb{R}^m\), \(\mathbf{v}_1,\dots,\mathbf{v}_r\) are orthonormal columns in \(\mathbb{R}^n\), and \(\sigma_i\) are the positive singular values. Each summand \(\sigma_i\mathbf{u}_i\mathbf{v}_i^{\top}\) is one singular component of \(\mathbf{A}\). The integer \(k\) will satisfy \(0\leq k\leq r\).

The residual is the part omitted from the truncation, \(\mathbf{E}_k=\mathbf{A}-\mathbf{A}_k\). For convenience, define \(\sigma_{r+1}=0\). This convention lets formulas for the largest discarded singular value also describe the case \(k=r\), when no positive singular values are discarded.

The Frobenius norm of a matrix \(\mathbf{B}=(b_{ij})\) is \(\|\mathbf{B}\|_F=\left(\sum_{i,j}b_{ij}^2\right)^{1/2}\). Its operator norm is \(\|\mathbf{B}\|_2=\max_{\|\mathbf{x}\|=1}\|\mathbf{B}\mathbf{x}\|\), where \(\|\mathbf{x}\|\) is the Euclidean norm. The Frobenius norm measures the combined size of all entries; the operator norm measures the largest amount by which the matrix stretches a unit vector.

Constructing and Checking a Truncation

A practical construction has three steps: order the singular values, retain the first \(k\) singular components, and subtract the retained matrix from the original to obtain the residual. Keeping whole components matters: the truncation is not made by deleting selected entries of \(\mathbf{A}\).

Rank and Error Formulas

Proof. The columns of \(\mathbf{A}_k\) lie in \(\operatorname{span}\{\mathbf{u}_1,\dots,\mathbf{u}_k\}\), because each summand has its columns proportional to one of those vectors. For \(1\leq j\leq k\), orthonormality of the \(\mathbf{v}_i\) gives \(\mathbf{A}_k\mathbf{v}_j=\sigma_j\mathbf{u}_j\). Since \(\sigma_j>0\) and the \(\mathbf{u}_j\) are orthonormal, these \(k\) images are linearly independent. Therefore the column space has dimension \(k\), so \(\operatorname{rank}\mathbf{A}_k=k\). If \(k=0\), \(\mathbf{A}_0=\mathbf{O}\) has rank \(0\), giving the same conclusion.

Write the residual as \(\mathbf{E}_k=\sum_{i=k+1}^r\sigma_i\mathbf{u}_i\mathbf{v}_i^{\top}\). In the Frobenius inner product, the inner product of two matrices \(\mathbf{B},\mathbf{C}\) of the same size is the sum of the products of corresponding entries. Direct expansion gives \(\langle\mathbf{u}_i\mathbf{v}_i^{\top},\mathbf{u}_j\mathbf{v}_j^{\top}\rangle_F=(\mathbf{u}_i^{\top}\mathbf{u}_j)(\mathbf{v}_i^{\top}\mathbf{v}_j)\). For \(i\ne j\), this is zero; for \(i=j\), it is \(1\). Consequently, the squared Frobenius norm of the residual is the sum of the squares of its coefficients, \(\|\mathbf{E}_k\|_F^2=\sum_{i=k+1}^r\sigma_i^2\), proving the stated formula, including the empty sum when \(k=r\).

For a unit vector \(\mathbf{x}\in\mathbb{R}^n\), \(\mathbf{E}_k\mathbf{x}=\sum_{i=k+1}^r\sigma_i(\mathbf{v}_i^{\top}\mathbf{x})\mathbf{u}_i\). Orthonormality of the \(\mathbf{u}_i\) gives \(\|\mathbf{E}_k\mathbf{x}\|^2=\sum_{i=k+1}^r\sigma_i^2(\mathbf{v}_i^{\top}\mathbf{x})^2\). Since \(\sigma_i\leq\sigma_{k+1}\), this is at most \(\sigma_{k+1}^2\sum_{i=k+1}^r(\mathbf{v}_i^{\top}\mathbf{x})^2\leq\sigma_{k+1}^2\), where the last inequality follows because the \(\mathbf{v}_i\) are orthonormal and \(\|\mathbf{x}\|=1\). If \(k<r\), taking \(\mathbf{x}=\mathbf{v}_{k+1}\) gives \(\|\mathbf{E}_k\mathbf{x}\|=\sigma_{k+1}\), so the upper bound is attained. If \(k=r\), the residual is zero and \(\sigma_{r+1}=0\). This proves the operator-norm formula in every case. \(\square\)

Ties and the Choice of Singular Vectors

If several singular values are equal, there may be more than one valid choice of singular vectors within the corresponding singular subspace. At a tie where the cutoff falls, different choices of \(k\) vectors can therefore produce different matrices \(\mathbf{A}_k\). The error formulas do not change: the discarded values, counted with multiplicity, still determine the two errors.

Common Pitfall: Adding Discarded Values

A common mistake is to add the discarded singular values and call that the Frobenius error. For the first example, this would give \(3+1=4\), but the Frobenius norm squares the entries before adding, so the correct result is \(\sqrt{3^2+1^2}=\sqrt{10}\). A related distinction is that the operator-norm error is not this square root; it is the largest discarded singular value, \(3\) in that example.

These formulas describe the error of the particular truncation built from the leading singular components. They do not, by themselves, compare it with every other matrix of rank at most \(k\). That optimality question requires a separate argument. For constructing and checking the truncated SVD, keep the two calculations distinct: combine discarded squares for the Frobenius norm, and select the largest discarded singular value for the operator norm.