Low-Rank Reconstruction in Principal Component Analysis

From Centered Observations to Principal Components

Principal component analysis (PCA) replaces a set of centered observations with coordinates along a selected collection of directions in feature space. Keeping the directions with the greatest variation gives a low-dimensional representation; projecting back onto those directions gives an approximation to the original observations. The singular value decomposition (SVD) makes both the representation and the information lost in reconstruction precise.

Let \(\mathbf{X}\in\mathbb{R}^{m\times n}\) be a centered data matrix: its \(m\) rows represent observations, its \(n\) columns represent features, and each column has mean zero across the observations. We assume \(m\ge2\), so the sample covariance matrix is defined. Write the transpose of row \(i\) as the column vector \(\mathbf{x}_i\in\mathbb{R}^n\). Thus a direction in feature space is a column vector in \(\mathbb{R}^n\).

Use an SVD with singular values in descending order. Set \(s=\min(m,n)\). The singular-vector expansion is

Here \(\mathbf{u}_1,\dots,\mathbf{u}_s\) are orthonormal vectors in \(\mathbb{R}^m\), and \(\mathbf{v}_1,\dots,\mathbf{v}_s\) are orthonormal vectors in \(\mathbb{R}^n\). Complete the right singular vectors to an orthonormal basis \(\mathbf{v}_1,\dots,\mathbf{v}_n\) of \(\mathbb{R}^n\). The added right singular directions, if any, lie in \(\operatorname{Nul}\mathbf{X}\). The expansion stops at \(s\): it does not require left singular vectors with indices greater than \(s\).

For an individual observation, the score vector is \(\mathbf{V}_k^{\top}\mathbf{x}_i\in\mathbb{R}^k\). Its reconstructed column is \(\mathbf{V}_k\mathbf{V}_k^{\top}\mathbf{x}_i\). The matrix formula \(\mathbf{X}_k=\mathbf{X}\mathbf{V}_k\mathbf{V}_k^{\top}\) applies this projection to every observation at once.

Projection, Variance, and Reconstruction Error

Because the columns of \(\mathbf{V}_k\) are orthonormal, \(\mathbf{V}_k^{\top}\mathbf{V}_k=\mathbf{I}_k\). Consequently, \(\mathbf{P}_k=\mathbf{V}_k\mathbf{V}_k^{\top}\) is symmetric and satisfies \(\mathbf{P}_k^2=\mathbf{P}_k\). It projects onto the span of the selected feature directions. For each observation, the residual \(\mathbf{x}_i-\mathbf{P}_k\mathbf{x}_i\) is orthogonal to that span, so the reconstruction is the closest point in the selected subspace. This also explains why the reconstruction error measures the variation discarded by the projection.

The sample covariance matrix of centered observations is \(\mathbf{C}=\frac{1}{m-1}\mathbf{X}^{\top}\mathbf{X}\). Its eigenvalues in descending order are \(\lambda_j=\frac{\sigma_j^2}{m-1}\) for \(1\le j\le s\), followed by zeros if \(n>s\). Thus the singular vectors on the right are covariance directions, and the squared singular values—not the singular values themselves—measure variance along those directions.

Proof. For any orthonormal basis \(\mathbf{q}_1,\dots,\mathbf{q}_k\) of a \(k\)-dimensional subspace, the total sample variance of the projected observations is \(\frac{1}{m-1}\sum_{\ell=1}^{k}\|\mathbf{X}\mathbf{q}_\ell\|^2=\sum_{\ell=1}^{k}\mathbf{q}_\ell^{\top}\mathbf{C}\mathbf{q}_\ell\). The covariance matrix has an orthonormal eigenbasis \(\mathbf{v}_1,\dots,\mathbf{v}_n\), with eigenvalues \(\lambda_1\ge\cdots\ge\lambda_n\ge0\), where \(\lambda_j=\sigma_j^2/(m-1)\) for \(j\le s\) and \(\lambda_j=0\) for \(j>s\). Define \(\alpha_j=\sum_{\ell=1}^{k}|\mathbf{v}_j^{\top}\mathbf{q}_\ell|^2\). The orthogonal projection onto the span of the \(\mathbf{q}_\ell\) cannot increase the length of a unit vector, so \(0\le\alpha_j\le1\); summing the squared coordinates of each \(\mathbf{q}_\ell\) in the orthonormal eigenbasis gives \(\sum_{j=1}^{n}\alpha_j=k\). The variance is therefore \(\sum_{j=1}^{n}\lambda_j\alpha_j\). If \(k=0\), this is zero. If \(k\ge1\), then \(\sum_{j=1}^{k}\lambda_j-\sum_{j=1}^{n}\lambda_j\alpha_j =\sum_{j=1}^{k}\lambda_j(1-\alpha_j)-\sum_{j=k+1}^{n}\lambda_j\alpha_j \ge\lambda_k\bigl(\sum_{j=1}^{k}(1-\alpha_j)-\sum_{j=k+1}^{n}\alpha_j\bigr)=0\). The last equality follows from \(\sum_j\alpha_j=k\). Thus no \(k\)-dimensional subspace retains more variance than the first \(k\) eigen-directions. Choosing those directions gives \(\alpha_j=1\) for \(j\le k\) and \(0\) otherwise, so the bound is attained. Substituting the eigenvalues gives the stated formula. If \(\mathbf{X}\ne\mathbf{O}\), total variance is positive and dividing the retained variance by it gives the stated fraction. \(\square\)

Proof. The columns of \(\mathbf{V}_k\) are orthonormal. If \(k\le s\), multiplying the singular-vector expansion of \(\mathbf{X}\) by \(\mathbf{V}_k\mathbf{V}_k^{\top}\) keeps the terms with \(j\le k\) and removes those with \(j>k\). Therefore \(\mathbf{X}-\mathbf{X}_k=\sum_{j=k+1}^{s}\sigma_j\mathbf{u}_j\mathbf{v}_j^{\top}\). The matrices \(\mathbf{u}_j\mathbf{v}_j^{\top}\) are orthonormal in the Frobenius inner product, since \(\operatorname{tr}((\mathbf{u}_i\mathbf{v}_i^{\top})^{\top}\mathbf{u}_j\mathbf{v}_j^{\top})=(\mathbf{u}_i^{\top}\mathbf{u}_j)(\mathbf{v}_j^{\top}\mathbf{v}_i)\), which is \(1\) if \(i=j\) and \(0\) otherwise. Hence the squared Frobenius norm of the residual is the sum of the squared coefficients, \(\sum_{j=k+1}^{s}\sigma_j^2\). If \(k>s\), all right singular directions not included among the first \(s\) are in \(\operatorname{Nul}\mathbf{X}\), so \(\mathbf{X}\mathbf{V}_k\mathbf{V}_k^{\top}=\mathbf{X}\); the error is zero, as is the stated empty sum. \(\square\)

Worked Reconstructions and Variance Calculations

Common Pitfalls

Do not confuse singular values with variance. In the first example, using \(\sqrt{6}/(\sqrt{6}+\sqrt{2})\) as the retained variance fraction is incorrect: singular values measure lengths, while variance uses their squares. The correct fraction is \(6/(6+2)=3/4\).

Do not create left singular vectors beyond the available dimensions. For the \(2\times3\) matrix in the rectangular example, a formula such as \(\sum_{j=1}^{3}\sigma_j\mathbf{u}_j\mathbf{v}_j^{\top}\) would require \(\mathbf{u}_3\in\mathbb{R}^2\) as a third orthonormal left singular vector, which cannot exist. The valid expansion stops at \(s=2\). For any \(k\le n\), the safe projection formula is \(\mathbf{X}\mathbf{V}_k\mathbf{V}_k^{\top}\); if \(k>s\), the additional right directions contribute no new reconstruction.

If \(\mathbf{X}=\mathbf{O}\), total variance is zero and a retained-variance fraction is undefined, not zero or one. The reconstruction error formula still applies and gives zero for every \(k\). In nonzero data, the retained fraction and squared error are complementary: the retained squared singular values plus the discarded squared singular values equal \(\|\mathbf{X}\|_F^2\).