Why PCA Maximizes Variance and Minimizes Reconstruction Error

Why These Two Optimality Claims Matter

Principal component analysis (PCA) represents cells using directions in gene-expression space. Its first direction is often described as capturing the greatest variation, while its first \(k\) directions provide a compact reconstruction of the data. These are precise mathematical claims: among all unit-length linear directions, the first principal component maximizes the sample variance of projected cell scores; among all approximations of rank at most \(k\), the PCA reconstruction has the smallest sum of squared residuals.

The claims concern a particular geometry. Each gene contributes according to the values in the input matrix, distances are Euclidean, and the data have been centered by gene. If the input has been scaled or transformed, these same results still apply to that input matrix—but the variance and reconstruction being optimized are then those of the transformed values.

Setup and Definitions

The unit-length requirement matters: without it, multiplying a direction by a large constant would multiply its scores and variance by the square of that constant. The sample covariance matrix of the centered columns is \(S=X^T X/(n-1)\). Since \(S\) is symmetric and positive semidefinite, it has orthonormal eigenvectors and nonnegative eigenvalues. Write its eigenvalues in descending order as \(\lambda_1 \ge \lambda_2 \ge \cdots \ge 0\), and let their corresponding orthonormal eigenvectors be \(v_1,v_2,\ldots\).

Equivalently, a singular value decomposition of \(X\) is \(X=U\Sigma V^T\). The right singular vectors in \(V\) are eigenvectors of \(X^T X\), and \(\lambda_i=\sigma_i^2/(n-1)\), where \(\sigma_i\) is the corresponding singular value. Thus the first principal direction is \(v_1\), and its cell scores are \(Xv_1\).

Why the First Direction Maximizes Variance

For any unit direction \(v\), the projected variance is \(\frac{1}{n-1}\|Xv\|^2=v^T S v\). Expand \(v\) in the orthonormal eigenvector basis: \(v=\sum_i a_i v_i\), where \(\sum_i a_i^2=1\). Then

The inequality holds because every \(\lambda_i \leq \lambda_1\) and every \(a_i^2\) is nonnegative. Equality is achieved by choosing \(v=v_1\), or by any unit vector in the leading eigenspace if the largest eigenvalue is tied. This proves the maximization claim. It also identifies what is being maximized: sample variance of the projected scores, not a measure of biological importance or a guarantee that the direction separates cell types.

Why the First \(k\) Components Minimize Reconstruction Error

Let \(V_k\) contain the first \(k\) right singular vectors of \(X\). Projecting each cell onto their span and mapping the scores back to gene space gives the reconstruction \(X_k=XV_kV_k^T\). Its rank is at most \(k\), and its squared reconstruction error is \(\|X-X_k\|_F^2\). The stronger optimality statement is that no other matrix of rank at most \(k\) can have smaller squared Frobenius error.

Here is a justification for arbitrary rank-\(k\) approximations, not just approximations written as PCA projections. Take any matrix \(A\) of rank at most \(k\), and let \(P\) be the orthogonal projection onto its column space. Then \(PA=A\). The two terms in \(X-A=(I-P)X+(PX-A)\) are orthogonal in the Frobenius inner product, so

Consequently, the error of \(A\) is at least the error from projecting \(X\) onto the column space of \(A\). The amount of squared data captured by that projection is

where \(u_i\) are the left singular vectors of \(X\). Set \(w_i=u_i^TPu_i\). Because \(P\) is an orthogonal projection, \(0\leq w_i\leq 1\), and \(\sum_i w_i\leq \operatorname{rank}(P)\leq k\). The weighted sum \(\sum_i \sigma_i^2 w_i\) is therefore at most \(\sum_{i=1}^k \sigma_i^2\): the largest possible sum assigns weight one to the \(k\) largest squared singular values and zero to the rest. Thus every rank-at-most-\(k\) approximation has error at least \(\|X\|_F^2-\sum_{i=1}^k\sigma_i^2=\sum_{i>k}\sigma_i^2\).

The truncated singular value decomposition \(A_k=U_k\Sigma_k V_k^T\) attains this bound, since its residual consists of the omitted singular-vector terms. Therefore it minimizes squared reconstruction error. The equivalent PCA projection \(XV_kV_k^T\) is the same truncated reconstruction. This argument also shows why the variance and reconstruction claims are related: both select the directions associated with the largest eigenvalues, or, equivalently, the largest squared singular values.

Assumptions, Diagnostics, and Limits

The results require a finite real-valued matrix and squared Euclidean error. For the projected sample variance interpretation, columns must be centered and \(n>1\); the divisor \(n-1\) is the sample-variance convention. A non-centered matrix can still be decomposed and its rank-\(k\) approximation can still minimize its Frobenius error, but its leading direction need not maximize the sample variance about the column means. For PCA as described here, centering is therefore not optional.

Useful checks include confirming that the input columns have means close to zero, that the selected directions have unit length and are mutually orthogonal, and that the eigenvalues are nonnegative and ordered from largest to smallest, allowing for numerical tolerance. For a proposed rank-\(k\) reconstruction, calculate the residual sum of squares and compare it with the sum of the omitted squared singular values. A materially smaller value signals a calculation or indexing error, because it contradicts the lower bound; a larger value may mean the reconstruction is not the truncated PCA solution.

Ties are not failures. If \(\lambda_1=\lambda_2\), multiple directions in that tied eigenspace achieve the same maximum projected variance, so a unique first direction is not determined. Similarly, a tied singular value at the rank-\(k\) boundary can permit multiple optimal \(k\)-dimensional subspaces. The optimal error remains the same. In single-cell analysis, sampling noise, highly correlated genes, and preprocessing choices can affect the estimated directions; the algebra proves optimality for the matrix supplied, not stability across data sets or biological relevance.

A common mistake is to say that a direction with the largest variance must be the most biologically meaningful. The mathematical result only says that it captures the most squared variation under the chosen representation and feature weighting. Another mistake is to claim PCA gives the best possible low-dimensional representation of every kind. The proof compares PCA with linear rank-\(k\) approximations under squared error. A nonlinear representation, or a different objective such as preserving selected pairwise distances, answers a different optimization problem and is not covered by this optimality claim.

For a fixed input matrix and rank, the main alternative is to use another rank-\(k\) linear approximation. It may be chosen for interpretability, biological constraints, or a different loss function, but it cannot improve on PCA's squared Frobenius reconstruction error without changing the assumptions or objective. The trade-off is explicit: PCA is optimal for this loss, but that does not make it optimal for every scientific purpose.