The Singular Value Decomposition Derivation of PCA

From a Centered Data Matrix to Principal Components

Principal component analysis (PCA) replaces a set of possibly correlated measurements with orthogonal directions through the data. The algebraic link between PCA and the singular value decomposition (SVD) makes this transformation concrete: the right singular vectors of the data matrix give the principal directions, while multiplying the data by those directions gives the component scores. The singular values also quantify the sample variation along those directions.

This derivation assumes a complete numerical data matrix whose columns have been centered, so each feature has sample mean zero. The earlier material on centering and scaling explains how choices such as standardizing columns affect the analysis. Here, take the analyzed columns as given and derive PCA for ordinary, equally weighted observations using Euclidean distance. The derivation is algebraic; it does not require normally distributed measurements.

For a proposed unit direction \(v\), the projected observations are the entries of \(Xv\). Since the columns of \(X\) are centered, these scores have mean zero. Their sample variance is

Thus the first principal direction maximizes \(v^TSv\) over all \(v\) with \(v^Tv=1\). Subsequent directions maximize the same quantity subject to orthogonality to the previously chosen directions.

Why the Eigenvectors Give the Directions

The covariance matrix \(S\) is symmetric and positive semidefinite: for every vector \(v\), \(v^TSv=\|Xv\|^2/(n-1)\geq 0\). Its eigenvalues are therefore real and nonnegative. Write its eigenvalues in descending order as \(\lambda_1\geq\lambda_2\geq\cdots\geq\lambda_p\geq0\), with corresponding orthonormal eigenvectors \(v_1,\ldots,v_p\).

The maximizing property follows directly by expressing any unit vector in that eigenvector basis. If \(v=\sum_{j=1}^p a_jv_j\), then \(\sum_j a_j^2=1\), and

Equality is attained by \(v=v_1\). After restricting \(v\) to be orthogonal to \(v_1\), its coefficient \(a_1\) is zero, so the same argument shows that the maximum is \(\lambda_2\), attained by \(v_2\). Repeating this reasoning establishes the successive directions. This is a complete variance-maximization justification, not an assumption about the data distribution.

Deriving PCA from the SVD

Take a full SVD of the centered matrix, with dimensions shown explicitly:

The columns of \(U\) and \(V\) are orthonormal. The diagonal entries of the rectangular matrix \(\Sigma\) are the singular values \(\sigma_1\geq\cdots\geq\sigma_m\geq0\), where \(m=\min(n,p)\); entries outside this diagonal are zero. Multiplying out the covariance matrix gives

Because \(U^TU=I\), the matrix inside this expression is diagonal: its first \(m\) diagonal entries are \(\sigma_j^2/(n-1)\), and any additional entries when \(p>n\) are zero. Consequently, the right singular vectors are eigenvectors of \(S\), and the corresponding eigenvalues are

The score matrix, with component columns in the same order as the columns of \(V\), is \(XV=U\Sigma\). For \(j\leq m\), its \(j\)th column satisfies \(Xv_j=\sigma_j u_j\). If \(p>n\), \(V\) has additional columns \(v_j\) for \(j>n\); these vectors lie in the null space of \(X\), so \(Xv_j=0\). They have no corresponding columns in \(U\). A zero singular value among the first \(m\) also gives a zero score vector. These distinctions matter whenever the matrix is rectangular or rank deficient.

Checks, Alternatives, and Limitations

The SVD derivation and direct eigendecomposition of the sample covariance matrix are mathematically equivalent: both return the same eigenvalues and principal directions, apart from sign choices and choices of basis in tied eigenspaces. The SVD works on \(X\) directly and is often preferable when \(p\) is large, particularly when \(p>n\); forming a \(p\times p\) covariance matrix can require substantial computation and storage. Computing the covariance matrix first can be convenient when \(p\) is modest, but it can magnify numerical conditioning problems because the eigenvalues of \(X^TX\) are squared singular values.

A few checks help detect mistakes. Confirm that the columns used really are centered and that the matrix orientation is observations by features. Check that the computed directions have unit length and are mutually orthogonal, and that \(Xv_j=\sigma_j u_j\) for the paired singular vectors where those vectors exist. The sample variance of each score column should equal \(\sigma_j^2/(n-1)\). Small discrepancies can arise from numerical rounding; a negative covariance eigenvalue beyond tiny rounding error signals a computational or data-handling problem because \(S\) is positive semidefinite.

PCA is sensitive to the units and scaling of the features: changing a feature's scale can change the directions. Scaling is therefore a modeling choice, not an automatic property of the SVD. Exact ties make individual directions within a tied subspace non-unique, and signs are always arbitrary: \(v_j\) and \(-v_j\) describe the same direction, with all scores changing sign accordingly. In addition, a zero score direction reflects no observed variation in that direction; it does not prove that the feature combination is universally constant outside this data set. Ordinary SVD-PCA also requires a complete matrix and does not, on its own, model missing values or outliers.

• Common mistake: “Every right singular vector has a corresponding left singular vector, so \(Xv_j=\sigma_j u_j\) for every column of \(V\).” This fails when \(p>n\), because \(V\) then has more columns than \(U\). The correct paired-column identity applies for \(j\leq\min(n,p)\); extra right singular vectors when \(p>n\) lie in the null space of \(X\) and produce zero scores.
• Another mistake: “The eigenvalue equals the singular value.” The eigenvalue of the sample covariance is \(\sigma_j^2/(n-1)\), not \(\sigma_j\). Squaring and applying the sample-covariance divisor are both necessary.
• Sign and ties: A software package may return a direction with the opposite sign, or a different orthonormal basis for tied eigenvalues. These can represent the same PCA solution and are not necessarily errors.