PCA Through Singular Value Decomposition

Why Use the SVD for PCA?

PCA is often introduced by finding the eigenvalues and eigenvectors of a covariance matrix. There is another route: apply the singular value decomposition (SVD) directly to the centered data matrix. The two routes give the same principal directions and component variances, with a direct conversion between singular values and covariance eigenvalues. That conversion is useful for understanding computations and checking that their scaling is correct.

The key point is that the data matrix contains the same variation summarized by the covariance matrix, but arranged by observation and variable. The SVD separates that variation into orthogonal directions. This tutorial derives the link and works through the conventions needed to interpret its quantities correctly. The connection is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: Let \(Z\) be an \(n \times p\) real data matrix with \(n\) observations as rows and \(p\) variables as columns. Assume each column has been centered by subtracting its sample mean, so every column sums to zero. The usual sample covariance matrix is \(S=Z^\mathsf{T}Z/(n-1)\). The singular value decomposition is \(Z=UDV^\mathsf{T}\), where the columns of \(U\) and \(V\) are orthonormal and \(D\) contains the nonnegative singular values.

For the SVD, write the singular values in descending order as \(s_1 \ge s_2 \ge \cdots \ge 0\). The columns of \(V\) are right singular vectors in the original variable space; the columns of \(U\) describe directions across the observations. A thin SVD may include only the singular vectors needed to represent the rank of \(Z\); a full SVD may include additional vectors associated with zero singular values.

The covariance convention matters. Here \(S\) is the sample covariance matrix and uses \(n-1\), not \(n\), in its denominator. The derivation below also indicates how the result changes if a calculation instead uses a divisor of \(n\).

Deriving the Covariance Eigenpairs

Start with a centered matrix \(Z\) and its SVD. Orthonormality of the columns of \(U\) means \(U^\mathsf{T}U=I\). Substituting the SVD into the sample covariance formula gives:

S=\frac{Z^\mathsf{T}Z}{n-1} =\frac{(UDV^\mathsf{T})^\mathsf{T}(UDV^\mathsf{T})}{n-1} =\frac{VD^\mathsf{T}U^\mathsf{T}UDV^\mathsf{T}}{n-1} =V\frac{D^\mathsf{T}D}{n-1}V^\mathsf{T}.

In a diagonal SVD representation, \(D^\mathsf{T}D\) has diagonal entries \(s_j^2\). Thus the covariance matrix is expressed using the right singular vectors and the squared singular values divided by \(n-1\). This is an eigen-decomposition: the right singular vectors are covariance eigenvectors, and the corresponding eigenvalues are the squared singular values divided by \(n-1\).

Theorem: For a centered data matrix \(Z=UDV^\mathsf{T}\), the sample covariance eigenpairs are \(v_j\) and \(\lambda_j=s_j^2/(n-1)\), where \(v_j\) is the \(j\)th column of \(V\) and \(s_j\) is its corresponding singular value. Therefore \(s_j=\sqrt{(n-1)\lambda_j}\). If the covariance convention uses \(Z^\mathsf{T}Z/n\) instead, its eigenvalues are \(s_j^2/n\).

This result explains why the SVD gives the principal directions: multiplying the covariance matrix by \(v_j\) yields \(\lambda_jv_j\). It also explains why the order of the singular values matters. Since squaring and dividing by the same positive number preserve order, the largest singular value corresponds to the largest covariance eigenvalue.

Component Scores and Variance

For a unit direction \(v_j\), the component score for each observation is the corresponding entry of \(Zv_j\). Using the SVD, \(Zv_j=s_ju_j\). The scores therefore form the vector \(s_ju_j\). Because \(u_j\) has squared length 1, the sum of squared scores is \(s_j^2\); because the data are centered, their sample variance is that sum divided by \(n-1\).

Zv_j=s_ju_j,\qquad \operatorname{Var}_{\text{sample}}(Zv_j) =\frac{(Zv_j)^\mathsf{T}(Zv_j)}{n-1} =\frac{s_j^2u_j^\mathsf{T}u_j}{n-1} =\frac{s_j^2}{n-1}.

This is a short independent check of the eigenvalue result: the component variance equals the covariance eigenvalue for that direction. The formula assumes centered data and a unit direction \(v_j\). If a direction is not unit length, the variance of its projected scores also depends on the squared length of that direction.

Worked Example: Converting Singular Values to Component Variances

A hypothetical analysis has \(n=4\) centered observations on three variables. The singular values of its data matrix are \(4\), \(2\), and \(0\). Find the sample covariance eigenvalues and the variance of the scores on the first two component directions. Use the SVD-to-covariance method, and check the result by summing the eigenvalues.

Method. There are \(n-1=3\) degrees of freedom in the sample covariance denominator. For each singular value, square it and divide by 3:

\lambda_1=\frac{4^2}{3}=\frac{16}{3},\qquad \lambda_2=\frac{2^2}{3}=\frac{4}{3},\qquad \lambda_3=\frac{0^2}{3}=0.

The score variance on the first component direction is \(16/3\), and on the second it is \(4/3\), because each score variance equals its covariance eigenvalue. The third direction has zero variance. The total variance is the sum:

\frac{16}{3}+\frac{4}{3}+0=\frac{20}{3}.

Check. The sum of the sample covariance eigenvalues is the trace of the sample covariance matrix, which is the total variance across the variables. The calculation is consistent provided that total variance is \(20/3\). The common mistake this avoids is dividing the squared singular values by \(n=4\); that would give population-divisor values, not the stated sample covariance eigenvalues.

Worked Calculations from Centered Data

The next example starts with the data matrix rather than a supplied list of singular values. It shows how the covariance eigenvalues and directions can be identified from the matrix product \(Z^\mathsf{T}Z\), then checked against the singular values.

Worked Example: A Correlated Two-Variable Data Matrix

Three hypothetical observations on two centered variables form the matrix

Z= \begin{pmatrix} -1&-2\\ 0&1\\ 1&1 \end{pmatrix}.

Find the sample covariance eigenvalues and the singular values. Then verify the leading component variance from its scores. The method is to compute \(Z^\mathsf{T}Z\), form the sample covariance matrix, and use its eigenvalues.

First, the column sums are \(-1+0+1=0\) and \(-2+1+1=0\), so the matrix is centered. The column sums of squares are \(2\) and \(6\), and their cross-product is \((-1)(-2)+(0)(1)+(1)(1)=3\). Thus:

Z^\mathsf{T}Z= \begin{pmatrix} 2&3\\ 3&6 \end{pmatrix}, \qquad S=\frac{Z^\mathsf{T}Z}{2} = \begin{pmatrix} 1&1.5\\ 1.5&3 \end{pmatrix}.

The trace of \(S\) is \(4\) and its determinant is \(1(3)-1.5^2=0.75\). Its characteristic equation is \(\lambda^2-4\lambda+0.75=0\), so:

\lambda_{1,2}=\frac{4\pm\sqrt{16-3}}{2} =\frac{4\pm\sqrt{13}}{2}.

Numerically, the eigenvalues are approximately \(3.8028\) and \(0.1972\). Since \(n-1=2\), the squared singular values are \(2\lambda_1=4+\sqrt{13}\) and \(2\lambda_2=4-\sqrt{13}\). The singular values are therefore approximately \(2.7578\) and \(0.6281\). Squaring either singular value and dividing by 2 recovers its covariance eigenvalue.

For an additional score check, a unit leading eigenvector is proportional to \((3,\,2+\sqrt{13})^\mathsf{T}\), normalized by its length \(\sqrt{9+(2+\sqrt{13})^2}\). Its projected scores are \(Zv_1\). By the derivation above, the sum of their squares is \(s_1^2=4+\sqrt{13}\), so their sample variance is \((4+\sqrt{13})/2=\lambda_1\). This check uses the score-vector identity rather than treating rounded score values as exact.

Check. The two eigenvalues sum to \(4\), the trace of \(S\). The mistake to avoid is reporting \(4+\sqrt{13}\) itself as the leading covariance eigenvalue: that is the squared singular value, and it must be divided by \(n-1=2\).

Worked Example: A Centered Matrix with a Zero Singular Value

Three hypothetical observations on three variables are centered as follows. Find the covariance eigenvalues, singular values, and variance of the first two component score vectors. Explain why the final singular value is zero.

Z= \begin{pmatrix} 1&0&1\\ 0&1&1\\ -1&-1&-2 \end{pmatrix}.

Each column sums to zero. Multiplying the matrix by its transpose gives:

Z^\mathsf{T}Z= \begin{pmatrix} 2&1&3\\ 1&2&3\\ 3&3&6 \end{pmatrix}, \qquad S=\frac{Z^\mathsf{T}Z}{2}.

The vectors \((1,1,2)^\mathsf{T}\), \((1,-1,0)^\mathsf{T}\), and \((-1,-1,1)^\mathsf{T}\) are mutually orthogonal. Multiplication by \(Z^\mathsf{T}Z\) gives, respectively, 9 times the first vector, 1 times the second, and 0 times the third. Thus the eigenvalues of \(Z^\mathsf{T}Z\) are \(9,1,0\); dividing by \(n-1=2\) gives covariance eigenvalues \(4.5,0.5,0\). The singular values are the square roots of the eigenvalues of \(Z^\mathsf{T}Z\), namely \(3,1,0\). Dividing their squares by 2 gives the same covariance eigenvalues.

Normalize the first eigenvector to \(v_1=(1,1,2)^\mathsf{T}/\sqrt{6}\). The scores are \(Zv_1=(3,3,-6)^\mathsf{T}/\sqrt{6}\). Their squared sum is \((9+9+36)/6=9\), so their sample variance is \(9/2=4.5\). For the second direction \(v_2=(1,-1,0)^\mathsf{T}/\sqrt{2}\), the scores are \((1,-1,0)^\mathsf{T}/\sqrt{2}\); their squared sum is \(1\), giving sample variance \(1/2=0.5\).

The third column of \(Z\) is the sum of its first two columns. The columns are linearly dependent, so the matrix has rank 2 and one singular value is zero. Equivalently, projection onto the normalized direction \((-1,-1,1)^\mathsf{T}/\sqrt{3}\) gives zero for every row. The zero eigenvalue represents no variation in that direction, not a negative or missing variance.

Conditions, Rank, and Common Traps

The algebraic link between an SVD and \(Z^\mathsf{T}Z\) holds for real matrices. To interpret its eigenvalues as variances in PCA, the matrix must represent centered numeric variables and the covariance denominator must be specified. No distributional assumption, such as normality, is needed for this matrix identity.

Centering is essential to the covariance interpretation. For uncentered \(Z\), the product \(Z^\mathsf{T}Z/n\) is a matrix of uncentered second moments, not the covariance matrix around the variable means. In particular, a large mean can contribute to this product even when the variable has little variation around its mean.

Centering also limits the rank. Since the rows of a centered matrix sum to zero, at most \(n-1\) rows can be linearly independent. Consequently, \(\operatorname{rank}(Z)\leq\min(n-1,p)\). Any remaining singular values are zero. If singular values are tied, their associated directions need not be uniquely determined within the tied subspace; changing the basis within that subspace does not change the covariance eigenvalues. The individual vector signs can also be reversed without changing the eigenpair.

A frequent exam trap is to mix up the three related quantities. A singular value is \(s_j\); its square \(s_j^2\) is an eigenvalue of \(Z^\mathsf{T}Z\); and the sample covariance eigenvalue is \(s_j^2/(n-1)\). Keep the denominator attached to the covariance convention being used. If the question specifies a divisor of \(n\), use that divisor instead. Never infer that the largest singular value is itself a variance.

Identify the matrix and convention.
Confirm that rows are observations, columns are variables, the columns are centered, and whether covariance uses \(n-1\) or \(n\).

Match directions.
The right singular vectors of \(Z\) are eigenvectors of its covariance matrix.

Convert and check.
Square each singular value, divide by the specified covariance denominator, and check that the resulting eigenvalues sum to the covariance trace.

Key takeaway: For centered data with sample covariance \(S=Z^\mathsf{T}Z/(n-1)\), the PCA directions are the right singular vectors of \(Z\), and each component variance is \(s_j^2/(n-1)\). The denominator is part of the conversion.

Check Your Understanding

For each problem, use the stated covariance convention and distinguish singular values from covariance eigenvalues.

A centered matrix has \(n=5\) observations and singular values \(3\) and \(1\), with all other singular values zero. What are its nonzero sample covariance eigenvalues? (Answer: \(9/4\) and \(1/4\).)
A centered matrix has \(n=6\) observations and singular values \(\sqrt{10}\) and \(\sqrt{2}\). Find the corresponding sample covariance eigenvalues. (Answer: \(2\) and \(0.4\).)
A centered data matrix has \(n=9\) observations. One sample covariance eigenvalue is \(5.5\). What is its corresponding singular value? (Answer: \(\sqrt{44}=2\sqrt{11}\).)
A centered matrix has \(n=4\) observations and singular values \(4\) and \(2\). A calculation reports covariance eigenvalues \(16/4\) and \(4/4\). Identify the convention error and give the sample covariance eigenvalues. (Answer: It used divisor \(n\); the sample covariance eigenvalues are \(16/3\) and \(4/3\).)
A matrix has columns with nonzero means, and its SVD is computed without centering. Can its singular values squared, divided by \(n-1\), automatically be called the sample covariance eigenvalues? Explain briefly. (Answer: No. The covariance identity applies to the centered matrix; without centering, the product includes contributions from the means.)

What You'll Learn