Tutorials › Exam SRM - Statistics for Risk Modeling › The Eigenvalue Decomposition of PCA

Home  /  Tutorials  /  Exam SRM / Principal Components Analysis

Exam SRM · Principal Components Analysis · Tutorial 669 of 926

The Eigenvalue Decomposition of PCA

Use the spectral decomposition of a covariance matrix to identify principal directions, check eigenvalues, and verify the full decomposition.

Intermediate 12 min read

What You'll Learn

  • State the spectral decomposition of a real symmetric covariance matrix using a complete orthonormal eigenbasis
  • Explain why covariance matrices are positive semidefinite and therefore have nonnegative eigenvalues
  • Find eigenvalues and normalized eigenvectors for small covariance matrices
  • Assemble and verify the matrix equation using eigenvectors and eigenvalues
  • Recognize zero eigenvalues, repeated eigenvalues, and common eigenvector calculation errors

Why Eigenvalue Decomposition Is Central to PCA

After a covariance matrix has been built, the next task in principal components analysis is to identify directions in the variable space that the matrix singles out. Eigenvalue decomposition provides those directions and an associated scalar for each one. It also gives a compact representation of the entire covariance matrix, provided the eigenvectors used form a complete basis.

The calculations are linear algebra, but the conditions matter. A covariance matrix is symmetric and positive semidefinite. These properties ensure that it has real eigenvalues, can be decomposed using an orthonormal basis of eigenvectors, and cannot have a negative eigenvalue. The earlier material on what principal components optimize connects the leading eigenvalue direction to the direction with the greatest variation.

Definition: For a square matrix \(S\), a nonzero vector \(q\) is an eigenvector if \(Sq=\lambda q\) for some scalar \(\lambda\), called its eigenvalue. In PCA, \(S\) is a covariance matrix and the eigenvectors are directions in the variable space. We use unit eigenvectors and, when ordering the principal components, list their eigenvalues from largest to smallest.

For a data matrix with \(n\) observations on \(p\) variables, let \(X_c\) be the \(n\) by \(p\) matrix after each variable has been centered by subtracting its sample mean. When \(n>1\), the sample covariance matrix is

$$ S=\frac{1}{n-1}X_c^T X_c. $$

This matrix is \(p\) by \(p\). The decomposition discussed here applies to any real symmetric covariance matrix, whether it is a sample covariance matrix or a population covariance matrix. No particular parametric distribution for the observations is required for the linear algebra result.

The Spectral Decomposition

The spectral theorem states that every real symmetric matrix has a complete orthonormal basis of eigenvectors. For a \(p\) by \(p\) covariance matrix, place those \(p\) unit eigenvectors in the columns of the square matrix \(Q\), and place their corresponding eigenvalues on the diagonal of \(\Lambda\). Then

$$ SQ=Q\Lambda,\qquad Q^TQ=QQ^T=I,\qquad S=Q\Lambda Q^T. $$

The two orthonormality equations both hold because \(Q\) is square and its columns form a complete orthonormal basis. This completeness is essential. From \(SQ=Q\Lambda\), right multiplication by \(Q^T\) gives \(SQQ^T=Q\Lambda Q^T\); it gives \(S=Q\Lambda Q^T\) because \(QQ^T=I\). An incomplete set of eigenvectors does not justify that equality.

If \(q_j\) is the \(j\)th column of \(Q\) and \(\lambda_j\) is the corresponding diagonal entry of \(\Lambda\), then \(Sq_j=\lambda_jq_j\). In PCA, each such eigenpair identifies one component direction and its associated eigenvalue. A direction is not an eigenvector merely because its entries look plausible: it must satisfy the eigenvector equation, and the eigenvectors in \(Q\) must be normalized and mutually orthogonal.

Why Covariance Eigenvalues Cannot Be Negative

The nonnegativity follows directly from the form of a covariance matrix. For any vector \(a\) of length \(p\),

$$ a^T S a =\frac{1}{n-1}a^T X_c^T X_c a =\frac{1}{n-1}(X_c a)^T(X_c a) =\frac{1}{n-1}\sum_{i=1}^{n}(X_c a)_i^2 \geq 0. $$

Thus \(S\) is positive semidefinite. If \(q\) is a unit eigenvector with eigenvalue \(\lambda\), substitute \(a=q\) into the inequality. Since \(Sq=\lambda q\) and \(q^Tq=1\),

$$ q^T S q=q^T(\lambda q)=\lambda q^Tq=\lambda\geq 0. $$

This is a useful diagnostic: a negative eigenvalue cannot belong to an exact covariance matrix. It may signal an arithmetic error, a matrix that is not a valid covariance matrix, or a numerical approximation that needs investigation.

Why the Eigenvectors Are Orthogonal

Symmetry explains why eigenvectors associated with distinct eigenvalues are orthogonal. Let \(u\) and \(w\) be eigenvectors of a real symmetric matrix \(S\), with eigenvalues \(\lambda\) and \(\mu\). Since \(S^T=S\),

$$ \lambda u^T w=(Su)^T w=u^T S^T w=u^T S w=\mu u^T w. $$

Therefore \((\lambda-\mu)u^T w=0\). If \(\lambda\ne\mu\), then \(u^T w=0\). When an eigenvalue is repeated, its eigenvectors need not be automatically orthogonal, but an orthonormal basis can be chosen within that eigenspace. This is why the spectral theorem can use a complete orthonormal eigenbasis even when eigenvalues repeat.

Worked Example: Decomposing a Two-Variable Covariance Matrix

A hypothetical portfolio-monitoring study produces the covariance matrix, in stated squared measurement units,

$$ S=\begin{pmatrix}4&2\\2&4\end{pmatrix}. $$

Find its eigenvalues and unit eigenvectors, order the eigenpairs, and verify the spectral decomposition. No parametric distribution assumption is needed.

Step 1: Find the eigenvalues. Solve \(\det(S-\lambda I)=0\):

$$ \det\begin{pmatrix}4-\lambda&2\\2&4-\lambda\end{pmatrix} =(4-\lambda)^2-4 =\lambda^2-8\lambda+12 =(\lambda-6)(\lambda-2)=0. $$

Thus the eigenvalues are \(6\) and \(2\), in descending order.

Step 2: Find and normalize the eigenvectors. For \(\lambda_1=6\), the first row of \((S-6I)q=0\) gives \(-2q_1+2q_2=0\), so a direction is \((1,1)^T\). Its length is \(\sqrt{2}\), giving \(q_1=(1,1)^T/\sqrt{2}\). For \(\lambda_2=2\), the equation \(2q_1+2q_2=0\) gives direction \((1,-1)^T\), so \(q_2=(1,-1)^T/\sqrt{2}\).

Step 3: Assemble and check. The vectors are unit length and \(q_1^Tq_2=(1-1)/2=0\). Hence

$$ Q=\frac{1}{\sqrt{2}}\begin{pmatrix}1&1\\1&-1\end{pmatrix}, \qquad \Lambda=\begin{pmatrix}6&0\\0&2\end{pmatrix}. $$

Multiplying by eigenvalue-weighted outer products gives

$$ Q\Lambda Q^T =6\left(\frac12\begin{pmatrix}1&1\\1&1\end{pmatrix}\right) +2\left(\frac12\begin{pmatrix}1&-1\\-1&1\end{pmatrix}\right) =\begin{pmatrix}4&2\\2&4\end{pmatrix}=S. $$

The decomposition checks exactly. The common mistake this calculation avoids is using unnormalized eigenvectors in \(Q\): that would make \(Q^TQ\ne I\) and invalidate this form of the decomposition.

Worked Example: Checking a Three-Variable Decomposition

For a hypothetical three-variable study, suppose the covariance matrix is

$$ S=\begin{pmatrix}4&2&0\\2&4&0\\0&0&1\end{pmatrix}. $$

Find a complete set of ordered eigenpairs and verify that the resulting matrix is the given \(S\). Again, this calculation requires no distributional assumption.

Step 1: Find the characteristic equation. The matrix is block diagonal, so its characteristic determinant factors into the determinant for the upper-left block and the final diagonal entry:

$$ \det(S-\lambda I) =(1-\lambda)\left((4-\lambda)^2-4\right) =(1-\lambda)(6-\lambda)(2-\lambda). $$

The eigenvalues in descending order are \(6,2,1\).

Step 2: Obtain the eigenvectors. The first two coordinates use the two-variable eigenvectors from the upper-left block. A complete orthonormal choice is

$$ q_1=\frac{1}{\sqrt{2}}\begin{pmatrix}1\\1\\0\end{pmatrix}, \quad q_2=\frac{1}{\sqrt{2}}\begin{pmatrix}1\\-1\\0\end{pmatrix}, \quad q_3=\begin{pmatrix}0\\0\\1\end{pmatrix}. $$

Each vector has length one. Their pairwise inner products are \(q_1^Tq_2=(1-1)/2=0\), \(q_1^Tq_3=0\), and \(q_2^Tq_3=0\). Three mutually orthonormal vectors in three-dimensional space form a complete basis.

Step 3: Reconstruct the matrix. With \(Q=(q_1,q_2,q_3)\) and \(\Lambda=\operatorname{diag}(6,2,1)\),

$$ Q\Lambda Q^T =3\begin{pmatrix}1&1&0\\1&1&0\\0&0&0\end{pmatrix} +\begin{pmatrix}1&-1&0\\-1&1&0\\0&0&0\end{pmatrix} +\begin{pmatrix}0&0&0\\0&0&0\\0&0&1\end{pmatrix} =\begin{pmatrix}4&2&0\\2&4&0\\0&0&1\end{pmatrix}. $$

This equals the supplied covariance matrix. The check confirms both the eigenvalue-vector pairings and the completeness of the basis.

Worked Example: A Zero Eigenvalue and a Redundant Direction

A hypothetical data set records two measurements that move identically after centering. Its covariance matrix is

$$ S=\begin{pmatrix}1&1\\1&1\end{pmatrix}. $$

Determine the eigenpairs, explain the zero eigenvalue, and verify the decomposition. No distributional assumption is needed.

Step 1: Calculate the eigenvalues.

$$ \det(S-\lambda I)=(1-\lambda)^2-1 =\lambda(\lambda-2)=0. $$

The ordered eigenvalues are \(2\) and \(0\). For \(\lambda_1=2\), the equation \(Sq=2q\) gives direction \((1,1)^T\); its unit version is \(q_1=(1,1)^T/\sqrt{2}\). For \(\lambda_2=0\), \(Sq=0\) gives direction \((1,-1)^T\), and \(q_2=(1,-1)^T/\sqrt{2}\) is unit length.

Step 2: Verify the full decomposition. These vectors are orthogonal, and two orthonormal vectors form a complete basis in two dimensions. Thus

$$ Q=\frac{1}{\sqrt{2}}\begin{pmatrix}1&1\\1&-1\end{pmatrix}, \qquad \Lambda=\begin{pmatrix}2&0\\0&0\end{pmatrix}, \qquad Q\Lambda Q^T =\begin{pmatrix}1&1\\1&1\end{pmatrix}. $$

The zero eigenvalue is consistent with positive semidefiniteness. Here the difference direction \((1,-1)^T\) has no variation because the two centered measurements are identical. A zero eigenvalue does not contradict the spectral theorem; it indicates a direction with no variation in this matrix.

Common Exam Traps

Several errors can produce a plausible-looking answer while breaking the decomposition:

  • Stopping before a complete basis is formed. An orthonormal set of some eigenvectors is not enough to conclude \(S=Q\Lambda Q^T\). The columns of \(Q\) must be a complete basis, so \(Q\) is square and \(QQ^T=I\).
  • Pairing an eigenvalue with the wrong vector. Each column \(q_j\) must satisfy \(Sq_j=\lambda_jq_j\) for the corresponding diagonal entry \(\lambda_j\).
  • Forgetting normalization. Eigenvectors can be multiplied by nonzero constants and remain eigenvectors, but the spectral decomposition uses an orthonormal \(Q\).
  • Accepting a negative covariance eigenvalue. Exact covariance matrices are positive semidefinite. Check the matrix construction and determinant calculation if a negative eigenvalue appears.
  • Treating a sign choice as a different direction. If \(q\) is an eigenvector, then \(-q\) is also an eigenvector with the same eigenvalue. Either sign gives the same outer product \(qq^T\).
Key takeaway: For a real symmetric covariance matrix, use a complete orthonormal eigenbasis. The resulting square matrix \(Q\) satisfies both \(Q^TQ=I\) and \(QQ^T=I\), so \(S=Q\Lambda Q^T\). Covariance structure also guarantees that every eigenvalue is nonnegative.

Check Your Understanding

For each problem, check both the eigenpair calculations and any stated conclusion about the decomposition.

  1. A hypothetical covariance matrix is \(\begin{pmatrix}9&0\\0&4\end{pmatrix}\). Give the eigenvalues in descending order and one matching complete orthonormal eigenbasis. (Answer: Eigenvalues \(9,4\); eigenvectors \((1,0)^T,(0,1)^T\); \(Q=I\) and \(\Lambda=\operatorname{diag}(9,4)\).)
  2. For \(S=\begin{pmatrix}5&1\\1&5\end{pmatrix}\), find the ordered eigenvalues and unit eigenvectors, then state the spectral decomposition. (Answer: Eigenvalues \(6,4\); eigenvectors \((1,1)^T/\sqrt{2}\) and \((1,-1)^T/\sqrt{2}\), respectively; \(S=Q\operatorname{diag}(6,4)Q^T\).)
  3. A centered-data covariance matrix is \(S=\begin{pmatrix}2&2\\2&2\end{pmatrix}\). Find its eigenvalues and explain why a zero value is allowed. (Answer: Eigenvalues \(4,0\); corresponding unit eigenvectors \((1,1)^T/\sqrt{2}\) and \((1,-1)^T/\sqrt{2}\); zero is allowed because the difference direction has no variation.)
  4. For \(S=\operatorname{diag}(3,3,1)\), are the first two eigenvectors uniquely determined by the matrix? State an orthonormal choice for the first two and the third. (Answer: No; the eigenvalue \(3\) is repeated, so any orthonormal basis in the first two coordinate directions works; one choice is \((1,0,0)^T,(0,1,0)^T\), with \((0,0,1)^T\) for eigenvalue \(1\).)
  5. A proposed covariance matrix is \(S=\begin{pmatrix}1&2\\2&1\end{pmatrix}\). Compute its eigenvalues and decide whether it can be an exact covariance matrix. (Answer: Eigenvalues \(3,-1\); it cannot be an exact covariance matrix because the negative eigenvalue violates positive semidefiniteness.)

Previous

Next

Back to Tutorials
Clinical Biostats