Eigenvectors and Eigenvalues in PCA

Why PCA Directions Are Eigenvectors

The earlier material on principal components as maximum-variance directions establishes the central optimization question: among all unit directions, which one gives the greatest variance after the observations are projected onto it? The covariance matrix turns that question into a precise calculation. Its eigenvectors identify directions where the projection behaves especially simply: projecting onto a unit eigenvector gives a variance equal to its associated eigenvalue.

This connection matters because it gives a practical way to find and interpret principal directions. Instead of testing every possible direction, one solves an eigenvalue problem for the covariance matrix. The result is useful only when the direction is normalized: the eigenvalue equals the variance of the projection for a unit eigenvector, not for an arbitrary multiple of that vector.

Definition: Let \(z_1,\ldots,z_n\) be centered observation vectors with \(p\) numeric variables, and let their sample covariance matrix be \(S=\frac{1}{n-1}\sum_{i=1}^{n}z_i z_i^T\). For a unit direction \(a\), where \(a^T a=1\), the projected value for observation \(i\) is \(a^T z_i\). The direction is an eigenvector of \(S\) with eigenvalue \(\lambda\) if \(Sa=\lambda a\).

Here the observations are centered, so each variable has sample mean zero. The covariance matrix \(S\) is symmetric. It is also positive semidefinite: for any vector \(a\), \(a^T S a\) is nonnegative. Its eigenvalues are therefore real and nonnegative. If PCA is performed using standardized variables, the same reasoning applies with the correlation matrix in place of the covariance matrix.

From Projected Variance to the Eigenvector Equation

For a direction \(a\), each projected value is \(a^T z_i\). Since the observations are centered, the projected values have mean zero too. Their sample variance is therefore the average squared projection, using the sample-variance denominator \(n-1\).

\begin{aligned} \operatorname{Var}(a^Tz) &=\frac{1}{n-1}\sum_{i=1}^{n}(a^Tz_i)^2\\ &=\frac{1}{n-1}\sum_{i=1}^{n}a^Tz_i z_i^T a\\ &=a^T\left(\frac{1}{n-1}\sum_{i=1}^{n}z_i z_i^T\right)a\\ &=a^T S a. \end{aligned}

The second equality follows because \(a^Tz_i\) is a scalar, so its square can be written as \((a^Tz_i)(z_i^Ta)=a^Tz_i z_i^Ta\). In the next equality, the fixed vector \(a\) is taken outside the sum. Thus the variance to maximize over unit directions is the quadratic form \(a^T S a\).

To find a direction at which this quadratic form is stationary under the constraint \(a^Ta=1\), introduce a multiplier \(\lambda\). The multiplier method gives the following short derivation.

Set up the constrained objective.
Maximize \(a^T S a\) subject to \(a^Ta=1\). Use the Lagrangian \(L(a,\lambda)=a^T S a-\lambda(a^Ta-1)\).

Set the gradient with respect to \(a\) to zero.
Because \(S\) is symmetric, the gradient of \(a^T S a\) is \(2Sa\), and that of \(a^Ta\) is \(2a\). Hence \(2Sa-2\lambda a=0\).

Obtain the eigenvector equation.
Rearranging gives \(Sa=\lambda a\). A stationary direction must therefore be an eigenvector of \(S\).

Identify the variance represented by the eigenvalue.
Multiply \(Sa=\lambda a\) on the left by \(a^T\). Since \(a^Ta=1\), \(a^TSa=\lambda a^Ta=\lambda\). The left side is the projected variance, so the eigenvalue is that variance.

Theorem: For a centered data set with sample covariance matrix \(S\), every unit eigenvector \(a\) with eigenvalue \(\lambda\) has projected sample variance \(a^T S a=\lambda\). The direction of greatest projected variance is an eigenvector associated with the largest eigenvalue of \(S\).

The last claim follows from the constrained optimization: the maximum of \(a^T S a\) over unit vectors occurs at a stationary direction, and for a symmetric covariance matrix the largest eigenvalue gives the maximum. A lower-eigenvalue eigenvector gives a direction with correspondingly lower projected variance. This tutorial uses the eigenvalues to interpret directions; comparing their shares of total variance is a separate calculation.

Worked Calculations

Worked Example: Two Eigen-directions and Their Variances

A hypothetical portfolio study has a covariance matrix for two centered numeric features \(S=\begin{pmatrix}4&2\\2&4\end{pmatrix}\). Find its eigenvalues and unit eigenvectors, then verify the projected variance for each direction. Use the covariance matrix as given; do not rescale its entries.

Method. Solve \(\det(S-\lambda I)=0\), then substitute each eigenvalue into \((S-\lambda I)a=0\). The covariance matrix is symmetric, and each direction will be normalized before interpreting its eigenvalue as a variance.

First compute the characteristic equation: \[ \det(S-\lambda I)=(4-\lambda)^2-2^2 =\lambda^2-8\lambda+12 =(\lambda-6)(\lambda-2)=0. \] Thus the eigenvalues are \(6\) and \(2\).

For \(\lambda=6\), the first row of \(S-6I\) gives \(-2a_1+2a_2=0\), so \(a_1=a_2\). A unit eigenvector is \(a_1=(1,1)^T/\sqrt{2}\), since its squared length is \(1/2+1/2=1\). Its projected variance is \[ a_1^T S a_1 =\frac{1}{2}(1,1) \begin{pmatrix}6\\6\end{pmatrix} =6. \]

For \(\lambda=2\), the first row of \(S-2I\) gives \(2a_1+2a_2=0\), so \(a_2=-a_1\). A unit eigenvector is \(a_2=(1,-1)^T/\sqrt{2}\). Since \(Sa_2=2a_2\), its projected variance is \(a_2^TSa_2=2a_2^Ta_2=2\). The calculations agree with the two eigenvalues. The first direction has the greater projected variance and is the maximum-variance direction. The normalization avoids mistaking the length of an eigenvector for a variance.

Worked Example: Check a Candidate Direction Against the Eigen-directions

For a different hypothetical pair of features, suppose the covariance matrix is \(S=\begin{pmatrix}5&2\\2&2\end{pmatrix}\). Find the eigen-directions and their variances. Then calculate the variance along \(c=(1,1)^T/\sqrt{2}\), which is not assumed to be an eigenvector.

Method. First solve the characteristic equation. Then use the quadratic form \(a^TSa\) for each unit direction, including the proposed direction \(c\). This checks the maximum-variance interpretation without assuming every candidate direction is an eigenvector.

The characteristic polynomial is \[ \det(S-\lambda I)=(5-\lambda)(2-\lambda)-4 =\lambda^2-7\lambda+6 =(\lambda-6)(\lambda-1). \] The eigenvalues are \(6\) and \(1\). For \(\lambda=6\), the equation \(-a_1+2a_2=0\) gives a direction proportional to \((2,1)^T\). Its length is \(\sqrt{2^2+1^2}=\sqrt{5}\), so the unit eigenvector is \(u=(2,1)^T/\sqrt{5}\). Its projected variance is \(u^TSu=6\).

For \(\lambda=1\), the equation \(4a_1+2a_2=0\) gives \(a_2=-2a_1\). The unit eigenvector is \(v=(1,-2)^T/\sqrt{5}\), and its projected variance is \(v^TSv=1\).

The proposed vector \(c\) is already a unit vector. Direct multiplication gives \(Sc=(7,4)^T/\sqrt{2}\), so \[ c^TSc=\frac{(1,1)(7,4)^T}{2}=\frac{11}{2}=5.5. \] This is below \(6\), the largest eigenvalue, as required. It is also not an eigen-direction because \(Sc=(7,4)^T/\sqrt{2}\) is not a scalar multiple of \(c\). The common mistake this avoids is labeling every unit direction’s variance as an eigenvalue: only eigenvector directions have that association.

Worked Example: Calculate the Covariance Matrix from Centered Observations

A hypothetical data set contains three observations on two numeric features: \((1,2)\), \((3,4)\), and \((2,3)\). Calculate the sample covariance matrix, find its eigen-directions, and verify the projected variance for each.

Method. Compute the variable means, center each observation, form \(S=\frac{1}{n-1}\sum_i z_i z_i^T\), and then solve the eigenvalue equation. This method keeps the sample denominator \(n-1\) consistent throughout.

The feature means are \((1+3+2)/3=2\) and \((2+4+3)/3=3\). The centered vectors are \(z_1=(-1,-1)^T\), \(z_2=(1,1)^T\), and \(z_3=(0,0)^T\). Their outer products are \[ z_1z_1^T=\begin{pmatrix}1&1\\1&1\end{pmatrix},\qquad z_2z_2^T=\begin{pmatrix}1&1\\1&1\end{pmatrix},\qquad z_3z_3^T=\begin{pmatrix}0&0\\0&0\end{pmatrix}. \]

There are \(n=3\) observations, so \(n-1=2\). Therefore, \[ S=\frac{1}{2} \begin{pmatrix}2&2\\2&2\end{pmatrix} =\begin{pmatrix}1&1\\1&1\end{pmatrix}. \] The characteristic equation is \(\det(S-\lambda I)=(1-\lambda)^2-1=\lambda(\lambda-2)=0\), giving eigenvalues \(2\) and \(0\). For \(\lambda=2\), the eigenvector equation gives equal coordinates; the unit eigenvector is \(a=(1,1)^T/\sqrt{2}\). For \(\lambda=0\), it gives opposite coordinates; the unit eigenvector is \(b=(1,-1)^T/\sqrt{2}\).

For the first direction, the three projected values are \(-\sqrt{2}\), \(\sqrt{2}\), and \(0\). Their mean is zero, and their sample variance is \((2+2+0)/2=2\). For the second direction, the projected values are \(0\), \(0\), and \(0\), so their sample variance is zero. Both variances match their eigenvalues. The zero variance occurs because the two centered features move together exactly in these observations; it is not a negative variance or a numerical error.

Common Exam Trap: Forgetting the Unit-Length Condition

The eigenvector equation is unchanged if an eigenvector is multiplied by a nonzero constant. Projected variance, however, is calculated for a specified projection scale. For a non-unit vector \(b\), the variance of \(b^Tz\) is \(b^TSb\), while the variance along its corresponding unit direction is \[ \frac{b^TSb}{b^Tb}. \] The denominator is necessary because \(b^Tb\) is the squared length of \(b\).

For example, in the first worked example take \(b=(2,2)^T\). Since \(Sb=(12,12)^T\), the unnormalized quadratic form is \(b^TSb=2(12)+2(12)=48\). A mistaken answer would call \(48\) the eigenvalue or the variance of the direction. But \(b^Tb=2^2+2^2=8\), so the variance along the unit direction is \(48/8=6\), the eigenvalue found above. Equivalently, the unit vector is \(b/\sqrt{8}=(1,1)^T/\sqrt{2}\).

Key takeaway: The eigenvector equation \(Sa=\lambda a\) identifies stationary directions of projected variance. When \(a\) is a unit vector, \(a^TSa=\lambda\), so its eigenvalue is exactly the variance of the centered observations projected onto that direction. The largest eigenvalue identifies the direction with greatest projected variance.

Check Your Understanding

For each problem, identify the eigenvalue or projected variance requested and keep the direction’s length in view.

For \(S=\begin{pmatrix}3&0\\0&1\end{pmatrix}\), find the eigenvalues and give a unit eigenvector for the direction with maximum projected variance. (Answer: eigenvalues \(3\) and \(1\); a maximum-variance unit eigenvector is \((1,0)^T\), with projected variance \(3\).)
For \(S=\begin{pmatrix}2&1\\1&2\end{pmatrix}\), verify that \(a=(1,1)^T/\sqrt{2}\) is an eigenvector and state its projected variance. (Answer: \(Sa=3a\), so the projected variance is \(3\).)
For \(S=\begin{pmatrix}4&0\\0&1\end{pmatrix}\), calculate the projected variance for the unit direction \(c=(1,1)^T/\sqrt{2}\). Is it an eigenvector direction? (Answer: projected variance \(5/2=2.5\); no, because \(Sc=(4,1)^T/\sqrt{2}\) is not a scalar multiple of \(c\).)
A covariance matrix has an eigenvector \(b=(3,4)^T\) with eigenvalue \(5\). What is the projected variance along the corresponding unit direction? (Answer: \(5\); \(b\) has length \(5\), and normalization does not change its direction or eigenvalue.)
Three centered observations in two variables are \(z_1=(1,1)^T\), \(z_2=(-1,-1)^T\), and \(z_3=(0,0)^T\). Using the sample covariance convention, find the covariance matrix and the projected variance along \((1,-1)^T/\sqrt{2}\). (Answer: \(S=\begin{pmatrix}1&1\\1&1\end{pmatrix}\); the projected variance is \(0\).)

What You'll Learn