Eigenvalues as Component Variances

Why Eigenvalues Measure Variability

In principal components analysis, a direction in the variable space defines a single numerical score for each observation. The variance of those scores tells us how much of the data’s variability lies along that direction. The eigenvalue result gives this variance directly: when the direction is a unit eigenvector of the covariance matrix, the variance of the projected scores is that eigenvalue.

This connection makes the eigenvalues more than an algebraic output. They quantify the variability associated with the principal component directions. The related total-variance result provides a useful check: when all eigenvalues are included, their sum is the total variance of the variables in the covariance matrix being analyzed.

Definition: Let \(n\) observations on \(p\) variables be centered, and arrange them as rows of an \(n \times p\) matrix \(X\). The sample covariance matrix is \(S = X^\mathsf{T}X/(n-1)\). For a direction \(a \in \mathbb{R}^p\), the score for observation \(i\) is \(t_i = x_i^\mathsf{T}a\), where \(x_i^\mathsf{T}\) is row \(i\) of \(X\). A unit direction satisfies \(a^\mathsf{T}a=1\). A principal component direction is a unit eigenvector of \(S\).

The results below use the sample covariance convention with divisor \(n-1\). Since the data are centered, the mean of the scores \(t_i\) is zero. The covariance matrix \(S\) is symmetric and positive semidefinite, so its eigenvalues are real and nonnegative, and it has an orthonormal set of eigenvectors. These are the usual conditions for covariance-based PCA.

Deriving the Variance of a Component

First consider any fixed direction \(a\), not necessarily an eigenvector. The vector of all scores is \(Xa\). Its sample variance is obtained by summing the squared deviations of the scores from their mean and dividing by \(n-1\). Because the observations have been centered, the score mean is zero.

\operatorname{Var}(t) = \frac{1}{n-1}(Xa)^\mathsf{T}(Xa) = \frac{1}{n-1}a^\mathsf{T}X^\mathsf{T}Xa = a^\mathsf{T}Sa.

This is the general directional-variance formula. Now suppose \(a\) is a unit eigenvector of \(S\), with eigenvalue \(\lambda\). By definition, \(Sa=\lambda a\). Substituting that equation into the general formula gives the component-variance result.

\operatorname{Var}(t) = a^\mathsf{T}Sa = a^\mathsf{T}(\lambda a) = \lambda(a^\mathsf{T}a) = \lambda.

The unit-length condition matters: the final equality uses \(a^\mathsf{T}a=1\). For an eigenvector that has not been normalized, the score variance is \(\lambda(a^\mathsf{T}a)\), not simply \(\lambda\). The direction is unchanged by rescaling, but the numerical scores and their variance are rescaled too.

Worked Example: Finding Component Variances From a Covariance Matrix

A hypothetical portfolio-monitoring analysis uses two centered measurements, each expressed in its original units. Its sample covariance matrix is

S = \begin{pmatrix} 4 & 2\\ 2 & 4 \end{pmatrix}.

Find the two eigenvalues, the variance of the scores along each unit eigenvector, and the total variance. Use the eigenvalue method, and check the result using the diagonal entries of \(S\).

Step 1: Find the eigenvalues. The characteristic equation is

\det(S-\lambda I) = (4-\lambda)^2-4 = \lambda^2-8\lambda+12 = (\lambda-6)(\lambda-2) = 0.

Thus the eigenvalues are \(6\) and \(2\).

Step 2: Find unit eigenvectors. For \(\lambda=6\), the vector \(a_1=(1,1)^\mathsf{T}/\sqrt{2}\) is an eigenvector: multiplying by \(S\) gives \(Sa_1=(6,6)^\mathsf{T}/\sqrt{2}=6a_1\). Its squared length is \((1^2+1^2)/2=1\). For \(\lambda=2\), \(a_2=(1,-1)^\mathsf{T}/\sqrt{2}\) is an eigenvector, since \(Sa_2=(2,-2)^\mathsf{T}/\sqrt{2}=2a_2\), and its squared length is also \(1\).

Step 3: Interpret the eigenvalues as score variances. For the unit direction \(a_1\), the score variance is \(a_1^\mathsf{T}Sa_1=6\). For \(a_2\), it is \(a_2^\mathsf{T}Sa_2=2\). Both equal their respective eigenvalues, as the derivation requires.

Step 4: Check total variance. The sum of the eigenvalues is \(6+2=8\). The trace, or sum of diagonal entries of \(S\), is \(4+4=8\), so the two calculations agree. Method: use the eigenvalues for the component variances and the trace for the total-variance check. Common mistake avoided: neither covariance entry \(2\) nor either individual variance \(4\) is the variance of a principal component score.

Total Variance and the Sum of Eigenvalues

The total variance represented by a covariance matrix \(S\) is the sum of its diagonal entries, \(\operatorname{tr}(S)\). Each diagonal entry is the sample variance of one of the original variables. The eigenvalue identity follows from the spectral decomposition of a symmetric covariance matrix.

Let the orthonormal eigenvectors of \(S\) be the columns of \(Q\), and let their eigenvalues be \(\lambda_1,\ldots,\lambda_p\). Then \(Q^\mathsf{T}Q=I\), and the spectral decomposition is \(S=Q\Lambda Q^\mathsf{T}\), where \(\Lambda\) is diagonal with entries \(\lambda_1,\ldots,\lambda_p\). The trace of a product is unchanged when its factors are cyclically rearranged. Applying that identity yields:

\operatorname{tr}(S) = \operatorname{tr}(Q\Lambda Q^\mathsf{T}) = \operatorname{tr}(\Lambda Q^\mathsf{T}Q) = \operatorname{tr}(\Lambda) = \sum_{j=1}^{p}\lambda_j.

This argument includes every eigenvalue of the \(p \times p\) covariance matrix. If some eigenvalues are zero, they contribute zero to the total but must still be accounted for when the complete list of eigenvalues is being checked. The identity applies to the matrix actually decomposed: for covariance PCA, it gives the sum of the original variables’ sample variances. If a different matrix is analyzed, its trace is the relevant total.

Worked Example: Checking Total Variance With Three Eigenvalues

A hypothetical risk study analyzes a centered data set using the covariance matrix

S = \begin{pmatrix} 5.5 & 2.5 & 0\\ 2.5 & 5.5 & 0\\ 0 & 0 & 1 \end{pmatrix}.

The directions \(a_1=(1,1,0)^\mathsf{T}/\sqrt{2}\), \(a_2=(1,-1,0)^\mathsf{T}/\sqrt{2}\), and \(a_3=(0,0,1)^\mathsf{T}\) are unit and mutually orthogonal. Determine their eigenvalues, the corresponding score variances, and the total variance. Use \(Sa_j\) to verify each eigenvalue.

For \(a_1\): Multiplication gives \(Sa_1=(8,8,0)^\mathsf{T}/\sqrt{2}=8a_1\). Thus its eigenvalue is \(8\), and its score variance is \(8\).

For \(a_2\): Multiplication gives \(Sa_2=(3,-3,0)^\mathsf{T}/\sqrt{2}=3a_2\). Thus its eigenvalue is \(3\), and its score variance is \(3\).

For \(a_3\): Multiplication gives \(Sa_3=(0,0,1)^\mathsf{T}=1a_3\). Thus its eigenvalue is \(1\), and its score variance is \(1\).

The sum of the eigenvalues is \(8+3+1=12\). The trace is \(5.5+5.5+1=12\), so the total-variance identity checks. Method: verify each eigenpair by multiplication, then sum the eigenvalues and compare with the trace. Common mistake avoided: the off-diagonal covariances \(2.5\) are not added when computing the trace.

Why Unit Length and the Full Eigenvalue List Matter

An eigenvector represents a direction, and multiplying it by any nonzero constant leaves that direction unchanged. But the score \(x_i^\mathsf{T}a\) changes by the same constant. Its variance therefore changes by the square of that constant. PCA uses unit eigenvectors so component variances have a standard scale and equal the eigenvalues directly.

The total-variance identity is also a diagnostic check. A discrepancy between the sum of eigenvalues and the trace can indicate that an eigenvalue was omitted, a covariance matrix was calculated with a different scaling convention, or arithmetic was incorrect. A zero eigenvalue is not an exception: it means there is no sample variation along that eigenvector, and including it leaves the sum unchanged.

Worked Example: Correcting a Nonunit-Eigenvector Calculation

For the covariance matrix \(S=\begin{pmatrix}4&2\\2&4\end{pmatrix}\), take the eigenvector direction \(a=(1,1)^\mathsf{T}/\sqrt{2}\), whose eigenvalue is \(6\). An analyst instead uses \(b=(2,2)^\mathsf{T}=2\sqrt{2}\,a\) and claims that the variance of the scores along \(b\) must still be \(6\). Find the actual variance and correct the claim.

Step 1: Check the scaling. The squared length of \(b\) is \(2^2+2^2=8\). Since \(b\) is an eigenvector in the same direction, \(Sb=6b\).

Step 2: Apply the general variance formula. The score variance for direction \(b\) is

b^\mathsf{T}Sb = b^\mathsf{T}(6b) = 6(b^\mathsf{T}b) = 6(8) = 48.

Thus the variance is \(48\), not \(6\). If the direction is normalized, \(b/\sqrt{8}=a\), its score variance is \(6\). Method: use \(a^\mathsf{T}Sa\) for any direction and use the eigenvalue equality only after verifying unit length. Common mistake avoided: an eigenvalue is the score variance for a unit eigenvector, not for every rescaling of that eigenvector.

Common Checks and Interpretation

For a covariance matrix, the eigenvalues are nonnegative, so each component variance is nonnegative. A negative eigenvalue in a purported covariance-matrix decomposition signals a computational or numerical issue, apart from tiny rounding artifacts in numerical software. In exact arithmetic, a covariance matrix is positive semidefinite.

Keep the two variance statements distinct. For a specified unit eigenvector, its score variance is one eigenvalue. Total variance is the sum of the variances of all original variables, equivalently the sum of all eigenvalues. These statements use the same covariance matrix and the same sample scaling. Do not compare eigenvalues from one matrix with the trace of another.

Check that the eigenvector is unit length before equating its score variance to its eigenvalue.
Use the general formula \(a^\mathsf{T}Sa\) when the direction is not an eigenvector or is not normalized.
For total variance, compare the sum of the complete eigenvalue list with the trace of the matrix decomposed.
Retain zero eigenvalues in the list when checking a full decomposition; they contribute zero but may reflect redundant directions or limited data rank.

Key takeaway: For a centered data matrix with sample covariance \(S\), the score variance in any direction \(a\) is \(a^\mathsf{T}Sa\). If \(a\) is a unit eigenvector with eigenvalue \(\lambda\), that variance is \(\lambda\). Across all \(p\) eigenvalues, \(\sum_{j=1}^{p}\lambda_j=\operatorname{tr}(S)\).

Check Your Understanding

For each problem, identify the applicable variance or trace calculation before checking the final answer.

A covariance matrix is \(S=\begin{pmatrix}7&3\\3&7\end{pmatrix}\). The direction \(a=(1,1)^\mathsf{T}/\sqrt{2}\) is a unit eigenvector. Find its score variance and the total variance. (Answer: score variance \(10\); total variance \(14\))
A covariance matrix has eigenvalues \(6,2,0\). Find its total variance, including the zero eigenvalue. (Answer: \(8\))
A covariance matrix is \(S=\operatorname{diag}(9,4,1)\), and \(a=(0,1,0)^\mathsf{T}\). Find the score variance in direction \(a\) and the total variance. (Answer: score variance \(4\); total variance \(14\))
A unit eigenvector has eigenvalue \(5\). A direction \(b\) is three times that unit eigenvector. Find the score variance along \(b\). (Answer: \(45\))
A centered data set has a \(4 \times 4\) correlation matrix. Without knowing its individual eigenvalues, find their sum. (Answer: \(4\))

What You'll Learn