Spectral Variance Decomposition and Total Variance

Why the Eigenvalues Account for Total Variance

Principal components are often summarized by saying that they explain a certain amount or proportion of a data set’s variation. That summary has a precise basis: the variance of a score along a principal direction is the corresponding covariance eigenvalue, and the eigenvalues together account for all the variance in the variables. This tutorial establishes those identities and shows how to use them without confusing total variance with the variance of a sum of scores.

Let \(X=(X_1,\ldots,X_p)^T\) be a random vector with finite second moments, mean vector \(\mu=E[X]\), and covariance matrix \(\Sigma=E[(X-\mu)(X-\mu)^T]\). The matrix \(\Sigma\) is symmetric and positive semidefinite, so it has an orthonormal eigenbasis. Write its eigenvalues in decreasing order as \(\lambda_1\geq\cdots\geq\lambda_p\geq 0\), and choose corresponding unit eigenvectors \(v_1,\ldots,v_p\).

Definition: The total variance of \(X\) is \(\operatorname{tr}(\Sigma)\), the trace of its covariance matrix. Since the diagonal entries of \(\Sigma\) are the marginal variances, \(\operatorname{tr}(\Sigma)=\sum_{j=1}^p\operatorname{Var}(X_j)\). The principal score in direction \(v_i\) is \(Z_i=v_i^T(X-\mu)\).

Total variance is therefore the sum of the variances of the original coordinates. It is also the sum of the variances of scores in any orthonormal coordinate system. Principal components are especially useful because their directions order this same total into nonincreasing amounts.

The Spectral Variance Decomposition

Theorem: Let \(X\) have finite second moments and covariance matrix \(\Sigma\), with eigenvalues \(\lambda_1,\ldots,\lambda_p\) and an orthonormal eigenbasis \(v_1,\ldots,v_p\). Then

\(\operatorname{tr}(\Sigma)=\sum_{i=1}^p\lambda_i\), and \(\operatorname{Var}(v_i^T(X-\mu))=\lambda_i\) for each \(i\). Moreover, distinct principal scores are uncorrelated. Consequently, for any \(k\) with \(1\leq k\leq p\), the first \(k\) principal directions account for \(\sum_{i=1}^k\lambda_i\) of total variance. If \(\operatorname{tr}(\Sigma)>0\), their proportion of total variance is \(\sum_{i=1}^k\lambda_i/\operatorname{tr}(\Sigma)\).

Here is a direct proof. The spectral decomposition is \(\Sigma=V\Lambda V^T\), where \(V\) has the eigenvectors as columns and \(\Lambda\) is diagonal with entries \(\lambda_i\). The \(j\)th diagonal entry of \(\Sigma\) is \(\sum_{i=1}^p v_{ji}^2\lambda_i\), so summing over \(j\) gives

\operatorname{tr}(\Sigma) =\sum_{j=1}^p\sum_{i=1}^p v_{ji}^2\lambda_i =\sum_{i=1}^p\lambda_i\left(\sum_{j=1}^p v_{ji}^2\right) =\sum_{i=1}^p\lambda_i,

because each eigenvector \(v_i\) has unit length, so \(\sum_j v_{ji}^2=1\). For a score \(Z_i=v_i^T(X-\mu)\), the variance rule for a linear combination gives

\operatorname{Var}(Z_i)=v_i^T\Sigma v_i =v_i^T(\lambda_i v_i) =\lambda_i v_i^Tv_i =\lambda_i.

For \(i\neq j\), the covariance of two scores is

\operatorname{Cov}(Z_i,Z_j) =v_i^T\Sigma v_j =v_i^T(\lambda_j v_j) =\lambda_j v_i^Tv_j =0,

since the eigenvectors are orthogonal. Thus the variance along each principal direction is its eigenvalue, and adding those variances gives the trace. Summing only the first \(k\) of them gives the stated amount and, when total variance is positive, dividing by the trace gives the stated proportion. \(\square\)

The same total-variance identity does not depend on choosing principal directions. If \(q_1,\ldots,q_p\) is any orthonormal basis, define \(W_i=q_i^T(X-\mu)\). Then

\sum_{i=1}^p\operatorname{Var}(W_i) =\sum_{i=1}^p q_i^T\Sigma q_i =\operatorname{tr}(\Sigma).

To verify the last equality, let \(Q\) be the matrix with columns \(q_i\). Orthonormality gives \(QQ^T=I\), and summing the diagonal entries of \(Q^T\Sigma Q\) gives \(\operatorname{tr}(Q^T\Sigma Q)=\operatorname{tr}(\Sigma QQ^T)=\operatorname{tr}(\Sigma)\). Principal directions therefore do not create total variance; they arrange its contributions along directions with the largest contributions first.

Worked Example: Two Correlated Variables

Suppose \(X\) is a centered bivariate normal random vector with covariance matrix

\Sigma= \begin{pmatrix} 4&1\\ 1&4 \end{pmatrix}.

The total variance is the trace, \(4+4=8\). To find the eigenvalues, solve

\det(\Sigma-\lambda I) =(4-\lambda)^2-1 =(3-\lambda)(5-\lambda)=0.

Thus \(\lambda_1=5\) and \(\lambda_2=3\), whose sum is \(8\), as required. Unit eigenvectors are \(v_1=(1,1)^T/\sqrt{2}\) and \(v_2=(1,-1)^T/\sqrt{2}\). For the scores \(Z_1=v_1^TX\) and \(Z_2=v_2^TX\), direct calculation gives

\operatorname{Var}(Z_1) =\frac{1}{2}(4+1+1+4)=5, \qquad \operatorname{Var}(Z_2) =\frac{1}{2}(4-1-1+4)=3.

The first direction accounts for \(5/8=0.625\), or \(62.5\%\), of total variance; the second accounts for \(3/8=0.375\), or \(37.5\%\). The two contributions add to \(100\%\).

Worked Example: The Sum Is the Same in Another Orthonormal Basis

Let \(X\) have independent, mean-zero coordinates with variances \(6,3,1\). Its covariance matrix is \(\Sigma=\operatorname{diag}(6,3,1)\), so its total variance is \(6+3+1=10\). Consider the following three directions:

q_1=\frac{(1,1,1)^T}{\sqrt{3}},\qquad q_2=\frac{(1,-1,0)^T}{\sqrt{2}},\qquad q_3=\frac{(1,1,-2)^T}{\sqrt{6}}.

Each vector has unit length. Their pairwise inner products are \(q_1^Tq_2=(1-1+0)/\sqrt{6}=0\), \(q_1^Tq_3=(1+1-2)/\sqrt{18}=0\), and \(q_2^Tq_3=(1-1+0)/\sqrt{12}=0\). They form an orthonormal basis. The variance of a score \(q^TX\) is \(q^T\Sigma q\), so the three score variances are

\operatorname{Var}(q_1^TX)=\frac{6+3+1}{3}=\frac{10}{3}, \qquad \operatorname{Var}(q_2^TX)=\frac{6+3}{2}=\frac{9}{2}, \qquad \operatorname{Var}(q_3^TX)=\frac{6+3+4}{6}=\frac{13}{6}.

Their sum is \(10/3+9/2+13/6=(20+27+13)/6=10\), exactly the total variance. These directions are not the eigenvectors of this diagonal covariance matrix, so their individual score variances need not be its eigenvalues. The sum remains the same because the directions are orthonormal.

Variance Proportions and Zero-Eigenvalue Directions

When \(\operatorname{tr}(\Sigma)>0\), the proportion \(\lambda_i/\operatorname{tr}(\Sigma)\) describes the fraction of total variance along the \(i\)th principal direction. These proportions are nonnegative and sum to one. For the first \(k\) directions, the cumulative proportion is \(\sum_{i=1}^k\lambda_i/\operatorname{tr}(\Sigma)\). If all eigenvalues are zero, then total variance is zero and such proportions are undefined: there is no positive total to use as a denominator.

An eigenvalue of zero has a precise interpretation. The score in its direction has variance zero and, being centered, equals zero with probability one. The variables have no random variation in that direction. This can occur when some coordinates are exact linear combinations of others; it does not invalidate the decomposition.

Worked Example: A Direction With Zero Variance

Let \(Z\sim N(0,1)\) and \(W\sim N(0,4)\) be independent, and set \(X=(Z,Z,W)^T\). Its mean is zero, and its covariance matrix is

\Sigma= \begin{pmatrix} 1&1&0\\ 1&1&0\\ 0&0&4 \end{pmatrix}.

The direction \(v_1=(0,0,1)^T\) has eigenvalue \(4\). The direction \(v_2=(1,1,0)^T/\sqrt{2}\) has eigenvalue \(2\), because multiplying \(\Sigma v_2\) gives \((2,2,0)^T/\sqrt{2}=2v_2\). The direction \(v_3=(1,-1,0)^T/\sqrt{2}\) has eigenvalue \(0\), because the first two coordinates of \(X\) are equal and \(\Sigma v_3=0\). These are orthonormal directions, with eigenvalues \(4,2,0\). Their sum is \(6\), matching the trace \(1+1+4=6\).

The corresponding centered scores are \(W\), \(\sqrt{2}Z\), and \(0\). Their variances are \(4,2,0\), respectively. The first direction accounts for \(4/6=2/3\) of total variance, the second for \(2/6=1/3\), and the third for none.

Common Mistake: Confusing Total Variance With a Score Sum

A common error is to say that total variance is the variance of the sum of the principal scores. That is generally false. Total variance is the sum of the score variances, not the variance of their ordinary sum. For example, in the first worked example the scores are uncorrelated with variances \(5\) and \(3\), so the variance of their sum is \(5+3+2\operatorname{Cov}(Z_1,Z_2)=8\). But in a different orthonormal basis the scores need not be uncorrelated, and the variance of their sum includes covariance terms. The invariant identity is the sum of variances across the full orthonormal basis.

A related mistake is to treat the eigenvalues as percentages without dividing by total variance. An eigenvalue is an amount of variance, in squared units of the variables. Only the ratio \(\lambda_i/\operatorname{tr}(\Sigma)\) is a proportion, and that ratio requires positive total variance.

Key takeaway: The trace of a covariance matrix is total variance. In an eigenbasis, each principal score contributes exactly its eigenvalue, and the full set of contributions sums to the trace. The first \(k\) eigenvalues give the variance accounted for by the first \(k\) principal directions.

Check Your Understanding

Use the definitions and identities above to answer the following questions.

For a covariance matrix with eigenvalues \(7,2,1\), what is the total variance, and what proportion is accounted for by the first two principal directions?
Why does the sum of score variances equal total variance for any orthonormal basis, not only an eigenbasis?
If an eigenvalue is zero, what does that imply about the centered score in its eigenvector direction?
Explain why the variance of the sum of two scores is not generally the definition of total variance.
What is the correct interpretation of \(\lambda_i/\operatorname{tr}(\Sigma)\), and under what condition is this ratio defined?

Spectral Variance Decomposition and Total Variance

What You'll Learn