Explained Variance and Its Decomposition

Why Explained Variance Is a Useful PCA Summary

Principal component analysis (PCA) replaces a set of measured variables with ordered directions of variation. Each component has an eigenvalue that measures the variance in its direction. To assess how much of the data’s total variation a component represents, compare its eigenvalue with the total variance across all variables.

That comparison depends on a fundamental identity: the total variance of a covariance matrix equals the sum of its eigenvalues. This tutorial derives the identity and uses it to calculate each component’s explained-variance proportion and the cumulative total through successive components. These calculations are aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: For a covariance matrix \(S\) of \(p\) numeric variables, the total variance is the sum of the variances of those variables. Since the variances appear on the diagonal of \(S\), total variance is \(\operatorname{tr}(S)\), the trace of \(S\). If the ordered eigenvalues are \(\lambda_1 \ge \lambda_2 \ge \cdots \ge \lambda_p \ge 0\) and the total variance is positive, the explained-variance proportion for component \(j\) is \(\lambda_j/\sum_{k=1}^{p}\lambda_k\). The cumulative proportion through component \(j\) is \(\sum_{k=1}^{j}\lambda_k/\sum_{k=1}^{p}\lambda_k\).

Total Variance and the Eigenvalues

Assume the data have \(n\) observations on \(p\) numeric variables. For covariance-based PCA, let \(X_c\) be the \(n \times p\) data matrix after subtracting each variable’s sample mean, and use the sample covariance matrix

S=\frac{1}{n-1}X_c^\mathsf{T}X_c.

This matrix is symmetric and positive semidefinite. Symmetry means \(S^\mathsf{T}=S\); positive semidefiniteness means \(z^\mathsf{T}Sz \ge 0\) for every vector \(z\). Consequently, its eigenvalues are real and nonnegative, and its eigenvectors can be chosen to form an orthonormal basis. These are the conditions needed for the decomposition below.

Let \(Q\) be the matrix whose columns are the orthonormal eigenvectors of \(S\), and let \(\Lambda\) be the diagonal matrix of the corresponding eigenvalues. Then \(S=Q\Lambda Q^\mathsf{T}\). The trace of \(S\) is the sum of its diagonal entries, which are the individual variable variances. The trace also equals the sum of the eigenvalues:

\begin{aligned} \operatorname{tr}(S) &=\sum_{i=1}^{p} S_{ii}\\ &=\sum_{i=1}^{p}\sum_{j=1}^{p}q_{ij}^{\,2}\lambda_j\\ &=\sum_{j=1}^{p}\lambda_j\left(\sum_{i=1}^{p}q_{ij}^{\,2}\right)\\ &=\sum_{j=1}^{p}\lambda_j. \end{aligned}

The second line follows from \(S=Q\Lambda Q^\mathsf{T}\). In the third line, the sums are rearranged. Each column of \(Q\) has length one, so \(\sum_{i=1}^{p}q_{ij}^{\,2}=1\), giving the last line. Thus the sum of the eigenvalues is exactly the total variance, not an approximation.

Theorem: For a real symmetric covariance matrix with positive total variance, the explained-variance proportion of component \(j\) is \(\lambda_j/\operatorname{tr}(S)\). The proportions are nonnegative and sum to 1. Their cumulative totals are nondecreasing and end at 1, apart from any displayed rounding.

A zero eigenvalue contributes no variance in its direction. If every eigenvalue is zero, then \(\operatorname{tr}(S)=0\), so explained-variance proportions are undefined: there is no variation to divide into proportions. In ordinary nondegenerate PCA, the total variance is positive.

From Eigenvalues to Proportions and Cumulative Totals

Once the eigenvalues are ordered from largest to smallest, each proportion uses the same denominator: their sum, which is the total variance. The cumulative proportion through component \(j\) adds the first \(j\) eigenvalues before dividing by that same total. Keeping the denominator fixed is important; the cumulative calculation is not a sequence of proportions with changing denominators.

\text{Explained proportion}_j =\frac{\lambda_j}{\sum_{k=1}^{p}\lambda_k}, \qquad \text{Cumulative proportion}_j =\frac{\sum_{k=1}^{j}\lambda_k}{\sum_{k=1}^{p}\lambda_k}.

The cumulative total gives a compact description of how the ordered components account for variation. It is a calculation, not by itself a rule for deciding how many components to retain; that decision depends on the analytical purpose and other considerations.

Worked Example: Checking the Decomposition for a Covariance Matrix

A hypothetical study records two centered measurements and a third centered measurement on each observation. Its sample covariance matrix is

S= \begin{pmatrix} 3&1&0\\ 1&3&0\\ 0&0&2 \end{pmatrix}.

Calculate the total variance from the diagonal, find the eigenvalues, and report each explained-variance proportion and cumulative proportion. Use proportions to four decimal places where needed.

Method. First calculate the trace. Then find the eigenvalues and verify their sum against the trace before calculating the proportions.

Total variance. The diagonal entries are 3, 3, and 2, so

\operatorname{tr}(S)=3+3+2=8.

Eigenvalues. The characteristic determinant separates into a two-by-two block and a one-by-one block:

\begin{aligned} \det(S-\lambda I) &=\big((3-\lambda)^2-1\big)(2-\lambda)\\ &=(2-\lambda)(4-\lambda)(2-\lambda). \end{aligned}

Therefore the eigenvalues, in descending order, are \(4, 2, 2\). Their sum is \(4+2+2=8\), which agrees with the trace.

Proportions and cumulative totals. Divide each eigenvalue by 8. The component proportions are \(4/8=0.5\), \(2/8=0.25\), and \(2/8=0.25\). Adding them in order gives cumulative proportions \(0.5\), \(0.5+0.25=0.75\), and \(0.75+0.25=1\). Thus the results are 0.5000, 0.2500, 0.2500 and cumulative totals 0.5000, 0.7500, 1.0000.

Check and trap avoided. The proportions sum to 1 and the eigenvalues sum to the matrix’s total variance of 8. A common error is to divide by the largest eigenvalue, 4, rather than by the total variance; that would incorrectly assign the first component a proportion of \(4/4=1\), instead of \(4/8=0.5\).

Correlation-Based PCA Has a Different Total Variance

For correlation-based PCA, each variable is standardized to have variance 1 before the correlation matrix is analyzed. A \(p \times p\) correlation matrix therefore has \(p\) diagonal entries equal to 1, and its trace—and hence the sum of its eigenvalues—is \(p\). This denominator differs from the trace of an unstandardized covariance matrix, whose total depends on the original variable variances.

The same proportion formula applies to either matrix: use the eigenvalues of the matrix actually analyzed, and divide by their sum. Do not use a covariance matrix’s total variance as the denominator for eigenvalues obtained from a correlation matrix, or vice versa.

Worked Example: Explained Variance for a Correlation Matrix

A hypothetical PCA uses the correlation matrix of three standardized measurements. Each pair has correlation 0.5, so the matrix is

R= \begin{pmatrix} 1&0.5&0.5\\ 0.5&1&0.5\\ 0.5&0.5&1 \end{pmatrix}.

Find the eigenvalues, each explained-variance proportion, and the cumulative proportions. Report proportions to four decimal places.

Method. The vector with all three entries equal has eigenvalue 2: each row of \(R\) sums to \(1+0.5+0.5=2\). Any vector whose entries sum to zero has eigenvalue 0.5, because \(R=0.5I+0.5J\), where \(J\) is the all-ones matrix, and \(J\) sends such a vector to zero. There are two independent directions whose entries sum to zero. Thus the ordered eigenvalues are \(2, 0.5, 0.5\).

Total and proportions. The matrix trace is \(1+1+1=3\), and the eigenvalues also sum to \(2+0.5+0.5=3\). Divide each eigenvalue by 3: \(2/3=0.6667\), \(0.5/3=0.1667\), and \(0.5/3=0.1667\), to four decimal places. The cumulative proportions are \(2/3=0.6667\), \(2.5/3=0.8333\), and \(3/3=1.0000\).

Check and trap avoided. The unrounded proportions sum to \(2/3+1/6+1/6=1\). A frequent mistake is to treat the first eigenvalue, 2, as the total variance and report \(2/2=1\). The denominator is the trace, 3, so the correct first proportion is \(2/3\).

Calculating a Full Explained-Variance Summary

In practice, an analysis may provide ordered eigenvalues rather than the full matrix. The calculation is then direct, but it is still useful to check that their sum agrees with the reported total variance. The example below also shows why it is helpful to keep unrounded values until the cumulative calculations are complete.

Worked Example: Building a Cumulative Summary from Reported Eigenvalues

A hypothetical covariance-based PCA uses five measurements whose sample variances are 3.0, 2.5, 2.0, 1.5, and 1.0. The reported eigenvalues, already ordered from largest to smallest, are 4.8, 2.7, 1.5, 0.7, and 0.3. Calculate the individual and cumulative explained-variance proportions. Give proportions to four decimal places and percentages to two decimal places.

Method and total check. The diagonal variances sum to \(3.0+2.5+2.0+1.5+1.0=10.0\). The eigenvalues also sum to \(4.8+2.7+1.5+0.7+0.3=10.0\), so the two calculations of total variance agree.

Individual proportions. Divide each eigenvalue by 10.0: the proportions are \(4.8/10=0.4800\), \(2.7/10=0.2700\), \(1.5/10=0.1500\), \(0.7/10=0.0700\), and \(0.3/10=0.0300\). As percentages, these are 48.00%, 27.00%, 15.00%, 7.00%, and 3.00%.

Cumulative proportions. Add the eigenvalues successively: \(4.8\), \(4.8+2.7=7.5\), \(7.5+1.5=9.0\), \(9.0+0.7=9.7\), and \(9.7+0.3=10.0\). Dividing these totals by 10 gives cumulative proportions of 0.4800, 0.7500, 0.9000, 0.9700, and 1.0000. Equivalently, the cumulative percentages are 48.00%, 75.00%, 90.00%, 97.00%, and 100.00%.

Check and trap avoided. The last cumulative proportion is exactly 1, and the individual proportions add to 1. Do not round intermediate cumulative sums and then reuse rounded values; calculate from the eigenvalues or unrounded proportions, and round only for presentation.

Common Interpretation and Calculation Traps

Using the wrong denominator. Divide by the sum of all eigenvalues, not by the largest eigenvalue or by the number of components. For a correlation matrix, that sum is \(p\); for a covariance matrix, calculate the trace from its variances.
Mixing matrices and eigenvalues. Eigenvalues from a correlation matrix describe standardized variables. They must be compared with the correlation matrix’s total variance, not with the original covariance matrix’s trace.
Ignoring component order. Cumulative proportions add eigenvalues in descending order. If values are not already ordered, sort them before forming a cumulative summary.
Expecting displayed rounded values to add exactly. Rounding can make displayed values sum to slightly less or more than 1. Check the unrounded values, and state that the discrepancy is due to rounding when relevant.
Treating a proportion as a causal explanation. An explained-variance proportion describes how much variance is represented by a component. It does not establish why the observed variables vary or imply a causal relationship.

Key takeaway: For a symmetric covariance or correlation matrix, total variance is the trace and equals the sum of its eigenvalues. Divide each ordered eigenvalue by that total for its explained-variance proportion; add the ordered eigenvalues first to obtain cumulative proportions.

Check Your Understanding

For each problem, calculate the requested values using unrounded quantities and report proportions to four decimal places where needed.

A covariance matrix has ordered eigenvalues 3.6, 1.2, and 0.2. Find the total variance, each explained-variance proportion, and the cumulative proportions. (Answer: total 5.0; proportions 0.7200, 0.2400, 0.0400; cumulative 0.7200, 0.9600, 1.0000.)
A correlation-based PCA has four ordered eigenvalues 2.4, 0.8, 0.5, and 0.3. Calculate the individual and cumulative proportions, and check the total against the matrix dimension. (Answer: total 4; proportions 0.6000, 0.2000, 0.1250, 0.0750; cumulative 0.6000, 0.8000, 0.9250, 1.0000.)
A covariance matrix has diagonal entries 5, 3, and 2, and ordered eigenvalues 6, 3, and 1. Calculate the total variance by both methods and then the proportions. (Answer: both totals are 10; proportions 0.6000, 0.3000, 0.1000; cumulative 0.6000, 0.9000, 1.0000.)
A four-variable covariance analysis reports eigenvalues 2.6, 1.2, 0.5, and 0.2. The trace calculated from the covariance matrix is 4.5. Find the individual and cumulative proportions. (Answer: proportions 0.5778, 0.2667, 0.1111, 0.0444; cumulative 0.5778, 0.8444, 0.9556, 1.0000.)
A correlation matrix has three variables, but a purported complete eigenvalue list is 2.1, 0.6, and 0.2. Check whether the list is consistent with the matrix’s total variance. (Answer: no; the eigenvalues sum to 2.9, whereas a three-variable correlation matrix has trace 3, so the list is inconsistent as stated.)

What You'll Learn