Choosing Components by Cumulative Variance

Why Use a Cumulative Variance Threshold?

PCA can replace a set of original variables with a smaller set of principal component scores. The practical question is often how many components to keep. A cumulative explained-variance threshold gives a clear numerical rule: retain enough leading components for their combined explained variance to reach a specified proportion of the total.

The threshold does not determine a universally correct answer. A higher threshold retains at least as much of the variance represented by the PCA as a lower threshold, but it will generally require at least as many components. The appropriate threshold depends on the purpose of the analysis and should be stated rather than assumed.

Definition: Let the eigenvalues of a PCA variance matrix be \(\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_p \geq 0\), where \(p\) is the number of variables. If the total variance \(\sum_{j=1}^{p}\lambda_j\) is positive, the explained-variance proportion of component \(j\) is \(\lambda_j \big/ \sum_{r=1}^{p}\lambda_r\). The cumulative explained variance through component \(k\) is the sum of the first \(k\) explained-variance proportions. For a stated threshold \(\tau\), retain the smallest \(k\) for which that cumulative proportion is at least \(\tau\).

From Eigenvalues to a Component Count

PCA orders components from largest eigenvalue to smallest. Each eigenvalue measures the variance along its corresponding component, using the covariance matrix for covariance-based PCA or the correlation matrix for correlation-based PCA. To calculate cumulative explained variance, first find the sum of all eigenvalues, then divide each eigenvalue by that sum, and finally add the proportions in component order.

For covariance-based PCA, the sum of eigenvalues equals the sum of the variances of the original variables. For correlation-based PCA with \(p\) variables standardized to unit variance, the sum is \(p\). These totals are not interchangeable: use the eigenvalues from the variance matrix actually used for the PCA. The eigenvalue proportions are unchanged if all eigenvalues are multiplied by the same positive constant.

T=\sum_{j=1}^{p}\lambda_j,\qquad e_j=\frac{\lambda_j}{T},\qquad C_k=\sum_{j=1}^{k}e_j=\frac{\sum_{j=1}^{k}\lambda_j}{T}

Here \(T\) is total variance, \(e_j\) is the proportion explained by component \(j\), and \(C_k\) is the cumulative proportion through component \(k\). For a threshold \(\tau\) between 0 and 1, the selected count is the first \(k\) with \(C_k \geq \tau\). If the threshold is stated as a percentage, convert it to a proportion before comparing it with \(C_k\).

Confirm the PCA variance matrix.
Determine whether the eigenvalues come from the covariance matrix or the correlation matrix, and use that set of eigenvalues consistently.

Find the total.
Add all eigenvalues, including those for components that may not be retained.

Calculate cumulative proportions.
Divide each leading eigenvalue by the total and accumulate the proportions in descending-eigenvalue order.

Apply the threshold.
Choose the first component count \(k\) for which \(C_k\) is at least \(\tau\), using adequate precision before making the decision.

Why the Calculation Works

The denominator is total variance because the component directions form an orthogonal coordinate system for the centered data. For a covariance matrix \(\Sigma\), its trace, the sum of its diagonal entries, is the sum of the original-variable variances. The spectral decomposition gives the same trace as the sum of the eigenvalues. Since component \(j\) has variance \(\lambda_j\), the first \(k\) components together account for \(\sum_{j=1}^{k}\lambda_j\) of the total \(\sum_{j=1}^{p}\lambda_j\).

\operatorname{tr}(\Sigma)=\sum_{j=1}^{p}\operatorname{Var}(X_j) =\sum_{j=1}^{p}\lambda_j \quad\Longrightarrow\quad \text{fraction represented by the first } k =\frac{\sum_{j=1}^{k}\lambda_j}{\sum_{j=1}^{p}\lambda_j}

This argument applies to a positive-semidefinite covariance or correlation matrix with positive total variance. It does not require the data to have a normal distribution. If all eigenvalues are zero, total variance is zero and the explained-variance proportions are undefined; there is no positive variance to summarize by this calculation.

Worked Example: Reaching a 90% Threshold

A hypothetical insurer summarizes four centered numeric indicators using covariance-based PCA. The eigenvalues, in descending order, are \(4\), \(2\), \(1\), and \(0.5\). Determine the smallest number of components that reaches a 90% cumulative explained-variance threshold. State the proportion retained and the proportion not retained.

Method. Divide cumulative eigenvalue sums by the sum of all four eigenvalues. This uses total variance as the denominator, rather than treating the threshold as a requirement for any single component.

Total variance. \(T=4+2+1+0.5=7.5\).

Individual proportions. The proportions are \(4/7.5=0.5333\), \(2/7.5=0.2667\), \(1/7.5=0.1333\), and \(0.5/7.5=0.0667\), to four decimal places.

Cumulative proportions. After one component, \(C_1=4/7.5=0.5333\). After two, \(C_2=(4+2)/7.5=6/7.5=0.8000\). After three, \(C_3=(4+2+1)/7.5=7/7.5=0.9333\). All four give \(C_4=7.5/7.5=1.0000\).

The first two components fall short of 0.90, and the first three exceed it. Therefore, retain three components. They represent \(93.33\%\) of the variance, leaving \(1-0.9333=0.0667\), or \(6.67\%\), outside the retained components. As a check, \(7/7.5+0.5/7.5=1\). A common mistake would be to select two components because they explain a large majority of the variance; the stated requirement is at least 90%, not merely a majority.

How the Threshold Changes the Decision

The component count is a function of both the eigenvalue pattern and the threshold. With the same eigenvalues, increasing the threshold cannot reduce the selected count: every cumulative proportion is fixed, so a stricter target can only be reached at the same or a later component. A small increase in the threshold can add a full component if the current cumulative value lies close to the new target.

The count also gives a simple measure of the dimension retained: keeping \(k\) components represents the data with \(k\) component scores rather than \(p\) original-variable coordinates. This is a statement about the number of coordinates and the variance fraction represented under the PCA calculation. It does not, by itself, establish that the chosen components are useful for a particular downstream task.

In R, when reading the output from prcomp, the component standard deviations can be squared to obtain the corresponding eigenvalues, and the explained-variance calculation can then be applied. Take care to interpret the output in light of whether the variables were centered only or also scaled: those choices determine which variance matrix the PCA represents.

Worked Example: Comparing Thresholds for Standardized Variables

A hypothetical portfolio review uses correlation-based PCA on five standardized numeric indicators. The eigenvalues are \(2.4\), \(1.3\), \(0.8\), \(0.4\), and \(0.1\). Determine the component counts for thresholds of 85%, 90%, and 95%, and report the retained dimension as a fraction of the five-variable dimension at each threshold.

Method. Because this is correlation-based PCA on five standardized variables, total variance is five. Confirming from the eigenvalues, \(2.4+1.3+0.8+0.4+0.1=5.0\).

Cumulative proportions. The first component gives \(2.4/5=0.48\). The first two give \((2.4+1.3)/5=3.7/5=0.74\). The first three give \(4.5/5=0.90\). The first four give \(4.9/5=0.98\), and all five give \(5/5=1.00\).

At 85%, two components are insufficient because \(0.74<0.85\), while three are sufficient because \(0.90\geq0.85\). At 90%, three components meet the threshold exactly. At 95%, three are insufficient and four meet the threshold. The counts are therefore 3, 3, and 4, respectively. Relative to the original five-dimensional representation, these are \(3/5=60\%\), \(3/5=60\%\), and \(4/5=80\%\) of the number of coordinates.

Check. Raising the threshold from 90% to 95% increased the selected count from three to four, while lowering it from 90% to 85% did not change the count. This illustrates why a threshold change does not necessarily change the selected dimension.

Worked Example: An Exact Threshold Boundary

A hypothetical maintenance study has five component eigenvalues \(3.6\), \(2.1\), \(1.5\), \(0.6\), and \(0.2\). Find the component count for a 90% threshold, then find it for a 91% threshold. Explain how rounding could mislead the decision.

Method. Calculate cumulative eigenvalue sums against the full total, retaining sufficient precision to distinguish 90% from 91%.

Total. \(T=3.6+2.1+1.5+0.6+0.2=8.0\).

Cumulative values. The successive cumulative sums are \(3.6\), \(5.7\), \(7.2\), \(7.8\), and \(8.0\). Dividing by \(8.0\) gives \(0.4500\), \(0.7125\), \(0.9000\), \(0.9750\), and \(1.0000\).

For a 90% threshold, the third component reaches exactly \(0.9000\), so the smallest qualifying count is three. For a 91% threshold, three are insufficient because \(0.9000<0.9100\); four qualify because \(0.9750\geq0.9100\). The answer for 91% is four.

A common mistake is to round a cumulative proportion prematurely and then treat “about 90%” as enough for a 91% requirement. The unrounded calculation shows that \(0.9000\) is below \(0.9100\), so four components are required. As a check, the fourth eigenvalue adds \(0.6/8=0.0750\), taking the cumulative proportion from \(0.9000\) to \(0.9750\).

Common Errors to Avoid

Using the wrong total. The denominator is the sum of all eigenvalues for the PCA variance matrix, not just the eigenvalues retained so far. In correlation-based PCA with \(p\) standardized variables, that sum is \(p\).
Stopping at a component’s individual proportion. A threshold applies to the cumulative proportion of the leading components, not necessarily to the proportion for the next component by itself.
Ignoring “at least.” If the cumulative proportion equals the threshold, that component count qualifies. Choose the first \(k\) with \(C_k\geq\tau\), not the first with \(C_k>\tau\).
Rounding before choosing. Keep enough precision for the comparison, particularly when the cumulative proportion is near the threshold. Round the reported percentage after deciding the count.
Mixing PCA inputs. Eigenvalues from a covariance matrix and a correlation matrix describe different variance scales. Do not use the total or interpretation for one matrix with eigenvalues from the other.

Key takeaway: Add eigenvalues in descending order and divide each cumulative sum by the total eigenvalue sum. Select the smallest component count whose cumulative proportion is at least the stated threshold, and report both the retained variance and the resulting dimension.

Check Your Understanding

For each problem, calculate cumulative explained variance before selecting the component count. Use the stated threshold exactly.

A covariance-based PCA has eigenvalues \(5\), \(3\), and \(2\). What is the smallest count reaching a 75% threshold, and what cumulative proportion does it retain? (Answer: 2 components; 80.00%)
A covariance-based PCA has eigenvalues \(1.8\), \(1.2\), \(0.6\), and \(0.4\). Find the smallest count reaching an 88% threshold. (Answer: 3 components; 90.00%)
A PCA has eigenvalues \(2.2\), \(1.7\), \(0.8\), and \(0.3\), and the threshold is 95%. Find the selected count and its cumulative proportion. (Answer: 4 components; 100.00%)
A PCA has eigenvalues \(4\), \(2\), \(1\), and \(1\). For a threshold of exactly 87.5%, determine the smallest count and the retained fraction of the original four-dimensional representation. (Answer: 3 components; 3/4, or 75% of the original dimension)
A correlation-based PCA on six standardized variables has eigenvalues \(2.1\), \(1.5\), \(1.0\), \(0.7\), \(0.5\), and \(0.2\). Find the smallest count reaching 90%, and state the cumulative variance proportion. (Answer: 5 components; 96.67%)

What You'll Learn