Why Component Selection Requires Judgment
Principal component analysis (PCA) orders components from the greatest variance explained to the least. The first component explains as much sample variation as possible among one-dimensional projections; each subsequent component explains the greatest remaining variation subject to orthogonality. The practical question is how many of these ordered components to retain. A scree plot and a cumulative explained-variance table help answer that question, but they do not produce a universally correct answer.
A defensible choice uses the analysis goal, the shape of the eigenvalues, and any stated variance-retention target. Different rules can suggest different counts. That disagreement is not necessarily an error: it can show that the data do not have a sharply defined cutoff.
Why Cumulative Explained Variance Uses Eigenvalue Sums
The rule follows from the relationship between a matrix’s trace and its eigenvalues. For a covariance matrix, the trace is the sum of the variances of the original variables, so it is the total variance in the data. The eigenvalue decomposition expresses that same total as the sum of the component variances.
Here \(S\) is the covariance matrix, \(Q\) is an orthogonal matrix of eigenvectors, and \(\Lambda\) is the diagonal matrix of eigenvalues. Because \(Q^{T}Q\) is the identity matrix, the trace of \(S\) equals the sum of the diagonal entries of \(\Lambda\). The first \(k\) principal components have variances \(\lambda_1,\ldots,\lambda_k\), so their combined share of total variance is their sum divided by the total. The same reasoning applies when \(S\) is a correlation matrix. The ratio is undefined if total variance is zero; ordinary PCA retention decisions assume there is positive total variance.
Three Common Ways to Choose a Count
No single retention rule is right for every PCA. Each rule answers a slightly different question, and each has limits.
| Method | How it is used | Main limitation |
|---|---|---|
| Scree plot | Look for a visible bend, or “elbow,” after a steep decline in eigenvalues. | The bend can be gradual or ambiguous, and its location involves judgment. |
| Cumulative variance | Retain the smallest \(k\) for which \(C_k\) meets a stated target. | The target is a convention tied to the task, not a universal statistical law. |
| Eigenvalue-greater-than-one rule | For a correlation matrix, retain components with eigenvalues greater than 1. | This is a rule of thumb, not a guarantee; it should not be mechanically applied to a covariance matrix with variables on different scales. |
The eigenvalue-greater-than-one rule is often called the Kaiser criterion. With a correlation matrix of \(p\) nonconstant standardized variables, the total eigenvalue is \(p\), so the average eigenvalue is 1. A component above 1 therefore explains more variance than the average single standardized variable. This interpretation does not make the cutoff a theorem about how many components are useful.
A cumulative target, such as 80% or 90%, should be stated rather than treated as an automatic default. A higher target generally retains more components. More retained components preserve more of the observed variation, but can reduce the compression or simplification that motivated PCA. The relevant balance depends on the intended use and the cost of discarding information.
Worked Example: Reading a Scree Plot and a Cumulative Table
A hypothetical insurer standardizes five claim-process measurements and performs PCA using their correlation matrix. The ordered eigenvalues are \(2.60, 1.25, 0.70, 0.30,\) and \(0.15\). The analyst wants to retain at least 90% of the total variance, but also wants to describe what the scree plot and Kaiser criterion suggest. Calculate the component proportions and cumulative proportions, then state a defensible choice. Name the method and a mistake to avoid.
Step 1: Check the total. The eigenvalues sum to \(2.60+1.25+0.70+0.30+0.15=5.00\). This agrees with the trace of a five-variable correlation matrix, which is 5.
Step 2: Calculate each component’s share. Divide each eigenvalue by 5.00: \(2.60/5.00=0.52\), \(1.25/5.00=0.25\), \(0.70/5.00=0.14\), \(0.30/5.00=0.06\), and \(0.15/5.00=0.03\). As percentages, the shares are 52%, 25%, 14%, 6%, and 3%.
Step 3: Accumulate the percentages. The cumulative values are 52%, \(52\%+25\%=77\%\), \(77\%+14\%=91\%\), \(91\%+6\%=97\%\), and \(97\%+3\%=100\%\).
Step 4: Compare the rules. The 90% target requires three components because \(C_2=77\%<90\%\) and \(C_3=91\%\geq90\%\). The Kaiser criterion retains two because \(2.60>1\) and \(1.25>1\), while the remaining eigenvalues are below 1. In a scree plot, the successive drops are 1.35, 0.55, 0.40, and 0.15. The steep initial decline and later flattening could support an elbow around the second component, although the bend is not perfectly sharp.
Conclusion and check. If meeting the stated 90% threshold is the priority, retain three components. This is a defensible choice even though the Kaiser rule and a possible scree elbow suggest two. Check: the first three eigenvalues sum to \(2.60+1.25+0.70=4.55\), and \(4.55/5.00=0.91\), or 91%. The method is to calculate the cumulative proportion and compare it with the specified target. The common mistake to avoid is reporting that two components meet 90% merely because they pass the Kaiser rule; their cumulative proportion is only 77%.
Interpreting an Elbow Without Treating It as a Test
In a scree plot, the largest eigenvalues usually appear on the left, followed by smaller values. An elbow is a point after which the plot becomes relatively flat: components before it account for comparatively large increments of variance, while later components contribute smaller increments. This is a visual aid, not a formal hypothesis test. A smooth, steady decline may have no convincing elbow. In that case, it is better to say the plot does not identify a clear cutoff than to claim an exact answer from an arbitrary bend.
Read the plot’s axes and scale before interpreting it. The horizontal axis should identify component number, and the vertical axis should identify eigenvalue (or, in some plots, explained-variance proportion). A plot of proportions has the same ordering and bend pattern as a plot of eigenvalues because every eigenvalue is divided by the same positive total. But its vertical values are proportions, not raw component variances.
Also distinguish a component’s individual contribution from its cumulative contribution. If component 3 explains 12%, that is the individual share for component 3; it does not mean that the first three components explain 12%. The cumulative amount is the sum of the first three shares.
Worked Example: A Covariance Matrix and a Target
A hypothetical risk team performs PCA on four measurements in their original units using the covariance matrix. The eigenvalues are \(36, 16, 8,\) and \(4\). A report requests at least 90% cumulative variance. Find the smallest eligible component count and explain whether the Kaiser criterion is appropriate. Name the method and a mistake to avoid.
Step 1: Find total variance. The sum is \(36+16+8+4=64\).
Step 2: Find individual shares. Divide by 64: \(36/64=0.5625\), \(16/64=0.25\), \(8/64=0.125\), and \(4/64=0.0625\). The percentages are 56.25%, 25%, 12.5%, and 6.25%.
Step 3: Find cumulative shares. After one component the total is 56.25%; after two it is \(56.25\%+25\%=81.25\%\); after three it is \(81.25\%+12.5\%=93.75\%\); all four sum to 100%. Thus two components fall short of 90%, while three exceed it. The smallest count meeting the target is three.
Step 4: Assess the other rule. The Kaiser criterion is not appropriate merely because these eigenvalues can be compared with 1. The analysis used a covariance matrix, and the variables are in their original units; the value 1 does not represent the average variance of one standardized variable here. A scree plot might show a substantial decline from 36 to 16 and a smaller decline thereafter, but that visual impression does not alter the arithmetic required by the stated 90% target.
Conclusion and check. Retain three components to meet the report’s target: \((36+16+8)/64=60/64=0.9375\), or 93.75%. The method is the cumulative-variance threshold, using the smallest \(k\) that reaches 90%. The common mistake to avoid is applying “eigenvalue greater than 1” to this unstandardized covariance analysis as if it were a universal cutoff.
Variance Retention Is Not Predictive Validation
PCA is an unsupervised procedure: it identifies directions that account for variation in the predictor measurements, without using a response variable to decide which directions matter for predicting that response. Consequently, a component with a small eigenvalue can still be relevant to a particular outcome, and components with large eigenvalues can capture variation that is not useful for that outcome. The scree plot and cumulative table describe variation in the variables supplied to PCA; neither directly measures predictive performance.
If the purpose is prediction, a retention choice based on explained variance should be treated as a candidate representation, not proof that the representation predicts well. Predictive usefulness needs to be assessed against the prediction objective using suitable held-out data or another appropriate evaluation procedure. This is a separate question from how much total variance the components retain.
Worked Example: A High Variance Share Does Not Settle Predictive Usefulness
A hypothetical actuary standardizes four policyholder measurements before using PCA as a possible input representation for a lapse model. The eigenvalues are \(1.80, 1.10, 0.70,\) and \(0.40\). The team’s exploratory rule is to retain at least 85% of predictor variance. In a separate held-out comparison for the lapse-model task, the prediction errors for representations using one, two, three, and four components are 14.2, 10.8, 10.5, and 10.7 units, respectively. Calculate the PCA retention count from the stated variance rule, then interpret what the held-out comparison adds. Name the method and a mistake to avoid.
Step 1: Check the eigenvalue total. \(1.80+1.10+0.70+0.40=4.00\), as expected for a correlation matrix of four standardized variables.
Step 2: Calculate shares and cumulative shares. The individual percentages are \(1.80/4.00=45\%\), \(1.10/4.00=27.5\%\), \(0.70/4.00=17.5\%\), and \(0.40/4.00=10\%\). The cumulative percentages are 45%, 72.5%, 90%, and 100%. The smallest count reaching 85% is three: \(C_2=72.5\%<85\%\), while \(C_3=90\%\geq85\%\).
Step 3: Interpret the held-out comparison. Among the four tested representations, three components have the lowest reported prediction error, 10.5 units. The one-component representation has error 14.2, and the four-component representation has error 10.7. Thus, for this hypothetical comparison, the three-component representation both meets the exploratory variance rule and has the lowest listed held-out error. That predictive result comes from the held-out comparison, not from the 90% variance figure.
Conclusion and check. Retain three components under the stated 85% rule. Separately, the supplied held-out results favor three components among those tested. The method is to keep the variance calculation and predictive comparison logically distinct. The common mistake to avoid is claiming that 90% explained variance proves predictive usefulness; it does not. The calculated share is \((1.80+1.10+0.70)/4.00=3.60/4.00=90\%\), while the predictive conclusion relies on the separately supplied errors.
Making and Reporting a Defensible Choice
When retention rules disagree, report the evidence and the reason for the final choice rather than presenting one convention as objectively decisive. For example, state that a cumulative target led to three components, that the scree plot suggested a possible bend after two, and that the Kaiser rule retained two because the analysis used a correlation matrix. If the objective is compression, the tradeoff between component count and retained variation matters. If the objective is prediction, predictive performance must be examined separately.
Identify whether the eigenvalues come from a covariance matrix or a correlation matrix. Do not apply a correlation-matrix cutoff without its standardized-variable interpretation.
Sum the eigenvalues, divide each eigenvalue by that total, then accumulate in the given component order.
Describe a visible elbow as a visual suggestion. For a numerical target, select the smallest \(k\) whose cumulative proportion meets or exceeds that target.
Explain that a retention criterion summarizes variation and does not by itself establish that the retained components are useful for a downstream prediction task.
A useful final report gives the chosen count, the rule that produced it, the cumulative variance retained, and any notable disagreement among rules. For instance: “Three components were retained to exceed the prespecified 90% threshold; they explain 91% of total variance. The scree plot suggests a possible bend after component 2, so the cutoff is not visually unambiguous.” This is more informative than saying only that three components are “correct.”
Check Your Understanding
For each problem, calculate or interpret the retention evidence and check the final answer.
- A correlation-matrix PCA has eigenvalues \(3.00, 1.50,\) and \(0.50\). Find the cumulative percentages and the smallest number of components meeting an 80% target. (Answer: Total \(=5.00\); cumulative percentages are 60%, 90%, and 100%; retain 2 components.)
- A correlation-matrix PCA with six standardized variables has eigenvalues \(2.40, 1.40, 0.90, 0.60, 0.40,\) and \(0.30\). Find the smallest count reaching 85% and state how many components the Kaiser criterion retains. (Answer: Total \(=6.00\); cumulative percentages are 40%, 63.33%, 78.33%, 88.33%, 95%, and 100%; retain 4 by the 85% rule and 2 by the Kaiser criterion.)
- A covariance-matrix PCA has eigenvalues \(20, 10, 5,\) and \(5\), and the requested target is 80%. What is the smallest count meeting the target, and should the eigenvalue-greater-than-one rule be applied mechanically? (Answer: Total \(=40\); cumulative percentages are 50%, 75%, 87.5%, and 100%; retain 3 by the target, and do not mechanically apply the Kaiser rule to this covariance matrix.)
- A correlation-matrix PCA has eigenvalues \(2.20, 1.30, 0.80, 0.50,\) and \(0.20\). A report requires at least 80% variance. Find the cumulative percentages and compare that count with the Kaiser count. (Answer: Total \(=5.00\); cumulative percentages are 44%, 70%, 86%, 96%, and 100%; retain 3 for 80%, while the Kaiser criterion retains 2.)
- A scree plot has no clear elbow, while a stated cumulative target is met by four components. What should the analyst report, and what does this establish about predictive usefulness? (Answer: Report four as the smallest count meeting the stated target, note that the scree plot has no clear elbow, and do not claim that the variance criterion establishes predictive usefulness.)