Two Different Goals for Keeping Components
Principal components summarize variation in the predictors. When the goal is prediction, however, the relevant question is not simply which directions vary most: it is which available components help predict the target. Those goals can agree, but they need not. A direction with little predictor variation may still be strongly associated with the target.
A component-retention rule based on explained variance is useful when the aim is to represent the predictor data compactly. A validation-based rule instead compares the predictions from candidate component sets on records not used to fit the predictive model. The choice should match the stated goal: a compact description of predictors and accurate predictions of a target are different objectives.
The eigenvalues describe variation in the predictors, not prediction error for the target. A threshold such as 80% is therefore a description-oriented convention, not a guarantee that the retained components are best for prediction. A predictive selection rule fits candidate prediction models using different component sets and compares their performance on validation data. For squared-error prediction, a common comparison is validation mean squared error (MSE); for a classification task, the metric should reflect the prediction objective, such as a suitable classification loss or a threshold-based measure when a threshold is part of the task.
Explained-Variance Rules
A cumulative-variance rule is applied to the eigenvalues from PCA. For a selected threshold \(c\), retain the smallest \(k\) satisfying \(\sum_{j=1}^{k}\lambda_j/\sum_{r=1}^{p}\lambda_r \geq c\). The threshold must be specified; different thresholds can produce different component counts. Such a rule does not use the target values, so it cannot directly determine whether an omitted direction would improve prediction.
Worked Example: Applying a Cumulative-Variance Threshold
A training set has three PCA eigenvalues, in descending order, of 6, 3, and 1. A team wants to retain enough components to explain at least 80% of the training predictor variance. Find the retained component count and check whether the last retained component is needed to meet the rule.
Method. Add the eigenvalues to obtain total predictor variance: \(6+3+1=10\). The individual explained-variance proportions are \(6/10=0.60\), \(3/10=0.30\), and \(1/10=0.10\). Thus the cumulative proportions are 0.60 after one component, 0.90 after two, and 1.00 after three.
After one component, 60% is below the required 80%. After two components, 90% reaches the threshold, so the smallest qualifying count is \(k=2\). The second component is needed: without it, the cumulative proportion remains 60%. The third component is not needed to reach 80%.
Check. The two retained eigenvalues sum to \(6+3=9\), and \(9/10=0.90\), which is at least 0.80. This rule answers how many components are needed to represent at least 80% of predictor variance; it does not establish that two components minimize prediction error. A common mistake is to interpret the 90% figure as 90% of the target variation explained. It refers only to predictor variance.
Why a Low-Variance Component Can Predict Well
A component's predictive value depends on its relationship with the target as well as its variance. In a simple linear prediction model, a component with small variance can have a large fitted slope if its covariance with the target is substantial. The resulting prediction improvement can exceed that of a higher-variance component whose target association is weaker.
The following justification uses a population least-squares projection with an intercept and one predictor score \(T\). Assume \(T\) and target \(Y\) have finite variances and \(\operatorname{Var}(T)>0\). For a prediction \(a+bT\), the intercept that minimizes mean squared error for a given slope is \(a=E[Y]-bE[T]\). After centering both variables, the error is \(Y-E[Y]-b(T-E[T])\). Its expected square is minimized by the slope shown below.
To see why, expand the centered squared error and differentiate with respect to \(b\). Setting the derivative to zero gives the least-squares slope. Substituting that slope into the error expression gives the reduction in mean squared error relative to predicting \(E[Y]\).
The reduction depends on squared covariance divided by the component variance, not on component variance alone. This result concerns a population linear projection with one score. It does not guarantee that an estimated model will improve on new data: estimated associations can be noisy, and validation is needed to assess generalization. With several mutually uncorrelated component scores in a linear model, their separate in-sample contributions to the least-squares projection can be assessed by the same covariance-over-variance calculation. Predictive performance must still be checked on data not used to fit that model.
Worked Example: A Small-Variance Component With Greater Target Value
Suppose two centered PCA scores \(T_1\) and \(T_2\) are uncorrelated, with \(\operatorname{Var}(T_1)=9\) and \(\operatorname{Var}(T_2)=0.25\). The target \(Y\) has variance 4, with \(\operatorname{Cov}(T_1,Y)=1.5\) and \(\operatorname{Cov}(T_2,Y)=0.4\). Compare their population linear least-squares contributions to prediction. Assume the stated variances and covariances describe the population, and use an intercept.
Method. For each score, calculate its least-squares slope and its reduction in mean squared error relative to predicting the target mean. For \(T_1\), the slope is \(1.5/9=1/6\), and the MSE reduction is \(1.5^2/9=2.25/9=0.25\).
For \(T_2\), the slope is \(0.4/0.25=1.6\), and the MSE reduction is \(0.4^2/0.25=0.16/0.25=0.64\). Although \(T_2\) has much smaller variance, its calculated MSE reduction is larger.
Because the scores are uncorrelated, the two-score linear projection has total MSE reduction \(0.25+0.64=0.89\). Its remaining MSE is \(4-0.89=3.11\). Using only \(T_1\) leaves MSE \(4-0.25=3.75\), while using only \(T_2\) leaves MSE \(4-0.64=3.36\). The calculations show that \(T_2\) contributes more to this population linear prediction than \(T_1\), despite explaining less predictor variance.
Check. The reductions are nonnegative and their sum, 0.89, is less than the original target variance of 4. A common mistake is to rank the scores by eigenvalue and conclude that \(T_1\) must be more useful. Eigenvalue order ranks predictor variance, not target association. This population calculation motivates predictive evaluation; it is not a substitute for validation on finite data.
Selecting Components With Validation
For predictive selection, define candidate component counts or component sets before comparing their out-of-sample performance. Fit each candidate prediction model using the training data and the specified component scores, then calculate its chosen metric on a separate validation set. For \(n_{\text{val}}\) validation records with observed targets \(y_i\) and predictions \(\hat{y}_i\), validation MSE is \(\sum_{i=1}^{n_{\text{val}}}(y_i-\hat{y}_i)^2/n_{\text{val}}\). Lower MSE is preferred when squared-error prediction is the objective.
The PCA transformation must be fitted using training predictors, and the same fitted transformation must be applied to validation predictors. The target values in the validation set are used to score predictions, not to fit the PCA. This is consistent with the training-only transformation principle in the earlier material on designing a training-only transformation pipeline. After component selection, a final evaluation should use data not used to select the component count; otherwise the reported performance may be optimistic.
Worked Example: Choosing a Component Count by Validation MSE
A fitted PCA has eigenvalues 8.5, 1.0, and 0.5. A cumulative 80% rule would retain how many components? Separately, three candidate regression models using the first \(k\) scores are evaluated on four validation records with targets 10, 12, 9, and 13. Their predictions are shown below. Choose the component count by validation MSE and compare it with the variance rule.
| Candidate model | Predictions for the four records |
|---|---|
| First component only, \(k=1\) | 10, 11, 10, 12 |
| First two components, \(k=2\) | 10, 12, 9, 12 |
| First three components, \(k=3\) | 11, 12, 9, 14 |
Step 1: Apply the variance rule. Total eigenvalue is \(8.5+1.0+0.5=10\). The first component explains \(8.5/10=0.85\), or 85%, so the 80% threshold retains one component.
Step 2: Calculate validation errors. For \(k=1\), the residuals (observed minus predicted) are \(0,1,-1,1\), so the squared errors are \(0,1,1,1\). Their sum is 3 and the MSE is \(3/4=0.75\). For \(k=2\), the residuals are \(0,0,0,1\), so the squared errors sum to 1 and the MSE is \(1/4=0.25\). For \(k=3\), the residuals are \(-1,0,0,-1\), so the squared errors sum to 2 and the MSE is \(2/4=0.50\).
Decision. The smallest validation MSE is 0.25, achieved by \(k=2\). The cumulative-variance rule selects \(k=1\), whereas validation selects \(k=2\). Thus, in this example, the second component improves validation predictions even though the first component already explains 85% of predictor variance. The decision uses the stated MSE objective and these validation results; it does not establish that the same count will be best in another sample.
Check. The squared-error sums are 3, 1, and 2, and dividing each by four gives the reported MSEs. A common mistake is to choose one component automatically because it passes the 80% threshold. That rule describes predictor variance; the validation comparison addresses predictive error.
Choosing and Reporting a Rule
The two approaches answer distinct questions, so report which objective governs the decision. Explained-variance retention is transparent and useful when dimensional compression or a compact summary of predictors is the priority. Validation-based selection is more directly aligned with predictive performance, but its result depends on the chosen metric, the validation sample, and the candidate models.
A sound comparison should consider the following points:
- Match the metric to the task. MSE is appropriate when squared errors are the objective; a different prediction task may require a different metric.
- Use a stated candidate set. Comparing a defined range of component counts makes the selection rule explicit.
- Do not select on the final test set. Repeatedly choosing components based on test performance turns the test set into selection data.
- Do not equate predictor variance with target value. PCA is formed without using the target, so its variance ordering is not a ranking of predictive usefulness.
- Interpret small differences cautiously. A single validation sample can favor a candidate by chance; report the metric and evaluation design rather than claiming universal superiority.
Check Your Understanding
For each problem, distinguish predictor-variance retention from predictive selection and check the arithmetic before interpreting the result.
- A PCA has eigenvalues 5, 3, and 2. What is the cumulative explained-variance proportion after two components, and how many are retained by a 75% threshold? (Answer: 80%; two components.)
- A centered score \(T\) has variance 0.5 and covariance 0.3 with a target \(Y\). Under a population linear projection with an intercept, find its least-squares slope and its MSE reduction relative to predicting the target mean. (Answer: slope \(0.3/0.5=0.6\); MSE reduction \(0.3^2/0.5=0.18\).)
- For five validation observations, the squared-error sums for models with one, two, and three components are 4.0, 2.5, and 3.0, respectively. Calculate each validation MSE and choose the model with lowest MSE. (Answer: 0.80, 0.50, and 0.60; choose two components.)
- A first component explains 92% of predictor variance. A second component explains 4%, but adding it reduces validation MSE from 1.20 to 0.90. State which choice is supported by an 80% cumulative-variance rule and which by validation MSE, and give the MSE reduction. (Answer: the variance rule supports one component; validation supports two; MSE falls by 0.30.)
- A candidate model has validation squared-error sum 6 over 8 records and test squared-error sum 3 over 4 records. A student chooses the component count by whichever of these two observed MSEs is smaller. Identify the flaw and calculate both MSEs. (Answer: both MSEs are 0.75, and choosing the count using test performance would use the test set for selection; select with validation and reserve the test set for final evaluation.)