Why a Low-Variance Component Can Matter
Principal component analysis (PCA) finds directions that capture variation among the predictors. In a classification problem, however, the goal is to distinguish outcome classes. Those goals need not agree: a direction can vary very little across observations yet separate the classes well, while a high-variance direction can reflect variation unrelated to class membership.
This matters when a model retains only the first few components because they explain a chosen percentage of predictor variance. Such a rule describes how much variation in the predictors is preserved; by itself, it does not describe how much information about the outcome is preserved. The issue is not that PCA is inherently unsuitable for prediction. It is that component retention based only on predictor variance may discard a useful direction.
Let \(X\) be a vector of centered predictors with covariance matrix \(S\). If \(a\) is a unit-length direction, the variance of the projected score \(a^\mathsf{T}X\) is \(a^\mathsf{T}Sa\). PCA ranks directions by this quantity. The class label, say \(C\), is not part of that calculation. Consequently, the variance ranking alone cannot establish whether a component is useful for predicting \(C\).
The following examples make the distinction concrete. Each specifies the scale or distribution used; that matters because PCA depends on whether it is applied to the covariance matrix of predictors in their stated units or to a correlation matrix after standardization.
What PCA Optimizes
The first component solves a constrained maximization problem: choose a unit vector \(a\) that maximizes the variance of the projected predictor values. A short Lagrange-multiplier argument gives the eigenvector rule.
For centered predictors with covariance matrix \(S\), the projected score has variance \(a^\mathsf{T}Sa\), with the constraint \(a^\mathsf{T}a=1\).
Maximize \(a^\mathsf{T}Sa-\lambda(a^\mathsf{T}a-1)\). Setting its derivative with respect to \(a\) equal to zero gives \(2Sa-2\lambda a=0\), or \(Sa=\lambda a\).
The maximizing direction is an eigenvector of \(S\) associated with its largest eigenvalue. No outcome label appears in this optimization, so maximizing predictor variance does not necessarily maximize class separation.
For \(p\) predictors, the eigenvalues of \(S\) give the component variances. If they are ordered \(\lambda_1\geq\lambda_2\geq\cdots\geq\lambda_p\), the proportion of predictor variance explained by component \(j\) is \(\lambda_j/\sum_{k=1}^{p}\lambda_k\). This proportion is not a classification accuracy, a probability of correct classification, or a measure of association with the class.
Worked Example: A Low-Variance Direction Separates Two Classes
A hypothetical risk-screening study has two centered predictor scores, \(Z_1\) and \(Z_2\), already expressed in the stated analysis units. The class \(C\) is equally likely to be \(-1\) or \(+1\). Assume \(Z_1\sim N(0,100)\), independently of \(C\) and \(Z_2\), and assume \(Z_2\mid C=c\sim N(0.6c,0.64)\). Compare a classifier using only the first principal component with one using the second principal component. The second argument of \(N(\mu,\sigma^2)\) denotes variance.
Step 1: Find the predictor variances. The marginal mean of \(Z_2\) is zero because the two classes are equally likely and their conditional means are \(0.6\) and \(-0.6\). By the variance decomposition, \(\operatorname{Var}(Z_2)=E[\operatorname{Var}(Z_2\mid C)]+\operatorname{Var}(E[Z_2\mid C])=0.64+0.36=1\). Independence gives \(\operatorname{Cov}(Z_1,Z_2)=0\). Thus the covariance matrix is diagonal, with eigenvalues \(100\) and \(1\), and the components are \(Z_1\) and \(Z_2\).
Step 2: Classify from PC1 alone. Since \(Z_1\) is independent of \(C\), observing \(Z_1\) does not change either class probability. With equal class priors, the best possible classification accuracy using only \(Z_1\) is \(0.5000\).
Step 3: Classify from PC2. The conditional means of \(Z_2\) are symmetric, and both classes have the same variance. The equal-prior Bayes rule assigns class \(+1\) when \(Z_2>0\) and class \(-1\) otherwise. For either class, the probability of correct classification is \(P(Z_2>0\mid C=+1)=\Phi(0.6/0.8)=\Phi(0.75)=0.7734\), with the same probability for class \(-1\). Here \(\Phi\) is the standard normal cumulative distribution function.
Conclusion and check. PC1 explains \(99.01\%\) of the predictor variance but offers only \(50.00\%\) accuracy, while PC2 explains \(0.99\%\) and gives \(77.34\%\) accuracy. The class-conditional means and the overall variance also check: \(0.64+0.6^2=1\). The mistake to avoid is interpreting the \(99.01\%\) figure as the percentage of class information preserved.
A Finite-Sample Calculation
A sample can show the same issue without a probability model. The next example uses centered component coordinates so that the covariance and the classification results can be checked directly. The labels are included to assess prediction; they do not determine the PCA directions.
Worked Example: An Explained-Variance Cutoff Drops a Perfect Separator
A hypothetical training sample has the following two centered predictor scores and binary class labels. PCA is applied to these scores using the sample covariance matrix, with divisor \(n-1\). A retention rule keeps components until at least \(80\%\) of predictor variance is explained.
| Observation | \(Z_1\) | \(Z_2\) | Class \(C\) |
|---|---|---|---|
| 1 | -3 | -1 | 0 |
| 2 | -3 | 1 | 1 |
| 3 | -1 | -1 | 0 |
| 4 | -1 | 1 | 1 |
| 5 | 1 | -1 | 0 |
| 6 | 1 | 1 | 1 |
| 7 | 3 | -1 | 0 |
| 8 | 3 | 1 | 1 |
Step 1: Calculate the covariance matrix. Both sample means are zero. The sums of squares are \(\sum Z_1^2=2(9)+2(1)+2(1)+2(9)=40\) and \(\sum Z_2^2=8\). In each pair of observations sharing a \(Z_1\) value, the products \(Z_1Z_2\) cancel, so \(\sum Z_1Z_2=0\). Therefore,
Step 2: Apply the retention rule. The eigenvalues are \(40/7\) for the \(Z_1\) direction and \(8/7\) for the \(Z_2\) direction. Their total is \(48/7\), so the first component explains \((40/7)/(48/7)=40/48=0.8333\), or \(83.33\%\). The \(80\%\) rule therefore retains PC1 alone and discards PC2.
Step 3: Compare classifications. Every value of \(Z_1\) occurs once with class 0 and once with class 1. Any deterministic classifier that uses \(Z_1\) alone must give the same prediction to the two observations in each pair, so it can correctly classify at most one of each pair: \(4/8=0.5000\). In contrast, \(Z_2=-1\) always has class 0 and \(Z_2=1\) always has class 1, so the rule “predict 1 when \(Z_2>0\)” correctly classifies \(8/8=1.0000\).
Conclusion and check. The component omitted by the variance cutoff perfectly separates the classes in this sample. The eigenvalues sum to the total sample variance \(48/7\), and the class counts are balanced at four observations each. The conclusion is about this constructed sample, not a guarantee that PC2 will generalize perfectly in new data.
Validate Predictive Value, Not Just Variance Retained
A component should be judged in the context of the prediction task. In practice, fit the PCA transformation using the training data, construct candidate classifiers under the intended retention choices, and compare their predictions on validation data not used to fit those classifiers. The earlier material on leakage-safe predictive pipelines is relevant: validation observations must not influence the fitted transformation or classifier. A high variance proportion is a description of the predictors, not evidence of good out-of-sample classification.
The appropriate comparison metric depends on the decision. Overall accuracy can hide weak detection of a minority class. Sensitivity is the proportion of actual positives correctly classified; specificity is the proportion of actual negatives correctly classified. When false-positive and false-negative costs differ, compare expected error costs as well as, or instead of, accuracy.
Worked Example: Choose Components Using Validation Outcomes and Error Costs
A hypothetical insurer is testing a classifier that flags accounts for manual review. PC1 explains \(96\%\) of the training predictors’ total variance, and PC2 explains \(4\%\). On a separate validation set of 30 accounts, 10 are actually positive and 20 are actually negative. Using PC1 alone, the classifier produces 3 true positives, 7 false negatives, 4 false positives, and 16 true negatives. Using PC1 and PC2, it produces 8 true positives, 2 false negatives, 3 false positives, and 17 true negatives. Assume a false negative costs \(\$500\) and a false positive costs \(\$100\); other outcomes have no stated error cost. Which model performs better for this stated decision?
Step 1: Check the validation totals. For PC1 alone, \(3+7=10\) actual positives and \(4+16=20\) actual negatives, for 30 accounts. For PC1 and PC2, \(8+2=10\) positives and \(3+17=20\) negatives, also for 30 accounts. The supplied counts are internally consistent.
Step 2: Calculate accuracy and class-specific rates. PC1 alone has accuracy \((3+16)/30=19/30=0.6333\), sensitivity \(3/10=0.3000\), and specificity \(16/20=0.8000\). With both components, accuracy is \((8+17)/30=25/30=0.8333\), sensitivity is \(8/10=0.8000\), and specificity is \(17/20=0.8500\). The second model improves each of these validation measures.
Step 3: Apply the stated costs. The PC1-only model has total error cost \(7(\$500)+4(\$100)=\$3{,}900\). The model using both components has total error cost \(2(\$500)+3(\$100)=\$1{,}300\). Dividing by 30 gives average costs of \(\$130.00\) and approximately \(\$43.33\) per account, respectively.
Conclusion and check. On this validation set, the model including PC2 is preferable under the stated costs: it reduces total error cost by \(\$2{,}600\). Its better result is not established by the fact that PC2 explains \(4\%\) of predictor variance; it is established here by the validation predictions and specified costs. This is validation evidence, not proof of performance in every future population.
Common Exam Trap: Treating Explained Variance as Predictive Information
A tempting but invalid argument is: “PC1 explains more than \(80\%\) of the predictor variance, so retaining PC1 must retain most of the classification signal.” The conclusion does not follow. PCA’s variance objective contains no class label, and the examples show that a low-variance direction can carry substantially more class information than a high-variance direction.
The correction is to separate two questions. First, how much variation in the predictors does a component explain? Calculate this from its eigenvalue and the total eigenvalue sum. Second, how well does a model using that component classify new observations? Assess this with an appropriate validation design and a metric suited to class balance and error costs. Do not substitute the answer to the first question for the answer to the second.
Also check whether the analysis uses a covariance matrix or a correlation matrix. Changing predictor scales or standardizing them can change the PCA directions and their variance ranking. It does not change the central principle: whichever PCA convention is chosen, component variance alone does not use the class labels. State the convention, fit the transformation without validation or test information, and compare predictive performance under the intended decision.
Check Your Understanding
For each problem, distinguish predictor variance from evidence about class prediction.
- A covariance-based PCA has eigenvalues \(81\) and \(9\), in that order. What proportion of total predictor variance does PC1 explain? Does this calculation establish that PC1 is more useful for classification than PC2? (Answer: PC1 explains \(81/(81+9)=0.9000\), or \(90.00\%\); no, the class labels are not used in that calculation.)
- A training set has two orthogonal component directions with eigenvalues \(18\) and \(2\). A rule retains enough components to explain at least \(90\%\) of predictor variance. Which components are retained, and what predictive question remains unanswered? (Answer: PC1 alone is retained because \(18/(18+2)=90\%\); whether PC2 improves class prediction remains unanswered.)
- Classes \(C=-1\) and \(C=+1\) have equal prior probabilities. Let \(Z_2\mid C=c\sim N(0.8c,0.36)\), and let a classifier predict \(+1\) for \(Z_2>0\) and \(-1\) otherwise. Find its accuracy. (Answer: \(Z_2\) has conditional standard deviation \(0.6\), so accuracy is \(\Phi(0.8/0.6)=\Phi(1.3333)\approx0.9088\).)
- A validation set contains 12 positives and 18 negatives. A model has 9 true positives and 3 false negatives, plus 6 false positives and 12 true negatives. Calculate accuracy, sensitivity, and specificity. (Answer: accuracy \(=(9+12)/30=0.7000\), sensitivity \(=9/12=0.7500\), and specificity \(=12/18=0.6667\).)
- On a validation set, a PC1-only model makes 5 false negatives and 2 false positives. A model using PC1 and PC2 makes 2 false negatives and 5 false positives. If a false negative costs \(\$400\) and a false positive costs \(\$100\), which model has lower total error cost? (Answer: PC1-only costs \(5(\$400)+2(\$100)=\$2{,}200\); the two-component model costs \(2(\$400)+5(\$100)=\$1{,}300\), so the model using both components has lower cost.)