Why PCA Pitfalls Matter
Principal component analysis (PCA) is a mathematical method, but its results depend on choices made before and after the calculation. A component can be dominated by a variable simply because that variable uses larger numerical units. A loading can be described as though it were an observation’s score. A sign reversal can be mistaken for a different result. And a component-retention rule can be presented as proof of predictive value when it only summarizes variation in the variables supplied to PCA.
These are not cosmetic issues. They can change the components, lead to incorrect descriptions of observations, or produce a misleading analysis. The aim here is to diagnose these errors and state what a PCA result does—and does not—establish.
PCA of centered variables uses the covariance matrix. PCA of centered and standardized variables uses the correlation matrix. Standardizing a variable means subtracting its mean and dividing by its standard deviation. Neither choice is automatically correct: it depends on whether the original measurement scales and variances should influence the analysis.
Scaling Is a Modeling Choice
Covariance-based PCA is sensitive to units. Converting a measurement from dollars to thousands of dollars multiplies its numerical values by \(1/1000\) and its variance by \(1/1{,}000{,}000\). The data have not substantively changed, but the covariance matrix has, and therefore its eigenvectors can change. Correlation-based PCA removes the effect of each variable’s scale by standardizing it first. It does not make every variable equally important in a substantive sense; it gives each variable unit variance before considering their dependence.
Worked Example: A Unit Change Alters Covariance PCA
A hypothetical analyst studies annual premium amount and claim frequency. The premium is recorded in dollars and has variance \(40{,}000\) dollars squared; frequency is a proportion and has variance \(0.04\). For this calculation, suppose the two variables have zero covariance. Compare the covariance PCA before and after recording premium in thousands of dollars, and then describe the correlation PCA.
Method. For a diagonal covariance matrix, the coordinate directions are eigenvectors, and the larger diagonal entry determines the first component.
In the original units, the covariance matrix is
Its eigenvalues are \(40{,}000\) and \(0.04\). Thus the first component points entirely in the premium direction. Its share of the total variance is
Now express premium in thousands of dollars. Dividing premium values by \(1{,}000\) divides its variance by \(1{,}000{,}000\), so the new premium variance is \(40{,}000/1{,}000{,}000=0.04\). The covariance matrix becomes
The eigenvalues are now both \(0.04\). There is no unique first direction: every unit direction has the same variance. The apparent dominance of premium in the original analysis was therefore a consequence of its units, not evidence that it is inherently the more important variable.
For correlation PCA, each variable has variance one after standardization. Under the stipulated zero covariance, the correlation matrix is the identity matrix, so it also has equal eigenvalues and no unique first direction. Check: both unit choices give equal variances after standardization, as expected. The common mistake is to treat the first covariance component as a scale-free result.
A practical R check is to inspect the units, means, and standard deviations before fitting. In prcomp, centering is on by default, while scaling is not; scale. = TRUE requests scaling by the column standard deviations. Scaling is not a button to press without thought. A constant column has standard deviation zero and cannot be standardized by division, so it must be addressed before requesting unit-variance scaling.
Loadings, Scores, and Directions Are Different
A component coefficient describes the direction used to form a component. An observation’s score is the result of applying that direction to the observation’s centered data. Some software and texts use “loading” for the eigenvector coefficients; others use it for the correlations between original variables and component scores. State which meaning is being used rather than relying on the word alone.
With standardized variables and an eigenvector \(v_j\), the score of observation \(i\) is \(t_{ij}=z_i^\mathsf{T}v_j\), where \(z_i\) is that observation’s vector of standardized values. If the eigenvalue for the component is \(\lambda_j\), the correlation of standardized variable \(k\) with that component’s score is \(\sqrt{\lambda_j}v_{kj}\). These quantities answer different questions: the eigenvector gives the component’s recipe, the score locates an observation along it, and the correlation loading describes a variable’s association with the component.
Worked Example: Calculating a Score From Component Coefficients
Suppose correlation-based PCA has two standardized variables with correlation \(0.8\). An eigenvector for the first component is \(v_1=(1/\sqrt{2},1/\sqrt{2})^\mathsf{T}\), and its eigenvalue is \(1.8\). An observation has standardized values \(z=(1,0.5)^\mathsf{T}\). Calculate its first-component score and the two variable-component correlations.
Method. Project the centered, standardized observation onto the eigenvector to obtain its score. For the variable-component correlations, use \(\sqrt{\lambda_1}v_{k1}\).
The score is
The correlation loading for each variable is
So the observation’s score is approximately \(1.0607\), while each variable’s correlation with the component is approximately \(0.9487\). The eigenvector coefficient for either variable is approximately \(0.7071\). These are three distinct values. Check: the score is positive because both coordinates of the observation align positively with the component direction. The common mistake is to report a loading as if it were the score of a particular observation.
A Sign Flip Is Not a Different Component
If \(v\) is an eigenvector, then \(-v\) is also an eigenvector for the same eigenvalue. Consequently, software may return either orientation. The scores reverse sign at the same time. Comparisons of PCA output should therefore focus on the direction and relationships, not on whether a particular coefficient or score is positive.
Worked Example: Checking a Sign Reversal
A component direction is \(v=(0.8,0.6)^\mathsf{T}\), and an observation’s centered coordinates are \(z=(1,2)^\mathsf{T}\). A second software run reports the direction with the opposite sign. Verify what changes and what remains the same.
Method. Compute the score with both directions, then compare the rank-one reconstruction \(tv\), which combines the score and direction.
With the first direction, the score is \(t=z^\mathsf{T}v=1(0.8)+2(0.6)=2\). With the reversed direction \(v'=(-0.8,-0.6)^\mathsf{T}\), the score is \(t'=z^\mathsf{T}v'=-2\). The score changes sign, but the reconstruction is unchanged:
Both runs describe the same component direction and reconstruct the same part of the observation. Check: reversing both the component and its score leaves their product unchanged. The common mistake is to interpret a sign reversal as evidence that the component’s meaning has reversed; the interpretation must account for the arbitrary orientation.
Keep the Response Out of Predictor PCA
PCA is unsupervised: it finds directions that explain variation in the variables supplied to it, without using an outcome. When PCA is used to summarize predictor variables, the response must not be included among those inputs. Otherwise the resulting components contain information from the outcome by construction. That makes them unsuitable as outcome-independent summaries of the predictors and can invalidate a predictive analysis.
Worked Example: Detecting Response Inclusion
A hypothetical four-record data set has two candidate predictors \(x_1\) and \(x_2\), and a response \(y\). After centering and standardizing, suppose the recorded values are \(x_1=(-1,-1,1,1)\), \(x_2=(-1,1,-1,1)\), and \(y=x_1\). Compare the correlation structure when PCA is applied to the predictors alone and when the response is mistakenly added.
Method. Calculate the pairwise correlations from the four rows, then inspect the eigenstructure of each correlation matrix.
The cross-products for \(x_1\) and \(x_2\) are \(1,-1,-1,1\), which sum to zero, so their correlation is zero. Since \(y=x_1\), the correlation of \(x_1\) with \(y\) is one, and that of \(x_2\) with \(y\) is zero. For predictors alone, the correlation matrix is
with eigenvalues \(1\) and \(1\): there is no unique first component. If the response is mistakenly included, the correlation matrix is
The vector \((1,0,1)^\mathsf{T}/\sqrt{2}\) has eigenvalue \(2\), because multiplying the matrix by \((1,0,1)^\mathsf{T}\) gives \((2,0,2)^\mathsf{T}\). This component directly combines \(x_1\) with \(y\). The remaining directions have eigenvalues \(1\) and \(0\), so the eigenvalues sum to \(3\), the number of variables in the correlation matrix.
The large first component here is not a predictor-only discovery: it uses the response, which duplicates \(x_1\). Check: excluding \(y\) gives two equal eigenvalues; including it creates a direction that explicitly contains \(y\). The common mistake is to run PCA on every numeric column without first separating predictors from the response.
Component Counts Need a Stated Criterion
A claim such as “retain three components because the elbow is obvious” is incomplete unless the evidence and decision rule are made clear. A scree plot, cumulative explained variance, the Kaiser rule for correlation-based PCA, and parallel analysis are different component-retention approaches. They can disagree because they ask different questions and use different reference points. None, by itself, establishes that retained components will improve prediction.
For a correlation matrix with \(p\) variables, the eigenvalues sum to \(p\). The proportion of total standardized variance explained by component \(j\) is \(\lambda_j/p\); the cumulative proportion through component \(k\) is \((\lambda_1+\cdots+\lambda_k)/p\). State the matrix being analyzed and the rule used before drawing a retention conclusion.
Worked Example: Reconciling Two Retention Rules
A correlation-based PCA of five variables has eigenvalues \(2.0, 1.3, 0.9, 0.5, 0.3\). A report claims that “two components must be retained because the eigenvalues exceed one.” Check that claim against the Kaiser rule and an 80% cumulative-variance criterion.
Method. First check the eigenvalue total, then apply each stated rule separately. The Kaiser rule for correlation-based PCA retains components with eigenvalues greater than one; the cumulative-variance criterion retains enough leading components to reach the specified percentage.
The eigenvalues sum to \(2.0+1.3+0.9+0.5+0.3=5.0\), as required for a five-variable correlation matrix. Under the Kaiser rule, only \(2.0\) and \(1.3\) are greater than one, so that rule selects two components. Their cumulative proportion is
Thus two components explain \(66\%\), not \(80\%\), of the total standardized variance. Adding the third component gives
The 80% criterion therefore selects three components. Neither calculation is arithmetically inconsistent: the criteria differ. Check: two components satisfy the stated Kaiser rule, while three are needed for at least 80% cumulative variance. The common mistake is to present one retention rule as a universal requirement or to claim the 80% threshold has been met with only 66%.
Implementation Checks Before Interpreting Output
A reliable interpretation begins by checking what was actually supplied to the PCA routine. In R, prcomp stores component directions in rotation and observation scores in x. For centered input, rotation contains the eigenvector directions, and the scores are obtained by projecting the centered data onto those directions. Do not describe a column of rotation as an observation score.
Before drawing conclusions, check that:
- The input columns are the intended numeric variables. Exclude identifiers and, when PCA is for predictor summaries, exclude the response.
- The scaling choice matches the question. Check whether variables have comparable units and whether equalizing their variances is justified.
- Centering and scaling were actually applied as intended. In
prcomp, scaling is not on by default; report the choice rather than assuming it. - Constant columns have been handled. A zero standard deviation makes standardization undefined.
- The same observations and column order are being compared. Otherwise scores or directions may be matched to the wrong records or variables.
- Sign differences are treated as orientation differences. Compare directions after allowing a whole component to be multiplied by \(-1\).
- Any component-count claim names its criterion. Report eigenvalues and, when relevant, cumulative proportions rather than relying on an unsupported label such as “enough components.”
PCA interpretation is strongest when it separates three questions: what variables were transformed, what mathematical direction was found, and what conclusion that direction supports. Explained variance describes variation in the input matrix. It does not, by itself, establish a causal explanation, an interpretable construct, or improved predictive performance. The central discipline is to make the preprocessing and retention choices explicit, then describe the output in terms those choices justify.
Check Your Understanding
For each item, identify the relevant PCA issue and check the calculation or interpretation.
- A variable has variance \(900\) when measured in dollars. What is its variance when its values are recorded in hundreds of dollars? (Answer: \(0.09\))
- A standardized observation is \(z=(2,-1)^\mathsf{T}\), and a component direction is \(v=(0.6,0.8)^\mathsf{T}\). What is the observation’s component score? (Answer: \(0.4\))
- A PCA routine returns direction \(v=(0.3,-0.954)^\mathsf{T}\). Another returns approximately \((-0.3,0.954)^\mathsf{T}\). What relationship should you check before concluding the results differ? (Answer: The directions are approximately sign reversals; compare the scores with the same sign reversal allowed.)
- A predictor-only correlation PCA has eigenvalues \(1.7,1.0,0.3\). What proportion of total variance is explained by the first two components, and how many satisfy the strict Kaiser rule? (Answer: \(2.7/3=0.9000\), or \(90.00\%\); one component satisfies the rule because \(1.0\) is not greater than one.)
- A team includes a binary outcome in a standardized matrix, runs PCA, then uses the resulting components as summaries of the predictors. What is the central flaw? (Answer: The components were constructed using the response, so they are not outcome-independent predictor summaries.)