Why PCA Stability Requires More Than Comparing Loading Tables
PCA creates predictors from directions of variation in the observed predictors. Those directions are estimated from a particular sample, so they can change when the sample changes. This matters when PCA scores enter a predictive model: a table of component loadings may look different across resamples even when the same information is represented, or may look reassuringly similar while the resulting predictions change.
There are three distinct objects to compare: individual component directions, the subspace retained for prediction, and the predictions made by the full modelling procedure. They answer different questions. Individual components can be sensitive to sign, ordering, and rotation; a retained subspace can remain stable despite changes to its chosen basis; and prediction stability must be checked directly rather than inferred from loadings alone.
In practice, a resample means that PCA is refitted on a different sampled training set, using the specified centering and scaling procedure. Each fit can produce different eigenvalues and eigenvectors. A comparison should respect the fact that an eigenvector and its negative describe the same direction: \(v_j\) and \(-v_j\) produce scores with opposite signs but represent the same PCA axis. This sign ambiguity is distinct from a genuine change in direction.
Comparing Components and Retained Subspaces
For a component with loading vectors \(v\) and \(w\) from two resamples, the absolute inner product \(|v^\mathsf{T}w|\) measures their directional agreement when both are unit vectors. A value of 1 means the axes agree, allowing for sign; a value near 0 means they are nearly perpendicular. If component ordering changes, match components by their directions or use a subspace comparison rather than pairing them mechanically by component number.
When \(k\) components are retained, their basis vectors are not always the most meaningful object to compare. Let \(V_A\) and \(V_B\) be matrices whose columns are orthonormal bases for the two retained subspaces. Their projection matrices are \(P_A=V_AV_A^\mathsf{T}\) and \(P_B=V_BV_B^\mathsf{T}\). If these projection matrices agree, the subspaces agree even if the bases differ. More generally, the singular values of \(V_A^\mathsf{T}V_B\) are the cosines of the principal angles between the subspaces. Principal angles near zero indicate close subspaces.
A small gap between adjacent eigenvalues is a warning about individual directions. When \(\lambda_j\) and \(\lambda_{j+1}\) are close, small changes to the estimated covariance matrix can substantially rotate their eigenvectors. If both directions are retained, that rotation can leave their joint subspace almost unchanged. If a cutoff retains one but excludes the other, the retained subspace itself may be sensitive to the cutoff.
Worked Example: A Near-Tie Rotates the First Component
Problem. Two training resamples produce the following covariance matrices for two centered predictors, measured on the same scales:
Compare the first-component directions and the subspaces retaining one and two components. Explain what a one-component cutoff implies.
Method. Find the eigenvalues and unit eigenvectors of each matrix, then compare directions using absolute inner products and compare retained spaces using their spans.
For \(\Sigma_A\), the diagonal entries give eigenvalues \(1.04\) and \(0.96\), with first eigenvector \(v_{A1}=(1,0)^\mathsf{T}\). For \(\Sigma_B\), vectors proportional to \((1,1)^\mathsf{T}\) and \((1,-1)^\mathsf{T}\) are eigenvectors. Direct multiplication gives eigenvalues \(1.04\) and \(0.96\), respectively. Thus \(v_{B1}=(1/\sqrt{2},1/\sqrt{2})^\mathsf{T}\).
The absolute agreement of the first directions is \[ |v_{A1}^\mathsf{T}v_{B1}|=|(1,0)(1/\sqrt{2},1/\sqrt{2})^\mathsf{T}|=1/\sqrt{2}\approx0.7071. \] Their angle is \(45^\circ\). Both eigenvalue gaps are only \(1.04-0.96=0.08\), so the leading direction is not strongly separated from the second.
If only the first component is retained, the retained directions differ by \(45^\circ\). If both components are retained, each basis spans the entire two-dimensional predictor space, so \(P_A=P_B=I_2\). The combined subspace is exactly the same.
Check. Each matrix has trace \(2.00\), equal to the sum \(1.04+0.96\) of its eigenvalues. Common mistake avoided: declaring that all PCA features are unstable because the first loading vector changed overlooks the stability of the two-component subspace.
Loadings, Scales, and Variable-Component Correlations
A loading table must be interpreted in light of how PCA was performed. For covariance-matrix PCA on centered, unstandardized predictors, the entries \(v_{ij}\) are coordinates of unit eigenvectors. They are not, by themselves, the correlations between the original predictors and the component scores.
To see the distinction, let predictor \(X_i\) have standard deviation \(\sigma_i\), and let \(Z_j=v_j^\mathsf{T}(X-\mu)\) be component \(j\)'s score. Since \(\Sigma v_j=\lambda_jv_j\), the covariance between \(X_i\) and \(Z_j\) is the \(i\)th entry of \(\Sigma v_j\), namely \(\lambda_jv_{ij}\). The score has standard deviation \(\sqrt{\lambda_j}\), so dividing the covariance by both standard deviations gives:
For PCA on standardized predictors, each standardized predictor has standard deviation 1, and the covariance matrix of those predictors is their correlation matrix. In that case, the variable-component correlation is \(\sqrt{\lambda_j}v_{ij}\). For covariance-matrix PCA, the additional division by the original predictor's standard deviation is necessary.
Worked Example: Interpreting a Covariance-PCA Eigenvector
Problem. Suppose the covariance matrix for two centered predictors is \(\Sigma=\operatorname{diag}(4,1)\). Find the first component score and the correlations of both predictors with it. A report instead scales the first eigenvector by \(\sqrt{\lambda_1}\) and calls the result the variable-component correlations. Check whether that report is correct.
Method. Find the leading unit eigenvector, form its score, and use the covariance-to-correlation formula with each predictor's standard deviation.
The eigenvalues are 4 and 1, so \(\lambda_1=4\), \(v_1=(1,0)^\mathsf{T}\), and the component score is \(Z_1=X_1\), since the predictors are centered. The predictor standard deviations are \(\sigma_1=\sqrt{4}=2\) and \(\sigma_2=\sqrt{1}=1\).
For \(X_1\), the correlation is \(\sqrt{4}(1)/2=1\). For \(X_2\), it is \(\sqrt{4}(0)/1=0\). These results also follow directly from \(Z_1=X_1\) and \(\operatorname{Cov}(X_1,X_2)=0\).
The proposed calculation \(\sqrt{4}v_1=(2,0)^\mathsf{T}\) is not a vector of correlations: its first entry is 2, which cannot be a correlation. The correction is to divide each entry by the corresponding predictor standard deviation, giving \((2/2,0/1)^\mathsf{T}=(1,0)^\mathsf{T}\). Check. Both corrected entries lie between \(-1\) and 1, as correlations must. Common mistake avoided: using the standardized-predictor formula for covariance-PCA variables without the scale adjustment.
Why Near-Tied Eigenvalues Create Directional Instability
An eigenvector is determined by the estimated covariance structure. If two eigenvalues are well separated, the corresponding directions are usually more clearly distinguished. If they are close, modest sampling variation can change which linear combination explains slightly more variance. The eigenvalue order can remain the same while the associated direction rotates substantially, as the preceding example showed.
This effect is not a software defect and does not imply that the underlying predictors have suddenly changed meaning. It reflects weak identification of a particular axis within a nearly tied group of directions. Consequently, component-by-component statements such as “predictor 1 dominates PC1” can be fragile. A combined statement about the space spanned by several near-tied components may be much more stable.
A short justification for using projection matrices makes this distinction precise. If \(V\) has orthonormal columns, the projection of a centered record \(x\) onto its column space is \(VV^\mathsf{T}x\). If another orthonormal basis is \(VQ\), where \(Q\) is an orthogonal matrix, then:
because \(QQ^\mathsf{T}=I\). Thus, changing or rotating the basis within a retained subspace changes the component coordinates but not the projection onto that subspace. This identity does not say that a predictive model using only some of those coordinates will always produce identical predictions: that also depends on how the model is fitted, regularized, or truncated.
Worked Example: A Rotated PC Changes a One-Component Prediction
Problem. A hypothetical response has linear signal \(Y=2X_1+X_2\). Two training resamples give the covariance matrices \(\Sigma_A\) and \(\Sigma_B\) from the first example. A model is fitted by population least squares using only the first PC. Compare its predictions for the record \(x=(1,2)^\mathsf{T}\). State what the comparison does and does not establish.
Method. In each resample, regress the response on its first-component score. For a unit direction \(v\), the score is \(Z=v^\mathsf{T}X\), and the population least-squares slope is \(\operatorname{Cov}(Z,Y)/\operatorname{Var}(Z)=v^\mathsf{T}\Sigma\beta/(v^\mathsf{T}\Sigma v)\), where \(\beta=(2,1)^\mathsf{T}\).
For resample A, \(v_{A1}=(1,0)^\mathsf{T}\) and \(\Sigma_A\beta=(2.08,0.96)^\mathsf{T}\). Therefore, the slope is \(2.08/1.04=2\). The record's score is \(1\), so the prediction is \(2(1)=2\).
For resample B, \(v_{B1}=(1/\sqrt{2},1/\sqrt{2})^\mathsf{T}\). Here \(\Sigma_B\beta=(2.04,1.08)^\mathsf{T}\), so \(v_{B1}^\mathsf{T}\Sigma_B\beta=3.12/\sqrt{2}\) and \(v_{B1}^\mathsf{T}\Sigma_Bv_{B1}=1.04\). The fitted slope is \((3.12/\sqrt{2})/1.04=3/\sqrt{2}\). The record's score is \(3/\sqrt{2}\), giving prediction \((3/\sqrt{2})(3/\sqrt{2})=9/2=4.5\).
The true linear signal for this record is \(2(1)+2=4\), between the two predictions. Both calculations use the stated covariance matrix and the corresponding population least-squares fit. The example shows that a one-component cutoff can lead to different predictions when the retained direction changes. It does not establish which resample's prediction is better on new observations; that requires an appropriate predictive evaluation. Check. The second fitted slope is \(3/\sqrt{2}\), and its score is also \(3/\sqrt{2}\), so the prediction is \(9/2\). Common mistake avoided: treating a changed loading vector alone as proof of a changed prediction—or assuming that a stable multi-component subspace guarantees stability after retaining only one direction.
A Practical Stability Assessment
A useful analysis compares components, subspaces, and predictions separately. The comparison should use a consistent predictor definition, centering and scaling rule, number of retained components, and downstream fitting procedure. The earlier material on foldwise PCA and transforming validation and test records establishes why PCA mappings must be learned from the relevant training data and then applied consistently. Here the focus is what to compare across the resulting fits.
For each resample, note the retained eigenvalues and gaps around the retention boundary. A small gap warns that individual directions near that boundary may be sensitive.
Account for sign changes and, where necessary, component reordering. Compare matched unit directions using absolute inner products; do not interpret a sign flip as a change in axis.
Use projection matrices or principal angles when several components are retained. This detects whether the combined directions are stable even if their individual coordinates rotate.
Apply each fitted PCA mapping and downstream model to the same evaluation records. Summarize prediction differences in the outcome's relevant units, alongside the component and subspace comparisons.
Distinguish instability in component labels or coordinates from instability in the retained space and from instability in predictions. These are not interchangeable conclusions.
For example, if two near-tied directions are both retained and their joint subspace is stable, their separate loading interpretations may be unreliable while the representation remains stable. If only one is retained, both the subspace and potentially the predictions can vary. Conversely, a visually stable loading table does not guarantee prediction stability: the downstream fit may also vary across resamples.
Common Exam Traps and Reporting
- Comparing signed loadings without sign alignment. A component and its negative describe the same axis. Use an absolute inner product or align signs before interpreting a loading comparison.
- Pairing components only by their numbers. Components are ordered by estimated eigenvalue, and close eigenvalues can make ordering or matching ambiguous. Check directions and eigenvalue gaps.
- Calling a stable basis necessary for a stable space. A rotation among retained components can change individual scores while leaving the projection onto the retained subspace unchanged.
- Assuming stable loadings prove stable prediction. The downstream model and the records being scored matter. Compare predictions directly on the same evaluation records.
- Confusing covariance and correlation PCA. For standardized predictors the variable-component correlations are \(\sqrt{\lambda_j}v_{ij}\); for covariance PCA divide additionally by the standard deviation of predictor \(i\).
- Interpreting resample variation as a formal uncertainty guarantee. Resampling reveals sensitivity under the chosen data-resampling scheme and modelling procedure. It does not, by itself, prove that future performance or every population-level direction has a particular stability.
Check Your Understanding
For each problem, identify whether the evidence concerns component, subspace, or prediction stability.
- Two unit first-component vectors have inner product \(-0.92\). What is their sign-adjusted directional agreement, and how should the negative sign be interpreted? (Answer: Agreement \(0.92\); the negative sign is a sign reversal, not a different PCA axis.)
- Two fits retain two components in three predictors. Their two-dimensional projection matrices are identical, but their first and second eigenvectors are rotated within that plane. What is stable, and what may differ? (Answer: The retained subspace and projected records are stable; the individual component scores and loading interpretations may differ.)
- For covariance PCA, \(\lambda_j=9\), \(v_{ij}=0.4\), and predictor \(i\) has standard deviation 2. Find its correlation with component \(j\). (Answer: \(\sqrt{9}(0.4)/2=0.6000\).)
- Two resamples retain only one component. Their leading eigenvectors have absolute inner product \(0.70\), and the predictions on the same evaluation record are 12 and 18. Give one conclusion that is supported and one that is not. (Answer: The leading directions and these predictions differ across the fits; the results alone do not establish which prediction is more accurate.)
- Two resamples each retain two components from a four-predictor space. The singular values of \(V_A^\mathsf{T}V_B\) are 1 and 0.8. What are the two principal angles, rounded to one decimal place, and what do they indicate? (Answer: \(\arccos(1)=0.0^\circ\) and \(\arccos(0.8)\approx36.9^\circ\); one direction agrees exactly, while the second differs appreciably, so the retained subspaces are not identical.)