Why PCA Must Be Refit Within Each Fold
When PCA scores are used as predictors in cross-validation, the PCA fit is part of the predictive procedure. Each fold has its own training records and assessment records. The means, scales, covariance structure, and component directions must be estimated from that fold’s training records alone. The assessment records are then expressed in the coordinate system fitted on that training fold.
This is more than a bookkeeping choice. A principal component depends on the data used to estimate it. If assessment records help determine the means, scales, or loadings, the assessment fold has influenced the features supplied to the model being evaluated. Conversely, fitting PCA separately but then using a different fold’s scaling or loadings gives assessment records scores in the wrong coordinate system.
Assume the predictors supplied to PCA are numeric and have any required missing-value handling completed by a method fixed without using assessment information. The number of retained components and the choice to center or scale must also be specified as part of the modeling procedure. The worked calculations below use two predictors and retain the first component.
Fitting and Projecting Within a Fold
Write \(x_{ij}\) for predictor \(j\) on training record \(i\). With centering and scaling, calculate the training-fold mean \(\bar{x}_{kj}\) and sample standard deviation \(s_{kj}\) for each predictor. Standardize both training and assessment values using those training-fold quantities:
The superscript \(k\) emphasizes that the standardization is fold-specific. Using the standardized training matrix \(Z_{T_k}^{(k)}\), calculate its sample covariance matrix and find its eigenvectors. If \(v_{k\ell}\) is the unit eigenvector for component \(\ell\), the score for record \(i\), including an assessment record, is:
The assessment score is not obtained by fitting a second PCA to the assessment records. It is a projection onto a direction already estimated from the training fold. Likewise, the assessment records do not get their own means or standard deviations for this calculation.
Identify \(T_k\) and \(A_k\), keeping assessment outcomes out of the PCA fit.
Calculate the chosen training-only centers and scales, then estimate the covariance matrix and component loadings from the transformed training records.
Apply the training-fold centers, scales, and loadings to each assessment record to obtain its component scores.
Use training-fold PCA scores to fit the predictive model and assessment-fold PCA scores to form assessment predictions.
Worked Example: Projecting an Assessment Record Onto a Training-Fold Component
A hypothetical pricing analysis uses two numeric predictors, \(X_1\) and \(X_2\). The training fold contains \((2,4)\), \((4,6)\), \((6,8)\), and \((8,10)\). An assessment record has values \((7,9)\). Center and scale each predictor using the training fold, and calculate the assessment record’s first principal-component score.
Method. Compute the training-fold means and sample standard deviations, form its standardized covariance matrix, and project the assessment record using the resulting loading. This avoids estimating a separate assessment-fold PCA.
For \(X_1\), the mean is \((2+4+6+8)/4=5\). The squared deviations sum to \(9+1+1+9=20\), so the sample variance is \(20/3\) and the sample standard deviation is \(\sqrt{20/3}\). For \(X_2\), the mean is \(7\); its deviations are also \(-3,-1,1,3\), so its sample standard deviation is also \(\sqrt{20/3}\).
After standardization, the two columns are identical. Each has sample variance 1, and their sample covariance is 1. Thus the covariance matrix and its leading unit eigenvector are:
For the assessment record, each standardized value is \(2/\sqrt{20/3}=\sqrt{3/5}\), approximately \(0.7746\). Therefore its first-component score is:
Check. Both standardized predictor values are positive and equal, so their projection onto the equal-weight loading is positive and larger than either value alone. The result is consistent with that geometry. The common mistake is to standardize this record using assessment-fold statistics; the prescribed score uses the training-fold mean and scale.
Fold-Specific Scales and Loadings
Different training folds can produce different PCA coordinate systems. Even when the same predictors and number of components are used, their centers, scales, and loadings can change. Scores from separate folds are therefore not automatically measurements on one shared scale or one shared axis.
Worked Example: One Assessment Record, Two Fold-Specific Mappings
Suppose two cross-validation folds produce the following training records. Fold A contains \((0,0)\), \((2,2)\), and \((4,4)\). Fold B contains \((8,5)\), \((10,3)\), and \((12,1)\). In both folds, use sample-standard-deviation scaling and retain the first component. Project the assessment record \((4,2)\) separately through each fold’s PCA mapping.
Method. Fit and apply each fold’s mapping independently. This avoids the mistake of reusing fold A’s parameters in fold B.
In fold A, the means are \((2,2)\). Each predictor has sample standard deviation 2. The standardized training rows are \((-1,-1)\), \((0,0)\), and \((1,1)\); hence the covariance matrix is \(\begin{pmatrix}1&1\\1&1\end{pmatrix}\). Choose the leading loading \(v_A=(1,1)^\mathsf{T}/\sqrt{2}\). The assessment record standardizes to \(((4-2)/2,(2-2)/2)=(1,0)\), so:
In fold B, the means are \((10,3)\), and both sample standard deviations are 2. The standardized training rows are \((-1,1)\), \((0,0)\), and \((1,-1)\). Its covariance matrix is \(\begin{pmatrix}1&-1\\-1&1\end{pmatrix}\), with leading loading \(v_B=(1,-1)^\mathsf{T}/\sqrt{2}\). The assessment record standardizes to \(((4-10)/2,(2-3)/2)=(-3,-0.5)\), giving:
Check. The fold A score is positive, while the fold B score is negative. That is not a calculation inconsistency: fold A’s training predictors move together, whereas fold B’s move in opposite directions, and the assessment record is centered differently in each fold. The incorrect calculation would apply fold A’s means, scales, and loading in fold B, producing \(0.7071\) there as well. That number is not the fold B score because it uses the wrong fitted mapping.
Why the Projection Formula Works
PCA chooses directions that capture large variation in the training data. For a unit direction \(v\), the variance of the training scores is \(v^\mathsf{T}Sv\), where \(S\) is the training covariance matrix. Maximizing this quantity subject to \(v^\mathsf{T}v=1\) gives the eigenvector equation \(Sv=\lambda v\); the direction with the largest eigenvalue is the first principal component.
Once a direction has been selected, the score formula has a direct geometric justification. For a standardized row vector \(z\) and a unit loading vector \(v\), approximate \(z\) by \(av\). The squared reconstruction error is \(\lVert z-av\rVert^2\). Expanding and differentiating with respect to \(a\) gives:
Setting the derivative to zero yields \(a=z^\mathsf{T}v\). The second derivative is \(2>0\), so this value minimizes the squared error. Thus projecting an assessment record by the training-fold loading gives its coordinate along that fitted direction. The justification depends on using the same feature definitions and scaling that were used to estimate the loading.
Component Signs and Predictive Meaning
An eigenvector and its negative describe the same PCA direction. If a numerical routine returns \(v\), another valid fit may return \(-v\). The resulting scores also reverse sign. This is not a change in the underlying direction, but it matters when scores are used by a predictive model: the fitted coefficient must be expressed in the same sign convention as the scores.
Worked Example: A Sign Reversal Does Not Change the Prediction
A standardized assessment record is \(z=(1,1)\). One valid first-component loading is \(v=(1,1)^\mathsf{T}/\sqrt{2}\); another is \(-v\). A linear predictive model using the first convention has linear predictor \(\eta=5+2t\). Show how to express the same prediction under the reversed convention and calculate the error from reversing the loading but not its coefficient.
Method. Calculate both scores and adjust the coefficient with the coordinate sign. This avoids treating a valid PCA sign reversal as a substantive change in the predictor.
With \(v\), the score is \(t=(1+1)/\sqrt{2}=\sqrt{2}\). The linear predictor is \(5+2\sqrt{2}\), approximately \(7.8284\). With loading \(-v\), the score is \(t'=-\sqrt{2}\). To preserve the same linear predictor, use coefficient \(-2\):
If the loading is reversed but the coefficient is incorrectly left at \(2\), the result is \(5+2(-\sqrt{2})=5-2\sqrt{2}\approx 2.1716\). The prediction changes by \(4\sqrt{2}\approx 5.6569\), entirely because the coefficient was not converted to the new sign convention.
Check. The product of the score and coefficient is unchanged when both signs reverse. The mistake is changing the PCA sign while interpreting the model coefficient as if its score convention had stayed fixed.
Implementation Checks and Common Traps
In R, the central operational rule is to call the PCA fitting routine separately on each training-fold predictor matrix. For example, a fold-specific fit can use the equivalent of prcomp on the training predictors with centering and scaling enabled. Use that fit’s rotation matrix and its training-derived center and scale when calculating assessment scores. Do not call PCA once on the full predictor table and then divide the resulting scores into cross-validation folds.
Before accepting a fold’s scores, check that:
- The training and assessment rows are assigned to the intended fold.
- Centers, scales, covariance information, and loadings were estimated from training rows only.
- Assessment predictors are transformed with that fold’s centers and scales, not statistics calculated from assessment rows.
- The assessment score uses the loading from the same fold’s PCA fit.
- Any comparison of component effects accounts for possible changes in loading signs and directions across folds.
Scaling is a modeling choice, not a universal requirement. If the specified procedure centers but does not scale, apply that choice consistently: estimate the centers on the training fold, leave the predictors unscaled, and project onto that fold’s loadings. If a training predictor has zero variance, its scale is undefined; a procedure that scales variables must handle such a predictor using a training-only rule, such as excluding it from that fold’s PCA. An assessment value cannot make a constant training predictor variable for purposes of fitting that fold’s components.
A further trap is to assume that component number alone guarantees a common interpretation. “First component” means the direction of greatest variance in a particular training fold. Its loadings can differ across folds, and its sign is arbitrary. Consequently, fold-specific scores are appropriate inputs to the model evaluated within that fold, but should not be treated as if they were necessarily a single globally fitted feature with identical coordinates.
Check Your Understanding
For each question, use the stated fold-specific PCA mapping rather than inventing statistics from the assessment records.
- A training fold has means \((1,2)\), sample standard deviations \((2,1)\), and first loading \((1,1)^\mathsf{T}/\sqrt{2}\). What is the first-component score for assessment record \((5,2)\)? (Answer: \(\sqrt{2}\approx 1.4142\))
- For the same assessment record and scaling as question 1, a valid PCA fit returns the loading with the opposite sign. What score does it return? (Answer: \(-\sqrt{2}\approx -1.4142\))
- A fold’s training means are \((10,4)\), its sample standard deviations are \((2,1)\), and its retained loading is \((0,1)^\mathsf{T}\). Find the retained score for assessment record \((12,7)\). (Answer: \(3\))
- A training-fold covariance matrix is \(\begin{pmatrix}2&1\\1&2\end{pmatrix}\), and the selected first loading is \((1,1)^\mathsf{T}/\sqrt{2}\). An assessment record, after applying that fold’s training-only centering and scaling, is \((1,0)\). Calculate its first-component score. (Answer: \(1/\sqrt{2}\approx 0.7071\))
- A procedure fits one PCA mapping using all records before making cross-validation folds. State the correction and identify which records should determine the means, scales, and loadings for one fold. (Answer: Refit PCA within each fold; that fold’s training records determine its means, scales, and loadings.)