Why Unsupervised Preprocessing Must Stay Inside Each Fold
PCA and clustering do not use the response variable when they are fitted, but they still learn from the distribution of the predictor data. A principal component depends on the observed predictor means, scales, and covariance structure. Cluster centers depend on which observations were available to the clustering algorithm. If assessment-fold observations contribute to those estimates, information crosses the training–assessment boundary.
That matters because cross-validation is intended to estimate how a modeling procedure performs on observations not used to fit it. The procedure includes preprocessing as well as the final predictive model. In each fold, fit scaling, PCA, or clustering using only that fold’s training observations. Then apply the learned transformation or assignment rule, without refitting it, to the fold’s assessment observations.
The Fold-Specific Workflow
Let fold \(k\) have training observations \(T_k\) and assessment observations \(A_k\). A transformation is estimated from predictors in \(T_k\) only. If it is denoted by \(\widehat{g}_k\), use \(\widehat{g}_k\) on both sets: transform \(T_k\) to fit the predictive model, and transform \(A_k\) to evaluate that model. The assessment set must not be used to re-estimate \(\widehat{g}_k\).
Form the training and assessment sets for the current fold before fitting any transformation.
Estimate means, standard deviations, PCA loadings, or cluster centers using the training fold only.
Use the training-derived constants and loadings on the training observations and the assessment observations. Do not recompute them for assessment.
Fit the downstream model using transformed training data, then predict for transformed assessment data and calculate the chosen performance measure.
The same rule is repeated independently for every fold. Thus, the PCA axes or cluster centers can differ between folds. That is expected: cross-validation evaluates the entire procedure, including how it learns a transformation from a training sample.
PCA: Learn the Coordinates From Training Data
PCA constructs directions that capture variation in the predictor data. For a training-fold predictor vector \(x\), suppose the chosen preprocessing subtracts the training mean \(\bar{x}_{T_k}\) and, if specified, divides each variable by its training standard deviation. Denote the resulting centered or standardized vector by \(z\). PCA estimates loading vectors from the training-fold covariance matrix of these transformed predictors. An observation’s score on a loading vector is its dot product with that vector.
A common convention is to retain the first \(r\) principal components. The downstream model then uses those \(r\) scores rather than the original predictors. The value of \(r\), the choice to scale variables, and any other preprocessing choices must be determined without using the assessment observations for the fold. For this tutorial, assume these choices are specified in advance.
Worked Example: Projecting an Assessment Observation Using Training PCA
A hypothetical pricing model uses two centered predictors. In one fold, the training observations are \((-\sqrt{5},0)\), \((0,0)\), and \((\sqrt{5},0)\). The assessment observation is \((2,1)\). Use the training-fold covariance matrix to find the first principal component and the assessment observation’s score on it.
Method. Compute the covariance matrix from the three training observations, find its leading loading, and use that loading to project the assessment observation. The assessment observation is not included when estimating the component.
The training means are both zero. Using the sample covariance denominator \(n-1=2\), the variance of the first predictor is \((5+0+5)/2=5\). The second predictor is zero for every training observation, so its variance is zero. The covariance is also zero. Therefore, the training covariance matrix is
The largest eigenvalue is \(5\), with corresponding unit loading \(a_1=(1,0)^\mathsf{T}\). The assessment observation is already expressed relative to the training means, so its first-component score is \((2,1)^\mathsf{T}(1,0)=2\). Its second coordinate does not contribute to this score.
Check. The training scores are \(-\sqrt{5}, 0, \sqrt{5}\), approximately \(-2.236, 0, 2.236\). Their sample variance is \((5+0+5)/2=5\), matching the leading eigenvalue. The assessment score is \(2\); it was calculated using the training loading, not by refitting PCA.
Common mistake avoided. Do not append the assessment observation to the training data and recompute the covariance matrix. That would allow its nonzero second coordinate to affect the component, even though the training-fold data contain no variation in that predictor.
Why the PCA Loading Maximizes Training Variation
The projection rule has a direct justification. Let \(S\) be the covariance matrix calculated from the preprocessed training observations. For any unit direction \(a\), a centered training observation \(z\) has score \(z^\mathsf{T}a\). The sample variance of these scores is \(a^\mathsf{T}Sa\). The first principal component chooses a unit direction that maximizes this variance.
To find the maximizing direction under \(a^\mathsf{T}a=1\), form the Lagrangian \(L(a,\lambda)=a^\mathsf{T}Sa-\lambda(a^\mathsf{T}a-1)\). Differentiating with respect to \(a\) and setting the derivative to zero gives \(2Sa-2\lambda a=0\), or \(Sa=\lambda a\). Thus a maximizing direction is an eigenvector of \(S\); the one associated with the largest eigenvalue gives the greatest training-score variance. This is why both the covariance matrix and its eigenvectors must be estimated within the training fold: the assessment data are not part of the sample whose variation the procedure is intended to learn.
Worked Example: Standardize and Apply a Training-Fold PC
In another hypothetical fold, training predictors \(x\) and \(z\) have rows \((0,0)\), \((2,2)\), and \((4,4)\). Use sample standard deviations to standardize each predictor, and suppose the assessment observation is \((6,2)\). Find the training-fold first-component loading and the assessment score.
Method. Estimate the means and standard deviations from the training rows, standardize the training rows, find the leading loading, and apply the same training estimates to the assessment row.
For each variable the training mean is \((0+2+4)/3=2\). The deviations are \(-2,0,2\), so each sample variance is \((4+0+4)/2=4\), and each sample standard deviation is \(2\). Standardizing gives training rows \((-1,-1)\), \((0,0)\), and \((1,1)\).
The sample covariance matrix of these standardized variables has both diagonal entries equal to \(1\). Its off-diagonal entry is \((1+0+1)/2=1\). Therefore,
The leading eigenvalue is \(2\), and a unit loading for it is \(a_1=(1/\sqrt{2},1/\sqrt{2})^\mathsf{T}\). Standardize the assessment observation using the training means and standard deviations: its \(x\) value becomes \((6-2)/2=2\), and its \(z\) value becomes \((2-2)/2=0\). Its score is therefore \((2,0)^\mathsf{T}a_1=2/\sqrt{2}=\sqrt{2}\), approximately \(1.4142\).
Check. The training scores are \(-\sqrt{2},0,\sqrt{2}\), with sample variance \((2+0+2)/2=2\), equal to the leading eigenvalue. The assessment score uses the training-derived standard deviations and loading. Recomputing either from the assessment row would define a different transformation.
Common mistake avoided. Standardize the assessment row with the training means and standard deviations. Do not standardize it relative to the assessment set or calculate a new standard deviation from the combined data.
Clustering: Fit Centers, Then Assign Assessment Observations
Clustering also learns structure from predictors. For a centroid-based method such as \(k\)-means, fitting the clusters estimates centers from the training observations. An assessment observation can then be assigned to the nearest training-derived center under the specified distance rule. If predictors are scaled before clustering, the scaling constants must also come from the training fold.
Cluster labels are arbitrary identifiers: “cluster 1” in one fold need not describe the same region as “cluster 1” in another. More importantly, the assessment observations must not be used to move the fitted centers. Some clustering methods do not define a natural rule for assigning new observations; in that case, the assessment assignment rule must be specified as part of the modeling procedure rather than silently refitting the clustering on all observations.
Worked Example: Assigning an Assessment Observation to Training Clusters
A hypothetical insurer groups training-fold risks using two predictors, with no scaling applied. The four training points are \((0,2)\), \((2,2)\), \((4,2)\), and \((6,2)\), and \(k\)-means with \(k=2\) has fitted the first two points to one cluster and the last two to the other. Assign the assessment point \((4,5)\) using the fitted centers.
Method. Calculate the training-cluster means to obtain the centers, then compare Euclidean distances from the assessment point to those centers. The centers remain fixed during assessment.
The first center is the coordinate-wise mean of \((0,2)\) and \((2,2)\): \(((0+2)/2,(2+2)/2)=(1,2)\). The second center is the mean of \((4,2)\) and \((6,2)\): \(((4+6)/2,(2+2)/2)=(5,2)\). As a check on the stated training assignment, each cluster has within-cluster squared distances \(1+1=2\).
The distance from \((4,5)\) to the first center is \(\sqrt{(4-1)^2+(5-2)^2}=\sqrt{9+9}=\sqrt{18}\approx4.2426\). The distance to the second center is \(\sqrt{(4-5)^2+(5-2)^2}=\sqrt{1+9}=\sqrt{10}\approx3.1623\). Since \(\sqrt{10}<\sqrt{18}\), assign the assessment observation to the cluster centered at \((5,2)\).
Check. The squared distances are \(18\) and \(10\), respectively, so the second center is nearer. The assessment point is assigned using those training centers; it does not contribute to their calculation.
Common mistake avoided. Keep both coordinate differences in the distance calculation. Omitting one difference or using a center recomputed with assessment observations can change the assignment.
What Cross-Validation Estimates
For each fold, the assessment loss is calculated after the fold-specific preprocessing and predictive model have been fitted. With \(K\) folds of equal size, the cross-validation estimate is the average of the \(K\) fold assessment losses. If fold sizes differ, an observation-weighted average is appropriate when the target metric is an average per observation: weight fold \(k\)’s mean loss by the proportion of assessment observations in that fold.
Here \(\widehat{L}_k\) is the mean loss among observations in assessment fold \(A_k\). Each \(\widehat{L}_k\) must come from a pipeline in which scaling, PCA, clustering, and the predictive model were learned without that fold’s assessment observations. The formula cannot correct leakage that occurred before the fold-specific evaluation.
It is also normal for PCA loadings and cluster centers to vary by fold. Component signs can reverse without changing the underlying PCA direction, and cluster labels can be permuted. Do not interpret raw component labels or cluster numbers across folds as though they were a single shared coordinate system. The relevant validation result is the performance of the full procedure on held-out observations.
Common Leakage Patterns and Their Corrections
- Fit PCA once on the entire predictor table, then cross-validate a model on the scores. The assessment predictors have influenced the component directions. Refit PCA within each training fold.
- Calculate scaling constants before making folds. Even without responses, assessment observations have influenced the means or standard deviations. Estimate them on each fold’s training data.
- Cluster all observations and then evaluate predictions by fold. Assessment observations have influenced the cluster structure. Fit clusters in the training fold and apply a defined assignment rule to assessment observations.
- Use different transformations for training and assessment. The model then receives predictors expressed in inconsistent coordinate systems. Apply the same training-derived transformation to both.
- Assume unsupervised means leakage-free. Leakage is about information crossing the evaluation boundary, not only about whether the response was used.
When the final test set is protected, the same principle applies after model selection: learn the final preprocessing rule using the permitted development data, then apply it unchanged to the test predictors. Do not use test results to revise the transformation and continue to treat that test set as untouched.
Check Your Understanding
For each problem, identify the training-only quantities and apply them without refitting on assessment data.
- A training fold has one predictor with values \(1,3,5\). Use the training mean and sample standard deviation to standardize an assessment value of \(7\). (Answer: mean \(=3\), sample standard deviation \(=2\), standardized value \(=2\).)
- A fold’s fitted cluster centers are \((0,0)\) and \((4,0)\). Assign assessment point \((3,4)\) using Euclidean distance. (Answer: distances are \(5\) and \(\sqrt{17}\approx4.1231\); assign to the center \((4,0)\).)
- Training PCA uses a loading \(a=(0.6,0.8)^\mathsf{T}\) after centering by training means \((2,1)\). Find the assessment score for \((5,3)\). (Answer: centered value \((3,2)\); score \(3(0.6)+2(0.8)=3.4\).)
- A two-fold evaluation has 20 assessment observations with mean loss \(0.30\) and 30 with mean loss \(0.20\). Find the observation-weighted cross-validation loss. (Answer: \((20/50)(0.30)+(30/50)(0.20)=0.24\).)
- A practitioner standardizes all predictors and fits PCA once before creating five folds, then fits a separate regression model in each fold. Is the resulting assessment performance a clean estimate for a procedure that learns PCA from training data? (Answer: No. Each assessment fold contributed to the scaling and PCA estimates; refit both within each training fold and apply them unchanged to its assessment fold.)