Why an Unsupervised Representation Can Still Leak
Principal component analysis (PCA) and other unsupervised representations do not use outcome labels when they estimate directions or coordinates. That does not automatically make them safe to fit before cross-validation. In an inductive prediction task, the model is meant to predict outcomes for new samples using information available when each prediction is made. If a held-out sample helps determine the representation used to predict its own outcome, the evaluation no longer follows that setup.
PCA makes the issue concrete. Its directions depend on the observed predictors: adding a sample can change the feature means, variances, covariance structure, principal directions, and the coordinates assigned to other samples. Fitting PCA once on all samples before making folds therefore allows held-out predictor values to influence the representation used in training and evaluation. The labels may remain hidden, but information from the held-out covariates has crossed the fold boundary.
This is a question about the evaluation target, not a claim that every all-sample PCA fit must increase a performance estimate. If the intended task is explicitly transductive—for example, predictions are made after the entire unlabeled prediction cohort is available and it is acceptable to use that cohort’s predictor distribution—using those predictors may be part of the method. Standard cross-validation for inductive prediction instead treats each held-out fold as unavailable while the prediction procedure is fitted.
Fitting PCA Within a Fold
Suppose the predictor matrix has \(n\) samples in rows and \(p\) features in columns. For one cross-validation fold, let \(X_{\mathrm{tr}}\) contain the training rows and \(X_{\mathrm{te}}\) the held-out rows. The feature definitions and their order must be the same in both matrices. Any decision to center only, or to center and scale, must follow the intended analysis and be carried out using training-fold estimates.
For centered PCA, calculate a training mean \(\hat{\mu}_{\mathrm{tr},j}\) for each feature \(j\). Subtract those means from training values, form the training covariance matrix, and find its eigenvectors. If features are to be standardized, estimate each feature’s scale \(s_{\mathrm{tr},j}\) from the training data and divide centered values by that scale before estimating the covariance matrix. The choice to standardize matters: it changes the relative influence of features and should not be chosen by inspecting the held-out fold.
Here \(s_{\mathrm{tr},j}=1\) if the analysis centers but does not scale. The columns of \(V_{\mathrm{tr}}\) are the principal directions, and the diagonal entries of \(\Lambda_{\mathrm{tr}}\) are their corresponding variances. If the first \(k\) components are retained, let \(V_{\mathrm{tr},k}\) contain the first \(k\) eigenvectors. A training sample’s score is its centered and, if applicable, scaled row multiplied by these eigenvectors.
The test data use the training-fold means and scales, and are multiplied by the training-fold loadings. Do not recenter the test fold by its own means or recompute its PCA. Doing so would give the test samples a different coordinate system from the one used to fit the prediction model.
Worked Example: An Outlying Test Sample Rotates the PCA Direction
Consider two unscaled features and the following hypothetical fold. The three training samples are \((-1,0)\), \((1,0)\), and \((0,1)\); the held-out sample is \((0,4)\). Use centered PCA without scaling. The training means are \((0,1/3)\).
For the training fold, the sums of squared deviations in the first and second features are \(2\) and \(2/3\). The sum of cross-products is zero. With \(n_{\mathrm{tr}}-1=2\), the sample covariance matrix is
The largest eigenvalue is \(1\), so the first principal direction is \((1,0)^\mathsf{T}\). The held-out sample is centered using the training mean, giving \((0,4-1/3)=(0,11/3)\). Its score on the training-fold first component is \((0,11/3)(1,0)^\mathsf{T}=0\).
Now consider the incorrect procedure of fitting PCA to all four samples. The overall means are \((0,5/4)\). The sum of squared deviations in the second feature is \(43/4\), so its sample variance is \((43/4)/3=43/12\). The first-feature sample variance is \(2/3\), and the cross-product is zero. The first principal direction is now \((0,1)^\mathsf{T}\), not \((1,0)^\mathsf{T}\). The held-out sample’s score is \(4-5/4=11/4\) on that component.
The outlying test value changed which direction PCA called “first.” The all-sample score is not the score that would result from fitting the representation on the training fold and then applying it to a new sample. This calculation demonstrates contamination of the representation; by itself, it does not imply that a particular performance estimate must be optimistic.
A Fold-Local Procedure and Its Justification
In each fold, fit the representation once on that fold’s training samples, then use the fitted mapping for both training and held-out samples. Any data-dependent choice involved in the representation—such as how many components to retain—must also be made without using held-out information. The prediction model is then fitted on the training scores and applied to the held-out scores.
Use the fold assignment appropriate to the prediction target and unit of validation. Mark the held-out samples as unavailable for fitting the representation.
Estimate any training-dependent centering, scaling, PCA directions, and component-retention decision from training samples only.
Apply the fitted training transformation to training samples and use its unchanged parameters to map held-out samples.
Fit the prediction model using training-fold representations, then calculate predictions and the chosen performance measure for held-out samples.
The justification follows directly from the evaluation design. For an inductive procedure, a prediction for a held-out sample must be produced by a rule fitted without that sample’s information. The training-fold mean, scale, and loading matrix define that rule for PCA. Projecting the held-out row through this fixed rule uses its predictors to make its prediction, as intended; using the row to estimate the rule would change what procedure is being evaluated.
The same logic applies to unsupervised representations other than PCA. A learned embedding, such as one estimated by a neural-network encoder or a data-adaptive manifold method, can depend on the sample set used to fit it. If the method offers an out-of-sample transformation, fit its representation on the training fold and use that transformation for held-out samples. Some methods do not provide a natural, fixed out-of-sample map. In that case, define an appropriate mapping method using training data, or use a validation design that matches the intended transductive task; do not silently fit the embedding jointly on training and held-out samples and report the result as inductive evaluation.
Worked Example: Scaling and Projecting a Held-Out Profile
Suppose a training fold contains three hypothetical samples with values \((10,100)\), \((12,120)\), and \((14,140)\) for two features. The training means are \(12\) and \(120\). The sample standard deviations are \(2\) and \(20\), respectively. After training-fold standardization, the rows are \((-1,-1)\), \((0,0)\), and \((1,1)\).
The standardized training covariance matrix is
Its leading eigenvector is \(v_1=(1,1)^\mathsf{T}/\sqrt{2}\), with eigenvalue \(2\). Now project a held-out profile \((16,180)\). Using only the training estimates, its standardized values are \(((16-12)/2,(180-120)/20)=(2,3)\). Its first-component score is
The score is in the training-defined component coordinate system. In particular, neither the test-fold mean nor a loading estimated after including \((16,180)\) appears in this calculation. The perfect correlation in this small training example makes the leading direction especially transparent; real data need not have such a simple covariance structure.
Worked Example: Choose the Number of Components Within the Fold
For a separate hypothetical training fold, suppose four centered training profiles are \((-2,-1)\), \((-2,1)\), \((2,-1)\), and \((2,1)\). Use centered PCA without scaling. The sample variances are \(16/3\) and \(4/3\), and the covariance is zero, so the first direction is \((1,0)^\mathsf{T}\). Its share of the total training variance is
If a pre-specified rule retains the smallest number of components explaining at least \(75\%\) of training variance, this fold retains one component. The held-out profile \((3,2)\), centered using the training mean \((0,0)\), has first-component score \((3,2)(1,0)^\mathsf{T}=3\). Both the variance calculation and the component-retention decision used only the training fold. If instead the number of components were chosen after looking at held-out outcomes or held-out performance, that choice would use validation information and invalidate this fold’s evaluation.
Diagnostics, Alternatives, and Limits
A useful audit is to list every quantity that helps define the representation and ask which samples were used to estimate it. For PCA, that list includes means, scales, loadings, and any data-driven decision about the number of components. For a learned embedding, it includes the fitted parameters and any settings chosen by examining validation performance. The following checks help identify problems:
- Confirm that each fold has its own fitted representation, rather than reusing an embedding estimated before splitting.
- Check that held-out scores were produced with training-fold means, scales, and loadings. Test-fold centering or a separate PCA fit changes the coordinate system.
- Verify that feature identities and order match between training and held-out matrices. A loading for one feature must not be applied to another.
- For PCA, inspect the training-fold eigenvalues and retained directions. Near-tied eigenvalues can make individual directions unstable; eigenvector signs are also arbitrary. Neither fact permits fitting on held-out samples.
- Check whether the embedding method has a defined out-of-sample transformation. A visualization method that only assigns coordinates jointly to the samples supplied to it may not directly support inductive prediction.
Different strategies answer different questions. Fold-local PCA is a direct option when a linear, training-estimated projection is suitable and an out-of-sample projection is needed. A fixed representation specified independently of the evaluation data can avoid refitting within folds, provided its provenance is genuinely external and its features align with the study data. A transductive embedding can be appropriate when the complete unlabeled prediction cohort is available at deployment and the evaluation design explicitly permits cohort-level information. These strategies should not be treated as interchangeable: each defines a different prediction procedure and potentially a different target for evaluation.
| Approach | What defines the representation | When it matches the evaluation |
|---|---|---|
| Fold-local PCA | Training-fold estimates; test samples are projected using the fitted map | Inductive prediction to samples not used to fit the representation |
| Fixed external representation | A mapping established independently of the evaluation samples | When the external mapping is appropriate and its provenance does not include held-out information |
| All-sample or transductive embedding | Training and held-out predictor distributions jointly influence coordinates | Only when the intended prediction setting permits access to the full unlabeled cohort and validation reproduces that setting |
Fold-local estimation does not solve every source of bias. It does not repair an unsuitable split, remove dependence between samples that should have been grouped, or justify choices made after inspecting held-out outcomes. Nor does an unsupervised embedding guarantee useful prediction: a direction explaining substantial predictor variance need not contain information about the outcome. The representation, fold design, and target must agree.
A common mistake is: “PCA did not see the labels, so fitting it once on all samples cannot cause leakage.” The correction is that validation protects the held-out samples from influencing the fitted prediction procedure, not only from entering a supervised label-based step. If held-out predictors alter the training representation, the fitted procedure has used information unavailable under the ordinary inductive target. The right question is not merely whether labels were used, but whether any held-out data helped construct the rule being evaluated.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In inductive cross-validation, why can PCA fitted to all predictor rows before splitting still cause leakage when no outcome labels were used?
- A training fold has feature mean \(5\), feature standard deviation \(2\), and a fitted one-dimensional loading of \(1\). What is the standardized PCA score for a held-out value of \(9\)?
- Why is it generally incorrect to center held-out samples using their own fold mean before projecting them through training-fold PCA loadings?
- When could an all-sample unsupervised embedding be suitable, and what must the evaluation design reflect?
Solutions. (1) PCA directions and other fitted representation quantities depend on predictor values. Held-out rows can therefore change the transformation used to fit and evaluate the prediction model, even without labels. (2) The training-based standardized value is \((9-5)/2=2\); multiplying by the loading \(1\) gives a score of \(2\). (3) That centers the test data by a quantity not available from the training fold and changes the coordinate system from the one used for training. Use the training mean. (4) It may be suitable for a transductive task in which the full unlabeled prediction cohort is available when predictions are made. Validation must allow the same cohort-level predictor information and evaluate that intended procedure.