Why Unsupervised Learning Can Leak
PCA and factor extraction do not use outcome labels, but that does not make them automatically safe to fit before a train-test split. These methods learn from the feature values: their means, scales, covariance structure, loadings, and sometimes the number or identity of retained features depend on the samples supplied. If held-out samples contribute to those estimates, the training procedure has received information about the evaluation data before it makes predictions.
For ordinary inductive prediction, the target is performance on new samples whose omics measurements are not available when the model is trained. The validation procedure should imitate that setting. Within each resampling split, learn every data-dependent representation from the training samples, then apply that learned representation to the held-out samples without refitting it. This applies even when the held-out outcomes are hidden.
Let \(X_{\mathrm{train}}\) denote the training feature matrix, with samples in rows and features in columns, and let \(X_{\mathrm{held}}\) denote the held-out matrix. A fitted representation has parameters \(\hat{\theta}_{\mathrm{train}}\), learned from \(X_{\mathrm{train}}\). Its intended use is
Here, \(\mathcal{A}\) is the fitting procedure and \(T\) is the transformation. The held-out matrix is an input to the final transformation, not to the fitting step. In contrast, estimating \(\hat{\theta}\) from the combined training and held-out matrices allows held-out measurements to influence the representation.
Fold-Local PCA and Its Projection Rule
PCA finds orthogonal directions that successively explain large amounts of variance in the data used to fit it. For the usual covariance-based PCA, the training features are centered by training means. If \(p\) features are retained, let \(\bar{x}_{\mathrm{train},j}\) be the mean of training feature \(j\), and let \(\hat{v}_k\) be the training-derived loading vector for component \(k\). For a sample with feature vector \(x\), its score on that component is
For held-out samples, the same training means and loadings must be used. If the training procedure standardizes features, it must also use training-derived scales when transforming held-out values. A held-out sample must not be recentered using a held-out-fold mean, and PCA must not be refitted on the held-out fold.
PCA results depend on choices such as whether to use a covariance or correlation matrix, which features are eligible, and how many components are retained. These choices are part of the representation procedure. If a choice is selected by looking at predictive performance, it must be made inside the training process rather than by inspecting the outer held-out results. The general requirement is not that one particular PCA variant be used, but that all learned choices respect the split.
Worked Example: Fit PCA on Training Samples, Then Project
Consider four hypothetical training samples measured on two features, \(A\) and \(B\): \((0,0)\), \((2,0)\), \((0,1)\), and \((2,1)\). Use covariance-based PCA without scaling. A held-out sample has values \((3,2)\). Find the first training principal component and project the held-out sample onto it.
Check the fitting data. The training means are \(\bar{x}_{A,\mathrm{train}}=1\) and \(\bar{x}_{B,\mathrm{train}}=0.5\). The centered training rows are \((-1,-0.5)\), \((1,-0.5)\), \((-1,0.5)\), and \((1,0.5)\). Their sums of squared deviations are 4 for \(A\) and 1 for \(B\); their cross-product sum is 0. With sample covariance denominator \(4-1=3\), the covariance matrix is
The largest eigenvalue is \(4/3\), with unit eigenvector \(\hat{v}_1=(1,0)^\mathsf{T}\). Thus the first principal component is the direction of feature \(A\). The held-out sample is centered using the training means, giving \((3-1,2-0.5)=(2,1.5)\). Its first-component score is
The held-out value of feature \(B\) contributes no score on this component. Refitting PCA with the held-out point included could change the fitted covariance and loading direction; that would be a different procedure from projecting with the training-derived component.
Other Learned Steps Belong Inside the Fold
A representation often includes steps before or after the decomposition. Any step whose result depends on the observed feature values must be fitted within the training fold. Examples include centering, scaling, variance filtering, choosing among features, estimating factor loadings, selecting a rotation, and learning a rule for calculating factor scores. Applying a training-derived transformation to held-out measurements is not leakage; learning or updating its parameters from those measurements is.
For factor analysis, a common model expresses a centered feature vector as \(x=\Lambda f+\epsilon\), where \(f\) is a vector of latent factors, \(\Lambda\) is a loading matrix, and \(\epsilon\) represents feature-specific variation. A standard identification convention sets the factor covariance to the identity and assumes the specific errors are uncorrelated with one another and with the factors. Estimation procedures may add distributional assumptions or use different fitting criteria. Whatever the chosen model and estimator, the means, loadings, uniqueness parameters, rotations, and score-calculation rule must be estimated on training samples only. A held-out factor score is then calculated using that fitted model, not by refitting the factor model to the held-out samples.
Worked Example: Training-Only Standardization Before a Representation
Suppose three hypothetical training samples have feature \(A\) values \(2,4,6\), and feature \(B\) values \(10,10,14\). A held-out sample has values \(8,16\). Standardize each feature using the training sample mean and sample standard deviation, then calculate the held-out standardized values.
For \(A\), the training mean is \(4\), and the sample standard deviation is \[ s_{A,\mathrm{train}}=\sqrt{\frac{(2-4)^2+(4-4)^2+(6-4)^2}{3-1}}=2. \] The held-out standardized value is \((8-4)/2=2\).
For \(B\), the training mean is \(34/3=11.333\). Its sample variance is \[ s^2_{B,\mathrm{train}}=\frac{(10-11.333)^2+(10-11.333)^2+(14-11.333)^2}{2} =\frac{10.667}{2}=5.333, \] so \(s_{B,\mathrm{train}}=\sqrt{5.333}=2.309\). The held-out standardized value is \[ \frac{16-11.333}{2.309}=2.021. \]
The standardized held-out vector is therefore approximately \((2,2.021)\). These values can be passed to a representation fitted on the standardized training data. If the held-out values had been used to recalculate either mean or standard deviation, the training representation and held-out transformation would no longer be based on the same training-only preprocessing rule.
A particularly clear leakage pathway is unsupervised filtering. Even when a filter never sees outcomes, it can use held-out feature values to decide which features appear variable enough to retain. That changes the input space given to the supervised model and can make evaluation optimistic or otherwise unrepresentative of deployment.
Worked Example: A Held-Out Value Changes a Variance Filter
Suppose a hypothetical feature \(B\) has training values \(0,1,0,1\). A prespecified filter retains a feature only if its sample variance is greater than 1. Calculate the training-only variance, then compare it with the variance obtained incorrectly by pooling in a held-out value of 100.
Using training samples only, the mean is \(0.5\), and the sum of squared deviations is \[ (0-0.5)^2+(1-0.5)^2+(0-0.5)^2+(1-0.5)^2=1. \] The sample variance is \(1/(4-1)=0.3333\), so the filter excludes \(B\).
If the held-out value is incorrectly included, the five values are \(0,1,0,1,100\), with mean \(102/5=20.4\). Their sum of squared deviations is \[ 2(0-20.4)^2+2(1-20.4)^2+(100-20.4)^2 =832.32+752.72+6336.16=7921.2. \] The pooled sample variance is \(7921.2/(5-1)=1980.3\). The filter would now retain \(B\), because \(1980.3>1\). Thus the held-out measurement changed the feature-selection decision. In a valid inductive evaluation, the training-fold filter excludes \(B\); the held-out value cannot reverse that decision.
Procedure, Diagnostics, and Alternatives
A reliable implementation treats each fold as a complete training-and-projection exercise. The fitted preprocessing and representation objects should be saved or passed forward as a single pipeline, so that the held-out data cannot silently trigger a refit.
Separate the training and held-out samples using the independent-unit and cohort structure required by the prediction target.
Estimate eligible-feature decisions, means, scales, PCA or factor parameters, and any score rules using training samples only.
Transform training samples with the fitted parameters and project held-out samples using those same parameters without refitting.
Train the supervised model on the transformed training data, then evaluate predictions on the held-out data. Repeat the full procedure separately for each fold.
Useful diagnostics include checking that the fitted means, scales, retained-feature list, loadings, factor parameters, and component count are all associated with the training fold. Confirm that transforming held-out data does not update those objects. A practical audit is to change or permute held-out feature values and verify that the fitted training parameters remain identical. The resulting projected scores should change when the held-out values change, but the transformation itself should not.
Compare the fold-local approach with its alternatives in light of the intended prediction setting:
| Approach | What it does | When it is appropriate |
|---|---|---|
| Fold-local unsupervised representation | Fits all data-dependent representation steps on training samples and projects held-out samples using fixed parameters. | Standard inductive validation when future samples arrive after model training. |
| Representation fitted before splitting | Uses samples from both training and held-out portions to estimate the representation. | Not appropriate for an inductive performance estimate when held-out samples are meant to represent unseen future data. |
| Fixed, prespecified representation | Uses an externally specified transformation that does not learn parameters from the evaluation samples. | Useful when the feature mapping is genuinely fixed in advance and available at deployment. |
| Transductive representation | Uses unlabeled evaluation-sample measurements to adapt the representation before prediction. | Only when deployment genuinely provides the relevant unlabeled samples in advance and evaluation is designed to estimate that setting. |
Transductive use is not automatically invalid, but it answers a different question. For example, if a deployment procedure will receive an entire batch of new samples before producing predictions and is explicitly allowed to adapt using all their unlabeled measurements, evaluation may reproduce that process. The evaluation must then keep outcomes unavailable during adaptation, define the batch-level procedure in advance, and avoid describing the result as performance for a single new sample processed independently.
Limitations and Common Mistakes
Fold-local fitting prevents one important source of optimism; it does not guarantee that the chosen representation is scientifically suitable, that the validation split matches deployment, or that the resulting performance estimate is precise. PCA prioritizes variance, not predictive relevance: a low-variance feature can carry outcome information, while a high-variance direction can reflect irrelevant biological or technical variation. Factor models also depend on their covariance structure, identification conventions, and estimation choices. These limitations remain even when fitting is leakage-free.
- Mistake: “It is unsupervised, so using all samples is harmless.” Correction: held-out feature values can alter learned means, scales, filters, loadings, and factor scores, which changes what the supervised model receives.
- Mistake: “I applied the same function to training and test data.” Correction: the key question is whether its parameters were estimated once on training samples or recalculated using held-out samples.
- Mistake: “Only the loadings need to be fold-local.” Correction: centering, scaling, feature filtering, component selection, rotations, and score rules may all be learned and must be treated consistently.
- Mistake: “A test set can be used to choose the number of components because no outcomes were used.” Correction: choosing based on held-out feature structure still adapts the prediction procedure to the evaluation samples and is not an inductive assessment.
The earlier material on feature selection inside resampling addresses the same split-respecting principle for outcome-informed selection. Here the essential point is that unsupervised feature learning also has fitted quantities, and those quantities belong inside the training fold. For any representation, ask which information determines its parameters, then ensure that held-out samples contribute only in ways permitted by the intended deployment procedure.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- Training values for one feature are \(2,4,6\), with training mean \(4\) and sample standard deviation \(2\). A held-out value is \(10\). What standardized value should be used if training-only standardization is required?
- Why can fitting PCA on all samples before cross-validation leak information even if PCA never uses outcomes?
- For the covariance matrix \(\begin{pmatrix}5&0\\0&2\end{pmatrix}\), what is the first principal direction, and why must held-out samples use it without refitting?
- A variance filter retains features above a fixed threshold. Can the threshold be fixed in advance while the filter is still fitted within each fold? Explain.
Solutions. (1) Use the training mean and scale: \((10-4)/2=3\). (2) The held-out feature values can change the estimated covariance matrix and therefore the component loadings and scores; labels are not required for this information pathway. (3) The first direction is \((1,0)^\mathsf{T}\), since 5 is the largest eigenvalue. Held-out samples should be projected on this training-derived direction so the evaluation represents application of a model learned without those samples. (4) Yes. The threshold can be prespecified, while the decision about which features exceed it is calculated from training-fold data only; held-out measurements must not change the retained-feature list.