Why “Target-Free” Does Not Mean “Leakage-Free”
A feature transformation can avoid the response variable and still use information that should be held out during model evaluation. The key question is not only whether the response was used. It is also which records were used to estimate the transformation. A mean used for imputation, a scale used for standardization, PCA loadings, and cluster centers are all learned from data. If validation or test predictors help estimate them, the resulting feature construction is not independent of those held-out records.
This matters because validation is intended to estimate how a modeling procedure will perform on new records. In the usual setting, the procedure receives a training sample, learns every required step from that sample, and then applies those learned steps to unseen records. Fitting an unsupervised transformation on the full data changes that procedure: the validation predictors have already influenced the representation in which the model is trained and evaluated. The response values may remain untouched, but the validation set is no longer fully held out.
The issue is not that a transformation learned from a larger sample is automatically worse or that every metric must become optimistic. The issue is that the evaluated procedure has used information unavailable under the intended prediction setting. That breaks the usual interpretation of the validation result as an estimate for a model trained without access to those records.
The Fit-Then-Apply Principle
For an ordinary train/validation split, let \(D_{\text{train}}\) denote the training records and \(X_{\text{val}}\) the validation predictors. A preprocessing rule should be estimated from training predictors only. Write that fitted rule as \(T_{\text{train}}\). The training and validation features are then constructed as \(T_{\text{train}}(X_{\text{train}})\) and \(T_{\text{train}}(X_{\text{val}})\), respectively. The same learned mapping is applied to both sets; it is not re-estimated on the validation set.
This notation highlights a distinction that is easy to overlook. A validation record may be transformed using a training-derived mean or loading; that is necessary to produce a prediction. What is not allowed in ordinary held-out evaluation is letting that record help determine the mean or loading in the first place.
Set aside validation records before estimating data-dependent transformations.
Estimate imputation values, scaling parameters, cluster centers, PCA loadings, and any other learned mapping from the training portion only.
Use the training-derived transformation on both training and validation predictors, without refitting it on validation records.
Fit the model using transformed training records and calculate the chosen validation metric using predictions for the transformed validation records.
In cross-validation, the same principle applies separately within each fold: fit the transformation on that fold’s analysis portion, then apply it to the assessment portion. If a preprocessing choice is being tuned, the choice and the transformation must remain inside the resampling process used for evaluation. This is consistent with foldwise clustering and PCA: the transformation is part of the modeling procedure, not a one-time operation performed before resampling.
Worked Example: Imputation Changes When a Holdout Record Is Included
Worked Example: Imputation Changes When a Holdout Record Is Included
A small hypothetical training set has response values \(y=0,2,4\) and predictor values \(x=0,2,\text{missing}\). A held-out record has predictor \(x=100\); its response is unavailable. Impute the missing training predictor with the observed mean, fit a least-squares line \(y=b_0+b_1x\), and predict the held-out response. Compare proper training-only imputation with the incorrect practice of estimating the imputation mean from all observed predictors.
Method. Estimate the imputation mean from the training records, replace the missing training value, and fit ordinary least squares. For the comparison, repeat the calculations using the held-out predictor in the mean. The common mistake is to treat a target-free imputation calculation as harmless even though the validation predictor contributes to the training data representation.
Training-only imputation. The observed training predictors are 0 and 2, so the mean is \((0+2)/2=1\). The completed training predictors are \(0,2,1\), paired with responses \(0,2,4\). Their means are \(\bar{x}=1\) and \(\bar{y}=2\). The centered cross-product sum is \((0-1)(0-2)+(2-1)(2-2)+(1-1)(4-2)=2+0+0=2\). The centered predictor sum of squares is \((0-1)^2+(2-1)^2+(1-1)^2=1+1+0=2\). Thus \(b_1=2/2=1\), and \(b_0=\bar{y}-b_1\bar{x}=2-1=1\). The prediction at \(x=100\) is \(1+1(100)=101\).
Incorrect full-data imputation. If the held-out predictor is included, the observed values are \(0,2,100\), with mean \((0+2+100)/3=34\). The completed training predictors become \(0,2,34\), still paired with responses \(0,2,4\). Now \(\bar{x}=12\), \(\bar{y}=2\). The centered cross-product sum is \((0-12)(0-2)+(2-12)(2-2)+(34-12)(4-2)=24+0+44=68\). The centered predictor sum of squares is \((0-12)^2+(2-12)^2+(34-12)^2=144+100+484=728\). Therefore \(b_1=68/728=17/182\), and \(b_0=2-(17/182)(12)=80/91\). The held-out prediction is \(80/91+(17/182)(100)=1860/182=10.2198\), rounded to four decimal places.
The two predictions differ because the held-out predictor changed the imputation value and therefore the regression fit. Neither calculation used the held-out response. The first is the valid held-out procedure; the second lets a validation record influence training. The numerical example is deliberately small and does not imply that leakage always moves predictions in a particular direction.
Worked Example: A Held-Out Record Can Rotate the PCA Representation
Worked Example: A Held-Out Record Can Rotate the PCA Representation
Suppose the training predictors are the two points \((1,0)\) and \((-1,0)\), and a held-out record has predictors \((0,10)\). Find the first principal component using the training records alone, then compare it with the first principal component obtained incorrectly from all three records. Use sample covariance, with divisor \(n-1\), and choose a positive direction for the first component.
Method. Center the predictors using the relevant sample mean, calculate the covariance matrix, and identify the direction of greatest variance. The common mistake is to calculate PCA once before the split and then assume that using no response in PCA makes the resulting evaluation independent.
Training-only PCA. The training mean is \((0,0)\). The centered points remain \((1,0)\) and \((-1,0)\). Their sample covariance matrix is \(\begin{pmatrix}2&0\\0&0\end{pmatrix}\), because the sum of squared first-coordinate deviations is \(1+1=2\), the second-coordinate deviations are both zero, and \(n-1=1\). The first component points along the first coordinate. The training scores are \(1\) and \(-1\).
Incorrect PCA on all records. The full-data mean is \((0,10/3)\). The centered points are \((1,-10/3)\), \((-1,-10/3)\), and \((0,20/3)\). The cross-product of the coordinate deviations sums to \(1(-10/3)+(-1)(-10/3)+0(20/3)=0\). The second-coordinate squared deviations sum to \(100/9+100/9+400/9=600/9=200/3\). After dividing by \(n-1=2\), the covariance matrix is \(\begin{pmatrix}1&0\\0&100/3\end{pmatrix}\). Its first component points along the second coordinate, not the first. Under this component, both training records have score \(-10/3\), while the held-out record has score \(20/3\).
The direction changed because the held-out predictor contributed substantial variation in the second coordinate. A predictive model trained on these scores would receive a different representation of the training records from the one it would have received had PCA been fit only on the training sample. The PCA calculation never used a response, but the validation predictors still influenced model training.
Worked Example: Full-Data Clustering Changes Training Features
Worked Example: Full-Data Clustering Changes Training Features
A one-dimensional training predictor has values 0 and 2. A held-out record has value 100. Create \(k=2\) cluster features using \(k\)-means, then compare fitting clusters on training data alone with fitting on all three records. Assume the global minimum within-cluster sum of squares is found.
Method. For each proposed partition, calculate the sum of squared distances from each value to its assigned cluster mean. The common mistake is to cluster all predictors before splitting because cluster fitting does not require a response.
Training-only fit. With two training values and \(k=2\), assigning 0 and 2 to separate clusters gives centers 0 and 2 and within-cluster sum of squares \(0+0=0\). Applying the learned centers to the held-out value 100 assigns it to center 2, since its distances are 100 and 98, respectively.
Incorrect full-data fit. One possible partition puts 0 and 2 together and 100 alone. The first cluster has center \((0+2)/2=1\) and sum of squares \((0-1)^2+(2-1)^2=2\); the singleton cluster contributes 0. If instead 0 is alone and 2 and 100 are together, the latter cluster has center 51 and sum of squares \((2-51)^2+(100-51)^2=2401+2401=4802\). The alternative grouping of 0 and 100 together has the same sum of squares, 4802. Thus the minimum is the partition with center 1 for the training pair and center 100 for the held-out singleton, with total sum of squares 2.
Under the full-data fit, the training values 0 and 2 both receive the same cluster feature, whereas under the training-only fit they receive different cluster features. The held-out predictor has changed the representation of the training data. The cluster IDs themselves are labels, so their numerical order is not meaningful; the consequential change is which records share a cluster.
Why the Evaluation Can No Longer Answer the Intended Question
The usual evaluation question is: how well does a complete modeling procedure perform when it is trained on one sample and applied to new records? Let \(A\) denote that procedure, including preprocessing and model fitting. For a training sample \(D_{\text{train}}\), a fitted procedure should be determined by \(D_{\text{train}}\). If instead preprocessing is estimated using both \(D_{\text{train}}\) and validation predictors, the fitted procedure depends on the validation set.
The reason is direct: data-dependent transformations are functions of their input records. For example, the sample mean changes when a new value is added; PCA depends on the covariance structure; and cluster centers depend on the records included in the clustering objective. If a held-out predictor can change the transformation, it can change training features, fitted model parameters, or both. Therefore the validation score no longer evaluates the same information flow as a model that had no access to those held-out predictors.
There is an important qualification. In a genuinely transductive prediction setting, the complete set of unlabeled future predictors may be available before predictions are made, and the intended algorithm may explicitly use that set to adapt its representation. In that case, using those predictors can be appropriate if it is part of the defined deployment procedure and the evaluation reproduces that procedure. It is not ordinary inductive holdout evaluation, where future records arrive as new observations and the fitted mapping is applied without refitting. The evaluation design must match the intended use.
This distinction is relevant to insurance and other actuarial prediction tasks. For example, a hypothetical insurer might build segments from applicant characteristics. If actual deployment means scoring each new application using fixed centers learned from historical training data, full-data clustering during validation does not reproduce deployment. If deployment instead provides a known unlabeled batch and the operational method explicitly reclusters that batch, the batch-level method must be assessed as such. Neither choice is universally correct; the prediction setting must be stated.
Practical Diagnosis and Common Traps
When reviewing a modeling workflow, identify every operation that estimates something from the observed predictors. Ask what records contributed to each estimate and whether those same records are supposed to represent unseen data. A step can leak even if it is described as exploratory, “unsupervised,” or performed before the model is fit.
- Imputation: Estimate a mean, median, or other imputation rule from the training portion, then apply it to validation records. Do not recompute the imputation value from the combined data.
- Scaling: Estimate centering and scaling parameters on training data. Apply those same values to the validation data rather than standardizing the validation records independently.
- PCA: Estimate the center, scale if applicable, and loadings using the training portion. Project validation records using those fixed training-derived quantities.
- Clustering: Estimate cluster centers on training data. Assign validation records to those centers rather than refitting the clustering with validation predictors included.
- Resampling: Repeat the fitting of each learned transformation inside each training fold. Transform that fold’s assessment records using its fitted mapping.
A frequent exam trap is to reason, “The response was not used, so there is no leakage.” The correction is to trace the data dependency: did a held-out predictor affect a transformation used to construct training features or predictions? If yes, the standard held-out evaluation has been compromised, even though the response did not enter that transformation.
Check Your Understanding
For each situation, identify whether the transformation is fitted correctly for ordinary held-out evaluation and give the requested calculation or correction.
- A training predictor has observed values 2 and 6, plus one missing value. A validation record has predictor value 16. What imputation value should be used for the missing training value, and what value would be obtained incorrectly by including validation? (Answer: Correct training-only mean: 4; incorrect full-data mean: 8.)
- Training records have values \((1,0)\) and \((-1,0)\); a validation record is \((0,8)\). If PCA is fit on training data only, which coordinate defines the first component? (Answer: The first coordinate.)
- A training clustering has one-dimensional values 1 and 5 with \(k=2\), and validation value 20. What are the training-only centers, and to which center is the validation value assigned? (Answer: Centers 1 and 5; validation value 20 is assigned to center 5.)
- A model workflow calculates a predictor’s mean and standard deviation on the full dataset, then splits records and fits a regression model on the training portion. Is the response required to have entered those calculations for leakage to occur? Give the correction. (Answer: No; full-data predictor statistics can leak validation information. Split first, estimate the mean and standard deviation on training data, and apply them unchanged to validation data.)
- In a three-fold cross-validation analysis, a single PCA basis is calculated from all records before the folds are formed. What should be done instead? (Answer: Within each fold, fit PCA on that fold’s training portion and project its assessment portion using those fitted PCA quantities.)