Why Preprocessing Belongs Inside the Resampling Fold
Cross-validation estimates how a modeling procedure performs on observations that were not used to build it. That procedure includes more than the final regression or classification model: it also includes any data-dependent preprocessing used to create its inputs. If an imputation value, scaling parameter, or dimension-reduction transformation is calculated using the full data set before the folds are formed, validation observations have already influenced the procedure being evaluated.
The influence can occur without using validation responses. A validation predictor value can affect a mean used for imputation, a standard deviation used for scaling, or a principal component direction. The resulting validation score is therefore not a clean evaluation of a procedure trained without access to that observation. The score need not be too favorable in every case; leakage can move it in either direction.
A preprocessing rule is fitted when its parameters are estimated from data. For example, calculating a mean to fill missing values is fitting an imputer. Replacing a missing validation value with that already calculated mean is applying the fitted imputer. The distinction is essential: validation data must be transformed, but must not help estimate the transformation.
The Fold-Specific Workflow
Suppose a data set is split into \(K\) folds. In turn, each fold serves as validation data while the other \(K-1\) folds form the training data. For each turn, fit every data-dependent preprocessing step using only the current training data. Apply the resulting transformations to the training data and the held-out fold, fit the predictive model on the transformed training data, and evaluate it on the transformed validation data.
Do not use its predictor values or responses to estimate preprocessing parameters or fit the model.
Estimate imputation values, scaling parameters, and any dimension-reduction directions from the training predictors alone.
Use the training-fold parameters without recalculating them when transforming validation predictors.
Fit it using transformed training data, then calculate the chosen validation loss from predictions for the held-out observations.
If a procedure contains several preprocessing steps, each data-dependent step follows the same rule. For example, an imputer and a scaler must both be fitted within the training fold. If principal component analysis (PCA) follows scaling, fit the scaler first using training data, transform the training data, and fit PCA to those transformed training predictors. Apply those same fitted operations, in the same order, to the validation predictors.
What Is Fitted in Common Preprocessing Methods?
| Method | Parameters estimated from the training fold | What is applied to validation data |
|---|---|---|
| Mean imputation | The observed-value mean for each predictor | Fill missing values using the corresponding training mean |
| Standardization | The mean and standard deviation for each predictor | Subtract the training mean and divide by the training standard deviation |
| PCA | The centering values and principal-component directions | Center using training values and project onto the training directions |
For a predictor \(X\) with training-fold values \(x_1,\ldots,x_n\), its training mean and population standard deviation are
When this convention is used, an observed value \(x\), including a validation value, is standardized as \(z=(x-\bar{x}_{\mathrm{tr}})/s_{\mathrm{tr}}\). Some software uses a sample standard deviation with denominator \(n-1\) instead. Either convention can be used if it is specified and applied consistently; the critical requirement is that the parameters come from the training fold.
Worked Example: Imputing a Missing Predictor Without Using Validation Data
A hypothetical claim-severity modeling data set is divided into a training fold with predictor values \(2,4,\text{missing}\) and a validation fold with values \(100,\text{missing}\). Use mean imputation, with the mean calculated from observed values. Find the correct imputed values and compare them with the values obtained by fitting the imputer before the split. The values are fixed for this example; no probability distribution is assumed.
Method. Fit the imputer on the observed training-fold values only. The training mean is \( (2+4)/2=3 \). The missing training value is therefore replaced by \(3\), and the missing validation value is also replaced by \(3\). The observed validation value \(100\) stays \(100\).
If the mean were incorrectly calculated before splitting, the observed values across both folds would be \(2,4,100\). Their mean would be \( (2+4+100)/3=106/3\approx35.333 \). Both missing values would then be filled with approximately \(35.333\). In particular, the validation value \(100\) has changed the imputed training value from \(3\) to approximately \(35.333\).
Check. With fold-specific fitting, the imputed training values are \(2,4,3\), whose mean is \(3\); the validation values are \(100,3\). With full-data fitting, the three observed values sum to \(106\), and dividing by \(3\) gives \(106/3\). The full-data imputer uses validation information. Common mistake avoided: calculating one imputation mean on all available rows and then splitting those rows.
Why Leaked Scaling Changes a Ridge Fit
For ordinary unpenalized regression with a single predictor, a consistent change of scale can leave fitted predictions unchanged. It does not follow that using validation data to scale is harmless. In penalized regression, the scale of predictors affects how the penalty acts on coefficients, so changing the scaling parameters can change the fitted prediction. The next example isolates that consequence. Ridge regression is used only to show the effect of the leaked transformation; the same fold-specific rule applies to any model.
Worked Example: Leaked Scaling Changes a Ridge Validation Error
Consider a hypothetical training fold with predictor-response pairs \((0,0)\) and \((2,2)\), and one validation pair \((10,10)\). Use the population standard deviation for standardization. Fit an intercept and a ridge coefficient by minimizing the sum of squared residuals plus \( \lambda\beta^2 \), with \( \lambda=1 \). The intercept is unpenalized. Calculate the validation squared error using (a) training-fold scaling and (b) scaling incorrectly fitted on all three predictor values. These are fixed illustrative data, not observations from an assumed distribution.
Method, with training-fold scaling. The training predictor mean is \(1\), and its population standard deviation is \( \sqrt{((0-1)^2+(2-1)^2)/2}=1 \). Thus the training standardized predictors are \(-1,1\), and the validation standardized predictor is \( (10-1)/1=9 \). The training response mean is \(1\). For centered predictors and an unpenalized intercept, the ridge coefficient is \( \sum z_j(y_j-\bar{y})/(\sum z_j^2+\lambda) \). Here its numerator is \( (-1)(-1)+(1)(1)=2 \), and its denominator is \(1+1+1=3\), giving \( \hat{\beta}=2/3 \). The fitted intercept on the standardized scale is \(1\). The validation prediction is \(1+(2/3)(9)=7\); its squared error is \( (10-7)^2=9 \).
Incorrect method, with full-data scaling. The predictor mean across \(0,2,10\) is \(4\). The population variance is \( ((0-4)^2+(2-4)^2+(10-4)^2)/3=56/3 \), so the standard deviation is \(s=\sqrt{56/3}\). The standardized training predictors are \(-4/s,-2/s\), and the standardized validation predictor is \(6/s\). The training response mean remains \(1\). The ridge coefficient numerator is \( (-4/s)(-1)+(-2/s)(1)=2/s \). Its denominator is \(16/s^2+4/s^2+1=20/(56/3)+1=15/14+1=29/14\). Therefore \( \hat{\beta}=(2/s)/(29/14)=28/(29s) \). The prediction is \(1+(28/(29s))(6/s)=1+168/(29(56/3))=1+63/203=266/203\). The validation squared error is \( (10-266/203)^2=(1764/203)^2\approx75.5101 \).
Check. Training-fold scaling produces validation prediction \(7\) and squared error \(9\). Leaked full-data scaling produces prediction \(266/203\) and squared error approximately \(75.5101\). The large difference arises because ridge penalizes the coefficient on the standardized scale, and the two scaling procedures produce different standardized training predictors. Common mistake avoided: assuming that a validation observation cannot affect a model fit when only predictor preprocessing, rather than response values, was leaked. These calculations illustrate one possible direction of change, not a rule that leakage always increases or always decreases validation error.
PCA Must Also Be Fitted Within the Fold
PCA uses the predictor values to estimate directions of variation. It does not need the response variable to leak information: a validation predictor can still change the direction onto which training and validation observations are projected. For a training-fold predictor vector \(x\), PCA centers using the training mean and projects onto directions estimated from the training-fold covariance matrix. Validation predictors are centered with that same training mean and projected onto those same directions. Recalculating the mean or directions using the validation fold changes the fitted transformation.
Worked Example: A Validation Point Rotates the Principal Direction
The training predictors in a hypothetical two-variable problem are \((-1,0)\) and \((1,0)\). The validation predictor is \((0,3)\). Use centered PCA with the population covariance matrix. Determine the leading principal direction when PCA is fitted correctly and when it is incorrectly fitted on all three observations, then compare the validation projections. No response values or probability distribution are needed.
Method, with training-fold PCA. The training mean is \((0,0)\). The training covariance matrix is \( \begin{pmatrix}1&0\\0&0\end{pmatrix} \), since the first coordinate has variance \((1+1)/2=1\) and the second is always zero. The leading principal direction is therefore the first coordinate axis, \((1,0)\). The validation point centered at the training mean is \((0,3)\). Its projection onto \((1,0)\) is \(0\).
Incorrect method, with full-data PCA. The mean across all three points is \((0,1)\). After centering, the points are \((-1,-1),(1,-1),(0,2)\). Their population covariance matrix is \( \begin{pmatrix}2/3&0\\0&2\end{pmatrix} \): the off-diagonal entries cancel, and the second-coordinate variance is \((1+1+4)/3=2\). The leading direction is now \((0,1)\). The validation point centered at the full-data mean is \((0,2)\), so its projection is \(2\).
Check. Training-only PCA gives the first-coordinate direction and validation score \(0\); full-data PCA gives the second-coordinate direction and validation score \(2\). The validation point has changed the representation even though it contributed no response value. Common mistake avoided: treating PCA as harmless because it is unsupervised. “Unsupervised” means the response is not used to fit the transformation; it does not mean that held-out predictors can be used without consequence.
Justification: Why the Validation Fold Must Be Held Out
The logic follows from what a validation score is intended to assess. In a resampling turn, the training data determine a complete fitted prediction rule, including the transformations required to prepare future predictor values. The validation observations then test that rule. If a validation predictor helps determine the rule, the validation observation is no longer independent of the fitting process in the intended sense.
More formally, let \(T\) denote the training fold and \(V\) its validation fold. Write a fitted preprocessing transformation as \(g_T\), emphasizing that it is estimated from \(T\). A prediction for validation predictor \(x\) is made using a model fitted to transformed training predictors and the transformed input \(g_T(x)\). This evaluates a procedure of the form “fit preprocessing on training data, then fit the model.” If instead preprocessing is estimated using \(T\cup V\), the transformation is \(g_{T\cup V}\). The resulting validation loss evaluates a different procedure, because \(V\) helped construct the inputs used in fitting. Thus the fold-specific workflow is required to assess the stated inductive procedure.
This argument does not say that \(g_T\) and \(g_{T\cup V}\) must produce different values on every data set. If they happen to agree, the numerical result may coincide. Nor does it say that every use of unlabeled future predictors is prohibited in every application. Some explicitly transductive tasks allow access to the batch of future predictors when making predictions. In that case, the evaluation design must reflect that intended information access. For ordinary prediction to new observations, fit preprocessing on the training fold alone.
Common Traps and Practical Checks
- Transforming once before cross-validation. A single full-data scaling, imputation, or PCA step before making folds allows each validation fold to affect its own transformation. Repeat fitting separately in every fold.
- Recomputing validation statistics. Do not center validation data by its own mean or fill its missing values by its own mean when evaluating an inductive procedure. Apply the training-fitted rule.
- Assuming only labels can leak. Predictor-only preprocessing can still pass information from validation predictors into model construction.
- Assuming leakage is always optimistic. The direction and size of its effect depend on the transformation, model, and data. The problem is that the score no longer evaluates the intended procedure, not that its error must always be smaller.
- Ignoring the order of operations. If scaling precedes PCA, each fold’s PCA must be fitted to that fold’s scaled training predictors. Apply the training scaling and training PCA to validation data in the same order.
A useful audit is to ask, for every quantity estimated from data: which observations were allowed to determine it? Means, standard deviations, imputation values, and PCA directions should be traceable to the training portion of the current fold. Repeating this audit for every fold gives a validation estimate aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Check Your Understanding
For each problem, assume ordinary inductive prediction unless stated otherwise. Work out which information may be used to fit the transformation.
- A training fold has observed predictor values \(1\) and \(5\), plus one missing value. The validation fold has an observed value \(9\) and one missing value. With mean imputation, what values replace the missing entries under the correct fold-specific procedure, and what value would an incorrectly full-data imputer use? (Answer: Correct training-fold mean \(3\), used for both missing entries; full-data mean \(5\).)
- A training fold has predictor values \(2,4\), and a validation value is \(8\). Using population standard deviations, find the validation standardized value using training-fold scaling. Then find it using full-data scaling. (Answer: Training-only mean \(3\), standard deviation \(1\), so \(z=5\); full-data mean \(14/3\), standard deviation \(\sqrt{56}/3\), so \(z=10/\sqrt{56}\approx1.3363\).)
- Training predictors are \((-2,0)\) and \((2,0)\); the validation predictor is \((0,4)\). With centered PCA and population covariance, what is the leading direction fitted on training data, and what is the leading direction if all three points are used? (Answer: Training-only covariance is \(\begin{pmatrix}4&0\\0&0\end{pmatrix}\), giving direction \((1,0)\); full-data mean is \((0,4/3)\), covariance is \(\begin{pmatrix}8/3&0\\0&32/9\end{pmatrix}\), giving direction \((0,1)\).)
- In a cross-validation fold, a standardizer is fitted on the training fold and then applied to validation predictors. Is that leakage? What if the standardizer is fitted once on the complete data set before folds are formed? (Answer: Applying the training-fitted standardizer to validation data is correct; fitting it on the complete data set before splitting leaks validation predictor information.)
- A team argues that PCA fitted on all predictors before cross-validation is safe because it never uses responses. What is the flaw in that reasoning? (Answer: Validation predictors can affect the estimated centering values and principal directions, so the evaluated transformation is not fitted on training data alone.)