Why Preprocessing Can Leak Information
A validation set is meant to represent observations the fitted procedure has not seen. But a model is often preceded by steps such as centering, scaling, clipping, or encoding predictors. If a step estimates its rule using the entire data set before the training-validation split—or before cross-validation folds are formed—then held-out observations have already influenced the procedure. The model may never have been fitted to their outcomes, yet the validation exercise is no longer a clean test of a procedure built from training data alone.
This is preprocessing leakage. It is a problem of information flow, not a guarantee that a particular validation score will be too favorable. Leakage can change a score in either direction, or have no measurable effect for a particular model and data set. The essential question is whether information from an observation that is supposed to be held out helped determine any part of the procedure being evaluated.
For a holdout evaluation, call the training data \(T\) and the held-out data \(V\). A learned preprocessing step has a fitting operation that estimates a rule from data, and a transformation operation that applies the rule to observations. The valid sequence is to estimate the rule using \(T\), transform \(T\) and \(V\) using that same rule, fit the prediction model using transformed \(T\), and evaluate predictions on transformed \(V\). The held-out outcomes are used to evaluate predictions, not to choose or fit any part of the procedure.
In \(K\)-fold cross-validation, this sequence is repeated for each fold: one fold is \(V_k\), and all remaining folds form \(T_k\). Every learned preprocessing rule is fitted separately on \(T_k\). The fact that a step uses only predictors, not outcomes, does not by itself make fitting it on all observations safe. The validation predictors are still meant to be unseen when the procedure is constructed.
A Score Can Improve Because of Leakage
The following hypothetical example uses a deliberately simple preprocessing rule so the information path and score difference can be calculated exactly. The rule clips a predictor at the largest value observed in the data used to fit the preprocessing step. Clipping replaces any value above that limit with the limit. In practice, other data-estimated cutoffs can be used; the point is that their estimates must respect the split.
Worked Example: A Full-Data Clipping Limit Makes the Holdout Look Better
Suppose the training observations are \((x,y)=(0,1),(1,3),(2,5)\), and the single held-out observation is \((3,7)\). Use the preprocessing rule \(z=\min(x,c)\), where \(c\) is the largest \(x\) value in the data used to estimate the rule. Then fit a least-squares line with an intercept using the transformed training predictor \(z\). Compare the absolute prediction error on the held-out observation under training-only preprocessing and full-data preprocessing.
Training-only rule. The training maximum is \(c=2\). The transformed training predictors are \(0,1,2\), so the fitted line is \(\hat y=1+2z\): it gives predictions \(1,3,5\) for the three training outcomes. The held-out predictor \(x=3\) is transformed using the training limit, giving \(z=\min(3,2)=2\). Its prediction is
Leaked full-data rule. If the limit is estimated before the split, the full-data maximum is \(c=3\). The training predictors remain \(0,1,2\), so the fitted line is still \(\hat y=1+2z\). But now the held-out predictor is transformed to \(z=\min(3,3)=3\), giving
The full-data procedure has used the held-out predictor to set a limit that determines how that same observation is represented. In this example, the resulting absolute prediction error falls from 2 outcome units to 0, even though no held-out outcome was used to fit the line. The zero error is not a valid estimate of the performance of the training-only procedure. It reflects information flow from the held-out predictor.
This example is illustrative, not a claim that clipping at the observed maximum is appropriate for every application. Nor does it show that every leakage mistake makes a score look better: a different held-out value or preprocessing rule could worsen the score instead.
The reason for the restriction can be stated directly. A validation result is intended to assess the complete procedure that would be constructed using training data and then applied to new observations. If a held-out observation helps estimate a preprocessing rule, the evaluated procedure has had access to information that would not be available when making a genuinely new prediction. Its validation result therefore answers a different question.
A useful distinction is between fixed operations and learned operations. A fixed operation is fully specified before looking at the data and does not estimate parameters from the sample. A learned operation obtains quantities or choices from data, such as a mean, standard deviation, cutoff, or set of categories. Learned operations belong inside the split. A software command may perform both kinds of operations, so the command’s name is not enough to decide whether it is safe; check what data it uses to determine its behavior.
Fold-by-Fold Audit
The same audit applies to a holdout split and to each cross-validation fold. For a given fold, write down the observations that form \(T_k\) and \(V_k\), then trace every learned rule from its inputs to the predictions. A fold is clean only if every such rule is estimated from \(T_k\), and the validation observations are transformed using those estimates without updating them.
Identify the training observations and the held-out observations for this evaluation. In cross-validation, repeat the check separately for every fold.
Include any operation whose values, parameters, categories, or choices are estimated from the data. Do not overlook steps applied before the model-fitting command.
Check whether each rule was estimated using training observations only. Ask whether validation predictors or outcomes could affect the rule.
Apply the training-fitted rule to the validation observations. Do not refit, update, or choose a rule after inspecting the held-out data.
If an operation used held-out observations to estimate a rule, rebuild the analysis so fitting that operation occurs inside each training split, then recalculate the validation results.
The checklist is about provenance: for each quantity used in prediction, be able to say which observations determined it. This includes quantities estimated from predictors alone. It also includes preprocessing choices made after looking at validation results; that is a separate way the held-out information can influence the evaluation, even if the transformation was initially fitted correctly.
Worked Example: Auditing a Fold’s Standardization
Consider a hypothetical four-observation predictor with values \(1,2,3,10\). In one two-fold split, the training observations have predictor values \(1,2,3\), while the validation observation has value \(10\). Suppose the preprocessing step standardizes by subtracting the mean and dividing by the sample standard deviation, using the training-only estimates.
For the training values, the mean is \(\bar{x}_T=(1+2+3)/3=2\). The sample standard deviation is
The correctly standardized validation value is therefore \((10-2)/1=8\). If someone standardizes all four observations before making the folds, the full-data mean is \(\bar{x}_{\mathrm{all}}=(1+2+3+10)/4=4\), and the full-data sample standard deviation is
That full-data procedure represents the held-out value as \((10-4)/4.082\approx1.470\), rather than 8. The held-out predictor has affected both the mean and standard deviation used to transform itself and the training observations. This fold fails the audit, even though the standardization did not use the outcome and even though this calculation alone does not show what direction the model’s validation score will change.
The correction is to estimate \(\bar{x}_T\) and \(s_T\) from the training observations, retain those estimates, and use them to transform the validation observation. For another fold, the training observations would differ, so its mean and standard deviation must be estimated again from that fold’s training portion.
What Leakage Does—and Does Not—Tell You
Leakage can make validation performance appear too good because the held-out data have helped prepare the procedure. But “leakage” does not mean “the score must be optimistic.” A preprocessing change may help or hurt particular predictions. In some settings, such as a model whose predictions are unchanged by a particular transformation, the leakage may not alter the score at all. The procedural problem remains: the evaluation did not follow the intended train-then-predict sequence.
Likewise, looking at validation predictors is still a possible leak even if no validation outcomes are used. For example, using all observations to decide how a predictor should be scaled allows the held-out predictor distribution to affect the representation supplied to the model. This is different from a fixed unit conversion, where no sample information is estimated. The relevant test is not “Did I use the outcomes?” but “Could a held-out observation influence a data-dependent choice or estimate?”
This matters because the validation score is used to judge how a prediction procedure may perform on observations not yet seen. If preprocessing has already incorporated information from the validation observations, that score does not faithfully assess the specified procedure. A clean data flow makes both the model and the preprocessing part of the procedure being evaluated.
Common Mistake
Correction: Predictor values in a validation fold are still held out. If they help estimate scaling parameters, the training portion has not been processed independently of the held-out portion. Estimate the scaler within each fold’s training data, then apply it unchanged to that fold’s validation data. Leakage need not use outcomes to be leakage, and it need not improve the score to be a flaw.
A practical record for an analysis can be short: name each learned preprocessing step, state which observations fitted it in each fold, and verify that the fitted rule was applied—not refitted—to validation observations. If a full-data preprocessing step has already been used, the remedy is not to explain away the resulting score; it is to rerun the validation procedure with the preprocessing inside the split.
Worked Example: Applying the Checklist to a Proposed Workflow
Suppose an analyst plans five-fold cross-validation. Before assigning folds, the analyst estimates a predictor’s mean and standard deviation from all observations, standardizes the entire predictor column, and then fits a separate regression model in each fold. The analyst does not use outcomes when standardizing. Decide whether the workflow passes the audit and state the correction.
Step 1: Mark each split. For fold \(k\), observations in \(V_k\) are held out and the other observations form \(T_k\). The evaluation asks how a procedure built from \(T_k\) predicts \(V_k\).
Step 2: Identify learned operations. The mean and standard deviation are estimated from observed predictor values, so standardization is data-dependent preprocessing.
Step 3: Trace the inputs. The analyst used all observations to estimate both quantities. Thus the observations in \(V_k\) contributed to the mean and standard deviation used for that fold. Not using outcomes does not remove this dependence.
Decision and correction. The proposed workflow fails the audit. For each fold, estimate the mean and standard deviation from \(T_k\) alone, standardize \(T_k\) using those estimates, and apply the same estimates to \(V_k\). Fit the fold’s model using the transformed training data and evaluate it on the transformed validation data. Repeat with fresh training-only estimates for each of the five folds.
Check Your Understanding
For each question, focus on which observations determine the preprocessing rule.
- In a holdout evaluation, a predictor’s mean is calculated from the full data set before splitting. Which part of the procedure has used held-out information, and how should it be changed?
- A fixed conversion from centimeters to meters is applied to every observation. Is this preprocessing leakage? Explain why or why not.
- In one fold, standardization is estimated from the training predictors only and then applied to the validation predictors without updating its parameters. Does this pass the preprocessing audit?
- Why can preprocessing leakage occur even when no held-out outcomes are used?
- Does leakage always make a validation score look better? Give a reason for your answer.