Why Preprocessing Must Respect the Fold
A prediction pipeline includes more than its final regression or classifier. Centering, scaling, nonlinear transformations with data-estimated parameters, and relative weights assigned to omics blocks all affect the representation on which the model is fitted. If any of those quantities are estimated using held-out samples, information from the evaluation data has entered the pipeline before prediction. The resulting performance no longer evaluates the procedure as it would be applied to new samples.
The rule is straightforward: fit every data-dependent preprocessing operation on the training portion of a fold, then apply those fitted operations without modification to that fold’s held-out portion. This applies whether the operation uses the outcome or only the predictors. A predictor-only calculation can still let held-out samples influence the representation learned from training data.
Suppose \(X^{(b)}\) is the predictor matrix for omics block \(b\), with samples in rows and features in columns. In one split, let \(I_{\mathrm{tr}}\) index training samples and \(I_{\mathrm{ho}}\) index held-out samples. A fitted preprocessing step can be represented as \(\widehat{\theta}_{b}=g(X^{(b)}_{I_{\mathrm{tr}}})\), followed by a transformation \(T(X^{(b)};\widehat{\theta}_{b})\). The held-out data are transformed using \(\widehat{\theta}_{b}\), not a separately calculated value \(g(X^{(b)}_{I_{\mathrm{ho}}})\).
The split must respect the independent observational units for the intended prediction task. For example, if the deployment target is a new patient, multiple samples from the same patient must not be divided between training and held-out data. Earlier material on choosing independent units for data splits and cohort-aware splits addresses that design question. Fold-local preprocessing prevents one kind of leakage; it cannot repair an inappropriate split.
Feature Transformations and Scaling
For a feature \(j\) in block \(b\), let the training-fold mean and sample standard deviation be \(\bar{x}_{bj,\mathrm{tr}}\) and \(s_{bj,\mathrm{tr}}\). Standardization uses these training estimates:
The same mean and standard deviation are used for every held-out sample. In particular, the held-out values do not have to have mean zero or standard deviation one after transformation. That is expected: the transformation is defined by the training data, not by a requirement that each new batch be recentered.
A fixed, prespecified operation such as taking a natural logarithm is not itself fitted to a fold. But any parameter estimated from the observed data must be fold-local. For example, a pseudocount chosen from the smallest observed value, a winsorization cutoff estimated from empirical quantiles, or a range used in min-max scaling must be estimated from training data only. If a transformation choice is selected after inspecting all samples, that choice is not independent of the held-out data either.
A training feature with standard deviation zero requires an explicit rule because division by zero is undefined. A pipeline might drop that feature or replace its scale with a prespecified value. The decision must depend only on the training fold, and the same feature handling must be applied to held-out data. A feature that varies in training but happens to be constant in the held-out fold is not a reason to refit its scale.
Worked Example: Standardizing a Held-Out Sample
Suppose a training fold has three hypothetical samples. For feature A, the training values are 1, 2, and 3; for feature B, they are 8, 10, and 12. A held-out sample has values A = 5 and B = 30. Calculate the standardized held-out values using the training-fold sample means and sample standard deviations.
For A, \(\bar{x}_{A,\mathrm{tr}}=(1+2+3)/3=2\), and \(s_{A,\mathrm{tr}}=\sqrt{[(1-2)^2+(2-2)^2+(3-2)^2]/(3-1)}=1\). Thus the held-out value becomes \((5-2)/1=3\).
For B, \(\bar{x}_{B,\mathrm{tr}}=(8+10+12)/3=10\), and \(s_{B,\mathrm{tr}}=\sqrt{[(8-10)^2+(10-10)^2+(12-10)^2]/2}=2\). The held-out value becomes \((30-10)/2=10\). It is not capped or rescaled: it lies ten training standard deviations above the training mean.
The calculation assumes the two features have the same meaning and measurement scale in training and held-out samples, and that the training standard deviations are nonzero. It demonstrates why transformed held-out values need not resemble the transformed training distribution.
Estimating and Applying Block Weights
When blocks contain different numbers of features or have different scales, a model may give one block disproportionate influence. One possible remedy is to assign each block a weight based on its training-fold variability. The weighting rule must be specified as part of the pipeline, and its parameters must be estimated anew within each training fold.
For a simple example, define the average feature variance in block \(b\) as \(v_b=p_b^{-1}\sum_{j=1}^{p_b}s_{bj,\mathrm{tr}}^2\), where \(p_b\) is the number of features in the block. If \(v_b>0\), define its weight as \(\gamma_b=1/\sqrt{v_b}\). After feature centering, multiply every feature in that block by \(\gamma_b\). This rule makes the average training-fold feature variance of the weighted block equal to one. It does not make each individual feature variance equal to one, and it does not guarantee that the block will have equal predictive importance.
Worked Example: Weighting Blocks by Training Variability
Consider hypothetical training-fold data with three samples. Block A has one centered feature with values \(-1,0,1\). Block B has two centered features: \((-2,0,2)\) and \((-1,0,1)\). A held-out sample has A = 2 and block B values \((2,-1)\). Use the average-feature-variance rule above.
The sample variance of A is \([(-1)^2+0^2+1^2]/2=1\), so \(v_A=1\) and \(\gamma_A=1/\sqrt{1}=1\).
The first feature in B has sample variance \([(-2)^2+0^2+2^2]/2=4\); the second has sample variance \([(-1)^2+0^2+1^2]/2=1\). Thus \(v_B=(4+1)/2=2.5\), and \(\gamma_B=1/\sqrt{2.5}\approx0.6325\). The average weighted training variance in B is \(0.6325^2(2.5)\approx1.000\), with the small discrepancy due only to rounding.
Apply the training weights unchanged to the held-out values. A becomes \(2(1)=2\). Block B becomes \((2(0.6325),-1(0.6325))\approx(1.265,-0.6325)\). The held-out values are not used to recompute either block’s variance or weight.
This calculation assumes each training-fold block variance is positive and that the listed features are the block’s complete feature set for this step. A block with zero average variance needs a prespecified training-only handling rule; the formula cannot be applied by dividing by zero. The choice to use average variance, total variance, or another scaling target is a modeling decision, not a universal standard.
A Data-Estimated Range Must Also Be Fold-Local
Leakage is not limited to means and standard deviations. Any parameter derived from the observed predictor values belongs inside the fold. Min-max scaling illustrates the point: the training minimum and maximum determine the map, and held-out values are allowed to fall outside the training range.
Worked Example: Applying Training Min-Max Scaling
Suppose a hypothetical training fold has values 2, 4, and 6 for one feature, and its held-out value is 10. Define the training-fold min-max transformation as \((x-\min_{\mathrm{tr}})/(\max_{\mathrm{tr}}-\min_{\mathrm{tr}})\).
The training minimum is 2 and maximum is 6, so the training values map to \((2-2)/(6-2)=0\), \((4-2)/(6-2)=0.5\), and \((6-2)/(6-2)=1\). The held-out value maps to \((10-2)/(6-2)=2\). The result 2 is valid: the fitted transformation does not promise that new observations fall in the interval from 0 to 1.
If the held-out value were included when fitting the range, the maximum would instead be 10. The training values would then map to 0, 0.25, and 0.5, while the held-out value would map to 1. That procedure lets the held-out sample alter the representation of training samples, so it evaluates a different pipeline.
Why the Fold-Local Rule Is Necessary
The evaluation target is the performance of a complete learning procedure on data not used to construct that procedure. Preprocessing contributes to what is learned: it determines the coordinates, scales, or relative block contributions on which the prediction model operates. If held-out covariates affect those choices, the fitted training pipeline has indirectly used information from the samples meant to assess it.
A short algebraic comparison makes the effect visible. In Worked Example 1, the training values for B have mean 10 and sample standard deviation 2. Fold-local scaling maps the held-out value 30 to \((30-10)/2=10\). If all four values 8, 10, 12, and 30 are used to estimate a single mean and sample standard deviation, their mean is \(60/4=15\), and their sample variance is \([49+25+9+225]/3=308/3\). The standard deviation is \(\sqrt{308/3}\approx10.13\), so the same held-out value becomes \((30-15)/10.13\approx1.480\). The two values differ because the second calculation used the held-out observation to set the transformation.
When cross-validation is used to select a model, each training fold must contain its own preprocessing fit. In nested cross-validation, preprocessing is fitted within the training portion relevant to each model fit; it is not fitted once on all data before the resampling procedure. The earlier material on nested cross-validation for model selection explains the resampling structure; the key point here is to treat preprocessing as part of the fitted procedure at every level where a model is fitted.
Implementation, Diagnostics, and Alternatives
Define the transformation and its parameter-estimation rule in advance, including what happens when a training feature or block has zero variance.
Estimate all data-dependent quantities, such as feature means, scales, ranges, and block weights, using training predictors only.
Apply the same parameters and feature ordering to training and held-out samples; do not refit on held-out values.
Fit the model using transformed training data, then predict for the transformed held-out data.
A practical diagnostic is to record the fitted preprocessing parameters for every split and verify that each was calculated from that split’s training indices. Another useful check is a perturbation test: changing held-out predictor values should not change the fitted training means, scales, ranges, or weights. It may change predictions, as it should, but it must not change the preprocessing object fitted on training data. Also confirm that feature columns are aligned between partitions and that no held-out-specific recentering or rescaling is performed.
| Approach | What determines the transformation? | Use in evaluation |
|---|---|---|
| Global, data-estimated preprocessing | Training and held-out samples together | Leaks held-out distribution information; generally unsuitable for estimating performance on new samples. |
| Fold-local preprocessing | Training samples in each split | Evaluates the pipeline that could be fitted without access to held-out samples. |
| Fixed external reference parameters | A prespecified reference available independently of the evaluation samples | Can be appropriate when deployment will use that reference, but answers a different, explicitly defined evaluation question. |
External parameters are not automatically leakage. For example, a transformation fixed using a separate reference population before the evaluation data are collected may be appropriate if the intended deployment procedure will use that same reference. The reference and its role must be defined independently of the held-out evaluation samples. Conversely, estimating a mean from all samples in a study and calling it a “reference” does not make it external.
Fold-local preprocessing has limits. It prevents held-out predictors from determining fitted transformations, but it does not guarantee transportability across populations, correct model specification, or good predictive performance. Nor does it resolve how to handle missing values, batch effects, or feature selection; those require their own carefully specified fold-local procedures. After evaluation is complete, the chosen pipeline can be refitted on all available development data for deployment. That final fit is distinct from the earlier held-out performance calculation.
Common Pitfalls
- Wrong reasoning: “Scaling uses no outcome, so it cannot leak.” Correction: Held-out predictor values can still affect training means, scales, ranges, and block weights. Fit those quantities on training samples only.
- Wrong reasoning: “A transformed test value outside 0 to 1 means min-max scaling failed.” Correction: The range is defined by training data; a new value may legitimately fall beyond it.
- Wrong reasoning: “Calculate block weights once because they do not use labels.” Correction: If weights are estimated from observed block variability, estimate them separately using each training fold.
- Wrong reasoning: “Refitting the scaler on the held-out fold makes its values comparable.” Correction: That changes the pipeline and uses the held-out distribution. Apply the training-fitted scaler unchanged.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A training fold has feature values 2, 4, and 6. What is the standardized value of a held-out value 8 using the training mean and sample standard deviation?
- Why can a held-out value transformed by training min-max scaling be greater than 1?
- For two blocks with training average feature variances 1 and 4, what are their inverse-square-root weights under the rule in this tutorial?
- Does a prespecified transformation based on an independent external reference necessarily violate fold-local evaluation? State the condition that matters.
Solutions. (1) The training mean is \((2+4+6)/3=4\), and the sample standard deviation is \(\sqrt{[(2-4)^2+0^2+(6-4)^2]/2}=2\). Thus the held-out standardized value is \((8-4)/2=2\). (2) The training minimum and maximum define the range; a new value above the training maximum maps above 1 when the fitted training range is applied unchanged. (3) The weights are \(1/\sqrt{1}=1\) and \(1/\sqrt{4}=0.5\). (4) No, provided the reference and parameters were defined independently of the held-out evaluation samples and the intended deployment procedure will use that same reference. If held-out samples helped estimate the parameters, the transformation is not independent of the evaluation data.