Why Scaling Must Be Fold-Wise
Genomic predictors can be measured on very different scales. One feature might be a small proportion, another a large count, and a third a transformed expression value. Scaling puts features on a chosen common scale, which can matter for methods that use distances or penalize coefficient size. But in cross-validation, the scaling parameters are part of the prediction procedure: they must be estimated using only the training samples in each fold.
A held-out fold represents samples whose outcomes and measurements are not available when a model is trained for evaluation. If the mean, standard deviation, or other cohort-level normalization parameter is estimated using those samples, information about the held-out covariates has entered the training process. This is leakage even if the outcomes were not used. It can make a validation result too favorable, although the effect is not guaranteed to improve performance in every data set.
Fitting and Applying Common Transformations
For z-score scaling, estimate a mean and sample standard deviation separately for each feature from the training samples. Use the same estimates for both the training and held-out values. Consequently, the transformed training values have mean zero and sample standard deviation one, when the training feature has nonzero standard deviation. The held-out values need not have mean zero or standard deviation one; forcing them to do so would use the held-out fold to fit another transformation.
Other transformations follow the same rule. For min-max scaling, estimate the minimum and maximum in training data, then apply them to held-out values. A held-out transformed value can fall below zero or above one; that is not an error, but a consequence of applying the training transformation without refitting it. Robust scaling uses training-fold quantities such as the median and interquartile range, again carried over unchanged to held-out samples.
Worked Example: Z-Score Scaling With a Held-Out Extreme
Suppose three hypothetical training samples, A, B, and C, have values \(0\), \(1\), and \(0.15\) for feature 1, and values \(0\), \(0.10\), and \(0.09\) for feature 2. A held-out sample Q has feature-2 value \(100\). Find the training estimates and compare Q with A and B on feature 2 after scaling.
For feature 1, the training mean is \((0+1+0.15)/3=0.3833\). The sum of squared deviations is approximately \(0.5817\), so the sample variance is \(0.5817/(3-1)=0.2908\), and the sample standard deviation is \(\sqrt{0.2908}=0.5393\).
For feature 2, the training mean is \((0+0.10+0.09)/3=0.06333\). The sum of squared deviations is approximately \(0.006067\). Thus the sample variance is \(0.006067/2=0.003033\), and the sample standard deviation is \(\sqrt{0.003033}=0.05508\).
Apply this feature-2 standard deviation to the difference between Q and A: \((100-0)/0.05508\approx1816\) standardized units. The difference between Q and B is \((100-0.10)/0.05508\approx1814\) standardized units. These are extremely large distances because Q is far from the training values. They are not a reason to recalculate the mean or standard deviation using Q; doing so would change the fitted procedure using held-out information.
Worked Example: Min-Max Scaling Outside the Training Range
A hypothetical training fold has feature values \(2\), \(4\), and \(8\). Its minimum is \(2\) and its maximum is \(8\). Use min-max scaling to transform these values and a held-out value of \(10\).
The training-fold transformation is \(x^*=(x-2)/(8-2)\). The training values become \((2-2)/6=0\), \((4-2)/6=0.3333\), and \((8-2)/6=1\). The held-out value becomes \((10-2)/6=1.333\).
The held-out result exceeds one because the sample is beyond the training maximum. Clipping it to one would be a separate modeling choice, not a consequence of min-max scaling itself. If clipping is part of a prespecified prediction pipeline, it must be applied consistently without estimating new parameters from the held-out outcomes.
Why Whole-Cohort Scaling Can Be Optimistic
The issue is not merely that a held-out standardized value may look unusual. Using held-out covariates to estimate scaling parameters can change which features dominate a model’s predictions. Since the held-out covariates can be associated with their outcomes, this can make predictions on that fold look better than predictions from a procedure whose preprocessing was fitted using training samples alone.
The following small hypothetical nearest-neighbor example shows one possible direction of the effect. It is deliberately simple so that the distance calculations can be checked directly; it does not imply that whole-cohort scaling always increases accuracy.
Worked Example: Leakage Changes a Validation Score
There are two training samples: A has features \((0,0)\) and class A; B has features \((10,1)\) and class B. Three held-out samples have features \((-100,0.8)\), \((0,0.8)\), and \((100,0.8)\), and all have class B. A 1-nearest-neighbor classifier predicts the class of the nearest training sample using Euclidean distance after feature-wise z-score scaling.
Proper fold-wise scaling. The training standard deviations are \(\sqrt{50}=7.071\) for feature 1 and \(\sqrt{0.5}=0.7071\) for feature 2. For the held-out sample Q with features \((0,0.8)\), the scaled distance to A is \(0.8/0.7071=1.131\). Its distance to B is \(\sqrt{(10/7.071)^2+(0.2/0.7071)^2}=\sqrt{2+0.08}=1.442\). Q is therefore classified as A, incorrectly.
The other held-out samples are classified by their distances in the same way. The sample with feature 1 equal to \(-100\) is nearer to A; the sample with feature 1 equal to \(100\) is nearer to B. Thus the fold-wise predictions are A, A, and B, giving \(1/3\) correct.
Leaky whole-cohort scaling. If the five samples are scaled together before splitting, feature 1 has mean \((0+10-100+0+100)/5=2\) and sample standard deviation \(\sqrt{20080/4}=70.85\). Feature 2 has mean \((0+1+0.8+0.8+0.8)/5=0.68\) and sample standard deviation \(\sqrt{0.608/4}=0.3899\). For Q, the distance to A is \(\sqrt{(0/70.85)^2+(0.8/0.3899)^2}=2.052\). The distance to B is \(\sqrt{(10/70.85)^2+(0.2/0.3899)^2}\approx0.532\), so Q is now classified as B.
With these whole-cohort scales, the samples at feature 1 values \(-100\) and \(100\) are also nearer to B. The resulting validation score is \(3/3\), compared with \(1/3\) under fold-wise scaling. In this constructed example, the held-out cohort changed the relative distance weights and happened to make all three held-out predictions correct. The difference illustrates a potential optimistic bias in the reported validation score; it does not establish that the leaked procedure will outperform the proper one in general.
A Fold-Wise Workflow and Its Assumptions
A scaling operation is suitable for this workflow when its parameters can be estimated from the training predictors and then used unchanged to transform new samples. The evaluation must also match how the eventual prediction will be made: a model intended to predict one new sample should not depend on the other held-out samples’ feature distribution. Some predeclared operations computed independently within each sample may be different from cohort-level scaling, but they must not use held-out outcomes or learn parameters across the training and held-out samples.
Use the chosen validation design to identify training and held-out samples for this fold.
For each feature, calculate the training-only mean and standard deviation, or the training-only parameters for the selected alternative transformation.
Apply the training-fold parameters unchanged to training and held-out predictors. Do not recenter or rescale the held-out fold.
Fit the model using transformed training predictors, then produce predictions for the transformed held-out predictors and calculate the chosen performance measure.
Re-estimate the transformation separately within each new training fold; parameters from one fold do not carry over to another.
This procedure estimates performance for a pipeline that receives training samples, fits its preprocessing, and predicts held-out samples. The transformation is not a one-time preliminary calculation for the entire data set. If a final model is later trained on all available samples, its scaling parameters can be estimated from that full training set; that is distinct from using the same samples to assess held-out performance.
Diagnostics, Alternatives, and Limitations
A practical diagnostic is to record the fitted parameters for each fold and verify that they were calculated from training rows only. Also check that the prediction code applies those exact values to the held-out rows. Transformed held-out values outside the training range, or held-out means that are not zero, are not by themselves signs of a bug. They can reveal distribution shift, which should be reported rather than hidden by refitting the transformation.
| Transformation | Training-fold parameters | Useful feature | Important caution |
|---|---|---|---|
| Z-score scaling | Mean and sample standard deviation | Centers features and measures variation in standard-deviation units | Sensitive to extreme training values; requires nonzero standard deviation |
| Min-max scaling | Minimum and maximum | Maps the training range to zero through one | Held-out values can fall outside that range; extreme training values set the scale |
| Robust scaling | Median and interquartile range | Less sensitive to extreme values than mean and standard deviation | Must still be fitted within each training fold; can fail when the interquartile range is zero |
A feature constant in the training fold has sample standard deviation zero, so the z-score formula is undefined. Do not divide by zero or estimate a replacement scale from the held-out fold. A defensible prespecified pipeline can remove that feature for the fold or use a defined fallback; whichever choice is used must be based on training data and applied consistently. A feature that varies in the held-out fold but not in training cannot be learned from that fold.
Different algorithms respond differently to scaling. Distance-based methods can change their notion of which samples are close; penalized models can change the effective penalty on coefficients. In contrast, some unpenalized models with an intercept are invariant to certain affine changes of predictors when the model is refitted consistently. This does not make whole-cohort scaling safe: the evaluation should reflect the actual pipeline and information available at prediction time. Fold-wise z-score scaling is also not a substitute for choosing a scientifically appropriate normalization; it only specifies where its data-dependent parameters are estimated.
Common Mistakes
- Scaling once before cross-validation. The wrong reasoning is that the transformation uses predictors, not labels, so it cannot leak. The correction is that held-out predictors still contribute information to estimated parameters; refit the transformation within each training fold.
- Standardizing each held-out fold to its own mean and standard deviation. This makes the held-out transformation depend on the composition of the evaluation cohort. Apply the training-fold parameters instead.
- Assuming held-out z-scores must have mean zero. Only the training values used to estimate the mean are centered to zero. Held-out values can have a nonzero mean, which may indicate a genuine shift.
- Clipping an out-of-range held-out value without specifying that rule. Min-max scaling alone does not clip. If clipping is chosen, define it as part of the pipeline and apply it consistently.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A training fold has mean \(5\) and sample standard deviation \(2\) for a feature. What is the transformed value of a held-out observation equal to \(9\), and should the held-out fold be recentered?
- Why can a min-max transformed held-out value exceed one?
- In the nearest-neighbor example, what is the proper fold-wise prediction for Q, and what distance comparison determines it?
- A feature has zero standard deviation in a training fold but varies in the held-out fold. What should not be done, and give one suitable fold-local response.
Solutions. (1) The transformed value is \((9-5)/2=2\); do not recenter the held-out fold. (2) The held-out value can exceed the training maximum, while the transformation uses the training minimum and maximum unchanged. (3) Q is classified as A because its fold-wise distances are \(1.131\) to A and \(1.442\) to B. (4) Do not estimate a scale using held-out data or divide by zero; a prespecified option is to remove the constant training feature for that fold or apply a defined training-only fallback.