Why Preprocessing Must Respect the Fold Boundary
Cross-validation estimates how a modeling procedure performs on observations not used to fit it. That procedure includes more than the final regression: it also includes any preprocessing whose settings are learned from data. If a fold's validation observations help determine those settings, the validation data have influenced the procedure being assessed.
This matters for elastic net models because scaling affects the relative penalties on coefficients, and imputation or clipping can affect the values the model receives. A validation fold must be processed using the parameters learned from its corresponding training fold, not parameters recalculated from all observations.
The governing distinction is whether an operation is fixed in advance or learned from the available data. Converting a prespecified measurement from days to years uses a fixed rule and does not estimate anything from a fold. By contrast, centering by the observed mean estimates a quantity from the data. The mean and any other learned preprocessing parameters must be estimated from the training fold alone.
The Fold-Specific Preprocessing Rule
For a given fold, let \(D_{\text{train}}\) denote its training observations and \(D_{\text{val}}\) its validation observations. Fit every data-derived preprocessing step using \(D_{\text{train}}\). Keep the resulting estimates fixed when transforming both the training observations and the validation observations. Then fit the model on the transformed training observations and evaluate it on the transformed validation observations.
Set aside the fold's validation observations. Do not use them to estimate preprocessing parameters.
Estimate quantities such as predictor means, standard deviations, or imputation values using training observations only.
Apply the training-fold transformation unchanged to its training observations and its validation observations.
Fit the model on transformed training data, predict transformed validation data, and calculate the chosen validation loss.
For example, if a predictor is standardized using the training-fold mean \(\bar{x}_{\text{train}}\) and standard deviation \(s_{\text{train}}\), every value in either part is transformed as follows:
The validation observations do not get their own mean or standard deviation. Nor should they be combined with the training observations to recalculate the quantities in this formula.
Worked Example: Standardizing Separately Within Each Fold
Worked Example: Standardizing Separately Within Each Fold
Six hypothetical risk records have predictor values \(1,3,5,7,9,11\). In a two-fold cross-validation split, Fold A uses \(1,3,5\) for training and \(7,9,11\) for validation; Fold B reverses those groups. For each fold, standardize using the training sample mean and sample standard deviation. Compare this with improperly calculating one mean and standard deviation from all six values.
Method. In each fold, calculate the training mean and sample standard deviation, then apply those same two values to the validation observations. For a sample of size \(n\), the sample standard deviation is the square root of the sum of squared deviations divided by \(n-1\).
Fold A training estimates. The training mean is \( (1+3+5)/3=3 \). The sum of squared deviations is \( (1-3)^2+(3-3)^2+(5-3)^2=4+0+4=8 \). Thus the sample variance is \(8/(3-1)=4\), and the sample standard deviation is \(s_{\text{train}}=2\).
Apply these training estimates to the validation values: \(7\) becomes \((7-3)/2=2\), \(9\) becomes \((9-3)/2=3\), and \(11\) becomes \((11-3)/2=4\). The Fold A validation values on the training scale are \(2,3,4\).
Fold B training estimates. Its training values are \(7,9,11\), so the mean is \((7+9+11)/3=9\). The squared deviations sum to \(4+0+4=8\), giving sample variance \(8/2=4\) and sample standard deviation \(2\). Its validation values \(1,3,5\) become \((1-9)/2=-4\), \((3-9)/2=-3\), and \((5-9)/2=-2\).
What the incorrect calculation does. Using all six values gives mean \(36/6=6\). Their squared deviations from 6 sum to \(25+9+1+1+9+25=70\). The full-data sample standard deviation is \(\sqrt{70/5}=\sqrt{14}\), approximately \(3.7417\). These parameters use each fold's validation values. For instance, Fold A's validation values would be transformed to approximately \(0.2673,0.8018,1.3363\), rather than \(2,3,4\).
Check. Under the correct Fold A transformation, the training values become \(-1,0,1\), which have mean 0 and sample standard deviation 1. Under the correct Fold B transformation, its training values also become \(-1,0,1\). Common mistake avoided: calculating the scaler once on the entire data set before creating the folds lets validation predictors influence preprocessing.
Imputation and Other Learned Transformations
Imputation follows the same rule. If missing predictor values are replaced by a training-fold mean, median, or another data-estimated quantity, estimate that quantity from the training fold. Apply it unchanged to missing values in both the training fold and the validation fold. Calculating an imputation value from the combined data uses information from validation observations, even when their outcomes are not used.
Worked Example: Imputing a Missing Predictor
A training fold contains four hypothetical claim records with duration values \(2,4,\text{missing},8\) months. Its validation fold contains values \(10,\text{missing}\) months. The prespecified imputation rule is to replace missing values by the observed training-fold mean. Find the imputed values and show what goes wrong if the observed validation value is included in that mean.
Correct training-only calculation. The observed training values sum to \(2+4+8=14\), and there are three observed values. The training-fold mean is \(14/3=4.6667\) months, rounded to four decimal places. Thus the missing training value is replaced by \(4.6667\), and the missing validation value is also replaced by \(4.6667\). The observed validation value \(10\) remains \(10\).
Incorrect pooled calculation. Including the validation value \(10\) gives an observed-value sum of \(2+4+8+10=24\) across four values, for a pooled mean of \(24/4=6\) months. That would replace the missing training value by \(6\), even though the validation record contributed to the replacement rule used to fit the model.
Check. Under the stated procedure, the training-fold imputation value is \(14/3\), not \(6\); the validation value \(10\) is not part of its calculation. Common mistake avoided: omitting validation outcomes does not make it acceptable to use validation predictors to estimate preprocessing parameters.
The same logic applies to a clipping or winsorization threshold estimated from the data. First specify the rule for estimating the threshold; then estimate it using the training fold only. The following example uses a prespecified rule of setting the upper cap equal to the training mean plus one training sample standard deviation.
Worked Example: Applying a Training-Fold Clipping Rule
A training fold has predictor values \(2,4,6\), and its validation fold has value \(10\). The prespecified upper clipping rule sets the cap at the sample mean plus one sample standard deviation. Values above the cap are replaced by the cap. Calculate the correct validation value after clipping, then compare it with the result from incorrectly estimating the cap using all four values.
Correct cap. The training mean is \((2+4+6)/3=4\). The squared deviations are \(4,0,4\), so the sample variance is \(8/2=4\) and the sample standard deviation is \(2\). The training-based cap is \(4+2=6\). Applying it unchanged to the validation value gives \(\min(10,6)=6\).
Incorrect pooled cap. The mean of \(2,4,6,10\) is \(22/4=5.5\). The squared deviations from \(5.5\) are \(12.25,2.25,0.25,20.25\), summing to \(35\). The pooled sample standard deviation is \(\sqrt{35/3}\), approximately \(3.4157\). The pooled cap is therefore \(5.5+\sqrt{35/3}\), approximately \(8.9157\), so the validation value would be clipped to approximately \(8.9157\).
Check. The correctly estimated cap is \(6\), based only on the training values; the pooled calculation instead lets the validation value raise its own cap. Common mistake avoided: a transformation remains data-derived even if its rule is written down before the fold is processed.
Why the Rule Holds
The purpose of a validation fold is to stand in for observations that were not used to construct the fitted predictive procedure. That procedure includes its preprocessing estimates. The fold-specific rule can be expressed by letting \(T_{\theta}\) denote a transformation with parameter \(\theta\), where \(\theta\) is estimated from data. For a training fold, the valid sequence is:
The key is that the same estimated parameter is applied to both parts. If instead the parameter is estimated as \(g(D_{\text{train}},D_{\text{val}})\), the fitted transformation depends on the validation observations. The validation loss then assesses a procedure that had access to information from the data it is supposed to evaluate. This is not the same procedure that would be applied to a genuinely unseen record using parameters learned from the training data.
This reasoning does not rule out every transformation before splitting. A fixed rule that uses no information estimated or selected from the observed data can be applied consistently before the folds are formed. Examples include converting a measurement using a prespecified unit conversion or applying a prespecified logarithm to positive measurements. In contrast, choosing which variables to transform after inspecting their distributions is data-derived: that choice must be made within each training fold. Any outcome-informed preprocessing, such as selecting predictors using their relationship with the response, must also use training-fold outcomes only.
Practical Checks and Common Traps
- Fit once, reuse everywhere. Within a fold, each learned transformation is fitted on that fold's training data and applied unchanged to its validation data.
- Do not recompute on validation. Separate validation-fold means, standard deviations, or imputation values do not represent the training-fitted procedure.
- Keep the complete pipeline inside the fold. Any data-selected feature set or data-estimated transformation belongs inside the training-fold process, not in a preliminary step based on all observations.
- Distinguish fixed choices from learned choices. A transformation specified without inspecting the data differs from a parameter, threshold, or choice estimated from the data.
- Remember the response variable. A preprocessing step that uses outcomes can leak information even if it is described as feature preparation rather than model fitting.
Check Your Understanding
For each problem, identify which data determine the preprocessing values and how they are applied.
- A training fold has predictor values \(4,6,8,10\); its validation value is \(12\). Standardize using the training mean and sample standard deviation. What is the standardized validation value, to four decimal places? (Answer: \(1.9365\))
- A training fold has observed claim durations \(1,2,5\) and one missing value. A validation fold has an observed value \(8\) and one missing value. If missing values are replaced by the observed training mean, what value replaces each missing entry? (Answer: \(8/3\), or approximately \(2.6667\) months, in both folds)
- In one fold, the training predictor values are \(0,2\) and validation values are \(4,6\). In the other fold, training values are \(4,6\) and validation values are \(0,2\). Using training sample means and sample standard deviations, what are the standardized validation pairs in fold order? (Answer: Fold one \((2.1213,3.5355)\); Fold two \((-3.5355,-2.1213)\))
- A model uses a logarithm chosen in advance for every positive measurement, followed by centering at the observed mean. Which step may be applied before the folds are formed, and which must be estimated within each training fold? (Answer: The prespecified logarithm may be applied before folding; the mean for centering must be estimated within each training fold and reused unchanged on its validation fold.)
- A team selects predictors using their correlations with the response, then performs cross-validation on the selected predictors. What must change to prevent leakage, and which response values may be used for selection in a fold? (Answer: Repeat predictor selection within each training fold; use that fold's training responses only, then apply the selected predictor set unchanged to its validation fold.)