Why Preprocessing Must Be Repeated Within Resampling
Tuning a penalized model means comparing candidate fits using resampling, such as cross-validation. But the model is not the only part of the analysis that learns from data. Imputation values, scaling parameters, bin cut points, and factor-level handling are also estimated or chosen from observed predictors. If assessment-fold observations contribute to those decisions, the assessment fold is no longer a clean evaluation of a process trained on the analysis fold.
The required discipline is simple: fit every data-dependent preprocessing step using only the analysis fold, then apply that fitted step to the assessment fold. Repeat this separately for every resample. The assessment fold may be transformed, but it must not determine how the transformation works.
For example, an analysis-fold median can fill missing predictor values in both folds. A separate median calculated from the assessment fold, or a median calculated once using the entire data set before splitting, cannot. The same rule applies to the mean and standard deviation used for scaling, cut points selected for binning, and any data-driven choice of predictor representation.
This restriction concerns information used to construct the model, not merely information in the response. Even if an assessment predictor value is available and its outcome is hidden, using that value to estimate a mean or choose a cut point changes the evaluation procedure. In ordinary cross-validation, preprocessing is learned from the analysis fold because that is the data the model-building procedure is allowed to use for that split.
The Fold-Specific Preprocessing Rule
Let \(A_k\) denote the analysis observations in resample \(k\), and let \(B_k\) denote its assessment observations. Write \(T_{A_k}\) for a preprocessing transformation whose parameters are estimated from \(A_k\). The assessment predictors are transformed using that same fitted transformation, not a newly estimated one.
The model is then fit using the transformed analysis predictors and analysis outcomes. Predictions for the assessment observations use the assessment predictors after the same transformation. The assessment outcomes are used only to calculate the chosen evaluation measure.
Identify the analysis and assessment observations for this fold before estimating any preprocessing quantities.
Estimate imputation values, scaling parameters, cut points, and factor handling using the analysis predictors only.
Apply the fitted transformation to the analysis predictors and assessment predictors. Do not refit it on assessment data.
Fit the penalized model on transformed analysis data, predict for transformed assessment data, and record the selected evaluation measure.
Each resample has its own fitted preprocessing parameters. After tuning is complete, refit preprocessing and the selected model using all development data.
Worked Examples
Worked Example: Imputation and Scaling Within One Fold
A hypothetical insurance team is comparing penalized prediction models for a numerical outcome. In one resampling split, the analysis-fold predictor values are \(2,4,6,\text{missing}\), and the assessment-fold values are \(10,\text{missing}\). Impute missing values using the analysis-fold mean, then standardize using the mean and sample standard deviation of the imputed analysis values. Transform the assessment values and show the resulting values, rounded to four decimals.
Method. Estimate imputation and scaling parameters from the analysis fold, in that order. The nonmissing analysis values sum to \(2+4+6=12\), so their mean is \(12/3=4\). Imputing the missing analysis value gives the analysis vector \(2,4,6,4\).
The mean of the completed analysis vector is \((2+4+6+4)/4=4\). Its sample variance is \[ \frac{(2-4)^2+(4-4)^2+(6-4)^2+(4-4)^2}{4-1} =\frac{4+0+4+0}{3} =\frac{8}{3}. \] Thus its sample standard deviation is \(\sqrt{8/3}\), approximately \(1.6330\).
The standardized analysis values are \((2-4)/\sqrt{8/3}=-1.2247\), \((4-4)/\sqrt{8/3}=0.0000\), \((6-4)/\sqrt{8/3}=1.2247\), and \((4-4)/\sqrt{8/3}=0.0000\). In the assessment fold, the observed value 10 becomes \((10-4)/\sqrt{8/3}=3.6742\). The missing assessment value is imputed to 4 using the analysis-fold mean, so its standardized value is 0.0000.
Check. The transformed analysis values have mean 0 and sample standard deviation 1, as required by this scaling convention. The assessment value 10 does not change either analysis-fold parameter. A common mistake is to include 10 when estimating the mean or standard deviation; that would let assessment data alter the transformation and would produce different values.
Worked Example: An Extreme Assessment Value Changes Global Scaling
In another hypothetical prediction problem, the analysis-fold predictor values are \(0,2,4\), while the assessment fold contains \(100\). Standardize using the analysis-fold mean and sample standard deviation. Then compare with the incorrect calculation that estimates both quantities using all four values. Round standardized values to four decimals.
Correct method. The analysis mean is \((0+2+4)/3=2\). The analysis sample variance is \(((0-2)^2+(2-2)^2+(4-2)^2)/(3-1)=8/2=4\), so the sample standard deviation is 2. The standardized analysis values are \(-1,0,1\), and the assessment value is \((100-2)/2=49\).
Incorrect global calculation. If the assessment value is included, the four-value mean is \((0+2+4+100)/4=26.5\). The sum of squared deviations is \(702.25+600.25+506.25+5402.25=7211\). The sample standard deviation is \(\sqrt{7211/3}\), approximately \(49.0272\). Under this incorrect transformation, the training values become approximately \(-0.5405,-0.4997,-0.4589\), and the assessment value becomes approximately \(1.4992\).
Check. The two calculations produce different representations of the same analysis observations because the assessment value 100 changed the global mean and standard deviation. The correct assessment value of 49 may look large, but it correctly indicates that 100 is far above the analysis-fold mean relative to its spread. A common mistake is to regard predictor-only preprocessing as harmless because no assessment outcome was used. In ordinary cross-validation, assessment predictors are still held out from fitting the preprocessing pipeline.
Worked Example: Binning and a Factor Level Not Seen in Training
A hypothetical pricing model uses a continuous predictor and a material-type factor. In one analysis fold, the observed continuous values are \(10,20,30,40\); the assessment values are \(25,35\). Define a two-bin predictor using the analysis median as the cut point, assigning values less than or equal to the cut point to “low” and larger values to “high.” The analysis fold has material types “bronze” and “silver”; the assessment fold has “bronze” and “platinum.” Describe the bins and how to handle the factor before making a penalized-model matrix.
Method for the numeric predictor. The analysis values in order are \(10,20,30,40\). With an even number of observations, their median is the average of the two middle values: \((20+30)/2=25\). The analysis values therefore map to low, low, high, high. Applying the same cut point to assessment values maps 25 to low, because the rule includes equality, and 35 to high.
Method for the factor. The analysis fold establishes bronze and silver as observed levels. Platinum is a novel assessment level. The preprocessing specification must decide in advance how novel levels are handled; it must not create a new model column based on seeing platinum in the assessment fold. One defensible policy is to use a predeclared “new” level, include that level in the fitted factor representation, and map any novel assessment level to it. If the policy instead rejects or maps novel levels to a reference category, that same policy must be applied consistently and its implications documented.
With a declared “new” level, bronze and silver retain their training representation, and platinum maps to “new.” The model matrix must have the same columns for analysis and assessment. If a “new” indicator has no analysis observations, the model has no evidence to estimate an effect for that category; mapping it to “new” prevents a schema mismatch but does not create information about its outcome. A common mistake is to inspect assessment levels and create a platinum indicator only for this fold. That makes the feature representation depend on assessment data and may leave different folds with incompatible columns.
Why the Rule Produces a Valid Fold Assessment
Cross-validation is intended to assess a model-building procedure as if it had been trained on the analysis portion and then applied to observations not used in building it. Preprocessing is part of that procedure. If a transformation is represented as \(T\), its fitted parameters must be determined by the data available to the procedure in that split.
For mean imputation, for example, let \(\bar{x}_{A_k}\) be the mean of the observed analysis-fold predictor values. The fitted imputation operation replaces a missing value by \(\bar{x}_{A_k}\). Applying that same operation to a missing assessment value gives \(\bar{x}_{A_k}\). If the assessment values were included in estimating the mean, the imputation value would instead depend on observations the procedure is supposed to be evaluating on. The assessment score would then measure a different procedure.
The same logic applies to standardization. If the analysis-fold mean and standard deviation are \(\bar{x}_{A_k}\) and \(s_{A_k}\), the transformed value is \[ z=\frac{x-\bar{x}_{A_k}}{s_{A_k}}. \] This formula uses those parameters because they define the coordinate system learned from the analysis data. Re-estimating them on the assessment fold would change the coordinate system for the observations being evaluated. Re-estimating them on the combined fold would allow assessment data to influence the fitted representation.
This matters especially for penalized models because a change in predictor scale changes the scale on which coefficients are represented and penalized. The earlier material on predictor standardization explains why scaling affects the penalty; the practical consequence here is that each fold must use its own analysis-derived scaling parameters. Using a single global set of parameters can make the resampling comparison inconsistent with the intended training procedure.
Implementing the Workflow in R
In R, the key is not a particular function name but where the preprocessing object is trained. For each split, create a preprocessing specification from the analysis data, estimate its parameters there, and use the trained object to bake both analysis and assessment data. For example, with the recipes workflow, a recipe can specify median imputation, normalization, and dummy encoding. Call prep() using the analysis fold as the training data, then call bake() separately for the analysis and assessment data.
Factor-level steps need deliberate ordering. A policy for missing factor values and novel levels should be specified before dummy encoding. For instance, a recipe may use step_unknown() for missing factor values and step_novel() for unseen levels before step_dummy(). Confirm that the resulting analysis and assessment model matrices have the same columns and that the novel-level policy behaves as intended. A reserved indicator with no training examples cannot estimate a novel category’s outcome effect.
The same discipline applies when constructing matrices for a penalized fit. Do not create a full-data model matrix, estimate its column means and standard deviations, and only then split the rows for cross-validation. Instead, make each fold’s matrix using transformations estimated on that fold’s analysis rows. When the model requires identical columns in both parts, the fitted preprocessing specification must provide that structure without using assessment observations to learn it.
A practical audit is to list every operation between raw predictors and the fitted model and ask, “Does this operation estimate anything from data, or make a data-driven choice?” If the answer is yes, fit it within the analysis fold. This includes imputation, normalization, data-driven bins, and predictor representations learned from the observed data. Fixed, prespecified operations such as converting dollars to thousands using a known factor of 1,000 do not estimate parameters from the fold, though they should still be applied consistently.
Common Traps and the Final Refit
- Preprocessing once before resampling. A global median, mean, standard deviation, or set of cut points has already used observations that become assessment data in some folds. Estimate these within each analysis fold instead.
- Recomputing assessment statistics. Standardizing the assessment fold by its own mean and standard deviation is not applying the learned transformation. Use the analysis-fold parameters.
- Assuming only outcome leakage matters. Assessment predictors can leak information into preprocessing even when their outcomes remain hidden.
- Allowing fold-specific columns to drift. A novel factor level must follow a prespecified policy; do not add a column after inspecting assessment data. Verify matching column names and ordering before fitting or predicting.
- Reusing one fold’s transformation for another. Each analysis fold may have different imputation and scaling estimates. Train a fresh preprocessing object in every resample.
After resampling has been used to select the tuning settings, the final model-building step is different from an individual fold assessment. Fit the preprocessing pipeline using all available development data, transform that data with the fitted pipeline, and fit the selected penalized model. If a separate final test set is being used, it remains outside this refit: apply the development-data pipeline to it without refitting preprocessing on test observations. This final refit does not undo the need for leakage-free transformations during tuning; it follows the tuning step and uses the larger development sample available for deployment.
Check Your Understanding
For each problem, identify which data may determine the transformation and apply it consistently.
- An analysis fold has observed predictor values \(3,5,7\) and one missing value. An assessment fold has values 11 and one missing value. Impute by the analysis mean and standardize by the completed analysis sample mean and sample standard deviation. Give the standardized assessment values to four decimals. (Answer: analysis mean \(=5\), sample standard deviation \(=2\); assessment values \(11\) and missing become \(3.0000\) and \(0.0000\).)
- Analysis values are \(1,3,5,7\); assessment values are \(9,100\). Use the analysis mean and sample standard deviation. What are the two assessment z-scores? (Answer: analysis mean \(=4\), sample standard deviation \(=\sqrt{20/3}\); scores approximately \(3.8730\) and \(74.0581\).)
- An analysis fold has factor levels “north” and “south.” An assessment observation has level “central.” A novel-level policy maps unseen values to a predeclared “new” category. What should happen to this observation, and may its presence determine a newly added assessment-only dummy column? (Answer: map “central” to “new” under the prespecified policy; do not add an assessment-only column. A “new” effect cannot be learned if the analysis fold has no “new” examples.)
- For one fold, an analyst estimates a median imputation value using all observations before constructing the split, then trains a penalized model only on the analysis rows. Is the assessment still independent of preprocessing, and what should be changed? (Answer: no; the global median used assessment-fold observations. Estimate the median from each analysis fold and apply that fold’s median to its assessment data.)
- A model uses analysis-fold values \(2,4,8,10\) to define a high-bin indicator equal to 1 when the value exceeds the analysis median. Assessment values are 6 and 9. What is the cut point and what are the two indicators? (Answer: the cut point is \((4+8)/2=6\); the indicators are \(0\) and \(1\), respectively.)