Why Preprocessing Must Respect the Resampling Split
A model is evaluated on validation observations to estimate how well it will predict data it did not use to learn. That estimate is meaningful only if the validation data remain outside the entire fitting process. This includes not just the final regression or tree, but also any step that estimates a value from the data: imputing missing values, estimating a scaling mean, deciding which factor levels to encode, or selecting predictors.
The central rule is simple: fit every data-dependent preprocessing step using the training portion of a resample, and apply that fitted step to the validation portion without refitting it. Repeat the process independently in each resample. A transformation such as subtracting a mean is not automatically safe because it is called “preprocessing”; if the mean was estimated using validation observations, information from validation has influenced the modeling process.
Write \(D_{\text{train}}\) for the training data in one resample and \(D_{\text{valid}}\) for its validation data. If a preprocessing step learns parameters \(\widehat{\theta}\), the proper sequence is \(\widehat{\theta}=\operatorname{fit}(D_{\text{train}})\), followed by transforming both portions with that same fitted rule. In particular, the validation observations may be transformed, but must not be used to update \(\widehat{\theta}\).
This applies to cross-validation as well as to a single train/validation split. Each fold has its own training observations, so each fold must have its own fitted imputer, scaler, encoder structure, and feature-selection decision. The validation score for a fold is then calculated from predictions made after applying that fold’s training-fitted transformations.
A Fold-by-Fold Pipeline
A pipeline is an ordered sequence of preprocessing and modeling steps. Its value for validation is procedural: it makes clear which operations are fitted and helps ensure that each fold uses only its own training data to learn them. The following sequence applies whether the final prediction model is a GLM, a penalized regression, or a tree-based method; the appropriate preprocessing steps depend on the model and the predictor types.
Separate the observations into that fold’s training portion and validation portion before estimating any preprocessing parameters.
Estimate imputation values and scaling parameters; establish the encoding structure; perform any data-dependent feature selection using the training portion only.
Apply the fitted training transformations to the training data and validation data. Do not recalculate the transformation parameters from validation observations.
Fit the predictive model using the transformed training portion, predict the transformed validation portion, and calculate the chosen validation metric.
Start with a new training portion and refit each data-dependent preprocessing step. Do not carry parameters learned in one fold into another fold.
Imputation and Scaling: Learn the Numbers on Training Data
For mean imputation, the fitted value for a numeric predictor is the mean of its observed training values. Missing values in both training and validation data are replaced by that value. For standardization, a common convention is to subtract the training mean and divide by the training sample standard deviation. If the training mean is \(\bar{x}_{\text{train}}\) and the training sample standard deviation is \(s_{\text{train}}>0\), a value \(x\) is transformed as follows:
The formula does not mean that each portion is standardized separately. The same training values are used to calculate \(\bar{x}_{\text{train}}\) and \(s_{\text{train}}\), and those values are then used for both portions. If the training standard deviation is zero, this formula is undefined; a pipeline needs a defined treatment for a constant predictor rather than attempting to divide by zero.
Worked Example: Impute and Standardize Within One Fold
A numeric predictor has training values \(2,4,6,\text{missing}\). In this example, use the training observed-value mean for imputation, then use the mean and sample standard deviation of the completed training values for standardization. A validation observation has value \(8\), and a second validation observation is missing. Find both transformed validation values. Round the final transformed values to three decimals.
Method. Fit the imputation value using the observed training entries only. Apply it to the training data, estimate the scaler from the completed training data, and apply that same scaler to validation. This avoids estimating either parameter from validation.
The observed training-value mean is \((2+4+6)/3=4\). Replace the missing training value by \(4\), giving completed training values \(2,4,6,4\). Their mean is \((2+4+6+4)/4=4\).
The sum of squared deviations from this mean is \((2-4)^2+(4-4)^2+(6-4)^2+(4-4)^2=4+0+4+0=8\). With four completed training values, the sample variance is \(8/(4-1)=8/3\), so the sample standard deviation is \(\sqrt{8/3}\approx1.633\).
For the validation value \(8\), the standardized value is \((8-4)/\sqrt{8/3}\approx2.449\). For the missing validation value, first impute the training mean \(4\); its standardized value is \((4-4)/\sqrt{8/3}=0\). Rounded to three decimals, the answers are \(2.449\) and \(0.000\).
Check. The imputed validation value is exactly the training mean, so its standardized value must be zero. The observed validation value is four units above the training mean, or approximately \(4/1.633=2.449\) training standard deviations above it.
This example also illustrates why the order of operations matters. The specified rule estimates the scaler after completing the training data by imputation. If a different preprocessing procedure is intended, such as estimating the scaling parameters from observed entries only, that procedure must be defined in advance and then applied consistently. It is not valid to choose a procedure by checking which one gives a better validation score and then report that same validation score as an unbiased assessment.
Encoding and Feature Selection Are Also Fitted Steps
Categorical encoding can involve learning which levels are present, how infrequent levels are handled, or which level is used as a reference. The encoding schema must be established from training data and then used to construct validation predictors. The coding convention itself may be specified in advance, but any data-dependent choices involved in setting up the encoding must not be learned by examining the validation portion.
Feature selection is also part of fitting. If a procedure chooses predictors using their association with the response, that association must be calculated from the training data in the current fold. Selecting variables once using the full data set and then cross-validating only the final model allows each validation fold’s responses to affect which predictors are evaluated.
Worked Example: Apply a Training-Defined Encoding
A training fold contains the territory values West, East, West, and Central. Use West as the reference level and create indicator variables for East and Central. A validation fold contains Central and East. State the encoded training and validation rows, using the column order East, Central.
Method. Establish the factor levels and indicator-column structure from the training portion, with the stated reference level West. Apply that same two-column structure to validation. No validation observations are used to redefine the encoding.
For West, neither indicator is one, so its row is \((0,0)\). For East, the East indicator is one and the Central indicator is zero, giving \((1,0)\). For Central, the row is \((0,1)\). Thus, in the order of the training observations, the encoded rows are West \((0,0)\), East \((1,0)\), West \((0,0)\), and Central \((0,1)\).
The first validation observation is Central, so its encoded row is \((0,1)\). The second is East, so its row is \((1,0)\). The result has the same predictor columns, in the same order, for training and validation.
Check. West is represented by both indicators being zero, as required for the reference level. Each non-reference training and validation level activates exactly its corresponding indicator. A common mistake is to create columns separately from the validation values; that can produce a different set or order of columns and no longer applies the training-fitted encoding consistently.
The example uses a fixed reference level and levels already seen in training. If an implementation must handle a validation level absent from training, it needs a prespecified rule for that case; the validation data must not be used to redesign the training encoding. Similarly, if categories are pooled based on their frequency, the frequencies used to decide the pooling must come from the current training portion.
Worked Example: Select Predictors Within the Fold
A training fold has response values \(y=(1,2,3,4)\), predictor \(x_1=(1,2,3,4)\), and predictor \(x_2=(1,3,2,4)\). Use a training-only selection rule that keeps a predictor when its absolute sample correlation with the response is greater than \(0.85\). Then fit ordinary least squares using the selected predictor and an intercept. A validation fold has \((x_1,x_2,y)\) rows \((5,100,5)\) and \((6,200,6)\). Find the validation RMSE. Round the final RMSE to three decimals.
Method. Calculate the correlations and make the selection using training data only. Fit the regression on the selected training predictor, then predict validation outcomes using the selected predictor and the fitted regression. The validation response is used to calculate the metric, not to change the selected set or fitted model.
The training means are \(\bar{y}=2.5\), \(\bar{x}_1=2.5\), and \(\bar{x}_2=2.5\). For \(x_1\), the centered values equal those of \(y\), so its sample correlation with \(y\) is \(1\). For \(x_2\), the centered values are \((-1.5,0.5,-0.5,1.5)\), while those of \(y\) are \((-1.5,-0.5,0.5,1.5)\). Their cross-product sum is \(2.25-0.25-0.25+2.25=4\). Each variable’s sum of squared deviations is \(5\), so the sample correlation is \(4/\sqrt{5\cdot5}=0.8\).
The selection threshold is strictly greater than \(0.85\). Therefore \(x_1\) is selected because \(1>0.85\), while \(x_2\) is not because \(0.8\) is not greater than \(0.85\). On the training data, the fitted least-squares line using \(x_1\) is \(\widehat{y}=x_1\): the four response values equal the four \(x_1\) values, so the intercept is \(0\) and the slope is \(1\).
The validation predictions are \(5\) and \(6\), from \(x_1=5\) and \(x_1=6\). The errors are \(5-5=0\) and \(6-6=0\), so the validation RMSE is \(\sqrt{(0^2+0^2)/2}=0\), or \(0.000\) to three decimals.
Check. If one improperly used validation responses to select a predictor, \(x_2=(100,200)\) would appear perfectly associated with the two validation responses \((5,6)\). That does not change the correct training-only decision: the stated rule selected \(x_1\) before validation outcomes were used to evaluate predictions.
Why the Rule Works
The validation metric is intended to describe performance on observations that did not help determine the fitted procedure. A preprocessing parameter estimated from validation data violates that separation, even if the final regression coefficients were fitted using training data alone. The issue can be seen directly from the operations: if the validation values contribute to a mean, a selected feature set, or an encoding decision, then the transformation applied to training and validation is partly determined by the validation sample.
For example, in mean imputation the fitted rule is “replace a missing value by \(\widehat{\mu}\).” If \(\widehat{\mu}\) is calculated from training observations only, then the imputation rule available when predicting new cases is reproduced: estimate from the data used to fit the model, then apply. If validation values are included in \(\widehat{\mu}\), the evaluation instead measures a procedure that had access to part of the held-out sample while being built. Repeating this mistake in every fold does not remove it; each fold’s validation portion must remain outside the fit for that fold.
This is why preprocessing belongs inside the resampling loop rather than being done once before the folds are created. The principle is aligned with the SOA Exam PA (Predictive Analytics) syllabus: the reported validation performance should correspond to the full modeling workflow that would be used on genuinely new data.
Common Exam Traps
- Scaling each fold separately. This is not the issue: each fold should have its own scaler, but its parameters must be learned from that fold’s training portion and then reused on its validation portion.
- Imputing before splitting. A mean computed over the full data set includes validation observations. Split first, then estimate the imputation value from training data in each resample.
- Selecting variables once before cross-validation. If selection uses response values, doing it before the folds lets validation responses influence the selected predictors. Repeat selection inside each fold.
- Refitting a transformation on validation. Applying a training-fitted mean and standard deviation is different from calculating new validation means and standard deviations. Use the former.
- Confusing sample and population standard deviation. State which convention is used. With training values \(1,3,5\), the mean is \(3\) and the sample standard deviation is \(\sqrt{[(1-3)^2+(3-3)^2+(5-3)^2]/(3-1)}=2\). A validation value \(7\) therefore standardizes to \((7-3)/2=2.000\), not a value calculated with a different denominator.
Check Your Understanding
For each problem, identify which values are learned from training data and apply them without refitting on validation.
- A training predictor has values \(10,20,30\). Standardize a validation value of \(40\) using the training mean and training sample standard deviation. Give the result to three decimals. (Answer: 2.000)
- A training predictor has values \(3,5,\text{missing}\). Impute the missing value using the observed training mean, then standardize using the completed training values’ mean and sample standard deviation. What is the transformed value of a validation value \(7\), to three decimals? (Answer: 3.000)
- Training levels are North, South, South. Use North as the reference and create one indicator column, South. What is the validation encoding for a South observation? (Answer: 1)
- A feature-selection rule uses training correlations and retains predictors with absolute correlation greater than \(0.7\). A training predictor has correlation \(0.68\) with the response, while its validation correlation is \(0.99\). Is it retained under the stated rule? (Answer: No; the training correlation is not greater than 0.7.)
- A five-fold validation procedure includes mean imputation, standardization, and predictor selection. How many separately fitted sets of preprocessing decisions are needed for the five validation folds, and which observations may be used to fit each set? (Answer: Five sets; each is fitted using only the corresponding fold’s training portion.)