Fitting Preprocessing Steps Inside Each Fold

Why Preprocessing Belongs Inside a Fold

Cross-validation estimates how a prediction procedure performs on observations it did not use to fit that procedure. A prediction procedure can include more than a regression model: it can also include steps such as centering or scaling predictors before fitting. If a step estimates a quantity from the data, that estimate is part of the procedure and must be calculated using the training portion of each fold.

For example, centering a predictor requires estimating its mean, and scaling it by its standard deviation requires estimating that standard deviation. In each fold, calculate these quantities from that fold’s training observations. Use them to transform both the training observations and the held-out observations. Do not recalculate them from the held-out fold: doing so would make the transformation used for validation depend on the validation data.

This tutorial focuses on the mechanics of that rule for centering and scaling. The calculations are simple, but their placement in the cross-validation sequence matters. A correctly calculated mean or standard deviation can still be used incorrectly if it is estimated from the wrong part of a fold.

The Fold-Specific Transformation

Suppose observation \(i\) has predictor \(x_{ij}\), the value of predictor \(j\). Let \(T_k\) denote the observation indices in fold \(k\)'s training set and \(V_k\) the indices in its held-out set. For predictor \(j\), calculate the mean and sample standard deviation using only observations indexed by \(T_k\):

Here \(|T_k|\) is the number of observations in the training fold. Centering subtracts the training-fold mean. Standardizing subtracts that mean and divides by the training-fold sample standard deviation. The same estimated quantities are used for every row, whether the row is in \(T_k\) or \(V_k\):

The superscript \(z\) denotes the standardized predictor value. The formula itself does not change between training and held-out observations; only the observation’s original value \(x_{ij}\) changes. In particular, a held-out value is not centered by subtracting the held-out fold’s mean, and it is not scaled by the held-out fold’s standard deviation.

These formulas require a nonzero training-fold standard deviation for standardization. If a predictor is constant in a training fold, its standard deviation is zero and division by that value is undefined. The analysis therefore needs a prespecified way to handle such a predictor, based on the training fold, rather than attempting the formula as written. Centering does not require a nonzero standard deviation.

Why the Training Values Have Mean Zero and Standard Deviation One

The transformation has useful, exact properties on the observations used to estimate it. These properties follow directly from the formulas; they are not assumptions about the distribution of the predictor.

For centering, the mean of the transformed training values is the mean of the original values minus the training mean:

For standardization, suppose \(s_{j,T_k}>0\). The training values’ sample variance after transformation is

Therefore the sample standard deviation of those standardized training values is \(1\). These statements concern the training fold because that is where the mean and standard deviation were calculated. They do not assert that the held-out values have mean zero or sample standard deviation one.

Where Preprocessing Fits in Cross-Validation

For each fold, preprocessing quantities are estimated before the model is fit. Then the transformation is applied to the training and held-out predictors, the model is fit using the transformed training data, and predictions are made for the transformed held-out data. The held-out outcomes, if available for evaluation, are used to calculate the fold’s prediction performance, not to estimate predictor means or standard deviations.

The model evaluated by cross-validation is the whole sequence, not just the final regression calculation. The sequence includes the rule for estimating and applying preprocessing quantities. Repeating that sequence in every fold makes the validation exercise represent how the procedure would treat new observations: estimate from available training data, then apply the resulting transformation to new predictor values.

Common Mistakes and Their Corrections

• Recalculating the mean or standard deviation on the held-out fold. This makes the held-out observations determine their own transformation. Instead, apply the quantities calculated from the corresponding training fold.
• Calculating one set of quantities before making the folds. This means that the quantities used in a fold were not estimated using only that fold’s training observations. Estimate them afresh within each fold.
• Expecting held-out standardized values to have mean zero and standard deviation one. Those properties hold for the training values used to calculate the quantities. The held-out values are transformed relative to the training distribution and need not have those summaries.
• Using the full-data mean because it seems more stable. Stability does not change which data the validation procedure is meant to use. For each fold, use the mean calculated from that fold’s training observations.
• Dividing by zero when a training predictor is constant. If \(s_{j,T_k}=0\), the standardization formula is undefined. Identify and handle the case according to a rule based on the training fold; do not silently substitute an arbitrary denominator.

The rule applies to estimated preprocessing quantities because their values influence the inputs presented to the fitted model. The particular arithmetic may be centering, scaling, or another allowed transformation, but in every case the fold’s training observations determine the quantities and the held-out observations receive the resulting transformation unchanged.