Keep Feature Engineering Inside Cross-Validation

Why Feature Engineering Belongs Inside Cross-Validation

Cross-validation estimates how well a modeling procedure predicts observations it did not use to develop its model. The procedure includes more than the final regression or classifier: it also includes every step that learns something from the data. If imputation values, scaling constants, selected predictors, or principal components are calculated using the full data set before the folds are formed, information from validation observations has already influenced the pipeline. The resulting validation score no longer measures the intended process.

The key distinction is between a fixed rule and a rule learned from data. A transformation specified in advance, such as taking the logarithm of a positive exposure amount, can generally be applied consistently. A median calculated from observed values, a set of categories discovered in the data, or a list of predictors selected by association with the outcome is learned from data. Estimate such quantities using the training part of each fold, then apply them unchanged to that fold’s validation part.

Definition: In a cross-validation fold, the training fold is the subset used to fit the preprocessing steps and predictive model. The validation fold is the held-out subset used to assess predictions. A leakage-free pipeline estimates every data-dependent preprocessing quantity using only the training fold, then applies the resulting transformations and fitted model to the validation fold.

Let \(D_{-k}\) denote the training data for fold \(k\), and let \(D_k\) denote that fold’s validation data. Write \(\widehat{T}_{-k}\) for all preprocessing operations fitted on \(D_{-k}\), and \(\widehat{f}_{-k}\) for the predictive model fitted using the transformed training data. For a validation predictor vector \(x\), the prediction is \(\widehat{f}_{-k}(\widehat{T}_{-k}(x))\). The same fitted transformations must be used for every validation observation in that fold.

\widehat{R}_{\mathrm{CV}} = \frac{1}{n}\sum_{k=1}^{K}\sum_{i\in D_k} L\!\left(y_i,\widehat{f}_{-k}\!\left(\widehat{T}_{-k}(x_i)\right)\right)

Here \(L\) is the chosen loss function, and each observation is scored in a fold where it was not used to fit either the preprocessing or the model. This expression assumes the folds form a validation partition, so each observation contributes once. For other resampling designs, use the corresponding weighting and aggregation convention. The central requirement remains the same: validation outcomes and validation predictor distributions must not be used to fit that fold’s pipeline.

A Fold-by-Fold Workflow

A complete pipeline treats preprocessing as part of model fitting. At each resampling split, repeat the data-dependent steps from the beginning. Do not calculate them once on the full data and pass the transformed data into cross-validation.

Form the split.
Use the intended resampling design to separate training observations from validation observations. If the design requires groups or time order, preserve those constraints when forming the split.

Fit preprocessing on training data.
Estimate imputation values, scaling constants, encoding rules, screening decisions, and dimension-reduction components using the training observations only.

Fit the predictive model.
Transform the training data using the fitted preprocessing, then estimate the model. Any choices made by looking at outcomes, such as supervised feature screening, belong inside this step.

Transform and predict validation observations.
Apply the training-fold preprocessing as-is. Do not re-estimate a median, mean, scale, encoding, or component using the validation fold.

Aggregate validation performance.
Calculate the chosen loss or score from the held-out predictions and combine fold results according to the resampling design.

Worked Examples

Worked Example: Imputation and Scaling in One Fold

A training fold has predictor values \(2,4,6\), with outcomes \(1,2,3\). Its validation fold has predictor values \(8\) and missing, with outcomes \(4\) and \(2\), respectively. Use the training-fold mean to impute missing values, and standardize using the training-fold mean and population standard deviation. Fit a linear regression to the standardized training predictor and calculate the validation RMSE. The model is an ordinary least-squares regression with an intercept.

Method and preprocessing fit. The training predictor mean is \((2+4+6)/3=4\). Its population variance is \(((2-4)^2+(4-4)^2+(6-4)^2)/3=(4+0+4)/3=8/3\). Thus the population standard deviation is \(\sqrt{8/3}\), approximately \(1.6330\). There are no missing training values to impute. The validation missing value is imputed with the training mean, 4; the validation predictors become 8 and 4.

Transform the training data. The standardized training values are \(z=(x-4)/\sqrt{8/3}\). They are approximately \(-1.2247, 0, 1.2247\). The corresponding outcomes are \(1,2,3\).

Fit the model. The training means are \(\bar z=0\) and \(\bar y=2\). The slope is \[ \frac{\sum (z_i-\bar z)(y_i-\bar y)}{\sum (z_i-\bar z)^2} = \frac{(-1.2247)(-1)+0+(1.2247)(1)}{1.5+0+1.5} = \frac{2.4494}{3} \approx 0.8165. \] The intercept is \(\bar y-0.8165\bar z=2\). Therefore the fitted model is approximately \(\widehat y=2+0.8165z\).

Predict and check. For \(x=8\), \(z=4/\sqrt{8/3}\approx2.4495\), so \(\widehat y\approx2+(0.8165)(2.4495)=4\). For the imputed value \(x=4\), \(z=0\), so \(\widehat y=2\). The validation RMSE is \(\sqrt{((4-4)^2+(2-2)^2)/2}=0\). The calculation checks: both predictions match the supplied validation outcomes.

Common mistake avoided: Do not recompute the imputation value or standardization constants using the validation values. That would make this fold’s transformation depend on the observations it is meant to evaluate.

Worked Example: Encoding a Validation Category Not Seen in Training

A training fold contains policy areas North, South, North, South, with response values \(1,0,1,0\). Fit an intercept-only linear regression with a North indicator, using South as the reference category. The validation fold contains North and East. The data-processing policy, specified before resampling from an external business rule, maps any area not represented in the training fold to the reference category. Construct the design rows and calculate predictions.

Method and training-fold encoding. Define \(I_{\mathrm{North}}=1\) for North and \(0\) for South. The training design rows are \((1,1),(1,0),(1,1),(1,0)\), where the first entry is the intercept. The North response mean is \((1+1)/2=1\), and the South response mean is \((0+0)/2=0\). The fitted intercept is 0 and the North coefficient is 1, giving \(\widehat y=0+1I_{\mathrm{North}}\).

Apply the same rule to validation. North receives design row \((1,1)\), so its prediction is \(0+1(1)=1\). East is not represented in the training fold; the specified mapping assigns it to the reference category, so its row is \((1,0)\) and its prediction is \(0+1(0)=0\). The predictions are therefore 1 for North and 0 for East.

Check and limitation. These predictions follow directly from the fitted training coefficients and the stated mapping rule. Mapping East to South is an operational assumption, not a fact learned from these four records; it must be appropriate to the application and defined without using validation outcomes. Without a defensible mapping, the pipeline needs another explicit policy for previously unseen levels.

Common mistake avoided: Do not inspect the validation fold to add an East indicator and then treat it as an ordinary fitted predictor. The training fold contains no East observations from which to estimate an East effect. A level dictionary may be specified independently in advance, but a model coefficient for an absent training level cannot be learned from that fold.

Worked Example: Fitting PCA Without the Validation Fold

A training fold has two observations on centered-to-be predictors \(x_1,x_2\): \((0,0)\) and \((2,0)\). Its validation observation is \((1,10)\). Fit a one-component principal component analysis (PCA) using the training fold, and compare its direction with the direction obtained by incorrectly fitting PCA to all three observations. Use the sample covariance matrix.

Method: train-fold PCA. The training mean vector is \((1,0)\). The centered training observations are \((-1,0)\) and \((1,0)\). Their sample covariance matrix is \[ \begin{pmatrix}2&0\\0&0\end{pmatrix}, \] because the sum of squared centered \(x_1\) values is 2 and the sample denominator is \(2-1=1\). The first principal component is the \(x_1\) direction, \((1,0)\). Center the validation point using the training mean: \((1,10)-(1,0)=(0,10)\). Its score on the training-fold first component is \((0,10)\cdot(1,0)=0\).

Calculate the incorrect full-data fit. The mean of all three observations is \((1,10/3)\). The centered \(x_1\) values are \(-1,1,0\); the centered \(x_2\) values are \(-10/3,-10/3,20/3\). Their cross-product sum is 0. The sample variances are \(2/2=1\) for \(x_1\) and \(((100+100+400)/9)/2=100/3\) for \(x_2\). The full-data covariance matrix is therefore \[ \begin{pmatrix}1&0\\0&100/3\end{pmatrix}. \] Its first principal component is the \(x_2\) direction, \((0,1)\), not the training-fold \(x_1\) direction.

Check and interpretation. The validation observation has a training-fold score of 0 on the selected component, but its presence changes the full-data first component to the \(x_2\) direction. The orientation change confirms that the validation predictor was able to affect a supposedly training-fitted feature. PCA, though unsupervised, must therefore be fitted within each training fold.

Common mistake avoided: “Unsupervised” does not mean “safe to fit on all observations.” PCA does not use outcomes, but validation predictor values can still change the components used to construct model features.

Screening, Tuning, and the Scope of the Score

Feature screening is also part of the pipeline when its decisions depend on the observed data. For example, selecting predictors by their association with the outcome uses outcome information. If the selection is performed once on all observations, then each validation outcome may influence which predictors are available to predict it. The screening rule must be re-estimated using only the training outcomes in each fold, and the selected training-fold variables must then be used for that fold’s validation predictions.

The same principle applies to predictor-only screening, such as removing variables because of their measured variance or correlation with other predictors. A criterion based on observed values can change when validation predictors are included. Fixing the criterion in advance does not make the resulting data-dependent decision fixed; the decision still needs to be learned within the fold.

If model settings are selected by trying alternatives and choosing the one with the best cross-validation result, the minimum of those results is a selection result, not an independent final assessment. A nested design can separate these tasks: use inner folds within an outer training set to choose settings and fit the pipeline, then assess the selected procedure on the outer validation fold. The same principle applies to preprocessing choices selected through validation performance.

Key takeaway: Validate the procedure you intend to use, not a version of the data preprocessed with information from every observation. In every fold, learn imputation, scaling, encoding decisions, screening, dimension reduction, and model parameters from training data; apply them unchanged to validation data.

Common Exam Traps

Preprocessing before splitting. Computing a median, mean, scale, or PCA basis once on the full data set leaks validation predictor information into the pipeline.
Treating outcome-free steps as automatically safe. Scaling and PCA do not need the outcome to leak information about the validation predictor distribution.
Using validation outcomes to screen variables. Any outcome-informed selection must happen inside the training fold.
Refitting transformations on validation data. Validation values must be transformed using training-fold quantities, not their own fold-specific mean or scale.
Ignoring levels absent from training. Define a defensible policy for unseen categories; do not pretend the fold can estimate a level effect from no training examples.
Reporting the best tuning score as an unbiased final assessment. Choosing settings to minimize a resampling score uses that score for selection; separate selection from assessment when an independent estimate is needed.

Check Your Understanding

For each problem, identify which quantities may be learned from the training fold and state the leakage-free validation procedure.

A training fold has observed predictor values \(3,5,7\); a validation fold has values \(9\) and missing. Impute using the training mean. What value replaces the missing validation value? (Answer: 5.)
A training fold has values \(1,3,5\). Standardize with the training mean and population standard deviation. What is the standardized value for a validation value of 7? (Answer: The training mean is 3, the population standard deviation is \(\sqrt{8/3}\), and the standardized value is \(4/\sqrt{8/3}\approx2.4495\).)
A supervised screen retains the two predictors with the largest absolute correlations with the outcome. A validation fold has unusually strong outcome associations. Where must the screening be performed? (Answer: Separately inside each training fold, using only its outcomes; apply the selected predictors to that fold’s validation observations.)
A training fold has numeric predictors with sample covariance matrix \(\begin{pmatrix}9&0\\0&1\end{pmatrix}\). A validation observation is \((4,100)\), and the training means are both zero. What is its score on the training-fold first principal component? (Answer: The first component is \((1,0)\), so the score is 4.)
A candidate pipeline tries 20 preprocessing-and-model settings and reports the smallest score from the same cross-validation runs used to choose the setting. What is the concern, and what design can provide a separate assessment? (Answer: The reported minimum is subject to selection optimism; use outer folds for assessment and inner folds within each outer training set for selection.)

What You'll Learn