Feature Selection Within Cross-Validation

Why Feature Selection Belongs Inside Cross-Validation

A model’s performance depends on more than its final fitting formula. If an analyst first uses the outcome to decide which predictors to retain, that decision is part of the modeling procedure. Cross-validation must evaluate the procedure that will actually be used, including how it selects predictors.

Suppose a study has many candidate predictors and the analyst screens them for association with the response before fitting a regression model. If screening uses every row, then the outcomes in a later assessment fold have already influenced which predictors are available to the model. The fold is no longer an independent assessment of the entire procedure. The reported error can be too optimistic, especially when many candidate predictors offer many opportunities to find chance associations.

Definition: Feature selection within cross-validation means fitting the outcome-based selection rule using only the training portion of each fold, then applying that fold’s selected features to its held-out portion. The selected feature set may differ from fold to fold. The assessment-fold outcomes are used to calculate performance, not to choose features.

Feature selection includes any data-dependent decision about which predictors enter the model. Examples include retaining predictors with the strongest marginal association with the outcome, using a significance-screening rule, or choosing a subset by a search procedure. The selection rule may be fixed in advance, or it may itself have settings to choose, such as how many predictors to retain.

The key test is whether a step uses information from the data. If the answer to that step could change when the assessment-fold outcomes change, the step must not use those outcomes during training. When selection depends on a training-set statistic, such as a correlation, the statistic and the resulting selected set are recalculated using each fold’s training data.

The Resampling Procedure

In \(K\)-fold cross-validation, the observations are divided into \(K\) folds. In turn, one fold is held out for assessment and the other \(K-1\) folds are used for training. For feature selection, the training operation has two linked parts: select features using the training data, then fit the predictive model using those selected features and the same training data.

Set aside one fold.
Keep its predictor values and outcomes out of every training-stage decision for this iteration.

Fit the selector on the other folds.
Use only their predictor values and outcomes to rank, screen, or otherwise select features.

Fit the model and predict.
Train the model using the training observations and the features selected in Step 2. Apply that fitted model to the held-out observations.

Repeat and summarize.
Repeat for every fold, then combine the held-out predictions to calculate the chosen performance measure.

The resulting cross-validation score assesses a learning procedure that includes feature selection. It does not require the same features to be selected in every fold. Fold-to-fold differences may be informative: they can indicate that the selection is sensitive to which observations happen to be in the training data.

For a squared-error assessment, let \(I_k\) be the set of observations in fold \(k\), let \(y_i\) be the observed outcome, and let \(\hat{y}_i^{(-k)}\) be the prediction for observation \(i\) from a model trained without fold \(k\). The pooled cross-validation mean squared error is

\operatorname{CV\ MSE} = \frac{\displaystyle\sum_{k=1}^{K}\sum_{i\in I_k} \left(y_i-\hat{y}_i^{(-k)}\right)^2} {\displaystyle\sum_{k=1}^{K}|I_k|}

This calculation uses each observation’s held-out prediction once. If fold sizes differ, averaging fold MSEs equally does not generally give this pooled result; each observation should receive equal weight unless a different weighting scheme is intended.

Worked Examples

Worked Example: Screening Predictors Separately in Four Folds

A hypothetical insurer is modeling claim severity with four candidate predictors, \(A,B,C,D\). The prespecified selection rule retains the two predictors with the largest absolute training-set correlations with severity. Four-fold cross-validation uses 25 assessment observations per fold. After fitting the selected-predictor regression in each training set, the assessment sum of squared errors (SSE) is 300, 325, 275, and 350, respectively. The selected sets are \(A,B\), \(A,C\), \(B,D\), and \(A,D\). Calculate the pooled cross-validation RMSE.

Method. In each fold, the two predictors are ranked using only that fold’s training outcomes and predictor values. The stated selected sets are therefore fold-specific. The assessment outcomes are used only in the fold’s SSE.

Combine the assessment errors. There are \(4\times25=100\) assessment observations in total. The total SSE is \(300+325+275+350=1{,}250\). Thus the pooled MSE is \(1{,}250/100=12.5\).

Take the square root. The pooled RMSE is \(\sqrt{12.5}=3.5355\ldots\), or 3.54 severity units when rounded to two decimal places.

Check. The fold-specific selected sets differ, but every observation is assessed using a model whose feature ranking did not use that observation’s outcome. The common mistake this avoids is selecting one set using all 100 outcomes before creating the folds.

Worked Example: Correcting a Full-Data Screening Error

In a hypothetical binary classification study, an analyst has 40 candidate predictors and uses outcome association to retain the single strongest predictor. In a three-fold evaluation, screening the full dataset first selects predictor \(Z\). The resulting fold RMSE values are 0.36, 0.40, and 0.38. When the same screening rule is instead fitted separately within each training fold, the selected predictors are \(Z\), \(W\), and \(Q\), and the fold RMSE values are 0.48, 0.45, and 0.51. Assume equal fold sizes. Compare the two reported mean fold RMSE values and identify the valid estimate.

Wrong calculation. The full-data-screened calculation is \((0.36+0.40+0.38)/3=1.14/3=0.38\). It is not a valid cross-validation estimate of the screening-and-modeling procedure: all assessment outcomes helped determine that \(Z\) was selected.

Correct calculation. With screening repeated within each training fold, the mean fold RMSE is \((0.48+0.45+0.51)/3=1.44/3=0.48\). This is the valid fold-based comparison for the stated procedure, because each fold’s outcome was excluded when that fold’s feature was selected.

Interpretation and check. The difference is \(0.48-0.38=0.10\) RMSE units. It is not evidence that within-fold selection makes the prediction method intrinsically worse; rather, it shows that the full-data screening calculation used information unavailable for a genuinely held-out prediction. The common mistake is treating a selected feature as though it had been fixed before seeing any outcomes.

Worked Example: Tuning the Number of Features Without Reusing Outer Outcomes

A hypothetical modeler is comparing two selection procedures: retain the top one or the top two predictors by absolute training-set correlation with a continuous response. The modeler uses three outer folds, each containing 20 observations. Within each outer training set, an inner cross-validation chooses between the two procedures. The mean inner RMSEs for top one versus top two are: outer fold 1, 5.0 versus 4.6; outer fold 2, 4.8 versus 5.1; outer fold 3, 4.7 versus 4.5. The chosen procedure’s outer assessment SSEs are 180, 220, and 200. Calculate the pooled outer RMSE and state which selection procedure is chosen in each outer iteration.

Choose within each outer training set. In outer fold 1, \(4.6<5.0\), so choose the top two. In outer fold 2, \(4.8<5.1\), so choose the top one. In outer fold 3, \(4.5<4.7\), so choose the top two. Each choice uses inner validation results from the outer training data only.

Evaluate the chosen procedure on the outer folds. The total outer assessment SSE is \(180+220+200=600\). The total number of outer assessment observations is \(3\times20=60\), so the pooled outer MSE is \(600/60=10\).

Calculate and check the RMSE. The pooled outer RMSE is \(\sqrt{10}=3.1622\ldots\), or 3.16 response units to two decimal places. Outer outcomes were used to calculate this score, not to choose whether to retain one or two predictors. The common mistake is to use the outer results to choose the preferred procedure and then report those same results as an unbiased assessment of that choice.

Why the Feature Selector Is Part of the Model

A useful way to understand the rule is to describe the fitted predictor as the output of a complete training algorithm. Given a training dataset \(D\), the algorithm first produces a selected feature set \(S(D)\), then estimates the model using \(D\) and \(S(D)\). Its prediction for new predictor values \(x\) can be written as

\hat{f}_D(x)=\operatorname{Fit}\bigl(D,S(D)\bigr)(x).

The selector \(S(D)\) is not an unrelated preliminary step: it changes the fitted predictor. In fold \(k\), the valid assessment prediction is therefore \(\hat{f}_{D_{-k}}(x_i)\), where \(D_{-k}\) excludes fold \(k\). This prediction depends on \(S(D_{-k})\), not on a feature set selected using the full dataset.

The justification for this procedure is the separation of information. Conditional on the training data, the fitted selector and model are determined without the assessment outcomes. Those outcomes can then provide an assessment of predictions made by the full training algorithm. If the assessment outcomes help determine \(S\), the assessed predictions are no longer predictions from a procedure that was independent of those outcomes.

Cross-validation is an estimate for a procedure trained on a fraction of the available data, such as \((K-1)/K\) of the observations in ordinary \(K\)-fold cross-validation. It is not automatically an exact estimate of the performance of a final model trained on every available observation. The important point here is that each fold must evaluate the same kind of complete training procedure, including selection.

When the Selection Rule Itself Is Chosen

A fixed selection rule can be repeated directly within each fold. For example, “retain the three predictors with the strongest absolute training correlations” is fixed if the number three and the ranking measure were specified in advance. By contrast, choosing the number of predictors after comparing several values, or choosing among screening methods based on validation performance, adds a tuning decision.

If that decision is made using the same fold outcomes later reported as the performance estimate, the estimate can again be optimistic. A sound approach is nested cross-validation: use inner resampling within each outer training set to choose the selection procedure or its settings, then evaluate the chosen procedure on the untouched outer fold. The third worked example illustrates this separation.

This is not a requirement to nest every fixed, prespecified rule. Nesting is needed when the data are used to choose among candidate procedures and an independent outer assessment is intended. If a final protected test set is available, it must also remain unused during screening, tuning, and other development decisions.

Implementation and Common Traps

In an R workflow, keep selection and fitting together inside the resampling loop. The following pseudocode emphasizes the data boundary; function names such as fit_selector stand for the selection method chosen for the analysis.

\texttt{for (k in seq\_len(K)) \{}

\texttt{\ \ train <- data[fold\_id != k, ]}

\texttt{\ \ assess <- data[fold\_id == k, ]}

\texttt{\ \ selector <- fit\_selector(train)}

\texttt{\ \ selected\_train <- apply\_selector(selector, train)}

\texttt{\ \ model <- fit\_model(selected\_train)}

\texttt{\ \ selected\_assess <- apply\_selector(selector, assess)}

\texttt{\ \ predictions[fold\_id == k] <- predict(model, selected\_assess)}

\texttt{\}}

The selector fitted in a fold must be applied consistently to both its training and assessment predictors. It is not refitted on the assessment data. If a predictor has to be imputed, scaled, encoded, or otherwise transformed as part of the selection process, any data-dependent quantities used in that step must also be learned from the training portion and then applied to the assessment portion.

Screening once before splitting. Even if the final model is refitted in every fold, the evaluation leaks if the outcomes were used to screen features before the folds were created.
Using assessment outcomes to settle ties or thresholds. A rule for handling ties or selecting a cutoff must be specified or learned from training data, not adjusted after inspecting assessment performance.
Assuming selected features must match across folds. They need not. The score evaluates the selection algorithm, not a requirement that every fold produce an identical list.
Choosing a feature rule from outer-fold performance and reporting that same performance as independent. If outer results influence the choice, they have become part of model development; use a further protected evaluation for a fresh assessment.

Key takeaway: In cross-validation, every outcome-based feature-selection decision belongs inside the training fold. If the choice of selection rule is itself data-driven, make that choice within an inner resampling process and reserve outer folds or a protected test set for evaluation.

Check Your Understanding

For each question, identify the valid procedure or calculate the requested result.

A researcher ranks 12 predictors by their correlation with a continuous outcome, using all observations, then performs five-fold cross-validation with the top three. What must change for the cross-validation to evaluate this screening procedure? (Answer: Recalculate the ranking and select the top three using only the training data in each fold; use that fold’s held-out outcomes only for assessment.)
In three equal-sized folds, the assessment SSEs from within-fold selection are 144, 180, and 156, with 20 observations per fold. Calculate the pooled RMSE. (Answer: Total SSE \(=480\), total observations \(=60\), MSE \(=8\), so RMSE \(=\sqrt{8}=2.83\) to two decimals.)
A fixed rule retains the five predictors with the strongest training-set association with the outcome. Must the analyst use inner cross-validation merely because features are selected? (Answer: No. If the rule and the number five are fixed in advance, apply that rule separately within each training fold; inner resampling is needed if the data are used to choose among rules or settings.)
Within an outer training set, inner validation selects the top-two-predictor procedure. May the analyst use the outer assessment outcomes to decide whether to retain two or three predictors? (Answer: No. That would use the outer assessment outcomes for tuning; the outer fold must remain untouched until assessment.)
A modeler selects the best feature set using a protected test set, refits the model, and reports performance on that same test set. What is the issue, and what is the correction? (Answer: The test set influenced feature selection, so it is no longer an independent final assessment; select features using training data and obtain a new untouched test set or an appropriate outer resampling assessment.)

What You'll Learn