Feature Selection Inside Training Folds

Why Feature Selection Must Follow the Folds

Genomic prediction commonly begins with far more candidate genes or variants than individuals. To make fitting manageable, or to reduce noise, an analyst may screen features by their association with the outcome and retain only the strongest candidates. That screening uses information about the outcome. If it is done before cross-validation, the held-out outcomes have already influenced which predictors are available to the model being evaluated.

Cross-validation is intended to estimate how well a specified learning procedure predicts outcomes not used to build its predictions. When feature selection is part of that procedure, it must be performed using only the training observations in each fold. The held-out observations are then used to assess predictions, not to decide which genes or variants enter the model.

Definition: Outcome-guided feature selection is any rule that uses the outcome values $Y$ to choose predictors, such as ranking genes by a case-control difference or a univariate association statistic. In $K$-fold cross-validation, fold-local selection applies that rule separately using the training portion of each fold. The resulting selected features may differ across folds.

Suppose the observations are assigned to folds $F_1,\ldots,F_K$. For fold $k$, the training indices are all observations not in $F_k$. If $S(\cdot)$ denotes a selection rule, the correct selected set is $S(X_{-k},Y_{-k})$, where the subscript indicates training observations only. A model fitted on that training set and selected set produces predictions for observations in $F_k$. Their outcomes are used afterward to calculate the chosen performance measure.

\widehat{R}_{\mathrm{CV}} = \frac{1}{K}\sum_{k=1}^{K} \operatorname{Metric}\!\left( Y_{F_k}, \widehat{f}_{-k}\!\left(X_{F_k};S(X_{-k},Y_{-k})\right) \right)

Here $\widehat{f}_{-k}$ is fitted without fold $k$, and the metric compares its predictions with the outcomes in that fold. The expression emphasizes the essential boundary: the outcomes $Y_{F_k}$ can score predictions, but cannot determine $S(X_{-k},Y_{-k})$. Fold-wise scaling and normalization are handled on the same training-only principle.

What Counts as Screening?

A screening rule includes more than an explicit significance test. Ranking genes by a univariate score and retaining the top $q$, keeping variants with the smallest association p-values, or selecting features that pass an outcome-dependent threshold are all outcome-guided selection. If the rule has a choice such as the number retained, that choice also needs to avoid using the held-out outcomes. This tutorial focuses on carrying out the screening rule within folds; it does not treat model tuning as a separate validation problem.

A prespecified feature set chosen independently of the study outcomes—for example, a fixed panel determined before looking at the data—does not create this particular form of leakage. In contrast, choosing that panel after examining all study outcomes does. A feature selection rule can be simple or sophisticated; the relevant question is whether the validation-fold outcomes helped determine the features used to predict them.

Worked Example: Screening Genes in Each Training Fold

Suppose a hypothetical genomic data set has 100 individuals, a binary disease outcome, and 20,000 measured genes. The analyst uses five-fold cross-validation and, in each fold, retains the 50 genes with the largest absolute case-control mean differences. Explain what is calculated in one fold.

Each held-out fold contains 20 individuals, leaving 80 for training. In the training set for fold 1, the analyst calculates the mean expression among cases and among controls for each gene. For gene $j$, the screening score is

D_j=\bar{x}_{j,1}-\bar{x}_{j,0},

where $\bar{x}_{j,1}$ and $\bar{x}_{j,0}$ are the training-fold means for cases and controls. The analyst ranks the 20,000 genes by $|D_j|$ and retains the top 50. For example, if one gene has mean expression 8.4 in training cases and 6.9 in training controls, its score is $D_j=8.4-6.9=1.5$ expression units. Its rank is determined relative to the other training-set scores.

The model is fitted using those 50 genes and the 80 training outcomes, then used to predict the 20 held-out individuals. The outcomes for those 20 individuals are compared with the predictions to calculate the fold's score. The procedure is repeated for the other four folds, recalculating all 20,000 screening scores each time. The selected sets need not be identical: each is determined by a different training sample.

The example assumes that the fold split is suitable for the prediction target and that the screening and model-fitting calculations use training observations only. It also assumes the expression values are available in a form appropriate for the stated mean-difference rule; any outcome-independent preprocessing that is estimated from the data must not use held-out observations if it can affect the predictors.

Why Global Screening Makes the Score Optimistic

If the analyst first screens all 100 individuals, the selected set depends on all 100 outcomes. Later, when one fold is called “held out,” its outcomes have already helped decide which features the model will use. Even if the model coefficients are refitted using only the training folds, that does not undo the earlier information flow. The held-out fold is no longer independent of the feature-selection decision.

The mechanism is particularly important when the number of candidate genes is large. Under a null association, random variation gives some genes apparently strong associations. Searching over many genes and retaining the strongest tends to select unusually large random associations. When the same observations are then used to evaluate prediction, the selected chance patterns can appear to generalize better than they truly do.

Justification: Suppose, as an approximation, that $m$ independent null screening scores are standard normal, $Z_1,\ldots,Z_m\sim N(0,1)$. Selecting the largest absolute score gives $M=\max_j|Z_j|$. For a threshold $t$, independence gives $$ P(M\le t)=\prod_{j=1}^{m}P(|Z_j|\le t) =\left(1-2\{1-\Phi(t)\}\right)^m, $$ where $\Phi$ is the standard normal cumulative distribution function. Thus the maximum grows with the number of searched features. A useful leading approximation is $E(M)\approx\sqrt{2\log(2m)}$; this describes selection of an extreme null score, not a universal formula for the bias in accuracy or another prediction metric.

This argument explains why screening on all outcomes makes the selected association look stronger, but it does not give a general numeric correction for cross-validated prediction error. The size of the performance bias depends on the outcome distribution, feature correlation, selection rule, model, and performance measure. A direct comparison on a hypothetical data set can quantify the discrepancy for that example, while not serving as a correction applicable to other data.

Worked Example: A Null Screening Score Can Look Extreme

Consider a hypothetical screening problem with 1,000 independent null genes. Assume their standardized screening scores are approximately independent $N(0,1)$ variables. The analyst selects the gene with the largest absolute score. Using the leading approximation,

E(M)\approx\sqrt{2\log(2m)} =\sqrt{2\log(2000)} =\sqrt{15.20} \approx 3.90.

Thus, even when every gene is null, the largest absolute score can be around 3.90 under this rough approximation. This is not a p-value and does not say that every search will produce exactly 3.90. It illustrates why a strong-looking association found by searching many features is not equivalent to a prespecified test of one feature.

The calculation relies on independent, standard-normal scores, an approximation that may be poor for correlated genomic features or non-normal screening statistics. It quantifies the tendency to select an extreme score; it does not directly imply a particular increase in cross-validated accuracy. Fold-local screening addresses the evaluation problem by preventing each validation fold's outcomes from contributing to the selected feature set.

Worked Example: Quantifying an Illustrative Accuracy Difference

Suppose a hypothetical null data set has 100 individuals split into five folds of 20. A model is evaluated in two ways. In the first, 50 genes are selected once using outcomes from all 100 individuals, then the model is refitted in each training fold. In the second, the same screening rule selects 50 genes separately within each training fold. Assume the recorded numbers correct in the five validation folds are as follows.

For global screening, the fold accuracies are $16/20=0.80$, $15/20=0.75$, $17/20=0.85$, $14/20=0.70$, and $18/20=0.90$. Their unweighted mean is

\frac{0.80+0.75+0.85+0.70+0.90}{5} =\frac{4.00}{5}=0.80.

For fold-local screening, suppose the numbers correct are 11, 12, 10, 11, and 11. The fold accuracies are $0.55, 0.60, 0.50, 0.55,$ and $0.55$, giving

\frac{0.55+0.60+0.50+0.55+0.55}{5} =\frac{2.75}{5}=0.55.

In this hypothetical comparison, global screening raises the reported mean accuracy by $0.80-0.55=0.25$, or 25 percentage points, relative to fold-local screening. The global-screening estimate is contaminated because each validation fold's outcomes helped choose the genes. The difference is an illustration of selection-induced optimism, not a general bias estimate. Under a null outcome with balanced classes, accuracy near 0.50 is a natural reference, but its exact variation depends on the sample and prediction procedure.

Implementation, Diagnostics, and Alternatives

A reliable implementation keeps the screening rule and its inputs explicit. For each fold, construct the training data, calculate outcome-guided scores on that training data, select features, fit the prediction model, and then predict the held-out observations. Save the selected feature set for each fold so that the analysis can be audited. A feature's selection frequency across folds can describe stability, but it does not change the performance estimate or justify replacing the fold-local sets with a globally selected set.

Define the selection rule.
Specify the screening statistic, direction or absolute-value rule, and number or threshold of features before scoring the validation folds.

Screen using training outcomes.
Within each fold, calculate scores and select features using only that fold's training observations and outcomes.

Fit and predict.
Fit the prediction model with the training data and fold-specific feature set, then generate predictions for the held-out observations.

Score only after prediction.
Use held-out outcomes to calculate the fold metric. Combine fold results according to the stated cross-validation summary.

Useful diagnostics include checking that selected features are not computed once before the fold loop, confirming that held-out outcomes are never passed to the screening function, and recording the number of features selected in each fold. Identical selected sets across folds are not automatically evidence of leakage—strong signals can recur—but they are a reason to verify the code and data flow. A more direct audit is to change or permute held-out outcomes while holding the training data fixed: the selected features and predictions for that fold should not change, although the scored metric can change.

Fold-local selection is not the only design. A fixed feature panel chosen independently of the evaluation data avoids outcome-guided selection within the study. A separate discovery sample can also be used to select features, with evaluation performed on individuals whose outcomes did not contribute to discovery. By contrast, selecting once on all samples and then cross-validating is not an acceptable alternative when the goal is an honest estimate of performance on new observations.

Fold-local screening has limitations. The resulting estimate evaluates a procedure trained on fewer individuals than the full data set, and selected genes can vary across folds, especially when signals are weak or correlated. A low selection frequency does not by itself show that a gene is uninformative, and a high frequency does not establish a causal role. Cross-validation assesses predictive performance for the defined target and split design; it does not establish biological mechanism.

Common Mistakes

Screening once, then refitting within folds. Refitting model coefficients does not remove the use of held-out outcomes in the global feature-selection step. Repeat outcome-guided screening within every training fold.
Calling screening “just preprocessing.” If the screening rule uses outcomes, it is part of model building and must respect the fold boundary.
Assuming that more candidate genes only improve discovery. Searching more features increases the chance of selecting extreme noise, particularly under null or weak-signal conditions.
Interpreting the illustrative accuracy gap as a correction factor. The 25-percentage-point difference above comes from one hypothetical set of fold scores; it is not a universal adjustment.
Treating selected features as causal findings. Outcome association and predictive usefulness do not by themselves establish a causal effect.

Key takeaway: If outcomes influence which genes or variants enter a model, selection must be repeated using only the training observations in each validation fold. Otherwise, the validation outcomes have helped build the predictor they are supposed to evaluate.

Check Your Understanding

Answer each question, then compare with the brief solutions.

In a five-fold analysis, a researcher ranks genes by case-control mean difference using all individuals, then fits a model separately in each training fold. Is the fold score protected from outcome-guided selection? Why?
For a particular training fold, gene A has mean expression 7.2 in cases and 5.7 in controls. What is its case-minus-control screening score?
Why does searching 1,000 independent null genes tend to produce a larger maximum absolute screening score than searching 10?
A feature is selected in every fold. Does that alone prove that screening used held-out outcomes?
Name one alternative to fold-local screening that avoids using evaluation outcomes to choose features.

Solutions: (1) No. The held-out outcomes helped determine the globally selected genes, even though model fitting was fold-specific. (2) $7.2-5.7=1.5$ expression units. (3) Taking a maximum over more random scores gives more opportunities for an extreme value; under the independent standard-normal approximation, the leading expected maximum absolute score grows roughly as $\sqrt{2\log(2m)}$. (4) No. A strong and reproducible signal could be selected in every fold; inspect the data flow to diagnose leakage. (5) Use a feature panel fixed independently of the evaluation outcomes, or select features in a separate discovery sample and evaluate on independent individuals.