Why Feature Selection Must Follow the Split
In multi-omics prediction, the number of candidate features can greatly exceed the number of independent participants. Screening genes by their association with an outcome, ranking metabolites by a univariate test, or selecting variables with a supervised algorithm can reduce the predictor set. But these procedures also learn from the observed outcomes. If they see outcomes belonging to an evaluation fold, the fold is no longer an independent test of the complete prediction procedure.
The key distinction is between evaluating a model with a fixed, prespecified feature set and evaluating a procedure that chooses features from data. The second procedure includes both selection and model fitting. Honest resampling must evaluate that whole procedure: the features used for each held-out participant must be chosen without using that participant’s outcome.
Let \(D_{\mathrm{tr}}\) denote the training data in one resampling split and \(D_{\mathrm{va}}\) its held-out data. Write \(S(D_{\mathrm{tr}})\) for the selected feature set and \(A\) for a model-fitting rule. The evaluated prediction procedure is \(A(D_{\mathrm{tr}}; S(D_{\mathrm{tr}}))\). Its predictions on \(D_{\mathrm{va}}\) are assessed only after selection and fitting are complete. The validation outcomes may be used to calculate the evaluation metric, but not to select features, change the model, or choose among alternative selection rules for that same evaluation.
Fold-Local Selection in Practice
For each outer split, perform supervised screening using only that split’s training outcomes, select the features according to the stated rule, fit the model on the training observations and selected features, and then predict the held-out observations using that same feature set. A feature absent from a particular training fold is absent from that fold’s fitted model; the set can differ across folds. This variation is part of the procedure being evaluated, not a reason to choose a single set using all outcomes.
If the selection rule itself has adjustable settings—such as the number of top-ranked features, a screening threshold, or a penalty parameter—choose those settings without using the outer held-out outcomes. When tuning is required, use inner resampling within each outer training set. The outer held-out fold then evaluates the complete inner-tuning, selection, and fitting procedure. This is consistent with the earlier material on nested cross-validation. Preprocessing learned from data must also be confined to the relevant training partition, as covered in the earlier material on fold-local preprocessing.
Specify the supervised score or algorithm, feature-count or threshold rule, and any tie-handling decisions before inspecting outer-fold performance.
Within each outer training fold, calculate outcome-based scores and select features. If settings are tuned, do so using inner resampling restricted to this training fold.
Fit the predictor using the selected features in the outer training fold, then apply the fitted procedure to its held-out fold. Do not revise the feature set after seeing held-out outcomes.
Combine outer-fold predictions or metrics according to the evaluation plan. The selected feature list may differ from fold to fold.
Worked Example: A Screen That Must Be Repeated
Suppose a hypothetical study has 120 independent participants and 5,000 transcript features. A five-fold outer resampling plan holds out 24 participants at a time, leaving 96 in each training fold. The planned selection rule keeps transcript features with a training-fold association-test \(p\)-value below .01, then fits a logistic regression using the retained features.
In one outer training fold, suppose 31 of the 5,000 features meet the threshold. The logistic regression is fit using those 31 features and the 96 training participants. Predictions are then made for the 24 held-out participants. Their outcomes are used to evaluate these predictions, not to recalculate the 5,000 association tests.
The procedure is repeated in each outer fold. Another fold might retain 27 features or 36 features because its training participants and outcomes differ. That is valid: every held-out participant is evaluated by a selection rule that did not use that participant’s outcome. In contrast, screening once on all 120 outcomes and using the resulting list in all five folds would expose each held-out outcome to feature selection.
How Selection Creates Optimism
Even when every candidate feature is unrelated to the outcome in the population, some features will appear associated by chance in a finite sample. If many candidates are examined and the most promising are retained, the selected association is systematically more extreme than an association chosen independently of the outcomes. Evaluating on the same outcomes used for selection therefore measures both any genuine signal and the advantage gained from searching.
A simple normal-score argument quantifies this mechanism. Assume a null setting with \(p\) mutually independent candidate features, all unrelated to the outcome. Suppose a feature’s standardized association score in a training partition is \(T_j\), and its score in an equally sized held-out partition is \(V_j\). For this illustration, assume the scores are independent across partitions and features, with \(T_j,V_j\sim N(0,1)\). The standardized score from the combined sample is \(S_j=(T_j+V_j)/\sqrt{2}\), also standard normal. These assumptions are an idealized approximation; real feature scores can be correlated, and finite-sample scores need not be exactly normal.
If features are selected by the largest absolute combined-sample score, the selected score is \(\max_j |S_j|\). Because \(V_j\) and \(S_j\) are jointly normal, their conditional mean is \(E(V_j\mid S_j)=S_j/\sqrt{2}\). Consequently, the expected held-out score oriented in the direction chosen using the combined outcomes is \(\operatorname{E}[\max_j |S_j|]/\sqrt{2}\), which is greater than zero despite the null assumption.
The result follows by conditioning on the selected full-sample scores. Given all \(S_1,\ldots,S_p\), the index \(j^*\) and its sign are fixed, and the conditional expectation of the oriented held-out score is \(\operatorname{sign}(S_{j^*})S_{j^*}/\sqrt{2}=\max_j|S_j|/\sqrt{2}\). Averaging over the scores gives the expression above. Thus, a held-out association score is no longer centered at zero when the held-out outcomes helped choose which score to report.
This calculation concerns a standardized association score, not a universal inflation amount for accuracy, area under the curve, or another prediction metric. The amount of optimism in a prediction metric depends on the selection rule, the prediction model, feature dependence, sample size, and metric. The general conclusion is that evaluation outcomes reused for selection cannot provide an unbiased assessment of the entire selection-and-fitting procedure.
Worked Example: Approximate Null Selection Optimism
Consider the idealized null model above with \(p=100\) independent candidate features and equal-sized training and held-out partitions. Suppose the feature with the largest absolute combined-sample score is selected. For independent standard normal \(S_j\), the expected maximum absolute score can be written as
where \(\Phi(t)\) is the standard normal cumulative distribution function. Numerical evaluation gives an expected maximum of approximately 2.75. Therefore the expected held-out score in the direction selected using the combined outcomes is approximately
The arithmetic shows that searching 100 null features can produce a selected held-out association score with an expected magnitude near 1.94 standard-score units in this idealized setting. It does not mean that every performance metric is inflated by 1.94 units, nor that an individual selected feature is certainly null. It demonstrates why using outcomes from an evaluation partition during screening can create an apparently favorable association even when no candidate feature carries population signal.
Worked Example: Comparing Leaked and Fold-Local Screening
Suppose a hypothetical split has two candidate features, A and B. Their standardized signed association scores are summarized as follows: on the training partition, feature A has score \(T_A=0.4\) and feature B has score \(T_B=1.2\); on the held-out partition, their scores are \(V_A=2.4\) and \(V_B=-0.2\). Assume equal partition sizes and use the combined-sample score \((T_j+V_j)/\sqrt{2}\) for the illustration.
If screening uses both partitions, the combined scores are
The full-data screen selects feature A because \(|1.98|>|0.707|\). Its held-out association score is \(2.4\), but that score contributed to selecting A, so it is not an independent validation result. If screening instead uses training data only, feature B is selected because \(|1.2|>|0.4|\). Its held-out score is \(-0.2\). This example does not establish that B is truly predictive or that the split score is a stable estimate; it shows that the selected held-out association can change when held-out outcomes are allowed to determine the feature.
Diagnostics, Alternatives, and Limits
A useful audit asks where each outcome-dependent decision occurred. Trace feature rankings, screening thresholds, feature counts, and any decisions made after inspecting fold performance. If a choice used all outcomes before splitting, or was changed after reviewing outer-fold results, the outer evaluation has been compromised. A clean implementation can record, for every outer fold, the training participant identifiers, the selected feature identifiers, and the rule that produced them.
Differences in selected features across folds are not themselves evidence of leakage. They may reflect sampling variability, correlated predictors that can substitute for one another, or genuinely weak signals. Conversely, a stable list does not prove that selection was fold-local: the same leaked list can appear in every fold. The decisive diagnostic is the data flow, not whether the selected lists look similar.
| Approach | What it evaluates | Main concern |
|---|---|---|
| Screen once using all outcomes, then resample | A model conditional on a feature set informed by the evaluation outcomes | Optimistic evaluation; held-out outcomes helped select predictors |
| Repeat screening inside each training fold | The specified selection-and-fitting procedure | Valid only if no held-out outcome influences selection or fitting |
| Use an externally prespecified feature list | A model using that fixed list | The list must genuinely have been fixed independently of the evaluation outcomes and target data |
A feature list established in independent data can be treated as prespecified for a new evaluation, provided the list and its construction were not informed by the evaluation outcomes. This is different from screening the current data once and then calling the resulting list fixed. Fold-local selection also does not eliminate uncertainty: with limited sample size, selected features can be unstable, and the estimated performance can vary across splits. Resampling evaluates the stated procedure under its assumptions; it cannot guarantee transport to a different population or repair a mismatch between the split and the intended deployment setting.
This tutorial concerns selection that uses outcomes. Predictor transformations or feature learning that do not use outcomes raise a separate question about what information their fitting uses; the relevant training-fold boundary still matters, but the outcome-guided selection argument above should not be treated as a complete analysis of those procedures.
Common Mistake
- Wrong reasoning: “The model was fitted only on the training fold, so the validation fold is independent.”
- Correction: If features were ranked or filtered using validation outcomes before fitting, the validation fold already influenced the model-building procedure. Repeat the supervised selection inside the training fold, then use the held-out outcomes only for evaluation.
- Wrong reasoning: “The features were selected before cross-validation, so they are fixed during cross-validation.”
- Correction: A feature set is not independent merely because it is held fixed after selection. If its selection used outcomes from the observations later treated as held out, evaluation is conditional on information from those outcomes.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A researcher screens 10,000 genes using outcomes from the full dataset, then runs five-fold cross-validation. Is the resulting cross-validation evaluation an honest assessment of the screen-and-fit procedure? Why?
- In an outer training fold, 18 features pass a prespecified outcome-association threshold. May the analyst use the outer held-out outcomes to decide to keep only the top 10?
- Under the idealized null-score derivation, why is the expected selected held-out score positive when feature selection uses the combined outcomes?
- A feature list was specified using a separate, independent dataset before the evaluation cohort was examined. Must that list be reselected in every fold of the evaluation cohort?
Solutions. (1) No. Outcomes from each held-out fold contributed to selecting the genes, so the held-out data are not independent of the full selection-and-fit procedure. The screen must be repeated in each training fold. (2) No. Choosing the feature count using outer held-out outcomes makes those outcomes part of model selection. Use only training data, with inner resampling if the count is to be tuned. (3) Selection favors features with large absolute combined scores. Because the held-out score contributes to each combined score, its conditional expectation in the selected direction is positive; specifically, it is \(\operatorname{E}[\max_j|S_j|]/\sqrt{2}\) under the stated assumptions. (4) Not necessarily. A genuinely independent, prespecified list can be treated as fixed for that evaluation, provided the evaluation outcomes did not influence its construction or choice.