Why Screening Can Make Null Features Look Predictive
Genomic prediction often starts with many candidate features and comparatively few individuals. When \(p\), the number of features, is much larger than \(n\), the sample size, a screening step can be useful: it ranks or filters features before a prediction model is fitted. But if the outcome is used to screen features, the screen itself learns from the outcomes. Evaluating a model after screening on all outcomes can therefore give an overly optimistic estimate of prediction performance.
The issue is not limited to a sophisticated model. Even a simple rule that chooses the feature most associated with the outcome can find an impressive-looking feature among many null features. If the same outcomes are then used to evaluate predictions, the evaluation reuses information that helped choose the feature. Cross-validation only separates training from validation for operations performed using the training data; it cannot undo a screen already informed by validation outcomes.
For a concrete setting, consider binary outcomes \(Y_i\in\{0,1\}\) and binary genomic features \(X_{ij}\in\{0,1\}\). A simple screen could rank features by their classification accuracy after choosing the better of the two possible orientations: predict \(Y=1\) when the feature is 1, or predict \(Y=1\) when it is 0. This is a deliberately simple screen; the same information problem arises with other outcome-based association scores.
A Null-Feature Calculation
Suppose the \(n\) outcomes are fixed and each candidate feature consists of independent fair binary values, independent of those outcomes and of the other features. Such a feature is null: its values carry no predictive information about \(Y\). For a feature to match all \(n\) outcomes in the same direction, its \(n\) values must equal the outcome vector, which has probability \(2^{-n}\). To match perfectly in the opposite direction, it must equal the complement of that vector, also with probability \(2^{-n}\). For \(n\geq 1\), these two events are disjoint. Thus
If the \(p\) features are independent under this setup, the probability that none matches perfectly is the probability that each avoids a perfect match. Consequently,
This calculation concerns chance matches under a specified null model. It does not say that a selected match reflects a true biological relationship. Nor does it require the analyst to have deliberately searched for a perfect feature: ranking thousands of features by association makes the most extreme chance results especially likely to rise to the top.
Worked Example: A Perfect Match Among 10,000 Null Features
Suppose \(n=8\) individuals have fixed binary outcomes and \(p=10{,}000\) independent null binary features follow the assumptions above. For any one feature, the probability of matching all outcomes either directly or in reverse is
The probability that all 10,000 features avoid a perfect match is
Therefore, the probability of at least one perfect match is
Under this hypothetical null setup, a perfect-looking feature is virtually certain to occur somewhere in the screen. Finding one in the full data is not evidence by itself that it will predict outcomes for new individuals.
Why Full-Data Screening Invalidates a Cross-Validation Score
In \(K\)-fold cross-validation, each fold is held out in turn. The model is fitted using the other folds, and predictions for the held-out individuals contribute to an estimate of performance. For that estimate to assess the specified training procedure, every outcome-guided step of that procedure must be repeated using only the current training folds. If feature screening is done once using all \(n\) outcomes, each held-out outcome has already had a chance to influence which features are retained. The validation predictions are no longer independent of the information used to select the predictors.
A high score produced this way does not demonstrate that the fitted model will generalize. It partly reports how well the screen was able to find features associated with the very outcomes later labelled “held out.” This is selection-induced optimism: the selected features tend to look better in the data used to select them than they would in fresh data.
Worked Example: Full-Data Screening Versus One Training Fold
Consider eight hypothetical individuals, with outcomes and two candidate features shown below. Use individuals 1, 2, 5, and 6 for training, and individuals 3, 4, 7, and 8 for validation. The screening rule chooses the feature with the greater accuracy when predicting \(Y=1\) for feature value 1.
| Individual | Outcome \(Y\) | Feature A | Feature B | Role |
|---|---|---|---|---|
| 1 | 0 | 0 | 0 | Training |
| 2 | 0 | 1 | 0 | Training |
| 3 | 0 | 0 | 1 | Validation |
| 4 | 0 | 0 | 1 | Validation |
| 5 | 1 | 1 | 1 | Training |
| 6 | 1 | 0 | 1 | Training |
| 7 | 1 | 1 | 0 | Validation |
| 8 | 1 | 1 | 0 | Validation |
Screened on all eight outcomes, Feature A is correct for individuals 1, 3, 4, 5, 7, and 8, giving \(6/8=0.75\) accuracy. Feature B is correct for individuals 1, 5, 6, and 7, giving \(4/8=0.50\). A full-data screen therefore selects Feature A. It is correct for all four validation individuals, so scoring that already selected feature on the validation fold gives \(4/4=1.00\). But this apparent validation success is not independent: validation outcomes helped select A.
Now screen using only the training individuals. For Feature A, training values are \(0,1,1,0\) and training outcomes are \(0,0,1,1\). Its accuracy is \(2/4=0.50\). For Feature B, the training values \(0,0,1,1\) match the training outcomes, so its accuracy is \(4/4=1.00\). Fold-local screening selects B. On the validation fold, B has values \(1,1,0,0\) against outcomes \(0,0,1,1\), so it gets \(0/4=0\) correct.
The low fold-local score is not a failure of the procedure: with only four training and four validation individuals, this particular split is highly variable. The key distinction is that the fold-local choice of B did not use validation outcomes. A high or low score can occur by chance; leakage is a defect in the evaluation design, not a promise that every honest split will score poorly.
Fold-Local Screening Procedure
The screening rule must be specified as part of the prediction procedure. For each fold, calculate its screening scores, choose features, fit the prediction model, and predict the held-out individuals using the chosen features. The validation outcomes are used to calculate the fold’s performance, not to decide which features enter that fold’s model.
Keep its outcomes unavailable to all outcome-guided training steps.
Compute the specified feature scores from training predictors and training outcomes, then apply the prespecified selection rule.
Fit the prediction model using the selected training features, then apply that fitted model to the held-out individuals.
Compare predictions with held-out outcomes to calculate fold performance, then repeat the entire screening-and-fitting procedure for each fold.
The screen’s specification includes more than the name of a score. It includes how features are ranked, how many are retained or what threshold is used, how ties are handled, and any outcome-informed decision about feature direction. These choices must not be made by looking at the current held-out outcomes. If a choice is revised after observing validation performance, those outcomes have also influenced the procedure being evaluated.
Worked Example: What a Fold-Local Score Means Under the Null
Suppose training-fold screening selects a feature and fits a classifier, while a held-out fold contains 20 independent individuals with balanced binary outcomes. Assume the fitted predictions are independent of these held-out outcomes, as would hold under the null model when the training process uses only training data. If 14 predictions are correct, the observed validation accuracy is
Conditional on the fitted predictions, each prediction has probability \(1/2\) of matching an independent balanced outcome. The expected number correct is therefore \(20(1/2)=10\), and the expected accuracy is \(10/20=0.50\). An observed \(0.70\) in this small fold can occur from sampling variability; it does not, by itself, show that leakage occurred or that the feature is genuinely predictive.
The balanced-outcome condition matters. If the held-out prevalence of class 1 is \(\pi\), and a fixed prediction is 1 with probability \(q\) independently of the outcome, the expected accuracy is \(\pi q+(1-\pi)(1-q)\), not necessarily \(1/2\). Thus chance performance depends on the outcome distribution and prediction rule; fold-local screening prevents a particular source of optimism but does not make every benchmark equal to 50%.
Assumptions, Diagnostics, and Limitations
The null-feature probability calculation assumed independent fair binary feature values, independent features, and outcomes fixed independently of those features. Real genomic predictors can be correlated, and their distributions need not be binary or fair. Correlation changes the probability calculation: many redundant features may provide fewer independent opportunities for extreme matches than the formula suggests. Nevertheless, the central evaluation rule does not depend on features being independent. Whenever outcomes help choose features, held-out outcomes must not participate in that choice.
Useful diagnostics begin with an information audit: list every step that uses outcomes, and ask whether it is repeated using only each training fold. A suspiciously strong score after ranking or filtering features once on the full data is a warning, not proof of signal. Another diagnostic is to compare the reported procedure with a faithfully fold-local implementation; a large drop can reveal that the earlier score depended heavily on selection. Such a comparison does not, on its own, identify the biological cause of any difference.
- Full-data screening followed by cross-validation: the validation outcomes influence feature choice, so the score is contaminated.
- Fold-local screening: the held-out outcomes are excluded from feature choice, so the evaluation matches a procedure trained without those outcomes.
- Prespecified features: if features were fixed independently of the evaluation outcomes, outcome-guided screening may not be needed; the evaluation still must respect the other data-splitting requirements of the study.
Fold-local screening is not a cure for every evaluation problem. A valid split must still reflect the intended prediction target and sampling structure. Small folds can produce unstable selections and noisy scores. Also, if the analyst uses validation performance to choose among screening rules or other analysis settings, that performance has become part of the choice; a single layer of cross-validation does not make such tuning harmless. Finally, fold-local evaluation estimates the performance of the stated training procedure, including its screening step. It does not establish a causal effect or prove that a selected feature has a biological mechanism.
Common Mistake
Wrong reasoning: “I used cross-validation, so the validation outcomes were independent of training, even though I screened all features once using the complete outcome vector.” The screen has already used those outcomes; dividing the data into folds afterward cannot reverse that information flow.
Correction: Repeat outcome-guided screening separately inside every training fold. Use the selected features to predict that fold’s held-out individuals, and use held-out outcomes only to assess those predictions. Interpret the resulting score with its sampling variability and the stated prediction target in mind.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- With \(n=6\) and \(p=500\) independent null binary features under the assumptions in the derivation, what is the probability that one feature matches the outcomes perfectly in either direction?
- A researcher screens all features using the complete outcome vector, then performs five-fold cross-validation. Why is the resulting score not an honest assessment of that screening-and-fitting procedure?
- In one training fold, Feature A has accuracy \(7/10\) and Feature B has accuracy \(8/10\). The prespecified rule retains the better feature. Which feature is retained, and what information may be used to score its predictions?
- Does a low score from fold-local screening prove that the screen was performed incorrectly? Explain.
Solutions. (1) For one feature, the probability is \(2^{1-6}=2/64=1/32=0.03125\), or about \(3.125\%\). (2) Each held-out outcome contributed to the full-data screen and could therefore affect which features were chosen; the fold is not held out from all outcome-guided training steps. (3) Feature B is retained because \(8/10>7/10\). Its held-out predictions are scored by comparing them with the held-out outcomes, but those outcomes must not influence screening, fitting, or other training decisions for that fold. (4) No. Small or variable folds can yield low scores even when the procedure is correctly fold-local. The defining question is whether held-out information influenced training, not whether the observed score is high.