Why Screening Must Respect the Split
A supervised predictor screen uses the outcome to decide which predictors appear useful. Examples include ranking predictors by their association with a numeric outcome, comparing outcome rates across predictor groups, or retaining predictors whose screening statistic exceeds a threshold. The screen is part of the model-building procedure because its result determines which information the fitted model may use.
That creates a specific risk during validation: if outcomes from a validation fold help choose predictors, the evaluation is no longer based on data that were kept separate from model development. Even if the final regression or other model is fitted only on the training rows, the screen may already have learned from validation outcomes. A validation score calculated afterward can then be too favorable.
The governing distinction is not whether the screen is simple or complex. It is whether the information used to make the screening decision would be available when predicting a genuinely new observation. A correlation, a p-value, a difference in group outcome rates, and a chosen screening threshold can all leak information if they are determined using held-out outcomes.
Fit the Screen Within Each Split
Suppose a data set is divided into a training set \(T\) and a validation set \(V\). Let \(S_T\) denote the set of predictors selected by applying a specified screening rule to the training data, including its outcomes. The fitted model uses \(S_T\), and the validation rows are evaluated using the corresponding predictor columns. The validation outcomes are used to calculate the performance measure only after the screening and fitting steps are complete.
Here \(X_{T,S_T}\) means the training predictors restricted to the selected set, and \(X_{V,S_T}\) means the validation predictors restricted to exactly that same set. Do not run a second outcome-informed screen on \(V\). If a screen selects no predictors, or selects predictors that cannot be fitted under the chosen model, that outcome must be handled by a prespecified rule rather than by looking for a more favorable validation result.
For \(K\)-fold cross-validation, the same logic is repeated \(K\) times. In fold \(k\), fit the screen on the other \(K-1\) folds, fit the model using the resulting predictors, and score predictions on fold \(k\). The selected sets may differ between folds. The cross-validation estimate summarizes the performance of this full procedure, including its screening step, rather than the performance of one screen created using all observations.
Identify the training rows and the held-out rows before calculating any outcome-informed screening statistic.
Calculate scores, apply thresholds or rankings, and record the selected predictors using training outcomes only.
Fit the model with the selected training predictors, then generate held-out predictions using the same predictor set.
Use held-out outcomes to calculate the fold's performance measure. Do not revise that fold's screen in response to its score.
Worked Example: Applying a Binary-Outcome Screen to Training Data
A hypothetical insurer is developing a model for whether a submitted small-equipment claim will require additional review. A training split contains eight records. For two binary candidate predictors, the training counts are as follows. The screening score is the absolute difference between the outcome rates for predictor values 1 and 0. Retain a predictor only if its score is at least 0.40.
| Predictor | Outcome 1 when predictor = 1 | Records when predictor = 1 | Outcome 1 when predictor = 0 | Records when predictor = 0 |
|---|---|---|---|---|
| A | 2 | 4 | 1 | 4 |
| B | 3 | 4 | 0 | 4 |
Method. Calculate each rate from the training counts, take the absolute difference, and compare it with 0.40. The outcome rates for A are \(2/4=0.50\) and \(1/4=0.25\), so its score is \(|0.50-0.25|=0.25\). Since \(0.25<0.40\), A is not retained.
For B, the rates are \(3/4=0.75\) and \(0/4=0.00\). Its score is \(|0.75-0.00|=0.75\). Since \(0.75\geq0.40\), B is retained. The selected set for this split is therefore \(\{B\}\).
Apply that decision to the validation rows by using their B values to form predictions. Do not calculate a validation outcome-rate difference for A or B and use it to change the selected set. Common mistake avoided: treating held-out outcomes as additional data for screening because they are available in the analysis file. They are reserved for evaluating the completed training procedure.
Why Full-Data Screening Contaminates Validation
A short probability argument explains why screening on all observations can create an overly favorable result. Assume, for illustration, that candidate predictors are unrelated to the outcome and that their standardized association scores are independent standard normal variables under the null. If the screen retains the predictor with the largest absolute score among \(m\) candidates, it is selecting the most extreme observed association, even though all candidates are noise.
For one candidate, the probability that the absolute standardized score exceeds 1.96 is approximately 0.05. For \(m\) independent candidates, the probability that at least one exceeds 1.96 is the probability that not all \(m\) scores remain within that range:
This calculation shows how selecting on observed outcomes can find an apparently strong association by chance. If the validation outcomes are included in that selection, the chosen predictor is partly selected for matching validation noise. Consequently, its validation performance is not an independent check of the selection decision.
By contrast, with an independent train/validation split, the selected predictor set is determined without using validation outcomes. Conditional on the training data and the resulting fitted model, the validation outcomes remain unseen during model development. Under the usual assumption that the split observations are representative draws from the same target population, the validation score then assesses the trained procedure on held-out cases. It may still be uncertain because the validation sample is finite, but it has not been optimized using those outcomes.
Worked Example: Quantifying the Chance of a Spurious Screen
A hypothetical analyst has 10 candidate predictors and plans to retain the one with the largest absolute standardized association with a numeric outcome. Assume all 10 predictors are truly unrelated to the outcome, and use the independent standard normal approximation described above. Calculate the probability that at least one candidate has an absolute score above 1.96.
Method. The probability that one null score is within the interval from \(-1.96\) to \(1.96\) is approximately \(0.95\). Under the stated independence assumption, the probability that all 10 scores are within the interval is \(0.95^{10}=0.5987369\), to seven decimal places.
Therefore, the probability that at least one exceeds the cutoff in absolute value is \(1-0.5987369=0.4012631\). To four decimals, this is 0.4013, or about 40.13%.
This is not the probability that a particular predictor is truly useful. It is the probability that the largest of these 10 null scores crosses the stated cutoff under the simplifying assumptions. If the screen uses all observations, including a validation fold, that fold's outcome noise contributes to choosing the extreme score. If the screen is fitted only on the training portion, the held-out outcomes do not contribute to that choice. Common mistake avoided: interpreting a large score selected from many candidates as if it had been specified in advance.
Fold-Specific Screens and the Final Fit
The predictors selected in one training fold do not have to match those selected in another. Each fold represents a new training sample, and the selection procedure is allowed to respond to that sample. The valid cross-validation result is obtained by applying the complete procedure afresh in every fold and aggregating the held-out scores. It is not necessary to force one globally selected set of predictors onto every fold.
Worked Example: Different Screens in Two Validation Folds
A hypothetical equipment-repair study uses two candidate binary predictors, A and B, for a binary outcome. In each fold, the screen retains a predictor when the absolute difference in training outcome rates between its two values is at least 0.50. The training counts for the two folds are shown below.
| Fold | Predictor | Rate for value 1 | Rate for value 0 |
|---|---|---|---|
| 1 | A | 4/5 = 0.80 | 1/5 = 0.20 |
| 1 | B | 3/5 = 0.60 | 2/5 = 0.40 |
| 2 | A | 2/5 = 0.40 | 3/5 = 0.60 |
| 2 | B | 5/5 = 1.00 | 0/5 = 0.00 |
Fold 1. A has score \(|0.80-0.20|=0.60\), which meets the 0.50 cutoff. B has score \(|0.60-0.40|=0.20\), which does not. Fold 1 retains A.
Fold 2. A has score \(|0.40-0.60|=0.20\), below the cutoff. B has score \(|1.00-0.00|=1.00\), which meets the cutoff. Fold 2 retains B.
The selected sets differ: fold 1 uses A and fold 2 uses B. Each set was determined using only that fold's training counts. The held-out rows in each fold are then predicted with the set selected for that fold, and their outcomes are used for the fold's performance score. Common mistake avoided: screening once on the combined data and then treating the resulting single set as if it had been determined independently of every fold.
After cross-validation has been used to assess a modeling procedure and the procedure has been finalized, a final model intended for future use can be developed using all available development data. At that stage, rerun the screen on all development outcomes and fit the model using the resulting predictors. If a separate test set is reserved for a final performance estimate, neither screening nor other model-development decisions may use that test set's outcomes. If test results are used to revise the screen or modeling choices, the test set has become part of development data and can no longer provide an untouched final assessment.
Common Leakage Traps
- Screening before creating folds. Selecting predictors once from all outcomes and then cross-validating only the model fit leaks fold outcomes into every selected set.
- Reusing a validation fold to revise the rule. Trying several screens or cutoffs and keeping the one that performs best on the same held-out outcomes makes those outcomes part of model development.
- Assuming a small screen is harmless. A simple correlation or rate difference still uses outcome information. Simplicity does not prevent leakage.
- Using a different feature set at scoring time. A held-out fold must be evaluated using the predictor set selected in its corresponding training fold, not a set selected using held-out outcomes.
The practical check is to ask, for every outcome-informed choice, “Could this decision have been made without seeing the held-out outcomes?” If not, move that step inside the training portion of each split. This disciplined workflow is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, identify the correct screening decision or validation procedure.
- A training split has a binary predictor with outcome rates \(6/10\) for value 1 and \(2/10\) for value 0. The retention cutoff is an absolute rate difference of 0.35. Calculate the score and state whether the predictor is retained. (Answer: Score \(=|0.60-0.20|=0.40\); retain it.)
- A screen ranks eight independent null predictors by absolute standardized association score. Using the 0.05 tail probability beyond 1.96 for one predictor, calculate the approximate probability that at least one score exceeds 1.96. Round to four decimals. (Answer: \(1-0.95^8=0.3366\).)
- In one fold's training data, predictor C has outcome rates \(3/6\) and \(1/6\); predictor D has rates \(5/6\) and \(2/6\). With a retention cutoff of 0.40, which predictors are retained? (Answer: C has score \(2/6=0.3333\) and is not retained; D has score \(3/6=0.5000\) and is retained.)
- An analyst calculates outcome correlations on all rows, keeps the top five predictors, and then performs five-fold cross-validation using only those five predictors. Identify the leakage and state the correction. (Answer: The full-data screen used each fold's held-out outcomes; rerun the screen using only the training portion in each fold, then apply that fold's selected set to its held-out rows.)
- After cross-validation is complete, a final model is fit using all development data. A separate test set has not been used for any decisions. May the screen be rerun on all development outcomes before fitting the final model? (Answer: Yes. Rerun it on all development data; keep the separate test outcomes untouched for final evaluation.)