Why Predictor Selection Must Stay Inside the Folds
Cross-validation estimates how well a modeling procedure predicts observations that were not used to fit that procedure. The procedure includes more than estimating model coefficients. If it chooses predictors by looking at their relationships with the response, that choice is part of the procedure too. Performing the choice before cross-validation lets outcomes from the validation folds influence which predictors are evaluated.
This is selection leakage. It can occur with a large or small predictor set, and with formal methods such as stepwise selection or informal screening such as keeping the predictors with the largest correlations. Even when each fold’s model coefficients are fitted using only that fold’s training data, the evaluation is compromised if the predictor set was selected using all the outcomes first.
The Fold-Contained Principle
Suppose a data set \(D\) contains predictors and a response. Let \(A\) denote a modeling procedure that includes both predictor selection and model fitting. In \(K\)-fold cross-validation, each observation is held out once, and the fitted procedure is assessed on that held-out fold. For the assessment to represent new observations, the entire procedure \(A\)—not just its final coefficient estimation—must be run separately using the training portion of each split.
For example, suppose a procedure first retains the five predictors with the strongest association with the response and then fits a regression model. In each outer training fold, it must determine those five predictors again using only that fold’s outcomes. It then fits the regression using those selected predictors and assesses predictions on the outer validation fold. The held-out outcomes must not determine the list of five.
Do not use its outcomes to select predictors, choose settings, or fit model parameters.
Apply the stated selection rule using only the outer training observations. If that rule itself uses cross-validation to choose among options, perform that inner cross-validation within the outer training data.
Fit the model using the selected features and all available outer training observations, then predict the untouched outer validation fold.
Use the losses on the outer validation folds to estimate the predictive performance of the complete selection-and-fitting procedure.
If the number of selected predictors is fixed in advance, the selection step still belongs inside each outer training fold. If the number is chosen by comparing candidate values, that comparison also must avoid the outer validation outcomes. Nested cross-validation is a standard way to do this: inner folds choose the predictors or settings within each outer training set; outer folds assess the resulting procedure.
Worked Example: Finding Leakage in a Screening Pipeline
A hypothetical insurer has 120 historical policy records, 60 candidate rating features, and a claim-cost response. An analyst ranks all 60 features by their absolute sample correlation with claim cost, keeps the top 8, and then performs 5-fold cross-validation. In each fold, the analyst refits the regression coefficients using the 96 training records and computes prediction error on the 24 held-out records. Is this a valid cross-validation estimate of the stated modeling procedure?
Method. Trace which observations influence each part of the pipeline. The question is not only whether the regression coefficients are estimated within each fold; it is also whether the validation outcomes influence feature selection.
Selection step. The analyst calculates all 60 correlations using 120 responses, including the 24 responses in the fold currently being treated as validation data. Those responses therefore help determine which 8 features appear in every fold’s regression.
Fitting step. Refitting coefficients on 96 records does not undo that dependence. Although those coefficient estimates exclude the 24 held-out records, the selected feature list was already informed by their outcomes.
Conclusion and correction. The estimate is not a clean assessment of the full selection-and-fitting procedure. In each fold, the analyst must rank the 60 features using only the 96 training records, keep the top 8 there, fit the regression on those records, and evaluate predictions on the 24 held-out records. The common mistake is to treat fold-specific coefficient fitting as sufficient protection; the correction is to repeat every outcome-informed selection step within the fold.
Why Choosing the Best Result Creates Optimism
Selection leakage can also arise when several candidate feature sets are each evaluated by cross-validation and the one with the lowest reported error is chosen. The same validation outcomes then serve two roles: they help choose the winner and they are used to report the winner’s performance. The reported minimum is generally more favorable than the performance of the selection procedure on new data.
A short derivation shows the source of the effect. Suppose two candidate procedures have the same true prediction risk \(R\), but their estimated risks are \(R+\epsilon_1\) and \(R+\epsilon_2\), where each error has mean zero. Choosing the smaller estimated risk produces an error term \(\min(\epsilon_1,\epsilon_2)\). Since \[ \min(\epsilon_1,\epsilon_2) =\frac{\epsilon_1+\epsilon_2-|\epsilon_1-\epsilon_2|}{2}, \] taking expectations gives \[ E[\min(\epsilon_1,\epsilon_2)] =-\frac{1}{2}E[|\epsilon_1-\epsilon_2|]\leq 0. \] If the estimates differ with positive probability, the inequality is strict. Thus, even when each candidate’s estimated risk is unbiased before selection, the minimum selected estimate tends to be too low. With different true risks, the direction and size of the total error also depend on how accurately the candidates are estimated; the key point is that choosing and evaluating on the same outcomes makes the reported minimum unreliable as an estimate for the selected procedure.
Worked Example: Comparing a Leaky Result With a Nested Estimate
A hypothetical claims study compares three feature-selection procedures, A, B, and C. An analyst has already computed the following mean squared prediction errors for each candidate on three validation folds. The errors are reported in thousands of squared dollars.
| Candidate | Fold 1 loss | Fold 2 loss | Fold 3 loss | Mean loss |
|---|---|---|---|---|
| A | 1 | 0 | 2 | 1 |
| B | 2 | 1 | 1 | 4/3 |
| C | 3 | 2 | 0 | 5/3 |
The analyst chooses the candidate with the smallest mean loss and reports that same mean as the estimated performance. A separate properly nested run of the full selection procedure gives outer-fold mean squared errors of 4, 3, and 5. Calculate both reported quantities and explain which one assesses the selection procedure.
Method. First calculate the candidate means used in the leaky comparison. Then calculate the mean outer loss from the nested run, where feature selection was repeated using only each outer training set.
For A, the mean loss is \((1+0+2)/3=3/3=1\). For B, it is \((2+1+1)/3=4/3\). For C, it is \((3+2+0)/3=5/3\). Since \(1<4/3<5/3\), the analyst selects A and reports a mean squared error of 1.
For the nested run, the outer-fold losses sum to \(4+3+5=12\), so their mean is \(12/3=4\). The nested value of 4 assesses the procedure that selects features within training data and then fits the model. The value 1 is the minimum among results evaluated on the same folds used to choose the winner, so it is not an independent estimate of the selected procedure’s performance. The common mistake is to report the winning candidate’s minimum as though the candidate had been specified in advance.
What Must Be Repeated Within Each Training Fold?
A useful audit is to list every decision that depends on the observed response or on the observed predictor distribution. Any such decision that is part of the intended modeling procedure must be repeated without access to the outer validation fold. Examples include:
- Screening predictors by correlation, statistical significance, or a univariate score involving the response.
- Choosing a subset size, a stepwise stopping point, or a threshold for retaining features.
- Comparing alternative feature sets or model specifications and keeping the one with the best validation result.
- Selecting a transformation or a tuning value based on predictive performance.
Some data-dependent operations do not use the response but still need to be estimated from training data. For example, a rule that removes near-zero-variance predictors or groups features based on their observed distributions can be influenced by validation predictors. Handling such operations within training folds is consistent with the broader principle of fold-contained preprocessing. Here, the central issue is selection informed by outcomes: a feature can look useful precisely because it happened to align with responses in the full data set.
The target should be clear. If the goal is to estimate the performance of a procedure that searches over predictors, each outer training set must perform that search. If the predictors were instead fixed using information genuinely available before the study and independently of the outcomes under evaluation, that fixed list can be treated as prespecified. The distinction is whether the held-out data had a role in making the choice.
Worked Example: Calculating an Outer-Fold Estimate After Inner Selection
A hypothetical portfolio study uses three outer folds. Within each outer training set, an inner cross-validation comparison chooses between feature sets A and B. The inner mean squared errors and the resulting outer validation mean squared errors are shown below. The outer losses are in thousands of squared dollars.
| Outer fold | Inner error for A | Inner error for B | Selected set | Outer validation MSE |
|---|---|---|---|---|
| 1 | 0.84 | 0.91 | A | 1.21 |
| 2 | 0.96 | 0.72 | B | 0.81 |
| 3 | 0.88 | 0.90 | A | 1.00 |
Calculate the mean outer-fold MSE and its square root. Assume that, for each fold, the chosen set was refitted using the entire outer training set before its outer validation predictions were made.
Method. Use the inner results only to select within each outer training set, then average the losses on the untouched outer folds.
In fold 1, \(0.84<0.91\), so A is selected. In fold 2, \(0.72<0.96\), so B is selected. In fold 3, \(0.88<0.90\), so A is selected. The outer losses are \(1.21\), \(0.81\), and \(1.00\); their sum is \(1.21+0.81+1.00=3.02\). The mean outer-fold MSE is \(3.02/3=1.0067\) to four decimal places. Its square root is approximately \(1.0033\), also to four decimal places, in the corresponding thousands-of-dollars scale. The calculation is valid because the inner comparisons choose the feature set without using that fold’s outer outcomes.
Check. The chosen sets differ across outer folds, which is allowed: the evaluated object is the selection procedure, not one feature set chosen using all observations. The common mistake is to choose a single set using the three outer losses and then describe their mean as an independent estimate.
Practical Checks and Exam Traps
When assessing a proposed resampling analysis, ask two questions: What was selected? and Which observations could influence that selection? A phrase such as “the model was refit in every fold” is not enough to establish validity. The selection rule itself must be traced.
- Trap: global screening followed by fold-specific fitting. Selecting features once on the full data still exposes the validation outcomes to the selection step. Refit the selection rule inside each training fold.
- Trap: taking the minimum of several cross-validation errors. If the same folds choose the winner and quantify its error, the reported minimum is subject to selection optimism. Use an outer evaluation layer to assess the search procedure.
- Trap: treating a chosen feature set as fixed after seeing the results. A feature set is not prespecified merely because it is fixed during the later coefficient-fitting step. If outcomes from the evaluation data helped choose it, it remains a data-dependent choice.
- Trap: assuming nested resampling guarantees zero uncertainty. Nested evaluation prevents this particular reuse of outer outcomes for selection. The resulting estimate still has sampling variability and reflects the training-set sizes and design used.
Check Your Understanding
For each situation, decide whether selection leakage occurs or calculate the requested outer estimate.
- An analyst uses all 180 responses to retain the 12 predictors with the largest absolute correlations, then runs 6-fold cross-validation, refitting coefficients within each fold. Is the reported cross-validation error a clean estimate of this selection-and-fitting procedure? (Answer: No; the held-out responses influenced the predictor list.)
- In a four-fold nested evaluation, the outer-fold mean squared errors are 2.4, 1.6, 3.2, and 2.8. Calculate the mean outer-fold MSE. (Answer: \((2.4+1.6+3.2+2.8)/4=10.0/4=2.5\).)
- A team compares 15 feature sets using the same five validation folds and reports the smallest of the 15 mean errors as its final performance estimate. What is the central problem, and what is the standard correction? (Answer: The same validation outcomes select and evaluate the winning set, making the minimum optimistic; use an outer validation layer and make the feature-set choice within each outer training set.)
- In one outer training set, inner cross-validation errors for feature sets R, S, and T are 0.73, 0.68, and 0.71. The selected set’s outer-fold MSE is 1.44. Which set is selected, and what loss contributes to the outer performance estimate? (Answer: S is selected because \(0.68\) is smallest; the outer-fold contribution is MSE \(1.44\).)
- Two candidate procedures have estimated errors \(R+\epsilon_1\) and \(R+\epsilon_2\), with equal true risk \(R\) and mean-zero errors. Why does selecting the smaller estimate tend to understate risk? (Answer: The selected error is \(\min(\epsilon_1,\epsilon_2)=(\epsilon_1+\epsilon_2-|\epsilon_1-\epsilon_2|)/2\), whose expectation is \(-E[|\epsilon_1-\epsilon_2|]/2\leq0\).)