Why Predictor Selection Belongs Inside Cross-Validation
Cross-validation is intended to estimate how well a predictive procedure will work on observations that were not used to build its predictions. In regression, the procedure may include more than fitting a model: it may first screen candidate predictors, keep some variables, and then fit the model using those variables. If the screening step uses the outcome values, it has learned from the data too. It must therefore be performed using only the training observations available in each fold.
This distinction matters when there are many plausible predictors. A variable can look useful in a particular dataset partly because of chance. If the same outcomes are used both to find apparently useful variables and to evaluate prediction error, the evaluation is no longer fully held out. Cross-validation can evaluate the complete selection-and-fitting procedure, but only if every data-driven step is repeated within each training fold.
The Fold-Wise Procedure
Suppose the observations have been divided into \(K\) folds. For fold \(k\), the training set contains all observations outside that fold, and the validation set contains the observations in the fold. The selection rule is part of the procedure being assessed. For example, it might retain predictors meeting a prespecified screening criterion, or it might select a fixed number of candidate predictors according to a prespecified ranking rule.
The rule itself should be specified before evaluating the folds. In each fold, apply that same rule to the training data, fit the regression model using the selected predictors and training observations, and predict the outcomes in the held-out fold. Once all folds have been processed, combine the held-out prediction losses using the chosen metric. The observations and outcomes in the held-out fold do not determine which variables are selected for that fold.
Set aside its predictor values and outcomes for evaluation.
Apply the prespecified screening or selection rule using only the observations outside the held-out fold.
Fit the model on that training set using the selected predictors, then predict outcomes in the held-out fold.
Repeat for every fold, then calculate the chosen performance measure from the held-out predictions and outcomes.
A useful way to describe what is being evaluated is as a whole procedure: given a training dataset, the procedure selects variables and fits a model. The selected set may differ from fold to fold. That is not a defect in cross-validation; it reflects that selection is uncertain and depends on which observations are available for training.
Worked Example: Keeping a Screening Rule Inside Each Fold
Suppose a hypothetical prediction task has candidate predictors \(X_1\), \(X_2\), and \(X_3\). Before examining the validation results, the analyst specifies this rule: within each training set, rank the candidates by the absolute sample correlation with the outcome and retain the two highest-ranked predictors. A regression model is then fitted using those two predictors.
In one fold, the training-set rankings are \(X_2\), \(X_1\), \(X_3\), so the model for that fold uses \(X_2\) and \(X_1\). In another fold, the rankings are \(X_3\), \(X_2\), \(X_1\), so that fold’s model uses \(X_3\) and \(X_2\). Each model is evaluated only on its own held-out observations.
This procedure is valid for estimating the predictive performance of the specified rule, because each validation outcome is excluded from its fold’s screening and fitting. It does not require the same two variables to be selected in every fold. The changing selections are part of the performance being assessed.
By contrast, ranking the predictors once using all outcomes and then carrying the resulting pair into every fold would allow each held-out outcome to affect which variables are used for its prediction. The fold labels would still exist, but the validation data would no longer be independent of the selection step.
Calculating a Cross-Validated Error After Selection
For a squared-error assessment, let \(y_i\) be the observed outcome and let \(\hat{y}_i^{(-k)}\) be the prediction for observation \(i\) from the model fitted for the fold that held out \(i\). The superscript indicates that this model was built without fold \(k\). The cross-validated mean squared error is the average of the squared held-out prediction errors:
Here \(F_k\) is the set of observation indices in fold \(k\), and \(n\) is the total number of observations. In this calculation, each prediction must come from the model whose variables were selected using that observation’s training set. If folds have unequal sizes, pooling the squared errors and dividing by \(n\) weights every observation equally; simply averaging the fold-specific means would instead give each fold equal weight.
Worked Example: Combining Held-Out Errors From Fold-Specific Models
In a hypothetical three-fold analysis, a screening rule selects \(X_1\) for the first fold’s model, \(X_2\) for the second fold’s model, and \(X_1\) and \(X_3\) for the third fold’s model. Suppose each fold contains two observations and the following are the held-out outcomes and predictions from those fold-specific models. The predictions are in outcome units.
| Fold | Held-out outcome \(y_i\) | Prediction \(\hat{y}_i^{(-k)}\) | Squared error |
|---|---|---|---|
| 1 | 10 | 9 | \((10-9)^2=1\) |
| 1 | 12 | 13 | \((12-13)^2=1\) |
| 2 | 8 | 6 | \((8-6)^2=4\) |
| 2 | 11 | 11 | \((11-11)^2=0\) |
| 3 | 9 | 10 | \((9-10)^2=1\) |
| 3 | 13 | 11 | \((13-11)^2=4\) |
The total squared error is \(1+1+4+0+1+4=11\), and there are \(n=6\) held-out predictions. Therefore,
The estimated mean squared prediction error is \(1.833\) squared outcome units. This calculation summarizes the predictions from the complete fold-wise procedure, including the variable selection performed separately in each training set.
Why Full-Data Selection Contaminates Validation
If the outcome is used to select variables on the full dataset, the held-out outcomes influence the selection rule’s result. A selected predictor can then partly reflect accidental features of those same held-out outcomes. Predictions for the held-out fold may look better than they would if the selection had been made without access to those outcomes. Thus the resulting error estimate does not cleanly represent prediction on data that played no role in building the procedure.
The reasoning behind this concern can be seen even before cross-validation. Suppose a researcher compares several candidate models on the same training observations and chooses the one with the smallest training error. The selected training error is the minimum of the candidate errors, so it cannot be larger than the training error of any one fixed candidate model. This selection advantage can make the chosen model’s training error look unusually favorable. A held-out fold avoids using its outcomes to make that choice only when the choice is repeated without those outcomes.
Here \(E_j\) is the error for candidate model \(j\) on the data used to compare candidates, and \(m\) is the number of candidates. This inequality is a direct consequence of taking a minimum. It does not say that every realized contaminated cross-validation estimate must be numerically lower than every valid estimate. It explains why reusing outcomes for selection can favor models that happen to fit those outcomes well, making an apparent performance estimate unreliable.
Worked Example: Comparing Full-Data Selection With Fold-Wise Selection
Consider a hypothetical four-observation validation exercise with several candidate predictors. An analyst first selects a predictor using all four outcomes, then reports predictions from that selected model as though the four observations had been held out. Suppose the observed outcomes and predictions from this contaminated approach are \(10, 12, 9, 11\) and \(9.5, 12.5, 8.5, 11.5\), respectively. The squared errors are \(0.25, 0.25, 0.25,\) and \(0.25\), giving
Now suppose the predictor is selected separately within the training set for each held-out observation, and the resulting predictions are \(8, 11, 10,\) and \(13\). The squared errors are \(4, 1, 1,\) and \(4\), so the fold-wise estimate is
The difference in these hypothetical results illustrates how strongly selection using the evaluation outcomes can distort an apparent score. It is not a rule that the contaminated estimate must always be smaller by this amount—or smaller in every dataset. The important point is that the first calculation cannot be interpreted as an independent held-out assessment: its predictor choice already used the outcomes it claims to predict.
What Must Be Repeated, and What Can Stay Fixed?
A predictor set chosen in advance for substantive reasons is not a data-driven selection step. If the analysis specifies the same predictors before examining the dataset and does not use the fold outcomes to revise that choice, there is no screening decision to repeat. In contrast, a rule that uses the observed data to decide which predictors to retain must be applied anew within each training fold.
This includes outcome-based screening, choosing predictors because they have strong sample associations with the outcome, or selecting variables after inspecting model results on the full dataset. The selection rule should also be distinguished from the selected variables: the rule may be fixed in advance, while its data-dependent output can vary across folds.
The practical consequence is that a cross-validation estimate describes the procedure actually evaluated. If deployment would begin with a training dataset, screen predictors using that dataset, and then fit a model, cross-validation should mimic those steps using each training fold. Evaluating a different workflow—such as selecting variables once with all available outcomes—answers a different and generally contaminated question.
Common Mistake
- Wrong reasoning: “I divided the data into folds, so my validation outcomes were held out. I can select the predictors once using the full dataset and still use the fold predictions for validation.”
- Correction: A fold’s outcomes are not fully held out if they helped decide which predictors its model uses. Repeat every data-driven selection step using only that fold’s training observations, then predict and score the held-out fold.
- Another pitfall: Treating different selected variables across folds as a failure. Different choices can be an honest indication that selection depends on the training sample; cross-validation assesses the procedure that produces those choices.
Check Your Understanding
Answer each question by identifying which data are allowed to influence selection and which observations are used for evaluation.
- A researcher ranks 30 candidate predictors by their correlation with the outcome using all observations, keeps the top five, and then runs five-fold cross-validation. What is the problem, and how should the workflow change?
- A predictor set was specified before any outcomes were examined and will be used unchanged in every fold. Is there a data-driven selection step that must be repeated? Explain.
- In a three-fold analysis, different predictors are retained in different folds. Does this alone make the cross-validation estimate invalid? Why or why not?
- Six held-out squared errors are \(1, 1, 4, 0, 1,\) and \(4\). Calculate the pooled cross-validated mean squared error.
- Why can choosing the candidate with the smallest error on the same data used for comparison produce an overly favorable assessment?