Why Feature Selection and Tuning Need Boundaries
A model-development process often involves more than fitting one fixed model. An analyst may choose which predictors to retain, compare several subsets, or tune a penalty parameter. Those choices affect the fitted model, so the data used to make them are part of the model-building process. The key question is not only which observations were used to estimate coefficients, but also which observations influenced the choice of the model.
A validation set may guide feature selection or tuning. That is its purpose. However, once its outcomes have guided a choice, its score is no longer a clean final estimate of performance: among the candidates considered, the chosen one was favored partly because it happened to score well on those particular observations. A test set reserved until all such choices are complete can provide a separate final evaluation.
This is a distinct issue from fitting a transformation or estimating a preprocessing quantity before splitting data. Here, the leakage occurs through a decision informed by outcomes: for example, ranking predictors by their association with the response on all observations, then claiming that a held-out subset gives an independent evaluation.
Which Data May Guide Which Decision?
The training set is used to fit candidate models. The validation set may compare those candidates and guide choices such as a predictor subset or tuning parameter. After the choices are settled, the selected procedure is fitted using the allowed development data, and the test set is used for final evaluation. The exact fitting sequence depends on the development design, but the separation of roles does not: outcomes reserved for evaluation must not help select the model being evaluated.
| Decision or task | Data that may guide it | What must remain separate |
|---|---|---|
| Estimate coefficients for a candidate model | Training data | Outcomes reserved for evaluation |
| Compare predictor subsets or tuning values | Validation data, or an internal resampling procedure within development data | The final test outcomes |
| Estimate final predictive performance | Test data not used in prior choices | All model-selection decisions must already be complete |
| Estimate performance by cross-validation | Fold-based evaluation of the complete model-building procedure | Each assessment fold must not guide its own model selection |
A useful way to define the object being evaluated is the complete modeling procedure. If the intended procedure searches over predictor subsets and chooses one using validation data, then an evaluation should assess that search-and-fit procedure, not merely the final coefficient-fitting step viewed in isolation.
Worked Example: Selecting a Predictor Subset on Validation Data
A hypothetical insurer is developing a model for a claim-severity measure. Three candidate predictor subsets were specified in advance. Each model was fitted using the training data, and the table gives its residuals on four validation observations, defined as observed outcome minus predicted outcome. Which subset should be selected by validation mean squared error (MSE), and what does that score tell us?
| Candidate subset | Validation residuals |
|---|---|
| A: exposure and territory | 2, −2, 1, −1 |
| B: exposure and policy class | 1, −1, 2, −2 |
| C: exposure, territory, and policy class | 0, 1, −1, 2 |
Method. Compute each candidate's validation MSE by squaring its four residuals, adding them, and dividing by four.
For subset A, the squared residuals are \(2^2=4\), \((-2)^2=4\), \(1^2=1\), and \((-1)^2=1\). Thus its sum is \(4+4+1+1=10\), and its validation MSE is \(10/4=2.5\).
For subset B, the squared residuals are \(1,1,4,4\). Their sum is \(10\), so its validation MSE is \(10/4=2.5\).
For subset C, the squared residuals are \(0,1,1,4\). Their sum is \(6\), so its validation MSE is \(6/4=1.5\). The validation rule therefore selects subset C, with a validation MSE of \(1.5\).
Suppose the selected subset C then has residuals \(1,-2,2,-2\) on four separate test observations. Its test squared errors are \(1,4,4,4\), with sum \(13\), so its test MSE is \(13/4=3.25\). The test calculation is \(3.25\), not \(1.5\). The difference does not prove that the validation procedure is always optimistic by this amount; it shows that the selected validation score and the separate test score are different quantities.
Common mistake and correction. It would be wrong to report \(1.5\) as an unbiased final performance estimate simply because subset C had the lowest validation MSE. That value helped select C. Use the validation score to make the selection, then use untouched test observations for final evaluation. The test residuals in this example are supplied for the selected model only.
Why the Best Validation Score Tends to Look Too Good
Even if every candidate has the same true prediction error, their observed validation errors can differ because of sampling variation. Choosing the smallest observed error favors a candidate with a favorable validation fluctuation. The minimum is therefore not, in general, an unbiased estimate of the selected candidate's true error.
This inequality compares the expected selected score with the smallest candidate expected score. It does not say that every particular validation minimum must be below every candidate's true error, nor does it quantify the optimism in a real application. The size of the effect depends on factors including the number of candidates, how variable their validation scores are, and how similar those scores are.
Worked Example: A Simplified Selection-Optimism Calculation
Consider three candidate models whose true prediction MSE is \(9.0\) for each candidate. For this simplified illustration, suppose each candidate's validation MSE equals \(9.0+E_j\), where the three errors \(E_1,E_2,E_3\) are independent and each is \(-0.6\) or \(+0.6\), with probability \(1/2\) for each value. Find the expected minimum validation MSE.
Method. The minimum is \(8.4\) whenever at least one candidate has error \(-0.6\), because \(9.0-0.6=8.4\). It is \(9.6\) only when all three errors are \(+0.6\), because \(9.0+0.6=9.6\).
Independence gives \(P(E_1=+0.6,E_2=+0.6,E_3=+0.6)=(1/2)^3=1/8\). Therefore the probability that the minimum is \(8.4\) is \(1-1/8=7/8\). The expected minimum is
The true MSE of every candidate is \(9.0\), while the expected selected validation minimum is \(8.55\). In this deliberately simplified model, the expected downward difference is \(9.0-8.55=0.45\). The calculation demonstrates the selection mechanism; the two-point error model is not a general formula for the magnitude of optimism in actual validation data.
This example's assumptions define the possible scores exactly: each individual candidate score can only be \(8.4\) or \(9.6\). Any other observed score would be inconsistent with this particular model. The calculation above is a theoretical expectation under those assumptions, not an observed validation table.
Tuning Parameters Are Selection Decisions Too
A tuning parameter controls some aspect of model complexity or fitting. Examples include a penalty strength, a tree-depth limit, or a choice among a finite set of candidate settings. Regardless of the parameter's meaning, if its value is chosen because it gives the lowest validation loss, that validation outcome has influenced the model. A larger search creates more opportunities to find a setting that happens to fit the validation sample well.
Worked Example: Choosing a Penalty Setting Without Spending the Test Set
A hypothetical risk team fits the same predictor set using four penalty settings, \(\lambda=0,1,4,16\), with coefficient estimation performed on the training data. Their observed validation MSEs are \(12.0,10.8,11.3,11.1\), respectively. A separate test set has not been examined. Select a setting and state the next step.
Method. Compare the four validation scores. The smallest is \(10.8\), corresponding to \(\lambda=1\), since \(10.8<11.1<11.3<12.0\). The validation-guided choice is therefore \(\lambda=1\).
The next step is to lock the choice and fit the selected modeling procedure using the development data allowed by the analysis plan. Then evaluate it on the untouched test set. The value \(10.8\) is the selected validation MSE, not the final test estimate.
Common mistake and correction. A tempting but incorrect report is: “The final expected MSE is \(10.8\), because it is the smallest score.” The calculation only establishes that \(10.8\) is the minimum among these four observed validation scores. It does not remove the selection effect or turn the validation set into independent test data. Keep the decision and the final evaluation distinct.
Selection Inside Cross-Validation
Cross-validation can estimate how a model-building procedure performs on data not used for fitting that particular model. But it is important to apply the procedure inside the resampling design. If predictor selection uses outcomes from all observations before cross-validation begins, then a fold's outcomes can influence which predictors are used to predict that same fold. The fold is no longer independent of the selection step.
For an outer-fold assessment, divide the development data into outer training and outer assessment portions. Any feature selection and tuning needed for the model should be carried out using only the outer training portion, often with an inner validation or resampling step. Fit the resulting model on the outer training portion and calculate its loss on the outer assessment portion. Repeat for the outer folds. The resulting estimate pertains to the whole selection-and-fitting procedure.
Worked Example: Calculating an Outer-Fold Error After Selection
A hypothetical actuary evaluates a procedure that selects predictors within each outer training portion and then fits a regression model. There are three outer assessment folds, each containing four observations. The residuals on the assessment observations are as follows: fold 1 has \(2,-2,0,0\); fold 2 has \(1,-1,2,-2\); and fold 3 has \(3,-1,1,-1\). Calculate the mean of the three fold MSEs. Assume the predictor selection for each fold used only that fold's outer training portion.
Method. Square and average the residuals separately in each fold. For fold 1, the squared residuals are \(4,4,0,0\), giving sum \(8\) and MSE \(8/4=2\).
For fold 2, the squared residuals are \(1,1,4,4\), giving sum \(10\) and MSE \(10/4=2.5\). For fold 3, they are \(9,1,1,1\), giving sum \(12\) and MSE \(12/4=3\).
The mean fold MSE is
Because all folds have four observations, this is also the pooled MSE: the total squared error is \(8+10+12=30\), and \(30/12=2.5\). The result assesses the selection-and-fitting process under the stated fold design, not just a fixed predictor subset.
If instead the analyst had screened predictors once using outcomes from all 12 observations and then used the resulting predictors in every outer fold, the assessment outcomes would already have influenced the predictor choice. The displayed fold residuals would then not represent an evaluation insulated from selection. The correction is to repeat the selection step using only each outer training portion.
Practical Checks Before Reporting a Score
Before interpreting a validation or resampling result, trace every choice back to the data that informed it. Ask whether response values were used to rank predictors, discard features, choose transformations, or tune a model before the evaluation split was applied. If so, the split does not protect the evaluation from that decision.
- List the choices. Include predictor screening, subset selection, tuning values, and any data-dependent model comparisons.
- Identify the information used. For each choice, record which outcomes and observations were available at that point.
- Keep evaluation outcomes out of choices. A validation set can guide development; a test set reserved for final evaluation should not.
- Evaluate the full procedure. In resampling, repeat selection and tuning within the training portion for each assessment fold.
- Report the role of each score. State whether a number is a training, validation, resampling, or test result, and whether it guided selection.
The central distinction is straightforward: using validation results to choose a model is a legitimate development step, but the selected validation score should not be presented as an independent final performance estimate. The model choice and the performance estimate answer different questions, and they require appropriately separated information.
Check Your Understanding
For each problem, identify the valid selection or evaluation step and check the stated calculation.
- A team compares three fixed predictor subsets. Their validation MSEs are \(7.2, 6.8,\) and \(7.0\). Which subset is selected, and should \(6.8\) automatically be reported as an independent final test estimate? (Answer: Select subset 2; no, \(6.8\) is the score that guided selection, not an independent final test estimate.)
- Two candidate models have validation residuals \((1,-2,1)\) and \((0,2,-2)\). Compute each MSE and identify the validation choice. (Answer: The first MSE is \((1+4+1)/3=2\); the second is \((0+4+4)/3=8/3\); select the first.)
- A predictor is ranked by its association with the response using all observations, after which the data are divided into training and test sets. Is the test evaluation insulated from the predictor-selection step? State the correction. (Answer: No; test outcomes influenced selection. Split first and perform response-based screening using training data only, with any further choices confined to development data.)
- In four-fold outer assessment, each fold has the same size and the fold MSEs are \(3,5,4,8\). Find the average fold MSE and describe the condition needed for it to assess a selection procedure. (Answer: \((3+5+4+8)/4=5\); feature selection and tuning must be repeated using only the corresponding outer training portion.)
- Three candidates have identical true MSE \(10\). In a simplified model, each candidate's validation MSE is \(10+E_j\), where independent \(E_j\) values equal \(-1\) or \(+1\) with equal probability. What is the expected minimum validation MSE? (Answer: The minimum is \(9\) unless all three errors are \(+1\), an event of probability \(1/8\); thus \(E[\min]=\frac{7}{8}(9)+\frac{1}{8}(11)=9.25\).)