Why Repeated Selection Makes Validation Scores Optimistic
A validation score is an estimate of performance, not a guaranteed measure of it. It varies with which observations happen to be in the validation sample. If one score is used to choose between several models or settings, the chosen score tends to include a favorable random fluctuation: among many noisy estimates, the largest is more likely to be unusually high than unusually low. For a loss metric, the same effect favors an unusually low observed loss.
This is a form of selection optimism. It can arise when choosing among algorithms, predictor sets, transformations, hyperparameters, or other analysis choices. The choices may be made by an automated search or by an analyst who repeatedly inspects results and revises the approach. In either case, if the same validation results influence the choices and then serve as the reported evaluation, the reported performance does not account for that selection.
The size of the effect depends on how noisy the validation estimates are, how many alternatives are considered, and how similar or different their true performance is. It is not generally possible to correct a selected score by subtracting a fixed amount. Instead, the evaluation design should separate the data used to select a model from the data used to evaluate the selection procedure.
A Simple Derivation of the Selection Effect
Consider two candidate models whose true AUCs are both \(\theta\). Suppose their validation AUC estimates are \(Q_1=\theta+\varepsilon_1\) and \(Q_2=\theta+\varepsilon_2\), where the errors are independent and each follows a normal distribution with mean 0 and variance \(\sigma^2\). These assumptions are a simplified model of validation noise, not a universal description of every resampling setting.
The analyst selects the larger observed AUC. The selected value is \(\max(Q_1,Q_2)\), so its excess over the common true AUC is \(\max(\varepsilon_1,\varepsilon_2)\). Use the identity \(\max(a,b)=(a+b+|a-b|)/2\). Since both errors have mean 0, the expected excess is one half of \(E[|\varepsilon_1-\varepsilon_2|]\). Their difference is normally distributed with mean 0 and variance \(2\sigma^2\). For a mean-zero normal variable with standard deviation \(\tau\), the expected absolute value is \(\tau\sqrt{2/\pi}\). Therefore:
The result is positive even though neither model is better in truth. The act of choosing the higher noisy estimate creates the optimism. Considering more alternatives generally creates more opportunity to select a favorable fluctuation, although the exact expected optimism depends on the joint distribution of their errors.
Worked Example: Quantifying Optimism for Two Candidate Models
A hypothetical model comparison uses validation AUC to choose between two models. Assume their true AUCs are equal and the validation error for each estimate is independent and normally distributed with standard deviation 0.020. Estimate the expected optimism in the selected validation AUC. Use the normal-error assumptions stated above.
Method. Apply the two-model result derived above. The expected selected-score excess is \(\sigma/\sqrt{\pi}\). The common true AUC, \(\theta\), cancels from the excess, so its numerical value is not needed.
Substitute \(\sigma=0.020\) and \(\sqrt{\pi}\approx1.77245\):
Rounded to four decimal places, the expected optimism is 0.0113 AUC. If the common true AUC were 0.8000, the expected selected validation AUC would be approximately \(0.8000+0.0113=0.8113\). This is an expectation under the stated assumptions, not a correction guaranteed to apply to a particular observed score.
Check. The selected value is the larger of two estimates centered at 0.8000, so an expected value above 0.8000 is consistent with selection. A frequent mistake is to treat 0.020 as the expected optimism; it is the standard deviation of each estimate, not the expected maximum’s excess.
Nested Resampling Separates Tuning From Evaluation
Nested resampling uses two resampling loops with different jobs. The inner loop selects a model or setting using only the data available for tuning. The outer loop evaluates the result on observations that did not participate in that inner selection. Repeating the outer loop gives an estimate of how the complete selection procedure performs across new samples.
For example, in nested cross-validation, each outer split divides the available development data into an outer training portion and an outer assessment portion. Within the outer training portion, inner cross-validation compares candidate settings. The selected setting is fitted using the outer training portion, and its predictions are evaluated on the outer assessment portion. The outer assessment results are not used to choose settings for that outer split.
Set aside an outer assessment portion. Treat it as unavailable for all choices made during tuning.
Use inner resampling to compare the specified candidate models or settings. Select according to the metric and rule decided in advance.
Fit the selected model using the outer training portion, following the defined fitting procedure.
Calculate the chosen metric from predictions for observations excluded from the inner tuning and model fit.
Repeat the outer split and average or otherwise summarize the outer assessment results according to the evaluation plan.
Any operation that learns from data must respect the split boundaries. For example, if preprocessing estimates quantities from observations, those quantities must be fitted using only the training data available at that stage and then applied to the assessment data. Otherwise, the assessment portion has influenced the fitted pipeline. The same principle applies inside the inner loop: an inner assessment fold must not influence preprocessing fitted for that fold.
Worked Example: Calculating an Outer-Resampling Estimate
A hypothetical actuarial team compares three settings for a claim-frequency model. It uses four outer folds of equal size. In each outer training portion, inner cross-validation selects one setting; the corresponding best inner RMSE values are 8.8, 9.1, 8.6, and 9.0. After refitting within each outer training portion, the RMSE values on the respective outer assessment folds are 11.2, 10.8, 11.6, and 11.0. Calculate the mean outer RMSE and compare it with the mean selected inner RMSE. Assume all four outer folds are to receive equal weight.
Method. The inner-loop values describe selected tuning results. The outer-loop values estimate performance after tuning, so calculate each mean separately and interpret the outer mean as the resampling evaluation.
First, add the outer assessment RMSE values:
Divide by four equally weighted folds:
Now calculate the mean selected inner RMSE:
The inner mean is \(11.15-8.875=2.275\) RMSE units lower than the outer mean. This difference is consistent with the inner values being selected from alternatives and measured on data involved in selection, while each outer value is calculated on its excluded assessment fold. The defensible estimate for the specified nested procedure is the mean outer RMSE, 11.15, not the mean best inner RMSE.
Check. The arithmetic uses equal fold weights as specified. Averaging fold RMSE values is not always identical to calculating one pooled RMSE from all outer predictions; those are distinct summaries. A common mistake is to report 8.875 as the expected new-data error because it is the smallest available average. It is a tuning summary, not an independent evaluation.
What Nested Resampling Does—and Does Not—Estimate
An outer assessment score estimates performance for the whole procedure used in that outer split: the candidate set, inner tuning rule, fitting steps, and metric. It does not estimate the performance of a single permanently fixed model independently of how it was chosen. Different outer training portions may select different settings; that variation is part of the procedure being evaluated.
After the nested evaluation is complete, a team may use the development data to select and fit a final model. The nested outer results can support an estimate of the selection procedure’s performance, but they are not an untouched test of every later decision. If the team inspects outer results repeatedly and uses them to change the candidate set, metric, or modeling strategy, those results have started to influence selection too. A final performance claim after such changes needs a new, genuinely untouched evaluation set or a new evaluation design that accounts for the additional selection.
Nested resampling reduces the direct reuse that causes the original optimism; it does not make every possible analysis choice immune to selection. Its interpretation depends on the analysis plan and on keeping the outer assessment information out of model development. The resampling structure should match the intended prediction setting, including the way observations are divided. For example, a split should not place closely related records on both sides if that would not reflect the intended use of the model.
Worked Example: Catching Outer-Fold Reuse
A hypothetical team performs five-fold nested cross-validation. The inner loop selects settings separately in each outer training portion. The team then examines the five outer AUC values and chooses the algorithm with the highest mean outer AUC from among four algorithm families. It proposes to report that highest mean as an unbiased estimate for the chosen family. Identify the problem and specify a defensible correction.
Method. Trace which results were used for each decision. The inner loop selected settings, but the team also used the outer results to choose among the four algorithm families.
Problem. The outer results now serve two roles: they evaluate candidates and select the reported winner. The winning mean is the maximum of four noisy outer estimates. Even if each mean were unbiased for its own family before selection, the maximum is selected partly because it had a favorable fluctuation. Reporting that maximum as an unbiased estimate for the winner reuses the evaluation results.
Correction. If the four algorithm families are the intended candidate set, include family choice within the inner selection process for each outer training portion. Then retain the outer assessment results solely to evaluate that complete inner selection procedure. Alternatively, if the family choice must be made after comparing outer results, use an additional untouched evaluation set for the chosen family and its final procedure.
Check. The issue is not that four families were considered; it is that the outer assessment results were used to choose among them and then reported as if they were independent of that choice. Nested resampling works only when the outer loop remains outside all tuning and selection decisions being evaluated.
Common Mistakes and Practical Checks
- Reporting the winning validation score as final performance. That value helped select the winner and is typically optimistic. Use a separate assessment stage for evaluation.
- Calling a repeated look a new independent validation. Recalculating a score on the same validation observations does not create new evidence. Revising a model after seeing the score increases the number of choices influenced by those observations.
- Tuning on outer results. Outer folds are not a second tuning set. If their results guide choices, the reported outer performance is also subject to selection.
- Leaving learned preprocessing outside the resampling loop. Fit data-dependent preprocessing using only the relevant training portion, then apply it to its assessment portion.
- Interpreting nested estimates as a guarantee. They are estimates for a specified procedure and data setting. They do not guarantee the same performance in every future population or under a changed prediction process.
A useful audit question is: Which data influenced this decision? For every reported performance number, identify whether the observations used to calculate it also influenced the choice of model, setting, or analysis plan. This discipline is aligned with the SOA Exam PA (Predictive Analytics) syllabus because it links model comparison, validation design, and defensible communication of performance.
Check Your Understanding
For each problem, distinguish tuning information from evaluation information and check the arithmetic where requested.
- Two models have the same true AUC. Their independent validation errors are normally distributed with mean 0 and standard deviation 0.030. Using the two-model result, calculate the expected optimism from selecting the larger AUC. Give four decimal places. (Answer: \(0.030/\sqrt{\pi}=0.0169\) AUC)
- In a nested comparison, three equally weighted outer-fold RMSE values are 14.0, 12.5, and 13.5. The best inner RMSE values are 10.0, 9.5, and 10.5. Calculate both means and state which is the evaluation estimate. (Answer: Mean outer RMSE \(=13.333\ldots\), or 13.33 to two decimals; mean inner RMSE \(=10.00\); the outer mean is the evaluation estimate.)
- Within each outer training portion, an analyst selects a penalty setting by inner cross-validation, refits using that outer training portion, and scores predictions on the outer assessment portion. Should the outer assessment score be used to choose a different penalty setting for that same fold? Explain. (Answer: No. The penalty is selected in the inner loop; using the outer score to change it makes the outer assessment part of tuning.)
- A team tries 30 predictor sets on one validation sample and reports the lowest validation RMSE. What feature of this design creates optimism, and what resampling structure would address it? (Answer: Selecting the minimum among many noisy scores favors a low fluctuation; tune predictor sets within an inner loop and evaluate the selected procedure in an outer loop.)
- After nested resampling, a team uses the outer-fold AUCs to choose one of three algorithms and reports the winning algorithm’s same outer mean AUC as final performance. Is that evaluation still protected from selection optimism? (Answer: No. The outer results were used to select the algorithm; include algorithm choice in the inner loop or evaluate the final choice on untouched data.)