Why a Validation Score Can Become Too Good to Trust
A validation result is an estimate of how a model will perform on new data, not a guarantee of that performance. If one candidate model is evaluated once, random variation in the validation sample can make its score somewhat better or worse than its true generalization performance. If many candidates are evaluated and the one with the most favorable score is selected, the selection itself tends to favor a candidate whose validation noise was unusually favorable.
This effect is called selection optimism. For a loss measure such as RMSE or log loss, smaller values are better, so optimism means the selected validation loss tends to be smaller than the selected model’s actual generalization loss. For a performance measure such as accuracy, where larger values are better, the direction is reversed: the selected validation accuracy tends to be too high.
The key issue is not merely that the validation estimate has sampling error. Sampling error is expected in finite data. The additional issue is selection based on that error: among several noisy estimates, the most favorable one is more likely to include favorable noise. The more candidates or adaptive decisions are considered, the more opportunities there are to select noise along with genuine predictive signal.
A Simple Derivation of Selection Optimism
Consider two candidate models with the same true loss \(L\). Suppose each validation estimate is \(L+\varepsilon_j\), where the errors \(\varepsilon_1\) and \(\varepsilon_2\) are independent and each follows a normal distribution with mean 0 and variance \(\sigma^2\). The selected validation loss is the smaller of the two estimates. This is a simplified model of validation noise; real model estimates need not be normal or independent.
This identity follows from \(\min(a,b)=(a+b-|a-b|)/2\). Taking expectations, the average error term has expectation zero. The difference \(\varepsilon_1-\varepsilon_2\) is normal with mean 0 and variance \(2\sigma^2\). For a mean-zero normal variable with standard deviation \(\tau\), the expected absolute value is \(\tau\sqrt{2/\pi}\). Therefore:
So, under these assumptions, the selected validation loss is lower than the common true loss by \(\sigma/\sqrt{\pi}\) on average. This is a justification of the direction and scale of the effect, not a universal correction formula. With unequal true losses, dependent errors, or a different error distribution, the amount of optimism changes. In particular, validation estimates for related models are often correlated, so the independent-error calculation should not be treated as an exact adjustment for a real analysis.
Worked Example: The Best of Two Noisy Loss Estimates
A hypothetical model-development team compares two forecasting methods for a claim-cost outcome. Assume both methods have true generalization RMSE \(L=0.200\), and each validation RMSE estimate has independent normal error with standard deviation \(\sigma=0.040\). What is the expected validation RMSE of the method selected as better? State the expected optimism.
Method: apply the two-candidate derivation. The assumptions are equal true RMSE, independent mean-zero normal validation errors, and error standard deviation 0.040. The selected estimate is the smaller validation RMSE.
First calculate \(\sqrt{\pi}\approx1.77245\). The expected optimism in loss units is:
The expected selected validation RMSE is therefore:
Rounded to four decimal places, the expected selected validation RMSE is 0.1774, and the expected optimism is 0.0226 RMSE units. The selected method still has true RMSE 0.200 under the stated equal-performance assumption. The calculation checks because selection favors a lower estimate, while the assumed validation errors average zero before selection. Common mistake avoided: treating the selected estimate, 0.1774, as the true RMSE. It is an expectation under the model for validation noise, not a new estimate of either method’s true performance.
How Reuse Creates Optimism in Practice
In an applied analysis, repeated use can take several forms. An analyst might compare many candidate predictors, thresholds, transformations, or model settings on one validation set. They might inspect the errors, revise the features, and evaluate the revision on that same set. Even if every individual comparison is reasonable, the final choice has been influenced by the validation outcomes. The validation set has then contributed to model development, not just to evaluation.
The amount of optimism cannot generally be read from the number of comparisons alone. It also depends on how variable the estimates are, how different the candidates are in true performance, and how strongly their validation errors move together. If every candidate is nearly identical, repeated comparisons may mainly select noise. If one candidate is truly much better, its advantage can dominate the noise. Either way, the selected validation score by itself does not reveal how much of its apparent advantage is real.
Worked Example: Separating a Selected Score From an Independent Result
A hypothetical insurer compares three claim-frequency models using one validation sample. Their true generalization log losses are unknown. The validation log losses are 0.160 for Model A, 0.140 for Model B, and 0.150 for Model C. The analyst selects the model with the smallest validation loss. On a separate evaluation sample that was not used for model development or selection, the selected model has log loss 0.171. Calculate the selected validation score, the independent evaluation score, and their observed difference.
Method: select by the prespecified lower-is-better loss, then compare its validation and independent evaluation results. The validation choice is Model B because \(0.140<0.150<0.160\). Its selected validation loss is 0.140. The independent evaluation loss for Model B is 0.171.
The observed gap is:
Thus, the independent evaluation loss is 0.031 higher than the selected validation loss. This is consistent with selection optimism, but it does not prove that the whole gap is caused by selection: both samples have finite-sample variation, and the data may differ in other relevant ways. Nor does this result establish that Model B is worse than Models A or C on the population; only Model B was evaluated independently in the information given. Common mistake avoided: reporting the minimum, 0.140, as an unbiased estimate of performance merely because it was calculated on a validation set. The set was used to choose the model, so that minimum has a selection effect.
The arithmetic checks directly: the selected model is B, its two reported losses are 0.140 and 0.171, and their difference is 0.031. Loss values are kept to three decimal places as provided.
Worked Example: Repeated Adaptation and an Outer Evaluation
A hypothetical analyst develops a claim-severity model through three rounds of changes. At each round, the analyst uses the same inner cross-validation results to decide which version to keep. The best inner cross-validation RMSE for the retained version is 12, 11, and 13 in three outer resampling runs. In each run, the entire selection process is repeated using only that run’s outer-training data, and the selected procedure is then scored on that run’s outer assessment data. The outer RMSE values are 14, 13, and 15. Compute the mean inner selected score and mean outer score, and explain which is more appropriate for assessing the selection process.
Method: average the scores at each level and distinguish selection data from assessment data. The mean inner selected RMSE is:
The mean outer RMSE is:
The mean inner selected score is 12, while the mean outer score is 14, a difference of \(14-12=2\) RMSE units. The inner scores were used to choose versions, so they are vulnerable to selection optimism. The outer scores assess the repeated process of choosing and fitting a model using the outer-training data; they are the more appropriate results here for estimating that process’s performance. The difference of 2 is an observed difference between these averages, not a general correction to subtract from future inner scores. Common mistake avoided: treating the observed gap as a guaranteed amount of bias. It can also reflect ordinary sample variation and differences across resampling runs. The calculations check: the inner scores sum to 36, the outer scores sum to 42, and each average uses three runs.
When an Independent Evaluation Is Needed
An independent evaluation is especially important when the reported result is meant to describe the performance of a selected model or a model-development procedure, and the same validation results have influenced that selection. Independence here means that the evaluation outcomes were not used to choose, revise, or otherwise adapt the modeling decisions being assessed. It does not mean the data are magically free of sampling error; it means the evaluation is not additionally selected for being favorable.
One approach is to reserve an evaluation sample that remains unused during model development. Another is nested cross-validation: within each outer training sample, use an inner resampling process to make the modeling choices, then evaluate the resulting selected procedure on the corresponding outer assessment sample. The outer results estimate the performance of that development procedure, rather than simply restating the best inner score. Which design is appropriate depends on the data and the intended use of the result.
The split must also respect the structure of the prediction problem. For future observations in a time series, an independent evaluation should occur later in time than the training information. If several records belong to the same person, group, or entity, those records should not be divided in a way that makes the evaluation artificially easy. These design conditions do not remove selection optimism by themselves, but a poorly chosen evaluation design can introduce other forms of misleading performance.
List the models, settings, features, thresholds, or revisions whose results influenced the final choice.
If it did, treat the selected score as a development result, not as an independent assessment.
Use untouched evaluation data or an appropriate outer resampling design, while respecting time order and entity structure.
State which results supported model selection and which results assessed the selected model or procedure.
Common Exam Trap
A common error is to compare several candidates, report the smallest validation loss, and call it an unbiased estimate because the models were evaluated on observations not used to fit their coefficients. That argument overlooks the selection step. A validation observation can be outside the fitting sample and still influence the final model choice. If it does, the selected validation score is no longer independent of the selection process.
For example, suppose a team evaluates 30 candidate models on the same holdout data and reports the lowest loss. The incorrect conclusion is: “The holdout was not used to fit the model, so this score is an unbiased final estimate.” The correction is: “The holdout was used to select among 30 models, so the winning score is subject to selection optimism. An independent evaluation is needed to assess the selected result.” The number 30 alone does not determine the size of the optimism, but it signals multiple opportunities to select a favorable noise realization.
The practical rule is simple: a score can be used for model development or for a clean final assessment, but repeatedly using it for both roles weakens its value as an independent assessment. This distinction is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, distinguish the score used for selection from the result that can support an independent performance claim.
- Two equally good models each have true RMSE 0.300. Their validation errors are independent normal variables with standard deviation 0.030. Using the two-model result, calculate the expected selected validation RMSE to four decimal places. (Answer: expected optimism \(=0.030/\sqrt{\pi}\approx0.0169\); expected selected RMSE \(\approx0.2831\).)
- A team evaluates four models on one validation set and selects the one with the lowest log loss, 0.092. It then reports a loss of 0.108 for that selected model on untouched evaluation data. What is the observed gap, and which score is more appropriate for assessing the selected model? (Answer: gap \(=0.108-0.092=0.016\); the independent evaluation loss, 0.108, is more appropriate.)
- An analyst selects the largest AUC after trying several predictor sets on one validation sample. Is the selected AUC more likely to be too high or too low as an estimate of generalization AUC? Give the reason. (Answer: too high on average, because selection favors predictor sets with unusually favorable validation noise.)
- In three outer resampling runs, the inner selected RMSE values are 8, 10, and 9; the corresponding outer assessment RMSE values are 11, 12, and 10. Calculate both means and their observed difference. (Answer: mean inner RMSE \(=9\); mean outer RMSE \(=11\); outer minus inner \(=2\).)
- A model is revised repeatedly after analysts inspect the same holdout errors, and the final score is reported from that holdout. What is the main concern, and name one design that could provide a less selection-affected assessment. (Answer: the holdout has influenced model development, so its final score is subject to selection optimism; use untouched evaluation data or an appropriate nested cross-validation design.)