Why a Winning Validation Score Can Mislead
A validation set is useful because it compares candidate models on observations not used to fit their coefficients or tree structure. But when analysts repeatedly inspect that same set while changing predictors, transformations, algorithms, tuning values, or classification thresholds, the validation results begin to influence the search. The winning result can then reflect both predictive signal and chance features of that particular validation sample.
This is adaptive overfitting: the search adapts to the validation data, even if the individual candidate models were fitted using only training data. A more complicated search creates more opportunities to find a candidate that happens to score well on validation. The concern is not limited to trying a large number of formally listed models. It also applies when a person informally revises a model after looking at its validation errors.
Here, “risk” means the expected value of a chosen loss function on future observations from the target population. For a classification error rate, lower risk means fewer misclassified observations. For a loss such as log loss or mean squared error, lower risk also indicates better performance. The metric must match the prediction task and decision being assessed; a favorable result on one metric does not automatically imply favorable performance on another.
Why Searching Creates Optimism
Suppose each candidate’s validation estimate is unbiased before selection: \(E[\widehat{R}_{j,V}]=R_j\). When the analyst reports the smallest estimated risk, the reported value is the minimum of several noisy estimates. For every candidate \(k\), the minimum is no greater than that candidate’s estimate. Taking expectations gives the key inequality:
Because this holds for every \(k\), the expected selected score is no greater than the smallest candidate risk:
This argument explains the direction of the bias: selecting the minimum favors negative estimation noise. It does not say that every selected validation score must be too low, or quantify the bias for a particular search. The amount depends on factors including the number and similarity of candidates, validation-set size, variability of the metric, and the search process. Candidates’ estimates are often correlated because the models use the same observations; correlation can change the magnitude of the effect, but does not make repeated selection harmless.
Worked Example: Selecting the Lowest Error From Many Candidates
A hypothetical insurer develops claim-risk classifiers using 100 validation observations. The analyst tries 20 combinations of predictors and tuning values. Assume, for illustration, that all 20 candidates have the same true classification error rate of 0.20. The lowest validation error is 12 errors out of 100; the next-lowest is 15 out of 100. An untouched test set contains 500 observations, on which the selected classifier makes 96 errors. Compare the scores and explain the selection effect.
Method. Calculate the selected validation error and the independent test error as observed proportions, then interpret the difference. The common mistake this avoids is treating the minimum score from a search as if it were an unbiased estimate.
Validation score. The selected candidate makes 12 errors among 100 observations: \(12/100=0.12\), or 12%. Its validation accuracy is \(1-0.12=0.88\), or 88%.
Test score. On the untouched test set, the selected candidate makes 96 errors among 500 observations: \(96/500=0.192\), or 19.2%. Its test accuracy is \(1-0.192=0.808\), or 80.8%.
Interpretation. The selected validation error is 7.2 percentage points lower than the test error: \(0.192-0.12=0.072\). The assumed true error of every candidate was 20%, so the selected validation result is unusually favorable in this hypothetical example. The test result, 19.2%, is much closer to that assumed value.
The calculation does not establish that the true error is exactly 19.2%; a finite test set also has sampling variability. It does show why choosing the minimum among 20 validation scores and reporting that same minimum is not an independent assessment. The test set provides a separate check because it did not determine which candidate won.
What Counts as Adaptive Search?
A candidate is not defined only by its algorithm name. Any choice influenced by validation performance is part of the search. For example, a search may include choices about:
- which predictors to include, and whether to create interactions or transformations;
- which model family to use, such as a tree, a forest, or a neighbor-based classifier;
- tuning values such as tree depth, number of neighbors, or a regularization parameter;
- the classification threshold, if the threshold is chosen to improve a validation metric; and
- which metric to emphasize, if that decision is made after comparing validation results.
Trying these options is not inherently wrong. It is a normal part of developing predictive models. The issue is whether the same data are then used to claim that the selected option will perform just as well on new observations. A validation score can guide development, but its role as a performance estimate becomes less credible as the analyst repeatedly adapts decisions to it.
A useful record of the search includes the candidates considered, the metric used, the selection rule, and any human decisions made after inspecting validation results. This makes clear what the eventual evaluation must assess: not just a model with a fixed label, but the process that selected it.
Worked Example: A Transformation Search and an Untouched Test Set
A hypothetical portfolio team predicts whether a policy will produce a claim. It compares four prespecified procedures using validation log loss, where lower is better. The procedures differ in predictor transformations and algorithm settings. After selection, the chosen procedure is evaluated once on a separate test set. The results are:
| Procedure | Validation log loss | Test log loss |
|---|---|---|
| A: linear predictors, fixed setting | 0.205 | 0.209 |
| B: transformed predictors, fixed setting | 0.196 | 0.214 |
| C: transformed predictors, tuned setting | 0.184 | 0.231 |
| D: alternative algorithm and setting | 0.191 | 0.225 |
Method. Select by the validation metric, then report the test metric for that selected procedure. The common mistake this avoids is choosing the test-set winner after inspecting all test scores.
Step 1: Select. Compare the validation values: \(0.205\), \(0.196\), \(0.184\), and \(0.191\). Their minimum is \(0.184\), so procedure C is selected.
Step 2: Assess. The selected procedure’s test log loss is \(0.231\). Procedure A has a test log loss of \(0.209\), but A is not substituted for C on the basis of this test result. The test set is being used to evaluate the selected procedure, not to conduct another selection.
Step 3: Interpret the gap. Procedure C’s test loss exceeds its validation loss by \(0.231-0.184=0.047\). This gap is consistent with the possibility that the validation search favored a procedure that fit peculiarities of the validation sample. It does not, by itself, prove that adaptive overfitting is the only cause; ordinary sampling variation can also contribute.
The final reported performance for the selected procedure is the test result, subject to test-sample uncertainty. Choosing A because its test score is smallest would use the test set for selection; a fresh independent evaluation would then be needed to make an unbiased assessment of that new selection.
Assessing the Whole Selection Procedure
An independent test set is one way to assess the selected procedure, provided it remains untouched during all model development. Another approach is nested validation. The outer folds assess the generalization performance of the full process, while the inner folds select predictors, transformations, algorithms, tuning values, and any other settings. The outer assessment observations must not influence those inner decisions.
The following sequence summarizes the separation of roles:
Keep the outer evaluation observations unavailable to every inner selection decision.
Use the inner procedure to compare the candidate models and choose the one that meets the stated selection rule.
Calculate the chosen performance metric on observations that did not influence the inner choice.
Combine the outer scores according to the metric and fold design, and describe the result as an estimate rather than a guarantee.
A nested estimate pertains to a procedure: how well the process of searching and fitting is expected to perform under the specified data-splitting design. It need not equal the performance of one final model refitted on all available development data. After evaluation, a final model may be selected and refitted using the development data, but its performance is not separately known unless it is assessed on independent data.
Worked Example: Computing an Outer-Fold Estimate
A hypothetical team has 300 development observations and uses three outer folds of 100 observations each. In each outer training set, an inner search selects among prespecified predictor and tuning configurations. The outer held-out sets contain 18, 21, and 20 classification errors, respectively. Calculate the pooled outer error rate and explain what it estimates.
Method. Pool the outer errors and divide by the total number of outer evaluation observations. The common mistake this avoids is reporting the best inner validation score as the nested estimate.
Step 1: Add the errors. The total number of outer errors is \(18+21+20=59\).
Step 2: Add the evaluation observations. There are \(100+100+100=300\) outer evaluation observations.
Step 3: Divide. The pooled outer error rate is \(59/300=0.196666\ldots\), which rounds to \(0.1967\), or 19.67%, to four decimal places as a proportion.
Interpretation and check. The estimate evaluates the process that selected a configuration within each outer training set and then predicted its corresponding held-out fold. It is not the inner search’s minimum validation error. The arithmetic check is \(300(0.196666\ldots)=59\) errors.
Common Traps and Practical Checks
- Calling repeatedly inspected data a test set. Once its results influence the choice of model or analysis, it has a validation role. A final evaluation then requires new untouched data or a properly nested design.
- Counting only formal tuning values. Trying different predictors, transformations, thresholds, or metrics after inspecting validation performance also adapts the search to that set.
- Reporting only the best score. The minimum hides the number and nature of the alternatives considered. State the selection process and use an independent assessment for performance claims.
- Confusing the selected model with the selection procedure. Nested validation assesses the procedure that chooses among candidates, not just a single fixed candidate chosen in advance.
- Claiming a test score is exact future performance. A test score is an estimate and is subject to sampling variation. It is more credible when the test observations represent the intended prediction population and remain independent of development decisions.
The central safeguard is a clear boundary between data used to choose and data used to assess. A single validation set may be sufficient for a modest, prespecified comparison, but repeated adaptive search weakens its role as an independent estimate. Nested validation or a genuinely untouched test set addresses that problem by reserving evaluation data from the decisions that created the winner.
Check Your Understanding
For each problem, distinguish selection evidence from an independent assessment.
- A team compares three fixed classifiers on 80 validation cases. They make 11, 8, and 10 errors. Which classifier wins, and what is its validation error rate? (Answer: The second classifier; \(8/80=0.10\), or 10%.)
- The team evaluates that selected classifier once on 250 untouched cases and observes 50 errors. What is its test error rate, and how many percentage points higher is it than the selected validation error in Question 1? (Answer: \(50/250=0.20\), or 20%; it is 10 percentage points higher.)
- An analyst tries 12 transformations against one validation set, chooses the transformation with the lowest validation log loss, and reports that same score as expected future performance. Identify the problem and one appropriate remedy. (Answer: Adaptive selection can make the minimum validation loss optimistic; use nested validation or an untouched test set to assess the selection procedure.)
- Four equally sized outer evaluation folds contain 22, 18, 24, and 16 errors, with 100 observations per fold. Calculate the pooled outer error rate. (Answer: \((22+18+24+16)/(4\times100)=80/400=0.20\), or 20%.)
- A model has test error 0.17. After seeing that result, an analyst changes the classification threshold to improve test accuracy and reports the new test accuracy. Is this still an independent test assessment? (Answer: No. The threshold was selected using test results, so the test set has become part of the adaptive selection process; a new untouched assessment is needed.)