Model Selection Bias From Repeated Validation

Why Repeated Validation Creates Optimism

A validation score is useful for comparing candidate ratemaking models, but it is not a neutral referee if the same sample is used repeatedly to invent, adjust, and select specifications. Each score varies because the validation sample is finite. When many candidates are tried, the selected model is often the one that benefited most from favorable sample noise, even if its underlying predictive performance is no better.

This is model selection bias from repeated validation: the score reported for the winner tends to be more favorable than the winner’s expected score on new data, because the winner was chosen partly for having an unusually favorable score on the reused validation data. The issue is not that comparing candidates on the same validation sample is inherently wrong; it is that the same scores are then treated as an independent estimate of the performance of the selected model.

Definition: Let $L_j$ be the validation loss for candidate $j$, with lower values indicating better predictive performance. If a candidate is selected as $j^*=\arg\min_j L_j$, the selected validation loss $L_{j^*}$ is subject to selection bias when the candidates or their specifications were repeatedly adjusted in response to those same validation results. Its apparent performance can be better than its expected performance on new, comparable data.

The effect can arise from trying different variables, interactions, transformations, rating territories, distributional assumptions, or cut points, as well as from explicitly fitting many candidate models. Even if each individual comparison is sensible, the total search matters. The larger and more adaptive the search, the greater the opportunity to select a model that happens to fit the particular validation observations unusually well.

A Short Derivation of the Optimism

A simple calculation shows the mechanism. Suppose two candidates have exactly the same true expected loss, $\mu$, and their validation scores are $L_1=\mu+\varepsilon_1$ and $L_2=\mu+\varepsilon_2$. Assume the score errors are independent and each has a normal distribution with mean zero and variance $\sigma^2$. These assumptions describe score noise for this illustration; they are not a claim that every validation score is normally distributed.

The selected score is the smaller of the two. Using $\min(a,b)=(a+b-|a-b|)/2$, and using the zero means of the errors, gives:

E[\min(L_1,L_2)] =\mu-\frac{1}{2}E[|\varepsilon_1-\varepsilon_2|].

Independence and the assumed normal distributions imply $\varepsilon_1-\varepsilon_2\sim N(0,2\sigma^2)$. For a mean-zero normal variable with standard deviation $\tau$, the expected absolute value is $\tau\sqrt{2/\pi}$. Here $\tau=\sqrt{2}\sigma$, so:

E[|\varepsilon_1-\varepsilon_2|] =\sqrt{2}\sigma\sqrt{\frac{2}{\pi}} =\frac{2\sigma}{\sqrt{\pi}}, \qquad E[\min(L_1,L_2)] =\mu-\frac{\sigma}{\sqrt{\pi}}.

Thus, even though neither candidate is truly better, the expected winning validation score is below the common true loss. With more than two candidates, the minimum has still more opportunity to be unusually low. The exact amount depends on the number of candidates, dependence among their scores, and the search process; it is not generally valid to apply the two-candidate expression as a correction for a larger search.

Worked Example: Two Equally Good Candidates and the Expected Winner

A hypothetical portfolio team compares two models using the same validation procedure. Assume each model’s true expected loss is $100 per exposure. For this calculation only, each validation score equals the true loss plus an independent normal score error with mean 0 and standard deviation $4 per exposure. Find the expected score of the model selected for having the lower validation loss, and identify the optimism. No interest or mortality assumptions apply.

Method. Apply the two-candidate derivation. The score errors have variance $4^2=16$, so their difference has distribution $N(0,32)$.

The standard deviation of that difference is $\sqrt{32}=5.6569$. Its expected absolute value is $5.6569\sqrt{2/\pi}=4.5135$ dollars per exposure. Half of this amount is the expected downward effect of selecting the lower score:

\frac{1}{2}(4.5135)=2.2568.

Therefore the expected winning validation score is $100-2.2568=\$97.7432$ per exposure, rounded to four decimals. The expected optimism is $100-97.7432=\$2.2568$ per exposure. This is a result under the stated equal-performance, independent-normal-error assumptions, not a universal adjustment for an actual model search.

Check. The expected selected score is lower than the common true loss, although the models have identical true performance. The gap is exactly $\sigma/\sqrt{\pi}=4/\sqrt{\pi}=\$2.2568$. A common mistake is to interpret a low winning score as evidence that one candidate must truly be better; the calculation shows that selection alone can create a favorable gap.

Use a Final Test Sample Only After Selection

A straightforward safeguard is to divide suitable data into a development portion, a validation portion, and a final test portion. Fit candidates using development data, use the validation data for all candidate comparisons and specification decisions, and then lock the selection. Only after the selection and scoring rules are fixed should the test portion be used to assess the selected model.

The test sample must not influence the decision being evaluated. If the test result prompts another model change, a new variable, or a revised selection, the test data have become part of the search. The resulting score is no longer an untouched assessment; a further independent test set or an appropriate resampling design is needed for a fresh estimate.

Where possible, assess the selected candidate and a predeclared reference model on the same test observations, using the same exposures, loss definitions, maturity rules, and scoring method. The paired comparison helps control for differences in the observations being scored. The comparison remains an estimate with sampling uncertainty; an untouched test score is not a guarantee of future performance.

Worked Example: Interpreting a Locked Test Result

A hypothetical insurer evaluates four pure-premium specifications. On a validation sample, their losses are $116, $113, $115, and $117 per exposure, respectively. A predeclared incumbent model has validation loss $120. The team selects candidate B, with validation loss $113, and locks that choice before opening a separate test sample. On the test sample, candidate B has loss $119 and the incumbent has loss $121. Calculate the apparent validation improvement and the test-sample improvement. Assume the scores are measured on the same exposure basis. No interest or mortality assumptions apply.

Method. Lower loss is better. Calculate each improvement as incumbent loss minus candidate loss, then divide by the incumbent loss for the relative improvement.

On validation, the absolute improvement is $120-113=\$7$ per exposure. The relative improvement is $7/120=0.058333\ldots$, or 5.8333% when rounded to four decimal places as a percentage.

On the untouched test sample, the absolute improvement is $121-119=\$2$ per exposure. The relative improvement is $2/121=0.0165289\ldots$, or 1.6529% to four decimal places as a percentage.

The test result supports an estimated improvement of $2 per exposure on these test observations, not the $7 suggested by the validation comparison. The results do not establish that the difference will persist in every future period. The common mistake is to report the validation improvement as an unbiased estimate after selecting the lowest-scoring candidate; the independent test result is the more appropriate estimate for that locked selection.

Nested Resampling Evaluates the Selection Process

A final test sample can be difficult to reserve when data are limited. Nested resampling offers another approach. In each outer split, the outer training portion is used for an inner selection process: candidate specifications are compared using only that training portion, and one candidate is selected. The selected candidate is then scored on the outer holdout, which did not participate in the inner selection for that split. Repeating this process produces outer scores that assess the complete selection procedure, rather than just one fixed candidate.

Define the procedure.
Specify the candidate set, fitting rules, inner validation method, selection rule, and scoring measure before evaluating outer results.

Select within each outer training set.
Use inner validation data to choose a candidate. Do not use the corresponding outer holdout to revise that choice.

Score on the outer holdout.
Evaluate the selected candidate on the outer observations, using the pre-specified measure and consistent exposure and loss treatment.

Summarize the outer results.
Combine the outer scores according to the resampling design. Interpret the result as an assessment of the selection procedure, not necessarily of one identical model in every split.

The split design must reflect how predictions will be used. For a future-period application, random splits can leak time-related information; use an appropriate time-ordered or rolling-origin design instead. Similarly, exposures, loss maturity, and scoring periods must be aligned so that the outer holdout represents the intended prediction task. Nested resampling limits selection bias only if the outer results are not themselves repeatedly used to redesign the procedure and then reported as an untouched final assessment.

Worked Example: Outer Scores for a Selection Procedure

A hypothetical carrier compares three candidate frequency models using nested resampling with three outer splits. For each split, the inner validation losses select the candidate shown below. The selected candidate’s outer loss and a predeclared incumbent’s outer loss are then recorded. Calculate the mean outer loss for each approach and the relative improvement of the selection procedure. Treat the three outer scores as equally weighted for this exercise. No interest or mortality assumptions apply.

Outer split	Inner losses for A, B, C	Selected model	Selected model outer loss	Incumbent outer loss
1	104, 102, 103	B	106	108
2	99, 101, 98	C	100	102
3	105, 103, 104	B	104	105

Method. The inner minimum is selected within each split: B is selected in split 1 because $102<103<104$; C in split 2 because $98<99<101$; and B in split 3 because $103<104<105$. Then summarize the outer losses, which were not used for those selections.

The mean outer loss for the selection procedure is $(106+100+104)/3=310/3=103.3333$. The mean incumbent outer loss is $(108+102+105)/3=315/3=105.0000$. The absolute improvement is $105-103.3333=1.6667$ loss units. Relative to the incumbent mean, the improvement is $1.6667/105=0.015873$, or 1.5873% to four decimal places as a percentage.

The result estimates how this inner selection procedure performed across these outer splits. Because the selected model differs between splits, it does not directly report the outer performance of one fixed model. With only three splits, the result also should not be treated as a precise estimate. A common mistake is to report the inner minimum scores as the procedure’s performance; the outer scores are used here because they were held out from each split’s selection.

Practical Controls and Common Traps

Repeated validation is easy to miss because the search may happen informally. A modeler might change a territory grouping after reviewing a score, retain the change when it helps, and continue trying alternatives on the same data. Although each iteration may look like ordinary development, the final reported score reflects all the opportunities to adapt to the sample.

Count the search broadly. Include manual specification changes and repeated diagnostic-driven adjustments, not just models produced by a formal automated search.
Lock the procedure before final assessment. Record the candidate set, selection rule, scoring measure, and any tie-breaking rule before evaluating an untouched test set or outer holdouts.
Do not keep checking the test set. Repeatedly comparing new candidates on a test set turns it into another validation set.
Respect the prediction setting. In time-based ratemaking, preserve the time ordering and ensure that predictor, exposure, and loss information would have been available at the prediction date.
Do not assume nested resampling eliminates all uncertainty. The result depends on the splits, data volume, and similarity of the resampled observations to future business.

Key takeaway: The minimum of many noisy validation scores is typically too favorable because selection uses the same noise that is later reported. Use validation for development, then assess the locked selection with an untouched test sample or use nested resampling to evaluate the complete selection procedure.

Check Your Understanding

For each problem, use the stated score direction and assumptions. Answers are given so you can check the calculations.

A fixed incumbent has validation loss 52. Three candidates have losses 50, 47, and 49. The lowest-loss candidate is selected. What is its apparent absolute and relative validation improvement over the incumbent? (Answer: 5 loss units; $5/52=9.6154\%$.)
Two candidates have equal true loss of 60. Their validation scores have independent normal errors with mean 0 and standard deviation 3. What is the expected lower validation score? (Answer: $60-3/\sqrt{\pi}=\$58.3074$, to four decimals.)
In three outer splits, a nested selection procedure has outer losses 12, 15, and 13. A fixed incumbent has corresponding outer losses 14, 14, and 16. With equal split weights, find both mean losses and the procedure’s absolute mean improvement. (Answer: procedure mean $13.3333$, incumbent mean $14.6667$, improvement $1.3333$ loss units.)
A candidate was selected using validation data. On a previously untouched test sample, its loss is 88, while a predeclared incumbent’s loss is 91 on the same observations. What is the relative test-sample improvement? (Answer: $3/91=3.2967\%$.)
A team uses a test sample to select among several revised specifications, then reports the winner’s score on that same sample. Is this an untouched final assessment, and what is the appropriate correction to the process? (Answer: No; the test sample has become part of the selection process. Lock the specification using development and validation data, then assess it on new untouched data or use an outer resampling design not used to revise the procedure.)

What You'll Learn