Why a Reused Test Set Can Look Too Good
A test score is useful because it estimates how well a fixed model predicts new observations. But if several models are evaluated on the same test set and the one with the lowest score is chosen, the test set has also helped make the decision. The winner is likely to be the model whose score benefited most from random variation in that particular set.
This is a selection effect, not evidence that any one score was deliberately manipulated. Each candidate’s score can be unbiased when considered on its own, while the lowest score among them is biased downward. The more candidates compared, the more opportunities there are for one to look unusually good by chance.
The qualification “fixed in advance” matters. For one prespecified candidate, random test-set variation can put its score above or below its true risk. After comparing candidates, however, the decision rule favors a low score. The selected score therefore tends to include a more negative-than-average test-score error.
A Derivation for Two Equally Good Candidates
The effect is especially clear when two candidates have the same true risk \(R\), and their test-score errors are independent and normally distributed, each with mean zero and variance \(\sigma^2\). Thus \(T_1=R+\varepsilon_1\), \(T_2=R+\varepsilon_2\), and \(\varepsilon_j\sim N(0,\sigma^2)\). Independence is an assumption for this calculation; scores from the same test set need not be independent in practice.
Use the identity \(\min(a,b)=(a+b-|a-b|)/2\). The difference \(\varepsilon_1-\varepsilon_2\) is normally distributed with mean zero and variance \(2\sigma^2\). For a mean-zero normal variable \(D\) with standard deviation \(\tau\), integrating its density gives \(E[|D|]=\tau\sqrt{2/\pi}\). Substituting \(\tau=\sqrt{2}\sigma\) yields the expected minimum:
The average of the two errors has expectation zero, but the absolute difference is nonnegative and is positive with probability one in this continuous model. Taking the minimum subtracts half of that difference. That is why the expected minimum falls below \(R\). The formula depends on the assumptions stated above; it is not a universal correction for any pair of test scores.
Worked Example: Two Equally Accurate Models with Normal Test Errors
A hypothetical pricing team compares two prediction models using test-set mean squared error (MSE). Suppose both models have true MSE \(100\), in squared-dollar units, and their test-score errors are independent \(N(0,100)\). Find the expected reported minimum MSE and the expected optimism relative to the selected model’s true MSE.
Method. Apply the derived two-candidate formula. Here \(R=100\) and \(\sigma=\sqrt{100}=10\), because the second parameter of \(N(0,100)\) is the variance.
Both candidates have true MSE \(100\), so whichever is selected has true MSE \(100\). The expected reported score is therefore \(100-94.3581=5.6419\) units below the selected model’s true MSE. To three decimal places, the expected minimum is \(94.358\), and the expected optimism is \(5.642\).
Check. Each fixed candidate’s expected test score is \(100\), as required by the zero-mean error assumption. The minimum’s expectation is lower, not equal to \(100\), because selection favors a negative error. A common mistake is to treat the selected score as though its model had been specified before seeing the test results.
Selection with Unequal True Risks
Candidates do not usually have identical true risks. The selected score can still be optimistic relative to the selected model’s true risk. It is useful to distinguish two comparisons: the lowest score versus the best candidate’s true risk, and the lowest score versus the true risk of whichever candidate the rule actually selects. These quantities need not be the same.
For a finite set of candidates with unbiased scores, the expected minimum score is no greater than the minimum of their expected scores. In symbols, \(E[\min_j T_j]\leq \min_j E[T_j]=\min_j R_j\). The selected model’s true risk is at least \(\min_j R_j\) for every selection outcome. Thus the expected minimum score is no greater than the expected true risk of the selected model. The gap can be zero in special cases, but choosing by the score creates a downward selection pressure.
Worked Example: Comparing Unequal Candidates with Discrete Errors
A hypothetical claims team compares two models. Candidate A has true MSE \(20\); candidate B has true MSE \(22\). Each test-score error is independently equally likely to be \(-4\) or \(+4\). Find the expected minimum test score and the expected true MSE of the selected model.
Step 1: List possible scores. Candidate A’s score is \(16\) or \(24\); candidate B’s score is \(18\) or \(26\). Independence gives four equally likely pairs:
- \((16,18)\): the minimum is \(16\), and A is selected.
- \((16,26)\): the minimum is \(16\), and A is selected.
- \((24,18)\): the minimum is \(18\), and B is selected.
- \((24,26)\): the minimum is \(24\), and A is selected.
Step 2: Average the selected scores.
Step 3: Average the selected models’ true risks. A is selected in three of the four cases, and B in one. Therefore:
The expected reported minimum is \(20.5-18.5=2\) MSE units below the expected true risk of the selected model. It is \(20-18.5=1.5\) units below the best candidate’s true risk, which is a different comparison. Check: each candidate’s score is unbiased on its own because its error averages to zero; the minimum is not unbiased after the choice is made.
More Comparisons Mean More Chances for a Lucky Score
When many candidates have similar performance, each additional comparison creates another chance that test-set noise will produce an unusually low score. The selected model may be genuinely better, may simply have had favorable test-set variation, or may reflect both. The observed minimum alone cannot separate these explanations.
The next example calculates this effect without assuming normally distributed errors. The error distribution is deliberately specified, so the calculation follows from counting possible outcomes rather than from applying a general correction formula.
Worked Example: Three Candidates with Discrete Test-Score Errors
A hypothetical underwriting group compares three models, each with true MSE \(40\). For each model, suppose its test-score error is independently equally likely to be \(-3\), \(0\), or \(+3\). Calculate the expected minimum test score and compare it with the expected true risk of the selected model.
Step 1: Find the minimum error’s probabilities. The minimum error is \(-3\) if at least one of the three errors is \(-3\). The probability is \(1-(2/3)^3=19/27\). The minimum is \(0\) if no error is \(-3\) and at least one is \(0\), with probability \((2/3)^3-(1/3)^3=7/27\). It is \(+3\) only if all three errors are \(+3\), with probability \((1/3)^3=1/27\).
Step 2: Compute the expected minimum error and score.
Step 3: Compare with the selected model’s true risk. Every candidate’s true risk is \(40\), regardless of which one is selected. Therefore \(E[R_J]=40\), and the expected minimum score is \(2\) units too low. Check: the minimum error cannot be positive unless all three errors are positive, while even one negative error makes it negative. The negative expected minimum is consistent with that imbalance.
How to Keep the Final Evaluation Informative
A test set ceases to be a clean final evaluation set when its results repeatedly influence which model, setting, or analysis is chosen. The resulting optimism comes from the selection rule, even if no model is refit on the test observations. Using the same observations to rank alternatives and then presenting the winning score as an untouched evaluation confuses selection with assessment.
A safer procedure is to make development choices using training data and, where needed, validation data, then reserve the test set for a final evaluation after those choices are fixed. If the test results are used to revise the analysis and choose again, that set has become part of the decision process; the revised result needs fresh independent evaluation to avoid reporting the selection-winner score as though it were independent. The target and loss measure should also remain consistent: selecting by one score and reporting another does not remove the fact that repeated test-based selection occurred.
Check Your Understanding
For each problem, make clear whether the requested quantity is a test score, a true risk, or their difference.
- Two equally accurate hypothetical models each have true MSE \(30\). Their independent test-score errors are \(N(0,9)\). Use the two-candidate formula to find the expected minimum test MSE and the optimism relative to the selected model’s true MSE. (Answer: expected minimum \(=30-3/\sqrt{\pi}\approx28.307\); optimism \(\approx1.693\) MSE units.)
- Candidate A has true risk \(10\), candidate B has true risk \(11\), and each has an independent test-score error equally likely to be \(-2\) or \(+2\). List the four score pairs and calculate the expected minimum score and expected true risk of the selected model. (Answer: expected minimum \(=9.25\); expected selected-model true risk \(=10.25\); optimism \(=1.00\).)
- Four candidates have the same true risk \(25\). Their independent test-score errors are each equally likely to be \(-1\) or \(+1\). Find the probability that the minimum error is \(-1\), then calculate the expected minimum test score. (Answer: probability \(=15/16\); expected minimum score \(=24.125\).)
- A team tests three fixed models on one set and records MSEs of \(14\), \(12\), and \(15\). It selects the second model. State the selected reported score and explain whether the three observed values alone determine the selection optimism. (Answer: selected reported MSE \(=12\); no, the observed scores alone do not specify the errors’ distribution or the selected model’s true risk.)
- Three equally accurate models each have true risk \(50\). Independent test-score errors are equally likely to be \(-2\), \(0\), or \(+2\). Calculate the probabilities of minimum errors \(-2\), \(0\), and \(+2\), then give the expected minimum score. (Answer: probabilities \(19/27\), \(7/27\), and \(1/27\); expected minimum score \(=50-4/3\approx48.667\).)