Why the Lowest Cross-Validation Score Can Mislead
Cross-validation is often used to compare model configurations, such as different elastic net values of \(\alpha\) and \(\lambda\). Each configuration receives a score, and the configuration with the lowest score is selected. But a score is an estimate, not the configuration’s exact prediction risk. Even if every candidate were equally good, random variation could make one score unusually low. Selecting that score also selects some favorable noise.
This tutorial isolates that selection effect. It does not imply that every cross-validation estimate is unbiased before selection: fold design, training-set size, preprocessing, and the chosen loss can create other issues. The key point is narrower. When several noisy estimates are compared and the minimum is reported, the minimum tends to be below the underlying best risk.
The distinction between \(R_j\) and \(\widehat R_j\) matters. A low estimate can result from a genuinely low-risk candidate, favorable estimation noise, or both. Tuning deliberately searches for low estimates, so the winner’s error is not like the error of a configuration chosen independently of those estimates.
A Selection-Over-Noise Derivation
First consider \(M\) candidates with the same true risk \(R\), and suppose their cross-validation errors are independent and identically distributed as \(N(0,\sigma^2)\). These are simplifying assumptions for an exact calculation, not a claim that real cross-validation errors must be independent or normal. Then \(\widehat R_j=R+\varepsilon_j\), and the selected score is \(R+\min_j\varepsilon_j\). For two candidates, the expected minimum has a simple form.
For any real numbers \(a,b\), \(\min(a,b)=(a+b-|a-b|)/2\). Apply this identity to the two errors.
Because each error has mean zero, \(E[\min(\varepsilon_1,\varepsilon_2)]=-E[|\varepsilon_1-\varepsilon_2|]/2\).
Independence gives \(\varepsilon_1-\varepsilon_2\sim N(0,2\sigma^2)\). If \(Z\sim N(0,\tau^2)\), then \(E|Z|=\tau\sqrt{2/\pi}\). Therefore \(E|\varepsilon_1-\varepsilon_2|=2\sigma/\sqrt{\pi}\).
Substitution yields \(E[\min(\widehat R_1,\widehat R_2)]=R-\sigma/\sqrt{\pi}\). The amount below \(R\), \(\sigma/\sqrt{\pi}\), is the expected optimism in this two-candidate model.
The result has two implications. First, more variability in the individual estimates (larger \(\sigma\)) increases the expected downward shift. Second, the effect comes from choosing the minimum; before selection, each estimate has expectation \(R\). For more than two independent, equally variable normal errors, the expected minimum is also below zero. Its size increases with the number of candidates, although it does not generally grow in direct proportion to that number.
There is also a broad argument that does not require normal errors. If the vector of estimates has means \(R_1,\ldots,R_M\), then the minimum is a concave function of that vector. Jensen’s inequality gives \(E[\min_j\widehat R_j]\leq \min_j E[\widehat R_j]=\min_jR_j\), provided the estimates have finite expectations. This compares the expected selected score with the best candidate’s mean score. It does not say that every selected score is too low, nor does it by itself quantify the difference.
Worked Example: Two Equally Good Candidates With Normal Noise
Suppose two hypothetical pricing models have the same true mean squared prediction error of \(0.18\) (in squared response units). Each cross-validation estimate has an independent error distributed as \(N(0,0.03^2)\). Estimate the expected score after choosing the lower of the two estimates.
Method. Use the two-candidate normal selection formula. The true risk is \(R=0.18\), and the error standard deviation is \(\sigma=0.03\).
The expected downward shift is \(\sigma/\sqrt{\pi}=0.03/\sqrt{\pi}\). Since \(\sqrt{\pi}\approx1.77245\), \(0.03/1.77245\approx0.01693\).
The expected selected score is therefore \(0.18-0.01693=0.16307\), rounded to five decimal places. This is an expectation across repeated realizations of the cross-validation noise; it does not predict the score from one particular run.
Check. Each candidate’s estimate separately has expectation \(0.18\), but the selected score is expected to be lower. The calculation avoids treating the minimum of two unbiased estimates as itself unbiased.
Candidate Count and Correlated Estimates
The independent-error calculation is a useful benchmark, but candidate cross-validation scores often share data and folds. Their errors may therefore be correlated. A shared fluctuation can move all scores up or down without changing which candidate wins. What drives selection is especially the variation in the differences between candidates.
For two equally good candidates with jointly normal errors, suppose each error has variance \(\sigma^2\) and their correlation is \(\rho\). The difference has variance \(2\sigma^2(1-\rho)\). Applying the minimum identity above gives \(E[\min(\widehat R_1,\widehat R_2)]=R-\sigma\sqrt{(1-\rho)/\pi}\). This formula assumes equal true risks, equal error variances, joint normality, and \(-1\leq\rho\leq1\). Positive correlation reduces the difference noise and hence reduces this particular selection effect; it does not automatically remove it.
Worked Example: Matched Scores With Positive Correlation
Two hypothetical model configurations have equal true risk \(R=0.25\). Their cross-validation errors are jointly normal, each with standard deviation \(0.04\), and their correlation is \(0.75\). Find the expected lower score.
Method. Use the correlated two-candidate formula. Here \(1-\rho=1-0.75=0.25\), so the expected downward shift is \(0.04\sqrt{0.25/\pi}=0.02/\sqrt{\pi}\).
Since \(0.02/\sqrt{\pi}\approx0.011284\), the expected selected score is \(0.25-0.011284=0.238716\), rounded to six decimal places.
For comparison, if the errors were independent with the same standard deviation, the shift would be \(0.04/\sqrt{\pi}\approx0.022568\), twice as large. The shared variation associated with positive correlation does not distinguish the candidates as much. This is a property of the stated model; the correlation alone does not establish that the real cross-validation scores satisfy these assumptions.
Check. As \(\rho\) approaches \(1\), the difference between the two errors becomes less variable and the formula’s selection shift approaches zero. The example avoids interpreting correlated estimates as if they supplied two fully independent chances to obtain a low score.
Worked Example: Selection From Discrete Noise
Two hypothetical claim-severity models have equal true prediction loss \(R=0.10\). For each model, suppose the cross-validation error is independently \(+0.04\) or \(-0.04\), each with probability \(0.5\). Calculate the expected selected score by enumerating the outcomes.
Method. Compute both estimates for each possible error pair. The four equally likely pairs are:
- \((-0.04,-0.04)\): estimates \(0.06,0.06\); selected score \(0.06\).
- \((-0.04,+0.04)\): estimates \(0.06,0.14\); selected score \(0.06\).
- \((+0.04,-0.04)\): estimates \(0.14,0.06\); selected score \(0.06\).
- \((+0.04,+0.04)\): estimates \(0.14,0.14\); selected score \(0.14\).
Thus the expected selected score is \((0.06+0.06+0.06+0.14)/4=0.32/4=0.08\). It is \(0.02\) below the common true risk of \(0.10\). Each individual estimate still has expectation \(0.10\): the selection effect appears only after taking the minimum.
Check. The minimum is \(0.06\) in three of the four cases and \(0.14\) in one. This enumeration avoids assuming normality where none was specified.
Unequal Risks: A Candidate Can Win on Noise
In practice, candidate risks need not be equal. A configuration with slightly higher true risk can win if its estimate receives a sufficiently favorable error. The selected score is then influenced both by which configurations are competitive and by the noise that ranks them. Do not interpret a winning configuration as proof that it has the lowest true risk.
Worked Example: A Higher-Risk Candidate Wins Once
Consider two hypothetical configurations with true risks \(R_1=0.200\) and \(R_2=0.210\). For each configuration, the cross-validation error is independently \(+0.010\) or \(-0.010\), each with probability \(0.5\). Find the expected selected score and the expected true risk of the selected configuration.
Method. Enumerate the four equally likely error pairs:
- \((-0.010,-0.010)\): estimates \(0.190,0.200\); configuration 1 wins, with score \(0.190\) and true risk \(0.200\).
- \((-0.010,+0.010)\): estimates \(0.190,0.220\); configuration 1 wins, with score \(0.190\) and true risk \(0.200\).
- \((+0.010,-0.010)\): estimates \(0.210,0.200\); configuration 2 wins, with score \(0.200\) and true risk \(0.210\).
- \((+0.010,+0.010)\): estimates \(0.210,0.220\); configuration 1 wins, with score \(0.210\) and true risk \(0.200\).
The expected selected score is \((0.190+0.190+0.200+0.210)/4=0.1975\). The expected true risk of the selected configuration is \((0.200+0.200+0.210+0.200)/4=0.2025\). Their difference is \(-0.0050\). Configuration 2, despite having higher true risk, wins in one of four cases because its error is favorable enough relative to configuration 1’s.
Check. The calculation separates the average reported score from the average true risk of the selected candidate. It avoids claiming that the same candidate must win every time or that the selected score equals the selected candidate’s true risk.
Common Mistake and Practical Interpretation
A common exam trap is to say, “Each cross-validation estimate is unbiased, so the minimum is unbiased.” The conclusion does not follow: unbiasedness applies to each estimate before selection, not to a value chosen because it is smallest.
The effect is important when interpreting a tuned score as a performance estimate. Trying more candidate configurations creates more opportunities for favorable noise; more variable estimates make the ranking less reliable; and correlation changes how much of that variation affects the ranking. These are tendencies, not a universal numerical correction. The formulas above quantify selection optimism only under their stated assumptions.
When reporting a result, distinguish the score used to choose a configuration from an assessment designed to estimate the performance of the selection procedure. A low tuning score is useful for comparison, but it should not be presented as though selection had not taken place. This distinction is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Check Your Understanding
For each problem, focus on what is being selected and on the assumptions stated.
- Two equally good hypothetical models have true risk \(0.30\). Their independent normal cross-validation errors each have standard deviation \(0.02\). What is the expected lower score? (Answer: \(0.30-0.02/\sqrt{\pi}\approx0.2887\).)
- Two candidates have the same true risk \(0.20\). Each has independent error \(+0.03\) or \(-0.03\) with equal probability. What is the expected minimum score? (Answer: \(0.185\).)
- Two equal-risk candidates have joint normal cross-validation errors with standard deviation \(0.05\) and correlation \(0.36\). If their common true risk is \(0.20\), what is the expected minimum? (Answer: \(0.20-0.05\sqrt{0.64/\pi}\approx0.1774\).)
- Candidate 1 has true risk \(0.40\), candidate 2 has true risk \(0.42\), and their independent errors are each \(+0.03\) or \(-0.03\) with equal probability. Enumerating the four error pairs, what are the expected selected score and expected true risk of the selected candidate? (Answer: \(0.390\) and \(0.405\), respectively.)