Why an Apparent Gain Needs an Uncertainty Estimate
Two candidate models can receive different average cross-validation scores even when their underlying predictive performance is similar. The difference may reflect a real improvement, or it may arise because the particular validation folds happened to be easier or harder for one model. A useful comparison therefore asks not only which average score is better, but also how much the estimated difference varies across resamples and whether its size matters for the decision.
The comparison should be paired: evaluate both models on the same validation splits, with the same outcome definition and scoring rule. This makes each split a matched comparison and helps cancel variation caused by the split’s difficulty. The approach below is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
The sign convention matters. A positive \(\bar d\) means B reduces loss relative to A; it does not mean B has a higher loss. If the metric is a score for which higher is better, either define the difference in the reverse order or convert the score to a loss. State the convention before interpreting results.
Pair the Models Before Measuring Variation
Suppose one validation split contains unusually difficult cases. Both models may have higher loss on that split. If the models are scored on different splits, this variation becomes mixed with the model difference. With matched splits, the difference \(d_j\) removes some of that shared fluctuation.
The reason can be expressed through the variance identity for two random quantities:
When both models tend to do well or poorly on the same folds, their losses have positive covariance. Subtracting the losses then reduces the variation in the comparison. This is why calculating uncertainty from two unpaired sets of scores can be needlessly noisy. Pairing does not, however, make the validation folds independent.
Worked Example: Why Matched Folds Can Clarify a Gain
A hypothetical pricing team compares two models using four matched validation folds. The losses are:
| Fold | Model A loss | Model B loss | \(d_j=L_{A,j}-L_{B,j}\) |
|---|---|---|---|
| 1 | 2.0 | 1.6 | 0.4 |
| 2 | 2.4 | 1.9 | 0.5 |
| 3 | 1.8 | 1.5 | 0.3 |
| 4 | 2.2 | 1.8 | 0.4 |
Step 1: Calculate the paired mean difference. The sum of the differences is \(0.4+0.5+0.3+0.4=1.6\), so \(\bar d=1.6/4=0.4\). On average, B reduces loss by 0.4 units.
Step 2: Calculate the sample variance of the paired differences. The deviations from 0.4 are \(0, 0.1, -0.1, 0\). Their squared values sum to \(0+0.01+0.01+0=0.02\). Thus \(s_d^2=0.02/(4-1)=0.006667\), and \(s_d=\sqrt{0.006667}=0.08165\).
Step 3: Compare with an unpaired calculation. Model A has mean loss \((2.0+2.4+1.8+2.2)/4=2.1\). Its deviations are \(-0.1,0.3,-0.3,0.1\), giving squared deviations that sum to \(0.20\); therefore \(s_A^2=0.20/3=0.06667\). Model B has mean loss \((1.6+1.9+1.5+1.8)/4=1.7\). Its deviations are \(-0.1,0.2,-0.2,0.1\), giving squared deviations that sum to \(0.10\); therefore \(s_B^2=0.10/3=0.03333\).
If the four scores per model were incorrectly treated as independent, the estimated standard error of the difference would be \(\sqrt{s_A^2/4+s_B^2/4}=\sqrt{(0.06667+0.03333)/4}=\sqrt{0.025}=0.1581\). The paired standard error before any cross-validation correction is \(s_d/\sqrt{4}=0.08165/2=0.04082\). Pairing captures the fact that the two models face the same folds. This uncorrected paired standard error is still not a reliable final uncertainty estimate for cross-validation, because the training sets overlap.
Method and trap avoided. The method is to form one difference per shared fold before measuring variation. Treating the model scores as unrelated discards the matching information; treating the resulting folds as independent would create a different, potentially serious, error.
A Cautious Standard Error for Cross-Validation Differences
A familiar standard error for a mean of independent observations is \(s_d/\sqrt{m}\). Cross-validation differences are not independent observations: the fitted models use overlapping training data, and repeated folds may reuse the same observations. The naive standard error can therefore be too small, producing an interval that looks more decisive than the evidence warrants.
One commonly used approximation for repeated train/test resampling is the corrected resampled standard error. Let \(m\) be the number of paired resamples, \(s_d^2\) the sample variance of their differences, \(n_{\text{test}}\) the validation-set size, and \(n_{\text{train}}\) the training-set size for a resample. The approximation is:
For ordinary \(K\)-fold cross-validation with folds of approximately equal size, \(n_{\text{test}}/n_{\text{train}}\) is approximately \(1/(K-1)\). With \(R\) repeats, \(m=RK\), so the correction accounts for more than the \(1/m\) term alone. It is an approximate correction, not an exact variance calculation for every data-generating process or resampling design.
An approximate interval can be formed as
where \(t_{m-1,\,1-\alpha/2}\) is the corresponding quantile of a \(t\) distribution with \(m-1\) degrees of freedom. Treat this as a diagnostic approximation, especially when there are few resamples or when the differences are strongly skewed. It is not a substitute for an independent external test set when a final assessment is required.
Worked Example: Repeated Five-Fold Differences
For two hypothetical claim-severity models, the differences in validation mean squared error across ten paired resamples are \(0.5, 0.3, 0.4, 0.2, 0.6, 0.1, 0.5, 0.3, 0.4, 0.2\). Each resample uses five-fold cross-validation. Positive values favor candidate B. Assess the estimated gain using the corrected approximation.
Step 1: Calculate the mean. The sum is \(3.5\), so \(m=10\) and \(\bar d=3.5/10=0.35\).
Step 2: Calculate the sample variance. Deviations from 0.35 are \(0.15,-0.05,0.05,-0.15,0.25,-0.25,0.15,-0.05,0.05,-0.15\). Their squares sum to \(0.225\). Therefore \(s_d^2=0.225/(10-1)=0.025\).
Step 3: Apply the correction. For five folds, \(n_{\text{test}}/n_{\text{train}}\approx1/(5-1)=0.25\). Thus \(\widehat{\operatorname{SE}}_{\text{corr}}=\sqrt{(1/10+0.25)(0.025)}=\sqrt{0.35(0.025)}=\sqrt{0.00875}=0.09354\).
Step 4: Form the approximate 95% interval. Using \(t_{9,0.975}=2.262\), the margin is \(2.262(0.09354)=0.2116\). The interval is \(0.35\pm0.2116=[0.1384,0.5616]\), or \([0.138,0.562]\) to three decimal places.
All ten differences are positive, which is evidence that B’s lower loss is consistent across these resamples. Yet if the prespecified minimum worthwhile reduction is 0.20 loss units, this interval does not establish that the gain exceeds that threshold: its lower endpoint is 0.138. The data suggest a directional advantage, but its practical size remains uncertain.
Method and trap avoided. The method uses paired differences from repeated matched splits and a dependence correction. Dividing \(s_d\) by \(\sqrt{10}\) alone would give \(\sqrt{0.025/10}=0.05\), substantially smaller than 0.09354, and would ignore the overlap structure.
From Variation to a Decision
A comparison should distinguish three questions. First, what is the estimated direction of the difference? Second, how uncertain is that estimate under the chosen resampling procedure? Third, is a gain of that size useful enough to justify adopting the candidate model? A small positive difference can be stable but operationally trivial; a larger estimated difference can be too uncertain to support a firm conclusion.
Set a practical threshold before interpreting the results. For a loss difference where positive values favor B, a threshold \(\Delta_{\min}>0\) is the minimum gain considered worthwhile. An interval entirely above zero supports a positive difference under the approximation. An interval entirely above \(\Delta_{\min}\) gives stronger evidence that the gain is not merely positive but also large enough to matter. If the interval includes zero or the threshold, report the uncertainty rather than converting the larger mean score into an unqualified winner.
The pattern of differences is useful alongside the interval. A mean driven by one unusually large gain among mostly small or negative differences deserves scrutiny. Conversely, mostly positive differences do not by themselves prove that the mean gain exceeds a practical threshold. Inspect the differences and report both their average and their spread.
Worked Example: Is the Gain Large Enough to Matter?
A hypothetical insurer compares two models using four matched folds. The paired loss differences, defined as baseline loss minus candidate loss, are \(0.8, 0.2, 0.6, -0.2\). Each fold uses 20% of observations for validation and 80% for training. The smallest worthwhile reduction in loss was set in advance as 0.30 units. Calculate an approximate corrected interval and interpret it.
Step 1: Calculate the mean difference. The sum is \(0.8+0.2+0.6-0.2=1.4\), so \(\bar d=1.4/4=0.35\).
Step 2: Calculate the sample variance. Deviations from 0.35 are \(0.45,-0.15,0.25,-0.55\). Their squares are \(0.2025,0.0225,0.0625,0.3025\), which sum to \(0.5900\). Thus \(s_d^2=0.5900/3=0.196667\), and \(s_d=\sqrt{0.196667}=0.44347\).
Step 3: Apply the correction. Here \(m=4\) and \(n_{\text{test}}/n_{\text{train}}=20/80=0.25\). Therefore \(\widehat{\operatorname{SE}}_{\text{corr}}=\sqrt{(1/4+0.25)(0.196667)}=\sqrt{0.50(0.196667)}=\sqrt{0.0983335}=0.31358\).
Step 4: Form an approximate 95% interval. With \(t_{3,0.975}=3.182\), the margin is \(3.182(0.31358)=0.9980\). The interval is \(0.35\pm0.9980=[-0.648,1.348]\) to three decimal places.
The estimated mean gain of 0.35 exceeds the practical threshold of 0.30, but the interval is wide and includes zero. The evidence is not sufficiently precise to conclude that the candidate improves loss, let alone that it reliably exceeds the worthwhile-gain threshold. The negative difference on one fold also shows that the improvement was not consistent across all four folds. With only four resamples, this approximate interval should be treated cautiously.
Method and trap avoided. The method compares the estimated gain with both zero and the prespecified practical threshold, while retaining uncertainty. Choosing the candidate solely because \(0.35>0.30\) would mistake a noisy point estimate for a dependable improvement.
Conditions, Limits, and Common Traps
The calculations are most informative when the comparison is designed consistently: both models use the same splits, scoring definition, target observations, and leakage-safe preprocessing. The training procedure, including any tuning, must not use the validation outcomes that are being scored. If model choice has already been repeatedly guided by these same validation results, the apparent gain can be optimistic; a simple interval around the selected difference does not undo that selection.
The corrected standard error is a rough adjustment, not a universal guarantee. Its usefulness depends on the resampling design and on having a meaningful set of paired differences. Unequal fold sizes, severe dependence, a highly skewed difference distribution, or very few resamples can make a simple interval unreliable. Repeating cross-validation can show sensitivity to how the data were partitioned, but it does not create new independent observations from the population.
- Wrong calculation: use \(s_d/\sqrt{m}\) and call it the cross-validation standard error. Correction: overlapping training sets create dependence; consider an appropriate correction and describe its limitations.
- Wrong comparison: evaluate A and B on different random splits. Correction: match the splits so that each difference compares models under the same validation conditions.
- Wrong conclusion: interpret a positive mean difference as proof of a practically meaningful gain. Correction: compare uncertainty with zero and with a practical threshold chosen for the application.
- Wrong claim: say repeated folds are independent because the random seed or repetition changed. Correction: resampled training sets can overlap even when their partitions differ.
Check Your Understanding
For each problem, use positive paired differences to mean that the candidate model has lower loss than the baseline. Unless stated otherwise, use the corrected resampled standard error and an approximate \(t\) interval.
- On four matched folds, the differences are \(0.3, 0.5, 0.1, 0.3\). Calculate \(\bar d\) and \(s_d^2\). (Answer: \(\bar d=0.30\), \(s_d^2=0.0267\).)
- In five-fold cross-validation with one repetition, \(s_d^2=0.04\). Using \(m=5\), calculate the corrected standard error. (Answer: \(\sqrt{(1/5+1/4)(0.04)}=0.1342\).)
- Ten paired resamples give \(\bar d=0.25\), \(s_d^2=0.09\), and each uses five-fold cross-validation. Using \(t_{9,0.975}=2.262\), calculate the approximate corrected 95% interval. (Answer: SE \(=\sqrt{(0.10+0.25)(0.09)}=0.1775\); interval \(=[-0.152,0.652]\), to three decimals.)
- A candidate has a positive mean difference of 0.18, and its approximate interval is \([0.04,0.32]\). The prespecified minimum worthwhile gain is 0.10. What can be concluded about direction and practical size? (Answer: The interval supports a positive difference, but its lower endpoint is below 0.10, so a gain exceeding the practical threshold is not established.)
- Two models have fold losses that rise and fall together, but their paired differences are nearly constant. Which is the more relevant source of variation for estimating the comparative gain, and why? (Answer: The variation in paired differences; shared fold difficulty is partly removed by pairing, whereas separate model-score variation includes that shared fluctuation.)