Why Compare Models on Matched Folds?
Two elastic net configurations can have similar cross-validation losses even when one is consistently a little better. The comparison is especially difficult when validation folds differ in difficulty: a fold with unusual observations may produce large losses for both models, while an easier fold may produce small losses for both. Comparing candidate models on the same folds controls for much of this fold-to-fold variation.
The basic method is to calculate one loss difference for each matched fold, then summarize those differences. A positive difference can be defined to mean that candidate A has greater loss than candidate B, and therefore performs worse on that fold. This tutorial develops that comparison, including its uncertainty summary and the limits of interpreting it.
The loss could be mean squared error, mean absolute error, or another measure selected for the modeling objective. The essential requirement for a direct paired comparison is that both candidates are scored on the same validation observations in each fold and that the same loss definition and scale are used for both. The candidate configurations might differ in the elastic net mixing parameter, the penalty parameter, or both.
The folds must be matched in their actual assignment, not merely in their number. If one candidate is evaluated on one random partition and the other candidate on a different partition, subtracting the fold-one scores does not create a meaningful pair. The fold labels then refer to different validation samples.
Paired Differences and Their Uncertainty
The mean paired difference is the average of the fold-level differences. Its sample standard deviation describes how much the observed differences vary across folds. A commonly used estimated standard error of the mean difference is the sample standard deviation divided by the square root of the number of folds.
A positive \(\bar{D}\) says that A had higher average loss than B across the matched folds; a negative \(\bar{D}\) says the reverse. The standard error summarizes variation in the paired differences, not variation in either candidate's losses considered alone.
Why Matching Can Reduce Noise
The advantage of pairing follows from the variance of a difference. For random fold-level losses \(L_A\) and \(L_B\), the identity \(\operatorname{Var}(L_A-L_B)=\operatorname{Var}(L_A)+\operatorname{Var}(L_B)-2\operatorname{Cov}(L_A,L_B)\) holds. When both models are scored on the same fold, the fold's difficulty often affects both losses in the same direction. This can make their covariance positive, reducing the variance of the difference.
For each fold, both configurations are scored on the same held-out cases. This makes a fold-level difference interpretable as a within-fold comparison.
Compute \(D_k=L_{Ak}-L_{Bk}\). Shared effects of an easy or difficult fold can cancel in this subtraction.
Calculate \(\bar{D}\) and \(s_D\) from the \(K\) differences, then use \(s_D/\sqrt{K}\) as the conventional estimated standard error.
This argument is a reason to prefer matched comparisons, not a guarantee that their variability will always be smaller. If the fold-level losses have negative covariance, the covariance term increases the variance of the difference. Matching also does not make the fold differences fully independent: training sets in ordinary \(K\)-fold cross-validation overlap. Consequently, the standard error above is a useful fold-level summary, but it should not automatically be treated as an exact measure of uncertainty for every cross-validation design.
Worked Example: Comparing Two Configurations on Five Matched Folds
A hypothetical pricing analysis compares elastic net configurations A and B using the same five folds and mean squared prediction error on each fold. The fold losses, in a common squared-loss scale, are:
| Fold | Loss for A | Loss for B | \(D_k=L_{Ak}-L_{Bk}\) |
|---|---|---|---|
| 1 | 1.8 | 1.4 | 0.4 |
| 2 | 2.4 | 2.5 | -0.1 |
| 3 | 1.9 | 1.6 | 0.3 |
| 4 | 2.6 | 2.6 | 0.0 |
| 5 | 2.3 | 1.9 | 0.4 |
Method. Calculate the paired differences first, with positive values indicating greater loss for A. Their sum is \(0.4-0.1+0.3+0.0+0.4=1.0\), so \(\bar{D}=1.0/5=0.2\). The mean loss for A is \(11.0/5=2.2\), and for B it is \(10.0/5=2.0\); their difference is also \(0.2\), which checks the paired mean.
Next, the deviations from the paired mean \(0.2\) are \(0.2,-0.3,0.1,-0.2,0.2\). Their squared values sum to \(0.04+0.09+0.01+0.04+0.04=0.22\). Therefore \(s_D^2=0.22/(5-1)=0.055\), and \(s_D=\sqrt{0.055}=0.2345\) to four decimal places. The estimated standard error is \(0.2345/\sqrt{5}=0.1049\), to four decimal places.
The observed average difference favors B in loss terms, because \(\bar{D}=0.2>0\). The fold differences nevertheless vary: one favors A, one is tied, and three favor B. This is a comparison summary, not proof that B will be better on every future sample. Common mistake avoided: the standard deviation and standard error are calculated from the five differences, not from the five A losses alone or the five B losses alone.
How Pairing Controls Fold Difficulty
A fold can be unusually difficult for both candidates because of its particular validation observations. If its losses rise together, comparing the candidates' separate fold-to-fold variation can obscure their relative performance. Subtracting the losses within each matched fold removes some of that shared movement. The next example quantifies the contrast between a paired standard error and an independent-sample calculation.
Worked Example: Paired Versus Unpaired Standard Errors
Two hypothetical configurations are scored on four identical folds. Their losses are \(A=(2,4,3,5)\) and \(B=(1.8,3.9,2.8,4.9)\). Use the paired method, and compare its standard error with the result from incorrectly treating the two sets of fold losses as independent.
Paired calculation. The four differences \(A-B\) are \(0.2,0.1,0.2,0.1\). Their mean is \((0.2+0.1+0.2+0.1)/4=0.15\). Deviations from \(0.15\) are \(0.05,-0.05,0.05,-0.05\), whose squared values sum to \(0.01\). Thus \(s_D=\sqrt{0.01/3}=0.0577\), and \(\widehat{\operatorname{SE}}(\bar{D})=0.0577/\sqrt{4}=0.0289\), to four decimal places.
Unpaired calculation for comparison. The means are \(\bar{L}_A=3.5\) and \(\bar{L}_B=3.35\). For A, the squared deviations from \(3.5\) sum to \(5\), so \(s_A^2=5/3=1.6667\). For B, the squared deviations from \(3.35\) sum to \(5.41\), so \(s_B^2=5.41/3=1.8033\). An independent-sample calculation would give \(\sqrt{(s_A^2+s_B^2)/4}=\sqrt{(1.6667+1.8033)/4}=0.9314\), to four decimal places.
The paired estimate is much smaller because the loss changes across folds are largely shared by the two candidates; their differences stay close to \(0.15\). The independent calculation discards that matching information and is inappropriate for these matched observations. Common mistake avoided: do not use the formula for two independent samples when each A loss and B loss come from the same validation fold.
Equal-Fold and Observation-Weighted Comparisons
When fold sizes are equal, averaging the fold-level mean losses gives the same result as pooling all validation observations. With unequal fold sizes, those summaries answer different questions. An equal-fold mean gives each fold equal weight. A pooled per-observation loss gives each validation observation equal weight, so larger folds receive more weight.
If fold \(k\) contains \(n_k\) validation observations and \(D_k\) is the difference in mean per-observation loss on that fold, the pooled difference is \(\sum_k n_kD_k/\sum_k n_k\). This weighting must be chosen consistently with the intended summary; one should not compare an equally weighted mean for one candidate with a pooled mean for the other.
Worked Example: Unequal Fold Sizes and the Target Average
A hypothetical data set is split into three matched validation folds of sizes \(2,3,\) and \(5\). The fold-level mean losses for A are \(4,2,\) and \(3\), and those for B are \(3,2.5,\) and \(2.8\). Find both the equal-fold mean difference and the pooled, observation-weighted difference, using \(D_k=L_{Ak}-L_{Bk}\).
The differences are \(4-3=1\), \(2-2.5=-0.5\), and \(3-2.8=0.2\). The equal-fold mean is \((1-0.5+0.2)/3=0.7/3=0.2333\), to four decimal places. It gives each of the three folds the same weight, regardless of fold size.
For the pooled comparison, weight each difference by its fold size: \((2(1)+3(-0.5)+5(0.2))/(2+3+5)=(2-1.5+1)/10=1.5/10=0.15\). This agrees with pooling the losses directly: A's weighted mean loss is \((2(4)+3(2)+5(3))/10=29/10=2.9\), while B's is \((2(3)+3(2.5)+5(2.8))/10=27.5/10=2.75\). Their difference is \(2.9-2.75=0.15\).
Both calculations are correct for their stated targets, and they differ because the smaller folds have larger differences here. Common mistake avoided: do not assume that an unweighted average of fold means equals the per-observation pooled loss when fold sizes differ.
Practical Checks and Common Traps
Before interpreting a matched-fold comparison, verify the direction of subtraction and the meaning of the average. If \(D_k=L_{Ak}-L_{Bk}\), then positive values favor B on loss, not A. Also verify that the folds really match and that the same metric is used for every candidate. A change in loss definition or scale makes the differences uninterpretable.
A particularly common arithmetic error is to use the number of folds instead of \(K-1\) in the sample variance, or to confuse the sample standard deviation \(s_D\) with the standard error \(s_D/\sqrt{K}\). For example, for exactly three differences \(-0.3,0,0.3\), the mean is \(0\). The squared deviations sum to \(0.09+0+0.09=0.18\), so the sample variance is \(0.18/(3-1)=0.09\), the sample standard deviation is \(0.3000\), and the estimated standard error is \(0.3000/\sqrt{3}=0.1732\), to four decimal places. There are three observations in this calculation; adding or assuming a fourth would change both summaries.
Matched-fold comparison is useful when candidate configurations have close validation scores because it focuses attention on how their relative performance changes across the same validation samples. It is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. The essential discipline is to retain the pairing from the initial fold assignment through the final calculation.
Check Your Understanding
For each problem, use the stated direction of subtraction and keep the matched-fold structure intact.
- Four matched-fold differences \(A-B\) are \(0.2,-0.1,0.1,-0.2\). Find their mean, sample standard deviation, and estimated standard error, rounding the last two quantities to four decimal places. (Answer: Mean \(0\); sample standard deviation \(0.1826\); standard error \(0.0913\).)
- On three identical folds, candidate A has losses \(1.2,1.5,1.1\), while B has losses \(1.0,1.6,0.9\). Find the mean paired difference \(A-B\) and say which candidate has the smaller average loss. (Answer: Mean difference \(0.1000\); B has the smaller average loss.)
- Three matched folds have sizes \(2,3,5\), and their mean-loss differences \(A-B\) are \(0.6,-0.2,0.1\). Find the pooled, observation-weighted difference. (Answer: \(0.1100\).)
- Three matched-fold differences \(A-B\) are \(-0.3,0,0.3\). Find their mean, sample standard deviation, and estimated standard error, rounding to four decimal places. (Answer: Mean \(0\); sample standard deviation \(0.3000\); standard error \(0.1732\).)
- The fold-level losses have variances \(1\) for A and \(1\) for B, and covariance \(0.8\) when evaluated on the same folds. Find the variance of the paired difference \(A-B\), and compare it with the variance calculated under an independence assumption. (Answer: Paired variance \(0.4\); independence-based variance \(2\).)