Why Compare Models on Shared Folds?
Cross-validation estimates how well a modeling procedure performs on data not used to fit it. When the goal is to compare two candidate models, the split itself can affect their measured performance: one validation fold may contain unusually difficult cases, while another may be relatively easy. Comparing the models on the same folds helps separate their relative performance from this variation in fold difficulty.
For a valid comparison, both candidates must be evaluated on the same validation observations in each fold. Each model is fitted using that fold’s training observations, and any learned preprocessing is fitted using training data only. The same performance measure and its direction must be used for both candidates. The split design must also respect the data structure: for example, use group-based folds for repeated entities or time-ordered folds when the prediction task requires them.
The subtraction order is a convention, but it must be stated and maintained consistently. Here the difference is A minus B, and lower loss is better. If a metric is a score for which larger values are better, such as a suitable discrimination score, define the difference or transform the score so that the interpretation is explicit. A numerical difference is not meaningful unless both models are assessed on the same observations using the same metric.
Calculate and Interpret Paired Differences
A basic summary is the average paired difference across folds. When folds have equal sizes, this is the arithmetic mean of the \(K\) fold differences. It describes the average difference in fold-level loss under the selected cross-validation split. The fold-level sample standard deviation describes how much those paired differences vary across the folds in that split.
These are descriptive summaries, not a guarantee that the observed advantage will persist on new data or under a different partition. Cross-validation folds are not independent replications: their training sets overlap, and the fitted models are therefore related. In particular, treating the \(K\) fold differences as independent observations and automatically using \(s_d/\sqrt{K}\) as a standard error can overstate the precision of the comparison.
Worked Example: Compare Two Losses on Five Shared Folds
A hypothetical pricing team compares two models using five shared validation folds. The reported losses are on the same scale, and lower is better. Model A has fold losses 12, 10, 14, 9, and 15; model B has losses 13, 11, 13, 12, and 16. Calculate the paired differences, their average, and their sample standard deviation. State which model has the lower average fold loss.
Method. Calculate A minus B separately within each shared fold. This pairing is important because both models face the same validation cases in a given fold.
The differences are \(12-13=-1\), \(10-11=-1\), \(14-13=1\), \(9-12=-3\), and \(15-16=-1\). Thus the paired-difference sequence is \((-1,-1,1,-3,-1)\).
The average difference is \((-1-1+1-3-1)/5=-5/5=-1\). Equivalently, A’s average loss is \((12+10+14+9+15)/5=60/5=12\), and B’s is \((13+11+13+12+16)/5=65/5=13\). Their difference is \(12-13=-1\), confirming the paired calculation.
For the sample standard deviation, the deviations from \(\bar d=-1\) are \(0,0,2,-2,0\). Their squared values sum to \(0+0+4+4+0=8\). Therefore \(s_d=\sqrt{8/(5-1)}=\sqrt{2}\), approximately 1.4142. A has one unit lower average fold loss, but the individual fold differences vary; the average alone does not show that A wins on every fold.
Check. The average of the five paired differences equals the difference between the two average fold losses. The negative sign is consistent with A’s lower average loss. The common mistake avoided is reversing the subtraction in some folds and then misreading the sign.
Why Pairing Can Make the Comparison More Informative
A fold can be easy or difficult for both models. Comparing each model on the same fold allows that shared difficulty to cancel in the difference. The variance identity below explains the potential benefit. It applies to the two losses measured on a common fold, provided their variances and covariance are defined under the same evaluation setup.
To justify the identity, write \(\operatorname{Var}(U-V)=E[((U-E[U])-(V-E[V]))^2]\). Expanding the square gives the squared centered term for \(U\), the squared centered term for \(V\), and minus twice their product. Taking expectations yields \(\operatorname{Var}(U)+\operatorname{Var}(V)-2\operatorname{Cov}(U,V)\). Set \(U=L_A\) and \(V=L_B\).
If the models’ losses tend to move together across folds, then \(\operatorname{Cov}(L_A,L_B)>0\), and the variance of their paired difference is smaller than it would be without the covariance term. This is why a shared-fold comparison can be more precise than comparing unrelated fold results. It is not a guarantee: the covariance could be small or negative, and paired folds do not make the results independent.
Worked Example: Read Fold Variation Without Overstating Certainty
On four shared folds, a hypothetical claim-severity model comparison gives paired differences \(d_k\) of \(-2,-1,0,-1\), again defined as A’s loss minus B’s loss. Calculate the average and sample standard deviation, then explain what they do and do not establish.
The average is \((-2-1+0-1)/4=-4/4=-1\). The deviations from \(-1\) are \(-1,0,1,0\); their squares sum to \(1+0+1+0=2\). Thus \(s_d=\sqrt{2/(4-1)}=\sqrt{2/3}\), approximately 0.8165.
A’s average fold loss is one unit lower than B’s, and three of the four differences favor A or tie, with two? Checking the signs directly, \(-2\), \(-1\), and \(-1\) favor A, while \(0\) is a tie. So A is lower on three folds and tied on one. The standard deviation summarizes dispersion of the four paired differences; it is not, by itself, a confidence interval or proof of a population-level advantage.
Common mistake avoided. Dividing \(s_d\) by \(\sqrt{4}\) and presenting the result as an automatically valid standard error assumes independent fold differences. Cross-validation folds share training data, so that assumption is generally unjustified. Treat the average and standard deviation as descriptive unless a suitable inferential method and its assumptions are established.
Variation Across Partitions
A single set of folds shows how the comparison varies across those particular validation folds. If several different valid partitions are run, each partition gives another summary of the comparison. For repeated \(K\)-fold cross-validation, a practical descriptive approach is to first calculate the mean paired difference within each repeat, then examine the distribution of those repeat-level means. The repeat-level summaries help reveal whether the conclusion is sensitive to how the observations were partitioned.
Do not regard all \(K\) fold differences from every repeat as independent data points. Repeats reuse the same observations, and folds within a repeat have overlapping training sets. A range or standard deviation of repeat-level means communicates observed partition sensitivity; it does not automatically provide a formal standard error or confidence interval.
Worked Example: Summarize Three Repeated Partitions
A hypothetical model comparison is repeated using three different four-fold partitions. Each number below is a paired difference, A minus B, in a lower-is-better loss. The fold differences are \((-2,-1,0,-1)\) in repeat 1, \((1,0,-1,0)\) in repeat 2, and \((-3,-2,-1,0)\) in repeat 3. Find each repeat’s mean, the average of the three repeat means, and their sample standard deviation.
Repeat 1: the sum is \(-2-1+0-1=-4\), so the mean is \(-4/4=-1\).
Repeat 2: the sum is \(1+0-1+0=0\), so the mean is \(0/4=0\).
Repeat 3: the sum is \(-3-2-1+0=-6\), so the mean is \(-6/4=-1.5\).
The average repeat mean is \((-1+0-1.5)/3=-2.5/3\), approximately \(-0.8333\). The deviations from this average are approximately \(-0.1667\), \(0.8333\), and \(-0.6667\). The sum of their squared deviations is approximately \(0.0278+0.6944+0.4444=1.1667\). The sample standard deviation of the repeat means is therefore \(\sqrt{1.1667/(3-1)}\), approximately 0.7638.
The average difference favors A, but the repeat means range from \(-1.5\) to 0, so the measured advantage varies by partition and one repeat shows a tie on average. Report both the overall direction and this sensitivity. With only three repeats, these summaries are descriptive; they do not establish a formal uncertainty interval.
Check. The mean of all 12 fold differences is also \(-10/12\), approximately \(-0.8333\), because each repeat has the same number of folds. The common mistake avoided is to present the 12 fold differences as 12 independent replications.
Unequal Fold Sizes and Aggregation
When folds differ in size, the ordinary average of fold-level losses gives every fold equal weight, not every validation observation. That may be an intentional summary, but it need not equal the loss over all out-of-fold predictions. For an additive observation-level loss, such as absolute error, the pooled mean loss weights each fold by its number of validation observations. Apply the same aggregation to the paired differences.
where \(n_k\) is the number of validation observations in fold \(k\). This formula applies when each \(d_k\) is a mean of the same kind of per-observation loss, and the target summary is the mean loss over all validation observations. It does not mean that every possible metric can be combined by this formula. For example, a mean of fold-specific RMSE values is not generally the pooled RMSE; define the target aggregation carefully.
Worked Example: Weight Additive Loss Differences by Fold Size
A hypothetical portfolio dataset has three validation folds of sizes 2, 3, and 5. Their mean absolute errors are 4, 5, and 3 for model A, and 3, 6, and 4 for model B. Find the pooled mean absolute error for each model and the pooled paired difference.
The fold differences, A minus B, are \(4-3=1\), \(5-6=-1\), and \(3-4=-1\). The weighted difference is \((2(1)+3(-1)+5(-1))/(2+3+5)=(2-3-5)/10=-6/10=-0.6\).
Check using total absolute errors. For A, the fold totals are \(2(4)=8\), \(3(5)=15\), and \(5(3)=15\), for a total of 38. Its pooled mean absolute error is \(38/10=3.8\). For B, the totals are \(2(3)=6\), \(3(6)=18\), and \(5(4)=20\), for a total of 44, giving \(44/10=4.4\). The pooled difference is \(3.8-4.4=-0.6\), as calculated.
A has the lower pooled mean absolute error by 0.6. The unweighted average of the three fold differences would be \((1-1-1)/3=-1/3\), which answers a different question by assigning the two-observation fold the same weight as the five-observation fold. The common mistake avoided is to confuse an average of fold metrics with the pooled observation-level metric.
Practical Reporting and Common Traps
A clear comparison states the split design, metric, difference direction, aggregation rule, and variation summary. For example, report that A had a specified average paired loss difference relative to B on shared folds, and then describe how the paired differences varied across folds or repeated partitions. Avoid saying that a model is definitively better solely because its average is slightly lower.
- Using different folds for each candidate. This mixes model differences with differences in validation cases. Compare on shared folds.
- Changing the sign convention. Define whether the difference is A minus B or B minus A, and state which sign favors which model.
- Ignoring fold sizes. For pooled averages of additive losses, use observation counts as weights. For other metrics, define the aggregation explicitly.
- Treating folds as independent replicates. Shared observations and overlapping training sets create dependence. Fold-level standard deviations describe variation but do not automatically justify conventional standard errors.
- Ignoring partition sensitivity. An average advantage can conceal repeats where the result is weak, tied, or reversed. Summarize repeat-level variation when repeated partitions are available.
Check Your Understanding
For each question, use A minus B as the paired difference unless stated otherwise. Assume lower loss is better.
- On four shared folds, A’s losses are 7, 9, 6, and 8; B’s are 8, 8, 7, and 10. Find the four paired differences and their mean. (Answer: \(-1, 1, -1, -2\); mean \(=-0.75\), favoring A on average.)
- Three paired differences are \(-2,0,2\). Calculate their sample standard deviation. (Answer: \(2.0000\).)
- Two folds have sizes 4 and 6, with paired mean absolute error differences of 1 and \(-2\). Find the pooled paired difference. (Answer: \((4(1)+6(-2))/10=-0.8\).)
- Three repeated-partition means are \(-1, 0.5,\) and \(-0.5\). Find their average and range, and state what the signs indicate. (Answer: average \(=-1/3\), approximately \(-0.3333\); range \([-1,0.5]\); A has a lower average loss overall, but partition means include one favoring B.)
- A report calculates a standard error by dividing the standard deviation of ten fold differences by \(\sqrt{10}\), then claims the result is automatically a valid uncertainty measure. Identify the issue. (Answer: the calculation assumes independent fold differences; overlapping training sets and repeated use of observations generally make that assumption unjustified.)