Why Compare Procedures on the Same Folds?
Cross-validation estimates how well a regression procedure predicts observations it did not use to fit the procedure. When comparing two procedures, however, their scores can vary not only because the procedures differ, but also because one particular partition of the data makes prediction easier or harder. Comparing them on identical folds helps separate these sources of variation: each procedure is evaluated on the same held-out observations each time.
The goal is not to make every fold equally easy. It is to make the comparison fair by giving both procedures the same evaluation conditions. If one fold contains unusually difficult cases, its scores may be high for both procedures. The difference between the two scores on that fold can be more informative about their relative performance than either score alone.
Set Up a Comparable Evaluation
A fold-level score summarizes how well a procedure predicts the held-out cases in one fold. The score might be an average prediction loss, for example, but the comparison does not depend on choosing a particular loss. What matters is that both procedures use the same score definition, outcome scale, held-out observations, and direction of preference.
A regression procedure means the complete rule used to produce predictions from a training set. For each fold, fit each procedure on the same training observations, then calculate its score on the common held-out fold. The cross-validation assignments should be fixed before comparing the scores. If the two procedures use different folds, a difference between their fold scores may reflect different held-out observations rather than a difference between the procedures.
For the simple average of fold differences to represent the average fold-level comparison, give each fold equal weight. If fold sizes differ and the goal is instead to compare average loss per held-out observation across the entire data set, weight each fold difference by its number of held-out observations. State which summary is being used; they need not be the same when fold sizes differ.
Worked Example: Calculate the Paired Differences
Suppose two hypothetical regression procedures are evaluated on the same four cross-validation folds. The score is an average prediction loss in outcome units, and lower is better. The fold-level scores are:
| Fold | Procedure A score | Procedure B score | \(d_k=L_{A,k}-L_{B,k}\) |
|---|---|---|---|
| 1 | 8 | 9 | -1 |
| 2 | 12 | 14 | -2 |
| 3 | 10 | 11 | -1 |
| 4 | 14 | 17 | -3 |
For example, on fold 2 the difference is \(12-14=-2\) score units. The mean paired difference is
Across these four folds, procedure A's score is lower by an average of 1.75 score units. This describes the observed cross-validation comparison; it does not by itself establish that A will always perform better on new data. The units are the units of the score, so an outcome-unit loss would make the difference 1.75 outcome units.
Why Matching Can Reduce Partition Variation
A common source of variation is fold difficulty. A fold may contain cases that are hard for either procedure to predict. If both scores rise together on difficult folds, taking their difference removes some of that shared movement. The relevant algebra is the variance of a difference:
This identity follows by expanding the squared deviation of the difference from its mean:
When matched folds make the two scores positively associated because they share held-out cases and fold difficulty, the covariance term reduces the variance of their difference compared with the sum of their separate variances. This is the mathematical reason a paired comparison can be less sensitive to partition randomness. It is not a guarantee: the degree of benefit depends on how similarly the procedures respond to fold difficulty.
Worked Example: Shared Fold Difficulty and the Difference
Use the four scores in the previous example. The means are \(\bar{L}_A=11\) and \(\bar{L}_B=12.75\). Their sample variances across folds are
The sample covariance, using deviations from the two means, is
Thus, the sample variance of the paired differences is
Directly checking the differences \(-1,-2,-1,-3\), their mean is \(-1.75\), and their sample variance is \[ \frac{(0.75)^2+(-0.25)^2+(0.75)^2+(-1.25)^2}{3} =\frac{2.75}{3}\approx 0.917. \] The match between these calculations illustrates how the shared variation in the scores enters the paired comparison. The calculation describes variation among these fold scores; it does not turn the folds into independent observations for a formal significance test.
Choosing and Reporting the Summary
With \(K\) folds, the equal-fold average paired difference is
If all folds have equal size, this is also the average difference per held-out observation across the folds. If fold sizes vary, an equal-fold average gives the same weight to a small fold and a large fold. To give each held-out case equal weight, let \(n_k\) be the size of fold \(k\), and calculate
For example, suppose three folds contain 2, 2, and 1 held-out cases, and the paired differences in average loss are 1, 3, and 10 units. The equal-fold average is
The case-weighted average is
Both calculations are correct for their stated targets: one summarizes the average fold difference, and the other the average difference weighted by held-out case count. Report which summary you used, the score and its units, the fold assignments, and the direction of subtraction. The sign alone is not enough if readers do not know whether the difference was A minus B or B minus A.
Worked Example: Interpret Differences That Change Sign
In another hypothetical comparison, the fold differences are \(-4,-2,1,\) and \(1\) score units, again defined as procedure A minus procedure B. Their mean is
On average across the four folds, procedure A has a score 1 unit lower. But A does not have the lower score on every fold: its difference is positive on folds 3 and 4, so procedure B scores lower on those folds. A careful report gives both the average and the fold-by-fold pattern, especially when the differences vary in sign. An average can conceal meaningful inconsistency across the held-out subsets.
What Matched Folds Do Not Establish
The paired summary is a useful description of performance under a chosen cross-validation setup. It does not automatically provide a confidence interval, a p-value, or proof that one procedure is superior in the population. In ordinary \(K\)-fold cross-validation, training sets overlap substantially, and the fold scores are not generally independent replicates. Therefore, treating the \(K\) differences as independent observations and applying an ordinary paired test without justification can give misleading uncertainty.
Matched folds also do not remove every source of uncertainty. Results can depend on the sample, the fold assignment, and the chosen score. If repeated cross-validation is used, the same principle applies within each repetition: compare procedures on the same folds of that repetition. Repeated splits can show how comparisons vary across assignments, but their dependence and the intended inference still need attention.
For a fair comparison, both procedures must be handled consistently. In each fold, train both on the same training cases, evaluate both on the same held-out cases, and calculate the same score. Any step that learns from data as part of producing predictions must be based only on the training portion for that fold. Otherwise, the held-out comparison may not represent prediction on genuinely unseen cases.
Common Mistakes
- Comparing scores from different folds as if they were paired. The difference may reflect which cases were held out. Use identical fold assignments before calculating \(d_k\).
- Forgetting the subtraction order. If \(d_k=L_{A,k}-L_{B,k}\) and lower is better, a negative difference favors A, not B.
- Reporting only the average. A mean difference can hide folds where the other procedure performed better. Inspect and report the fold-level pattern when it matters.
- Assuming matching proves a real-world advantage. Matching reduces the influence of shared partition variation; it does not eliminate uncertainty or establish general superiority.
- Using equal fold weights without stating the target. Equal weighting summarizes fold averages. If fold sizes differ, case weighting answers a different question.
Check Your Understanding
Answer these questions using the definitions and calculations in this tutorial.
- Procedure A has a score of 6 and procedure B has a score of 8 on the same fold. If lower is better and \(d_k=L_{A,k}-L_{B,k}\), what is the difference and which procedure has the lower score?
- Why is it useful to give both procedures the same held-out cases in each fold?
- Two procedures have fold-level scores that rise together on difficult folds. How can positive covariance affect the variance of their paired differences?
- When fold sizes differ, what is the distinction between an equal-fold average difference and a case-weighted average difference?
- Why should fold-level differences from ordinary \(K\)-fold cross-validation not automatically be treated as independent observations in a standard paired test?