Why Compare Models Fold by Fold?
When two genomic prediction procedures are evaluated by nested cross-validation, each procedure produces a prediction for the same held-out individuals in each outer fold. Comparing their losses fold by fold preserves this matching: both procedures encounter the same held-out cases, including the same unusually easy or difficult cases. The resulting comparison can be more informative than looking at two separate lists of fold scores.
The comparison is meaningful only if the procedures are evaluated under a fair, prespecified design. They should use the same outer splits, outcome definition, loss function, and evaluation units. Each procedure—including any hyperparameter selection and preprocessing—must be fitted using only the data permitted by that outer fold. The earlier material on nested cross-validation and hyperparameter selection optimism explains why outer-fold results must not be used to tune a procedure whose performance is then reported on those same results.
The sign convention matters: for a loss, a positive average difference favors B, not A. For a performance measure where larger is better, either define the difference in the opposite order or explicitly explain which sign represents better performance. Losses must also be on the same scale and have the same interpretation in every fold. For example, comparing mean absolute errors measured in expression units is different from comparing squared errors measured in squared expression units.
Conditions for a Fair Paired Comparison
Pairing is not just a calculation made after fitting models. It comes from the evaluation design. For each outer fold, both procedures should be evaluated on the same held-out individuals, with training data drawn from the same complementary part of the split. This makes differences within a fold a like-for-like comparison. If family, site, or other dependence determines the correct unit of splitting, both procedures must use the appropriate shared split design.
The procedures being compared should be defined in full before interpreting their outer-fold performance. If one procedure includes tuning, all tuning must be carried out inside the outer training data according to its specified pipeline. Fold-local feature selection, scaling, imputation, and other learned transformations must likewise be fitted without access to the outer test fold. If a transformation or selection step has seen test-fold information, matching the splits does not repair the leakage.
The loss must be computed consistently. A fold-specific loss should use the same definition and units in every fold; for example, each fold's mean absolute error is in the units of the outcome. When folds are different sizes, the analyst must also be clear about the summary being reported. An unweighted average gives each fold equal weight; a case-weighted summary gives each held-out individual equal weight. These summaries can differ, so the choice should match the intended performance summary and should not be changed after inspecting which procedure wins.
Worked Example: Interpreting Matched Mean Absolute Errors
Suppose two hypothetical procedures predict a continuous genomic phenotype. The table gives each procedure's mean absolute error (MAE), in phenotype units, for the same five outer test folds. Assume all five folds have equal size.
| Outer fold | Procedure A MAE | Procedure B MAE | \(\Delta_k=L_{Ak}-L_{Bk}\) |
|---|---|---|---|
| 1 | 0.80 | 0.75 | 0.05 |
| 2 | 1.10 | 1.00 | 0.10 |
| 3 | 0.90 | 1.05 | -0.15 |
| 4 | 1.30 | 1.20 | 0.10 |
| 5 | 0.90 | 0.85 | 0.05 |
The mean fold MAE for A is \((0.80+1.10+0.90+1.30+0.90)/5=5.00/5=1.00\) phenotype units. For B it is \((0.75+1.00+1.05+1.20+0.85)/5=4.85/5=0.97\) phenotype units. The paired differences sum to \(0.05+0.10-0.15+0.10+0.05=0.15\), so their mean is \(\bar{\Delta}=0.15/5=0.03\) phenotype units.
In this particular set of outer folds, B has an average MAE 0.03 phenotype units lower than A. B has lower loss in four folds; A has lower loss in fold 3. This is a descriptive comparison of these cross-validation results, not evidence that B is certainly better in the population or on every future sample.
Why Folds Are Not Independent Replicates
The outer-fold results share data. In ordinary \(K\)-fold cross-validation, each fitted model is trained on a large fraction of the same data used to train models for other folds. The fitted models therefore have overlapping training sets. Even though each individual is held out in one outer fold, the fold losses are not independent draws from a sequence of independent experiments: their fitted predictors depend on many of the same observed individuals and outcomes.
Pairing handles a different issue. It matches the two procedures within each fold, which controls for the fact that one fold may be harder than another. It does not make the folds independent of each other. In particular, the standard error \(s_{\Delta}/\sqrt{K}\), where \(s_{\Delta}\) is the sample standard deviation of the fold differences, is not automatically a valid standard error for a population-level performance difference. A conventional paired t test on the \(K\) fold differences relies on assumptions about independent, appropriately sampled differences that ordinary cross-validation folds do not satisfy.
A short variance identity shows why matching can help while leaving this dependence problem unresolved. Within a fold, the variance of a difference is
If both procedures tend to do well on easy folds and poorly on difficult folds, their losses have positive covariance. Subtracting matched losses can then remove some shared fold difficulty. But this identity concerns the two losses within a fold; it does not imply that the differences \(\Delta_1,\ldots,\Delta_K\) are independent across folds. Dependence among those differences remains a reason not to treat \(K\) folds as \(K\) independent studies.
Worked Example: Pairing Removes a Shared Difficulty Shift
Consider hypothetical losses, in a common loss scale, from four matched folds:
| Fold | Procedure A loss | Procedure B loss | \(\Delta_k\) |
|---|---|---|---|
| 1 | 2 | 1 | 1 |
| 2 | 8 | 7 | 1 |
| 3 | 4 | 5 | -1 |
| 4 | 10 | 9 | 1 |
The mean losses are \((2+8+4+10)/4=6.00\) for A and \((1+7+5+9)/4=5.50\) for B. The paired differences are \(1,1,-1,1\), with mean \((1+1-1+1)/4=0.50\). Thus B's average loss is 0.50 units lower across these four matched folds.
Folds 2 and 4 have much larger losses than fold 1 for both procedures, suggesting that some folds are harder for both. Pairing makes this shared level of difficulty visible: the comparison is based on the within-fold gaps, not on whether a fold is generally easy or hard. However, these four differences are still not four independent replications. The calculation supports a descriptive statement about these folds, not a t-test-based claim of statistical significance.
Summarizing and Diagnosing the Comparison
A useful report gives the fold-level losses or differences, the direction of the difference, and a clearly defined summary such as \(\bar{\Delta}\). The sign and size of the summary should be interpreted in the loss's units. Also report how many folds favored each procedure, but do not treat a count of wins as an independent-significance test. A large mean difference driven by one fold has a different pattern from a modest, consistently signed difference; both the average and the fold-level pattern help describe what happened.
Before drawing a conclusion, check the design and the arithmetic. The paired folds must really contain the same test cases for both procedures. The losses must use the same outcome scale and metric. Fold-specific values should be traced back to their predictions, and the difference should be computed in the stated order. If a fold is dramatically different, investigate whether it has a small or unusual test set, a site or family composition that differs from the others, or a data-processing problem. Such an investigation can explain the observed pattern, but it should not become a post hoc reason to omit an inconvenient fold.
The following comparisons clarify what paired outer-fold results can and cannot do:
| Approach | What it contributes | Main caution |
|---|---|---|
| Paired outer-fold differences | Directly contrasts procedures on the same held-out cases in each fold. | Fold differences remain dependent; their spread is not automatically a conventional inferential standard error. |
| Separate fold averages without matching | Summarizes each procedure's observed losses. | Conceals the within-fold pairing and can make the comparison less informative. |
| Ordinary paired t test on fold differences | Would quantify uncertainty under suitable independent-difference assumptions. | Those assumptions are generally not justified by a single ordinary cross-validation partition. |
Paired outer-fold comparison is therefore best treated as a structured descriptive comparison of two prespecified prediction procedures under a specified validation design. The result reflects the data, split design, training-set size, and evaluation metric used. It does not establish that a procedure will perform better for every cohort, population, or future training sample. Nor does repeatedly choosing the apparent winner after examining many procedures turn the same outer-fold results into an independent confirmation.
Common Mistakes
- Calling folds independent replicates. Their test sets may be disjoint, but their fitted models share much of their training data. A conventional t test based on the fold count is not automatically justified.
- Reversing the sign. With \(\Delta_k=L_{Ak}-L_{Bk}\) for a loss, positive values favor B. State the definition before interpreting the result.
- Pairing mismatched test sets. Losses from different held-out individuals do not form the intended within-fold comparison. Use the same outer splits for both procedures.
- Overstating a small average difference. A positive \(\bar{\Delta}\) describes which procedure had lower average loss on these folds; by itself it does not prove a general performance advantage.
- Ignoring leakage or post hoc choices. Matching cannot fix preprocessing that used held-out data or an evaluation design altered after seeing the results.
Check Your Understanding
Answer these questions using the loss-difference convention in this tutorial.
- If \(\Delta_k=L_{Ak}-L_{Bk}=-0.20\) for MAE, which procedure had lower MAE in that fold, and by how much?
- For three equal-sized folds, the paired differences are \(0.4,-0.1,0.3\) loss units. Calculate \(\bar{\Delta}\) and interpret its direction.
- Why is it invalid to assume that disjoint outer test folds make the fold differences independent?
- Two procedures are evaluated on different outer splits, but the same metric is used. What is missing from the comparison design?
- A researcher computes \(s_{\Delta}/\sqrt{K}\) and reports an ordinary paired t-test p-value. What assumption needs justification, and why is it problematic here?
Solutions. (1) A had lower MAE because the difference is negative; its MAE was 0.20 units lower than B's. (2) \(\bar{\Delta}=(0.4-0.1+0.3)/3=0.6/3=0.20\) loss units, so B had lower average loss by 0.20 units across these folds. (3) The fitted models for different folds use overlapping training sets, so their losses can be dependent even when their held-out individuals are disjoint. (4) The procedures were not tested on matched held-out cases, so fold-by-fold differences would not be like-for-like; use common outer splits. (5) The calculation and test require suitable independence assumptions for the fold differences. Ordinary cross-validation folds share training data, so those assumptions are not supplied merely by having \(K\) folds.