Why Repeat the Partitions?
A \(K\)-fold cross-validation estimate depends partly on which observations happen to be assigned to each fold. In genomic prediction, a small number of cases, an imbalanced outcome, or influential observations can make this dependence practically important: one valid partition may produce a better or worse estimate than another. Repeating cross-validation with new random partitions lets us describe that split sensitivity for the data currently in hand.
The interpretation must remain narrow. Repeating partitions does not collect new individuals, sites, or cohorts. It therefore describes how the evaluation changes when the partition changes, conditional on the observed dataset and the validation design. It does not, by itself, quantify how performance would vary if a new study population were sampled.
What Is Repeated?
In \(R\)-repeat \(K\)-fold cross-validation, each repeat uses a new partition of the same \(n\) observations into \(K\) folds. Within a repeat, each observation is held out once and used to evaluate a model trained on the other folds. The full analysis pipeline must be fitted using training data only, including preprocessing, feature selection, and any hyperparameter tuning. When tuning is part of the procedure, it must remain inside each outer training set, as in nested cross-validation.
The folds should respect the intended validation design. For example, if related individuals must stay together, they must remain together in every repeat; if folds are stratified, the stratification rule should be applied in every repeat. The same partitions should be used for competing methods when the goal is to compare their performance, so that partition difficulty is not needlessly confounded with method.
For a metric calculated separately on each fold, use the same prespecified fold-aggregation rule in every repeat. For an additive loss such as mean squared error, a fold-size-weighted average is natural. Other metrics may need a different well-defined aggregation; the important point here is to calculate one estimate per repeat using a consistent rule, rather than treating every fold from every repeat as a separate study.
Summarizing Partition Variation
A useful report includes the repeated-CV mean and a description of the spread, such as the minimum and maximum or a standard deviation. These are descriptive summaries of the chosen partitions, not automatically a confidence interval for a population performance parameter. A plot of the repeat-level estimates can also reveal skewness, clusters, or an unusually influential partition. The number of repeats, number of folds, split constraints, metric, and aggregation rule should be stated so the summary can be interpreted.
Worked Example: Repeat-to-Repeat Variation in Mean Squared Error
Suppose a hypothetical genomic prediction analysis uses the same dataset and a valid five-fold design in each of three repeats. The fold-level mean squared errors, in squared outcome units, are as follows. Assume each fold has the same size, so the unweighted fold mean is also the fold-size-weighted mean.
| Repeat | Five fold-level mean squared errors | Repeat estimate |
|---|---|---|
| 1 | 0.82, 0.91, 0.76, 1.01, 0.90 | 0.880 |
| 2 | 1.10, 0.80, 0.95, 0.85, 0.90 | 0.920 |
| 3 | 0.70, 0.88, 0.94, 0.98, 0.85 | 0.870 |
For repeat 1, the estimate is \((0.82+0.91+0.76+1.01+0.90)/5=4.40/5=0.880\). The corresponding calculations give \(4.60/5=0.920\) for repeat 2 and \(4.35/5=0.870\) for repeat 3. Thus the repeated-CV mean is \((0.880+0.920+0.870)/3=0.890\) squared outcome units, and the range is \(0.870\) to \(0.920\), a spread of \(0.050\).
The sample standard deviation of these three repeat estimates is \(\sqrt{[(0.880-0.890)^2+(0.920-0.890)^2+(0.870-0.890)^2]/(3-1)}=\sqrt{0.0007}=0.0265\) squared outcome units. It summarizes the observed repeat-to-repeat spread. It is not justified to interpret it as a standard error for performance in a newly sampled population.
Why Split Sensitivity Is Not Population Uncertainty
The distinction can be stated with a variance decomposition. Imagine that \(D\) itself could vary because a study population is sampled, and let \(Z\) denote the random partition generated for a given dataset. For a fixed analysis procedure, the law of total variance gives:
The second term describes average partition variation conditional on the data. Repeated cross-validation on the observed \(D\) gives information about this conditional variation. The first term describes how the expected estimate changes across datasets sampled from the target population. Repartitioning one fixed dataset does not observe that source of variation.
This decomposition is a conceptual justification, not a claim that a few repeats precisely estimate either term. It also assumes the random sources are clearly defined: if a learning algorithm is stochastic, its random seed can be fixed to isolate partition variation, or deliberately varied as an additional source of variability. The chosen approach should be reported.
Worked Example: Same-Data Repeats Versus New Study Samples
Consider a hypothetical study dataset whose three repeat estimates are the values above: \(0.880\), \(0.920\), and \(0.870\) squared outcome units. Their mean is \(0.890\). Suppose, separately, that a second hypothetical study sample drawn from the target population produces repeat estimates of \(0.980\), \(1.020\), and \(0.990\), with mean \((0.980+1.020+0.990)/3=0.9967\) squared outcome units.
The first set describes partition variation in the first dataset; the second describes partition variation in the second dataset. The difference between their means, \(0.9967-0.890=0.1067\) squared outcome units, illustrates a possible between-sample difference, but two hypothetical datasets are not enough to quantify population uncertainty. The key point is that no number of new partitions of the first dataset can substitute for observing new study samples.
Using Repeats Carefully
Repeated cross-validation is especially useful when a single partition could be unrepresentative. A wide spread warns that a reported estimate is sensitive to the allocation of observations, even if the mean across repeats seems favorable. Inspect repeat-level estimates and, when useful, fold-level behavior to look for partitions with unusually poor performance. In genomic data, also check that group, family, site, and outcome constraints were followed in every repeat.
A narrow spread is not proof that the estimate is unbiased or transportable. All repeats may share the same sampling limitations, cohort composition, measurement problems, or mismatch with the intended prediction setting. Repeated partitions cannot correct leakage, an invalid split design, or a systematic difference between the study data and the future population. They also cannot make a very small or unrepresentative study informative merely by generating many random seeds.
The main alternatives answer different questions. A single cross-validation run is cheaper but gives no direct view of partition sensitivity. Repeated cross-validation makes that sensitivity visible at greater computational cost. Evaluation on an independent external cohort addresses performance on that particular cohort rather than random repartitions of the development dataset. These approaches are not interchangeable: choose based on whether the concern is random-split dependence or evaluation on genuinely new data.
A Practical Workflow and Failure Checks
Choose the number of folds, split constraints, metric, and aggregation rule to match the intended prediction task.
Generate new valid partitions and refit all training-dependent steps, including any inner tuning, within each outer training set.
Report the mean and a descriptive measure of spread, with the number of repeats and units. Do not count the folds or repeat estimates as independent studies.
Look for invalid group assignments, unstable estimates, rare outcomes concentrated in folds, or a mismatch between the validation design and the intended deployment setting.
A warning sign is a change in the validation rule between repeats—for example, allowing related individuals to cross folds in some repeats but not others. Another is a high-performing repeat produced by an accidental leakage path or an extreme fold composition. These are design failures to investigate, not simply inconvenient values to discard. If algorithmic randomness is also varied, report that fact because the spread then reflects more than partition sensitivity alone.
Worked Example: Interpreting a Wide Range
Suppose a hypothetical classifier is evaluated with repeated, stratified five-fold cross-validation on a fixed genomic dataset. Its repeat-level accuracies range from \(0.71\) to \(0.84\), with mean \(0.78\). The range is \(0.84-0.71=0.13\), or 13 percentage points.
This result indicates substantial observed split sensitivity for accuracy under this design. It does not mean that the classifier’s population accuracy has a 13-point uncertainty interval, and it does not establish that the mean \(0.78\) will hold in a new cohort. The next checks are whether every repeat obeyed the stratification and grouping rules, whether a small number of cases drive the low-performing partitions, and whether accuracy is an appropriate metric for the outcome balance. If those checks pass, the spread should be reported as a practical limitation of the estimate rather than concealed by reporting only its mean.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A fixed dataset gives repeat-level losses of \(2.1\), \(2.5\), and \(2.2\). What is their mean, and what does the range describe?
- Why does increasing the number of random partitions not create more independent participants?
- A researcher changes from individual-level folds to family-wise folds halfway through the repeats. Is the resulting spread a clean measure of split sensitivity under one design? Explain.
- What additional source of variation is not assessed by repeating partitions of one fixed dataset?
- Suppose an algorithm uses random initialization. Name two defensible choices for handling that randomness when assessing split sensitivity.
Solutions. (1) The mean is \((2.1+2.5+2.2)/3=6.8/3=2.27\) loss units, and the range \(2.1\) to \(2.5\) describes variation across partitions of the fixed dataset. (2) The same participants appear repeatedly in different training and test folds, so new partitions do not add new sampled participants. (3) No. The spread combines different validation designs, so it cannot be attributed to random partitions under a single fixed design. (4) Variation from sampling a new study dataset from the target population. (5) Fix the initialization seed to isolate partition variation, or vary it deliberately and report that the observed variation includes both partition and algorithm randomness.