Repeated Cross-Validation and Performance Variability

Why Repeat Cross-Validation?

A single K-fold cross-validation run produces a performance estimate for one particular assignment of observations to folds. A different random assignment can produce a different estimate: some partitions may place unusually difficult observations together, or change which observations are used to fit each fold-specific model. Reporting only one score can conceal this sensitivity.

Repeated cross-validation addresses that issue by carrying out the same K-fold procedure several times with different fold assignments. Its purpose is to describe how the estimated performance varies with those assignments. It does not create new observations, and it does not by itself measure every source of uncertainty in future performance.

Definition: In repeated K-fold cross-validation, the available data are divided into \(K\) folds for each repeat. For each fold, the model is fitted using the other \(K-1\) folds and scored on the held-out fold. This is repeated for \(R\) different fold assignments. Let \(\widehat{L}_r\) denote the overall validation loss from repeat \(r\), where lower values indicate better performance under a loss measure.

The target, prediction unit, model specification, fitting procedure, and scoring rule should be held fixed across repeats if the goal is to isolate the effect of fold assignments. Any preprocessing or model fitting that uses data must be done within the training portion of each fold; otherwise information can leak from held-out observations into the fitted predictions. The validation design must also respect the structure of the data and the intended prediction task.

For each repeat, every observation is held out once. The repeat therefore produces one overall score, typically calculated by combining the held-out predictions and outcomes from all \(K\) folds. If folds have equal sizes, averaging their fold scores gives the same result as pooling the underlying per-observation losses. With unequal fold sizes, the aggregation rule matters, as shown below.

Summarizing the Distribution of Scores

After \(R\) repeats, report the repeated scores as a distribution, not merely as a mean. Useful descriptive summaries include the mean, median, minimum and maximum (or range), and sample standard deviation. A small standard deviation indicates that the estimate was relatively stable across the sampled partitions; a large one indicates greater sensitivity to the partitioning.

\overline{L}=\frac{1}{R}\sum_{r=1}^{R}\widehat{L}_r, \qquad s_L=\sqrt{\frac{1}{R-1}\sum_{r=1}^{R}(\widehat{L}_r-\overline{L})^2}

Here \(s_L\) is the sample standard deviation of the repeat-level scores. It is a descriptive measure of their spread, not automatically a standard error for performance on new data. In particular, it is generally not justified to report \(s_L/\sqrt{R}\) as though the repeats were independent experiments: they reuse the same observations, and their training sets overlap substantially.

A useful interpretation separates two sources of variation. Let \(D\) denote the observed data and \(A\) the random fold assignment. Variation in the score can arise because the data set itself differs from a future sample, and because different assignments \(A\) produce different fitted training sets and validation results for the same \(D\). Repeating the splits primarily explores the second source, conditional on the observed data.

Key distinction: The spread of repeated cross-validation scores describes sensitivity to the fold assignments used on the available data. It is not, on its own, a confidence interval for the model's performance in a future population, nor a measure of all uncertainty due to sampling, time changes, or model choice.

Worked Examples

Worked Example: Summarizing Repeat-to-Repeat Variability

A hypothetical insurer evaluates one frequency model using the same five-fold procedure in four independent random fold assignments. The selected loss measure is fixed, with lower values preferred. The four repeat-level losses are \(0.118,\ 0.126,\ 0.121,\) and \(0.135\). Calculate the mean, median, range, and sample standard deviation, then interpret them.

Method. Treat each complete repeat as one score. No parametric distribution for claims or losses is assumed; these are empirical validation scores. The common mistake is to use the \(K R=20\) fold scores as if they were 20 independent repeat-level estimates. The supplied values are already the four overall repeat scores, so the summaries use \(R=4\).

Mean. The sum is \(0.118+0.126+0.121+0.135=0.500\), so \(\overline{L}=0.500/4=0.125\).

Median. In sorted order, the scores are \(0.118,\ 0.121,\ 0.126,\ 0.135\). The middle two average to \((0.121+0.126)/2=0.1235\).

Range. The minimum is \(0.118\), the maximum is \(0.135\), and the range is \(0.135-0.118=0.017\).

Sample standard deviation. Deviations from \(0.125\) are \(-0.007,\ 0.001,\ -0.004,\ 0.010\). Their squared values sum to \(0.000049+0.000001+0.000016+0.000100=0.000166\). Thus the sample variance is \(0.000166/(4-1)=0.000055333\), and \(s_L=\sqrt{0.000055333}=0.00744\), rounded to five decimal places.

The mean loss is \(0.125\), while the observed repeat scores span \(0.017\). The standard deviation summarizes their partition sensitivity; it is not a confidence interval. Check: the mean lies between the minimum and maximum, and the squared deviations sum to the stated \(0.000166\).

Worked Example: Comparing Two Models on Matched Repeats

Two hypothetical claim-frequency models are evaluated with the same five-fold assignments in each of five repeats. Their losses are:

Model A: \(0.142,\ 0.151,\ 0.147,\ 0.139,\ 0.156\).
Model B: \(0.150,\ 0.149,\ 0.162,\ 0.144,\ 0.158\).

Calculate each mean loss and the mean paired difference \(B-A\). State which model has the lower average loss and how consistent that comparison is across repeats. Lower loss is preferred, and no probability distribution for the scores is assumed.

Method. Compare the models within each matching repeat, since they were tested on the same held-out partitions. The common mistake is to compare unpaired summaries while ignoring which partition produced each score.

The sum for A is \(0.142+0.151+0.147+0.139+0.156=0.735\), so its mean is \(0.735/5=0.1470\). The sum for B is \(0.150+0.149+0.162+0.144+0.158=0.763\), so its mean is \(0.763/5=0.1526\). On average, A has the lower loss by \(0.1526-0.1470=0.0056\).

The paired differences \(B-A\) are \(0.008,\ -0.002,\ 0.015,\ 0.005,\ 0.002\). Their sum is \(0.028\); their mean is \(0.028/5=0.0056\). A has lower loss in four of the five repeats; B has lower loss in the second repeat.

Thus A has the lower average loss, and the direction of the difference is fairly consistent but not universal. This is a descriptive comparison, not proof that A will outperform B on every future sample. Check: the mean paired difference equals the difference of the two means, \(0.1526-0.1470=0.0056\).

Worked Example: Combining Unequal Fold Scores Correctly

A hypothetical model is assessed in one three-fold repeat. The validation folds contain 20, 50, and 80 policies. Their total per-policy losses are respectively 3, 2, and 16. Calculate the overall per-policy loss for the repeat and compare it with the unweighted mean of the three fold-average losses.

Method. The repeat-level target is average loss per policy, so pool total loss across all policies and divide by the total policy count. No claim distribution is assumed; the fold totals are the given empirical losses. The common mistake is to average the three fold averages equally even though the folds contain different numbers of policies.

The fold-average losses are \(3/20=0.1500\), \(2/50=0.0400\), and \(16/80=0.2000\). The pooled total loss is \(3+2+16=21\), and the total number of policies is \(20+50+80=150\). Therefore, the overall loss is \(21/150=0.1400\).

The unweighted mean of the fold-average losses is \((0.1500+0.0400+0.2000)/3=0.3900/3=0.1300\), which differs from the pooled value. The correct aggregation depends on the defined scoring target; for per-policy average loss, each policy receives equal weight, so the pooled value \(0.1400\) is appropriate. Check: the pooled result is the size-weighted fold average: \((20(0.1500)+50(0.0400)+80(0.2000))/150=21/150=0.1400\).

Why the Mean and Spread Must Be Read Together

The mean summarizes the average performance estimate across the sampled assignments. The spread indicates whether that estimate changes materially when the folds change. Two models can have similar means but very different stability, or one model can have a slightly better mean while winning in only a few repeats. Those distinctions may matter when deciding whether a small apparent improvement is robust enough to investigate further.

A short justification explains why additional repeats can stabilize the reported average without providing new data. Conditional on a fixed observed data set, imagine that each random fold assignment yields a score from the same assignment-generating procedure, with conditional mean \(m(D)\). If assignments were independent draws, the average of \(R\) scores would have conditional expectation \(m(D)\): by linearity of expectation,

E\left[\frac{1}{R}\sum_{r=1}^{R}\widehat{L}_r\middle|D\right] = \frac{1}{R}\sum_{r=1}^{R}E[\widehat{L}_r\mid D] = \frac{1}{R}\sum_{r=1}^{R}m(D) = m(D).

This shows why averaging many assignments can reduce Monte Carlo noise in estimating the assignment-conditional mean. It does not show that \(m(D)\) equals the true future-population performance: the fixed data set remains fixed, and the fitted models still depend on it. Also, the empirical repeat scores need not be independent in every practical implementation. The equation justifies the conditional averaging idea, not an automatic standard error or confidence interval.

Fix the evaluation question.
Use the same prediction target, prediction unit, scoring rule, and modeling procedure across repeats.

Repeat the partitioning procedure.
Generate new fold assignments and refit the model separately for every training fold; keep held-out information out of fitting and preprocessing.

Calculate one score per repeat.
Combine fold results in a way consistent with the chosen scoring target, accounting for fold sizes when needed.

Report center and spread.
Give the repeated scores or a distribution summary such as mean, median, range, and sample standard deviation, then interpret the spread as partition sensitivity.

Common Traps and Practical Interpretation

Calling the standard deviation a confidence interval. It describes the spread of the observed repeat scores. It does not automatically quantify uncertainty about a future population.
Dividing the spread by the square root of all folds. Fold scores are not independent replications: their training sets overlap, and the same data are reused. Use repeat-level scores for descriptive summaries, and do not treat the fold count as an independent sample size.
Reporting only the mean. A single average hides whether performance estimates were stable or varied materially with the partitions. Include at least one measure of spread, and inspect the individual repeat scores when the number of repeats is modest.
Changing the scoring target between repeats. A change in weighting or aggregation can alter the score for reasons unrelated to fold assignment. Define the target and aggregation rule first.
Selecting the smallest observed score without noting the search. If many models or settings are compared, the best observed score can partly reflect favorable partition noise. Repeated cross-validation describes that variability but does not erase the effects of extensive selection.

Repeated cross-validation is most informative when the score distribution is considered alongside the average and when the validation design matches the intended use. A narrow spread supports the statement that results were relatively insensitive to the tested assignments—not the stronger claim that performance is certain or universally stable. If observations are dependent, the splitting design must respect that dependence; randomly splitting related records can give an overly favorable assessment.

Key takeaway: Repeat K-fold cross-validation with different assignments, calculate one comparable overall score per repeat, and report both its center and spread. Interpret the spread as fold-assignment variability on the available data, not as a stand-alone measure of future-performance uncertainty.

Check Your Understanding

For each problem, interpret repeated scores as empirical results from the stated procedure. Lower loss is better where losses are given.

A model has four repeat-level losses: \(0.204,\ 0.198,\ 0.211,\ 0.207\). Find the mean, median, and range. (Answer: mean \(0.2050\), median \(0.2055\), range \(0.013\).)
Three unequal folds contain 20, 50, and 80 policies and have total losses 3, 2, and 16. Find the pooled per-policy loss and the unweighted mean of fold-average losses. Which is correct if every policy is to receive equal weight? (Answer: pooled loss \(0.1400\); unweighted fold mean \(0.1300\); the pooled loss is correct.)
Two models have matched repeat scores A: \(0.31,\ 0.34,\ 0.32\), and B: \(0.33,\ 0.33,\ 0.35\). Find their mean losses, the mean paired difference \(B-A\), and the number of repeats won by A. (Answer: mean A \(0.3233\), mean B \(0.3367\), mean \(B-A=0.0133\); A has lower loss in 2 of 3 repeats.)
A procedure produces repeat losses \(0.06,\ 0.08,\ 0.10,\ 0.12\). Find the mean and range, and explain what the range measures. (Answer: mean \(0.09\), range \(0.06\); it describes the observed spread across fold assignments, not a confidence interval for future performance.)
Two candidate models have repeat losses A: \(0.11,\ 0.11,\ 0.11,\ 0.15\), and B: \(0.12,\ 0.12,\ 0.12,\ 0.12\). Compare their mean losses and describe the stability difference. (Answer: both means are \(0.12\); B is more stable across these repeats, while A has greater partition-to-partition variability.)

What You'll Learn