Why Repeat K-Fold Cross-Validation?
In K-fold cross-validation, the observations are divided into \(K\) folds. Each fold serves once as the validation set, while the other \(K-1\) folds are used to fit the model. The resulting validation losses are combined into one cross-validation score. With elastic net, this calculation is repeated for each tuning setting, such as a pair \((\alpha,\lambda)\), where \(\alpha\) controls the penalty mix and \(\lambda\) controls the amount of shrinkage.
A single set of folds can make a tuning setting look unusually good or poor simply because of which observations landed together in the validation folds. Repeated K-fold cross-validation makes this sensitivity visible: keep the observed data fixed, generate new random fold assignments, and repeat the full K-fold procedure. The purpose is to assess how results change with the partition, not to create new observations or new data sets.
Assume the observations are suitable for random K-fold splitting: in particular, individual rows can be assigned to folds without breaking a required grouping or time order. If observations are clustered, repeated within individuals, or time-ordered, ordinary random folds may be inappropriate. The folds must respect the data structure. Within each training fold, any preprocessing steps that estimate quantities from the data must be fitted using that training fold alone, then applied to its validation fold.
Calculate a Score for Each Repetition
Let \(L_{rk}(\alpha,\lambda)\) be the validation loss for fold \(k\) in repetition \(r\), at tuning setting \((\alpha,\lambda)\). If the folds have the same size, a repetition-level score is the mean of the \(K\) fold losses. If fold sizes differ and the goal is average loss per observation, weight each fold loss by its validation-set size \(n_{rk}\). Here \(n\) is the total number of observations.
For each fixed tuning setting, the repeated-CV summary is the mean of the \(R\) repetition-level scores. It is important to calculate a score within each repetition first: the folds within one repetition together form a complete cross-validation run.
The tuning setting with the smallest mean score is the setting with the lowest average validation loss across the repetitions. The loss function and the direction of preference matter: for losses such as mean squared error, smaller is better. Keep the same folds across candidate settings within a repetition when comparing those settings, so that their scores reflect the same partition.
Worked Example: Summarizing Partition Variation for One Setting
A hypothetical analyst evaluates one elastic net setting using 3-fold cross-validation repeated four times. The loss is mean squared error, so smaller values are preferred. Within each repetition, the three folds have equal size. The fold losses are:
- Repetition 1: \(0.18,\ 0.24,\ 0.21\)
- Repetition 2: \(0.20,\ 0.22,\ 0.24\)
- Repetition 3: \(0.19,\ 0.23,\ 0.21\)
- Repetition 4: \(0.22,\ 0.20,\ 0.24\)
Method. Average the fold losses within each repetition, then summarize the four repetition-level scores. This describes sensitivity to fold assignment for this observed data set.
For repetition 1, the score is \((0.18+0.24+0.21)/3=0.63/3=0.21\). For repetition 2, it is \((0.20+0.22+0.24)/3=0.66/3=0.22\). For repetition 3, it is \((0.19+0.23+0.21)/3=0.63/3=0.21\). For repetition 4, it is \((0.22+0.20+0.24)/3=0.66/3=0.22\).
The mean across repetitions is \((0.21+0.22+0.21+0.22)/4=0.86/4=0.215\). The range is \(0.22-0.21=0.01\). The sample standard deviation of the four repetition scores, as a descriptive measure, is found from deviations \(-0.005,\ 0.005,\ -0.005,\ 0.005\). Their squared deviations sum to \(4(0.005^2)=0.0001\), so the sample variance is \(0.0001/(4-1)=0.00003333\ldots\) and the sample standard deviation is approximately \(0.0058\).
Check. The four scores are all between the smallest and largest fold losses, \(0.18\) and \(0.24\), and their mean, \(0.215\), lies between the smallest and largest repetition scores, \(0.21\) and \(0.22\). The standard deviation is a descriptive summary of partition variation here, not an estimate of variability across newly sampled data sets.
Common mistake avoided: Do not treat the 12 fold losses as 12 independent repetition-level results. First combine the three folds into one score for each complete repetition.
Compare Tuning Choices Across Repetitions
Repeated cross-validation can show two related features of tuning: which setting has the lowest mean score, and how often each setting is selected as best within an individual repetition. These are different summaries. The overall winner by mean score need not be the most frequent repetition-level winner, especially when scores are close. A tie-breaking rule should be stated if two settings have the same score.
Worked Example: Comparing Three Elastic Net Settings
A hypothetical prediction study compares three settings, A, B, and C, using 2-fold cross-validation repeated three times. Each fold contains the same number of observations, and the loss is mean squared error. The fold losses are shown below.
| Repetition | Setting A fold losses | Setting B fold losses | Setting C fold losses |
|---|---|---|---|
| 1 | 0.12, 0.16 | 0.11, 0.15 | 0.15, 0.17 |
| 2 | 0.14, 0.16 | 0.13, 0.15 | 0.15, 0.17 |
| 3 | 0.11, 0.15 | 0.14, 0.18 | 0.13, 0.17 |
Method. Calculate each setting’s mean fold loss within each repetition, then calculate its mean across the three repetitions. For each repetition, also identify the setting with the lowest score.
In repetition 1, A scores \((0.12+0.16)/2=0.14\), B scores \((0.11+0.15)/2=0.13\), and C scores \((0.15+0.17)/2=0.16\). B is the repetition-level winner.
In repetition 2, A scores \((0.14+0.16)/2=0.15\), B scores \((0.13+0.15)/2=0.14\), and C scores \((0.15+0.17)/2=0.16\). B is again the winner.
In repetition 3, A scores \((0.11+0.15)/2=0.13\), B scores \((0.14+0.18)/2=0.16\), and C scores \((0.13+0.17)/2=0.15\). A is the winner.
Across repetitions, A has mean score \((0.14+0.15+0.13)/3=0.42/3=0.14\). B has mean score \((0.13+0.14+0.16)/3=0.43/3\approx0.1433\). C has mean score \((0.16+0.16+0.15)/3=0.47/3\approx0.1567\). Thus A has the lowest overall mean score. The repetition-level winners are B, B, and A: B wins two of three repetitions, while A wins one of three.
Check. The smallest of the three mean scores is \(0.14\), belonging to A. The most frequent repetition-level winner is B. The difference is not an arithmetic error; it shows why the average score and selection frequency should be reported as distinct summaries.
Common mistake avoided: Do not report “the most often selected setting” as though it must also minimize the average cross-validation score. State which summary is being used.
Why the Repetition-Level Average Is Appropriate
For a fixed tuning setting, each repetition produces one complete K-fold estimate of validation loss for the observed data. The mean across repetitions averages these estimates over the random partitions used. This explains why the repetition, rather than the individual fold, is the natural unit for describing how the full cross-validation result changes with fold assignment.
The equality follows by substituting the fold-weighted score for each repetition and distributing the factor \(1/R\) across the sum. Each repetition contributes total fold weight \(\sum_k n_{rk}/(Rn)=1/R\); across \(R\) repetitions, the weights sum to 1. Thus the repeated-CV mean is an average of complete partition-level scores. It does not turn the repeated validation predictions into observations from independent new data sets.
Even if the random-number generator uses a different seed for each repetition, the repetitions use the same observed cases. Their training sets overlap, and their validation losses are based on those same cases. Variation across repetition scores therefore describes sensitivity to the random fold assignment conditional on the data. It should not be presented as variation caused by drawing a new sample from the population.
Worked Example: Weighting Unequal Validation Folds
A hypothetical data set has 10 observations and is divided into three validation folds of sizes 4, 3, and 3. For one fixed elastic net setting, the fold mean squared errors are \(0.10,\ 0.20,\ 0.30\) in the first repetition and \(0.15,\ 0.15,\ 0.25\) in the second. Calculate the observation-weighted score for each repetition and their average.
Method. Weight each fold mean squared error by the fraction of observations in that fold. This makes each observation contribute equally to the repetition-level loss.
For repetition 1, the weighted score is \((4/10)(0.10)+(3/10)(0.20)+(3/10)(0.30)=0.04+0.06+0.09=0.19\).
For repetition 2, the weighted score is \((4/10)(0.15)+(3/10)(0.15)+(3/10)(0.25)=0.06+0.045+0.075=0.18\).
The mean across the two repetitions is \((0.19+0.18)/2=0.185\). As a check, the fold weights in each repetition sum to \(4/10+3/10+3/10=1\), and the two repetition-level scores lie within the ranges of their respective fold losses.
If the three fold losses in repetition 1 were instead averaged equally, the result would be \((0.10+0.20+0.30)/3=0.20\), not \(0.19\). Equal-fold averaging is appropriate when the fold sizes are equal; with unequal sizes, use observation weights when the target is average loss per observation.
Common mistake avoided: Do not assume that averaging fold losses equally always gives the observation-level cross-validation loss. Check fold sizes and use the weighting that matches the stated target.
What to Report—and What Not to Claim
For each candidate tuning setting, useful summaries include the mean repetition-level score and a descriptive measure of its variation across repetitions, such as the range or standard deviation. For a tuning search, also report the selected setting by mean score and, when helpful, the frequency with which each setting wins within a repetition. These summaries answer practical questions about partition sensitivity and selection stability; they are not all the same question.
The standard deviation across repetitions describes how much the score changed under the fold assignments actually tried. It is not a standard error for performance on a new sample. In particular, dividing this standard deviation by \(\sqrt{R}\) and interpreting the result as a generalization-performance standard error would require an appropriate independent sampling argument that repeated partitions of the same data do not provide.
Repeated cross-validation also does not remove all sources of variation or bias. It does not add information to the data, and a small spread across repetitions does not prove that performance would be stable on new data. It only indicates that, for the observed data and the chosen validation design, the calculated scores changed little across the partitions tried. Keep the loss function, fold design, preprocessing procedure, and tuning grid fixed when the aim is to assess fold-assignment variation.
Check Your Understanding
For each problem, treat the stated scores as validation losses, so smaller is better. Distinguish repetition-level summaries from fold-level values.
- A setting is evaluated with 2-fold cross-validation twice. The fold losses are \(0.12,\ 0.20\) in the first repetition and \(0.14,\ 0.18\) in the second. Find each repetition score, their mean, and their range. (Answer: Scores \(0.16\) and \(0.16\); mean \(0.16\); range \(0\).)
- A fixed setting has repetition scores \(0.21,\ 0.18,\ 0.24\). Calculate its mean score and range, and state what the range describes. (Answer: Mean \(0.21\); range \(0.06\); the observed score variation across the fold assignments tried for the same data.)
- Two settings have repetition scores X: \(0.10,\ 0.14\) and Y: \(0.11,\ 0.12\). Which has the lower mean score, and how often does each win across the two repetitions? (Answer: X mean \(0.12\), Y mean \(0.115\), so Y has the lower mean; X wins once and Y wins once.)
- In one repetition, validation-fold sizes are 2, 3, and 5, with losses \(0.10,\ 0.20,\ 0.30\), respectively. Find the observation-weighted score. (Answer: \((2/10)(0.10)+(3/10)(0.20)+(5/10)(0.30)=0.02+0.06+0.15=0.23\).)
- An analyst proposes dividing the standard deviation of repetition-level scores by \(\sqrt{R}\) and calling the result a standard error for performance on new data. What is the key flaw? (Answer: The repetitions reuse the same observed cases and describe fold-assignment variation, not independent new data samples; that calculation does not by itself justify a new-data performance standard error.)