Repeated K-Fold Cross-Validation

Why Repeat K-Fold Cross-Validation?

In K-fold cross-validation, each observation is used for validation once: the data are divided into \(K\) folds, and a model is fitted \(K\) times, each time leaving out a different fold. The result is a performance estimate based on predictions for all observations. But a particular random assignment to folds is only one way to divide the data. Another assignment can produce somewhat different training sets and validation scores.

Repeated K-fold cross-validation runs the K-fold procedure several times, using a new random fold assignment for each run. It provides a way to summarize how the estimated performance changes across those assignments. Repeating the procedure does not add new observations: every repetition reuses the same dataset.

Conditions for Interpreting the Estimate

Before calculating or interpreting a cross-validation score, specify what prediction task it is meant to represent. Suppose the data are \((x_i,y_i)\), where \(x_i\) denotes the predictor information for observation \(i\) and \(y_i\) its outcome. The prediction procedure is the complete rule that uses training data to produce predictions. The target is its expected performance on new observations from a specified population, when trained on data of roughly the size used in each fold.

The usual interpretation relies on several conditions. The observed cases should be suitable representatives of the target population, and the way cases are assigned to folds should respect the study design. For example, if several rows belong to the same person, assigning that person's rows to different folds may not represent prediction for a new person. The validation cases must not contribute information to fitting the model or other data-dependent parts of the prediction procedure. Finally, the procedure and performance measure should be defined consistently across folds and repetitions.

These conditions are not guaranteed just because software reports a cross-validation score. They are part of deciding whether the score answers the intended question. Repetition changes the random partition, not the target population or the underlying dataset.

Calculating and Combining Scores

Let \(F_{rk}\) be the set of observations in fold \(k\) of repetition \(r\), where \(r=1,\ldots,R\) and \(k=1,\ldots,K\). Write \(\hat{y}_i^{(-r,k)}\) for the prediction for observation \(i\) from a model fitted without fold \(F_{rk}\). A loss function \(\ell(y,\hat y)\) measures the error of a prediction. For example, squared-error loss is \((y-\hat y)^2\), and absolute-error loss is \(|y-\hat y|\).

The mean loss for fold \(k\) of repetition \(r\) is the average loss over that fold. To combine the folds into a repetition-level score, weight each fold's mean loss by its fraction of the observations. This gives each validation observation equal weight, including when fold sizes differ.

Here \(M_r\) is the cross-validation score for repetition \(r\). The overall repeated K-fold estimate is the mean of those scores:

For a loss such as mean squared error (MSE) or mean absolute error (MAE), smaller scores indicate better predictive performance. Their units differ: MSE has the square of the outcome's units, whereas MAE has the outcome's units. If the desired measure is root mean squared error (RMSE), take the square root of the relevant mean squared error. Averaging fold-level RMSE values is generally not the same as first combining squared errors and then taking the square root.

Why Weighting Matters When Fold Sizes Differ

Fold sizes may differ by one observation when \(n\) is not divisible by \(K\). In that case, the unweighted mean of fold scores gives a small fold the same influence as a larger fold. The weighted formula instead equals the mean loss over all validation observations in that repetition. To see why, substitute the fold score definition:

In this derivation, \(\ell_i\) is the validation loss for observation \(i\) under the model fitted without its fold in repetition \(r\). The final equality holds because the folds partition the data: each observation appears in exactly one validation fold during that repetition. The predictions, and therefore the losses, can differ from one repetition to another, but each observation contributes once per repetition.

What Variation Across Repetitions Means

A useful summary can include both the overall mean score and the observed spread of the repetition-level scores. A sample standard deviation across the \(R\) scores describes how much those scores varied across the random partitions that were run. It is a descriptive measure of split-to-split variation for this dataset and this procedure.

Those scores are not results from independent replications of a study. Every repetition uses the same observed cases, and the fitted models and validation predictions overlap substantially across partitions. Consequently, the variation across repetitions should not automatically be treated as the standard error from \(R\) independent datasets, or used by itself to form a confidence interval for population performance. Repeating K-fold cross-validation can show sensitivity to fold assignment; it cannot replace collecting new independent data.

Common Mistake: Mistaking Repeated Splits for New Data

A common mistake is to say, “I ran ten repetitions, so I have ten independent estimates of model performance.” The repetitions do give ten scores, but each score reuses the same observations. A different random partition changes which observations are grouped together, not which people or cases were sampled. The mean can be a more stable summary across the partitions tried, but the number of repetitions is not the number of independent samples.

Another mistake is to average fold scores without checking fold sizes. Use a plain mean only when the folds have equal sizes; otherwise, weight each fold score by its size divided by \(n\). In either case, report the performance measure and its units, the number of folds and repetitions, and the mean score. If reporting the spread across repetitions, label it as variation across random partitions rather than as uncertainty from independent replication.