Uncertainty in Cross-Validation Estimates

Why Variation in Cross-Validation Scores Matters

A cross-validation estimate is not a single, perfectly certain description of a model’s predictive performance. In \(K\)-fold cross-validation, the model is fitted several times, with a different fold held out for evaluation each time. The resulting fold scores can differ. If the scores are similar, the overall evaluation may be relatively stable across those particular partitions; if they differ substantially, the reported average can conceal sensitivity to which observations were held out.

That variation deserves attention, but it is easy to overinterpret. The folds are not independent replications: their training sets overlap, and the observations and fitted models are connected across folds. Consequently, the standard deviation of the fold scores divided by \(\sqrt{K}\) is generally not a sound standard error for the cross-validation estimate. It can be a useful descriptive calculation, but it does not automatically quantify uncertainty about performance in a new population.

Summarizing Fold-Score Variation

For a lower-is-better error measure, such as mean absolute error (MAE), smaller scores indicate better predictions. A basic summary of \(K\) fold scores is their mean, range, and sample standard deviation. These provide different information: the mean gives the central score, the range shows the distance from the smallest to the largest score, and the standard deviation summarizes their spread around the mean.

With equal-sized folds and an additive measure such as MAE, the ordinary mean of the fold MAEs is also the pooled MAE across all held-out observations. With unequal fold sizes, a size-weighted mean is needed to obtain that pooled MAE. For a nonlinear measure such as root mean squared error (RMSE), averaging fold RMSEs generally does not equal the RMSE calculated from all held-out prediction errors pooled together. State which summary is being reported rather than treating these rules as interchangeable.

The standard deviation of the fold scores is descriptive. It answers, “How spread out were these scores in this run?” It does not, without further justification, answer, “How precisely have we estimated the model’s performance for a new population?” The latter question concerns sampling and the evaluation procedure more broadly.

Why Fold Scores Are Dependent

In one \(K\)-fold run, each observation is held out once, but each fitted model is trained on most of the same data. For example, the model evaluated on fold 1 and the model evaluated on fold 2 both use observations from folds 3 through \(K\) for training. The held-out folds are disjoint, but the training sets are not. A shared training set can make the fitted models, and therefore their scores, related.

The overlap is not always an identical fraction if fold sizes differ. If there are five equal-sized folds, training sets that omit two different folds share three of the five folds’ observations; this is three-quarters of either training set. With unequal fold sizes, the shared fraction depends on the sizes of the omitted folds, so it need not be three-quarters.

Repeated cross-validation changes the partitions and lets us see how the summary varies when the splits change. It does not create new independent data sets: the same observed cases are reused across repetitions. Likewise, repeated random holdout evaluations on the same data share observations and often share training cases. Repetition can reveal sensitivity to random partitioning, but the repeat scores are not automatically independent population-level replications.

Why the Naive Standard Error Can Mislead

If \(K\) scores were independent observations with a common variance, the standard error of their mean would be estimated by \(s_{\text{fold}}/\sqrt{K}\). That familiar formula relies on independence. To see what changes when scores are dependent, let \(S_1,\ldots,S_K\) be fold scores with common variance \(\sigma^2\), and suppose each pair has the same covariance \(c\). The variance of their mean is

where \(\rho=c/\sigma^2\) is the common correlation, assuming \(\sigma^2>0\). The independence formula corresponds to \(c=0\), or \(\rho=0\). When the scores are positively correlated, the variance of their mean is larger than the independent-score formula suggests. The actual dependence pattern in cross-validation is not usually known well enough to replace the independence formula with this simplified expression as a routine correction. The derivation shows why independence matters; it does not provide a universal cross-validation standard error.

There is a second limitation: fold scores are produced by particular training and test subsets of one observed data set. Their spread reflects differences among those subsets and fitted models in that run. It is not a direct measure of all uncertainty that would arise from collecting a new sample from the target population. A narrow spread therefore does not guarantee a precise estimate of future performance, and a wide spread does not by itself identify the reason for the instability.

Variation Across Repeats

A repeated procedure can be summarized by its score in each repetition. Variation among those repeat-level scores describes sensitivity to the random partitions used on the same data. It is often more useful to report that variation transparently than to attach an unjustified confidence interval to the overall average.

The meaning of a “repeat score” should be clear. For repeated \(K\)-fold cross-validation, it is commonly the summary across the \(K\) folds within one complete repetition. The repeat-level summaries can then be compared. Because every repetition reuses the same observations, their variation is still not equivalent to variation across independent samples from the population.

How to Report the Uncertainty Honestly

A clear report separates the estimate from the evidence about its stability. It should identify the performance measure and units, state the cross-validation scheme and number of folds or repetitions, give the summary score, and describe the observed spread in scores. If a naive standard error is included, label it as independence-based and explain that the fold scores are dependent; do not treat it as a conventional standard error without a suitable justification.

For instance, a report might say: “The five-fold cross-validated MAE was 6.0 outcome units, with fold MAEs from 4 to 8 units. The fold scores varied across partitions and are dependent, so their standard deviation divided by the square root of five is not interpreted as a standard error.” This wording communicates both the point estimate and an important limit on its precision.

Variation does not itself say why scores differ. Differences may reflect which observations happened to be held out, changes in fitted models, or features of the evaluation data. The scores are a prompt to assess stability, not a diagnosis of its cause. A small spread is reassuring only about the variation represented by the partitions examined; it cannot rule out other forms of uncertainty.