Uncertainty Intervals for Cross-Validated Performance

What Does an Uncertainty Interval Describe?

A cross-validated loss is an estimate, not a fixed property of a fitted model. Its value depends on the observed sample and on how that sample is divided into folds. An uncertainty interval is useful only after specifying which source of uncertainty it is intended to describe. In genomic prediction, the independent sampling unit might be an individual, a family, or a site, depending on the study design. Treating related individuals as independent can make an interval too narrow even if the cross-validation code is otherwise correct.

Let \(D\) denote the observed data, and let \(\hat L_{\mathrm{CV}}(D)\) denote the cross-validated loss computed using a validation design chosen in advance. The design includes the number and construction of folds, the loss function, any stratification or grouping, and whether hyperparameters are selected within each training set. The earlier material on nested cross-validation establishes why tuning must be repeated inside the training data when the reported performance is intended to account for tuning.

This distinction matters because \(K\)-fold cross-validation typically trains each model on about \((K-1)n/K\) observations, not on all \(n\) observations. Its performance estimate therefore most directly concerns the learning procedure at that smaller training size. The final model fitted to all available data may perform differently. The interval also does not automatically cover distribution shifts, such as a change in the genomic cohort or clinical setting.

Why Fold-Based Standard Errors Are Usually Misleading

A tempting calculation is to find the standard deviation of the \(K\) fold losses and divide it by \(\sqrt K\). That formula is a standard error for the mean of \(K\) independent, identically distributed observations with finite variance. Fold losses generally do not satisfy that assumption. Training sets overlap heavily: in ordinary \(K\)-fold cross-validation, two fitted models share much of their training data. Test folds also come from the same original sample, and the fitted models can respond similarly to changes in that sample. Thus the fold losses are dependent, and \(K\) is not the number of independent data sets.

Unequal fold sizes, stratification, clustering, and an irregular loss distribution can add further complications. Even when folds have equal size, a small standard deviation across folds does not establish that the estimated loss would be stable in a new sample. Conversely, a large fold-to-fold spread can reflect which observations happened to be held out, rather than the full sampling uncertainty of interest.

A Resampling Interval That Repeats the Analysis

A more defensible approach is to resample the independent sampling units and repeat the entire validation procedure in each resample. If individuals are independent, resample individuals. If related individuals or repeated observations form clusters, resample the appropriate clusters instead. The procedure must preserve the original evaluation design: for example, keep families or sites together when the intended validation split keeps them together.

In each bootstrap replicate, rerun all steps that are part of the learning procedure, including fold-local preprocessing, feature selection, and inner-loop tuning where applicable. Reusing the original selected features or tuned hyperparameters would omit their contribution to uncertainty and can make the interval too optimistic. The bootstrap should also reproduce the planned outer-fold design. Where the evaluation target requires stratification or group-aware folds, apply those rules in each replicate.

A practical complication is that a bootstrap sample can contain an original individual more than once. Those copies must not be split between training and test folds: otherwise the same person can appear on both sides of a split, creating information leakage. Keep all copies of an original individual in the same fold, and account for their bootstrap multiplicity when computing the replicate loss. If the data are clustered, apply the same principle at the cluster level. A prespecified plan is also needed for replicates in which the desired folds cannot be formed, such as a class being absent from a training fold; silently dropping such replicates changes the resampling distribution.

Justification and Interpretation

The logic of the bootstrap is to approximate how the estimator would vary across repeated samples from the target population. The empirical distribution of the observed independent units is used as a stand-in for that population. Resampling from it creates datasets with variation in which units are present and how often they appear. Rerunning the complete analysis maps that data variation through the same procedure used to calculate the reported estimate.

This justification is approximate, not an exact coverage theorem for every cross-validation estimator. It relies on the empirical distribution adequately representing the target population and on the estimator behaving regularly enough for resampling to approximate its sampling distribution. Small samples, rare outcomes, highly unstable feature selection, or few independent clusters can undermine that approximation. Increasing the number of bootstrap replicates reduces Monte Carlo error in estimated quantiles, but it does not create more independent information or repair a poor design.

Diagnostics, Alternatives, and Limitations

Before reporting an interval, check whether the resampling procedure actually matches the scientific sampling process. Examine whether replicates preserve family, site, or other dependence; whether every replicate can form valid folds; and whether the entire tuned pipeline was rerun. Inspect the bootstrap distribution for skewness, extreme values, or multiple modes. These features may signal a genuinely unstable analysis, sparse outcomes, or a resampling scheme that is poorly suited to the data. Report the number of failed or modified replicates and how they were handled.

The main alternatives answer different questions. A fold-based standard error is simple but generally lacks a sound independence justification. Repeated cross-validation summarizes sensitivity to random splits conditional on the observed sample. A bootstrap that reruns cross-validation targets sampling variation under its resampling assumptions, while an external validation sample directly assesses performance in that sample and population but may itself have substantial sampling uncertainty. None of these approaches automatically accounts for population shift or repairs leakage.

A common mistake is to treat the \(K\) fold losses as \(K\) independent observations and report their standard deviation divided by \(\sqrt K\) as the standard error. The correction is to recognize that folds share training data and come from one dataset. Define the uncertainty target, resample the actual independent units, and rerun the full procedure when sampling uncertainty is the goal. If there are very few independent clusters or the bootstrap distribution is erratic, report that limitation rather than implying precise coverage.