Uncertainty In Held-Out Performance Estimates

Why a Held-Out Score Needs an Uncertainty Interval

A held-out test set gives an estimate of how a fixed prediction pipeline performs on new units from a target population. Its score is not usually the population performance itself: a different sample of subjects could produce a different accuracy, area under the ROC curve, or other metric. An uncertainty interval describes this test-sample variability. It does not measure every source of uncertainty in a prediction study, and it cannot correct a test set that has already been reused to choose a model or threshold.

In multi-omics studies, the row used for analysis is not always the independent unit. A subject may contribute multiple specimens, visits, or technical measurements; subjects may also be grouped within cohorts or sites. The interval must reflect the sampling structure and the population to which performance is meant to generalize. Resampling individual rows when those rows are correlated can make an interval substantially too narrow.

What the Interval Conditions On

For a basic held-out bootstrap, regard the prediction pipeline as fixed. The observed test units are treated as a sample from the target population, and each bootstrap replicate is a new sample of the same size drawn with replacement from those units. Calculate the metric anew in every replicate. For a metric such as accuracy, this means recalculating the fraction correct; for AUC, it means recalculating AUC from the resampled predictions and outcomes.

This procedure estimates variation from which independent test units happened to be observed, conditional on the fitted pipeline and the chosen evaluation design. It does not, by itself, include variation from fitting a different training set, selecting different features, tuning different hyperparameters, or choosing a different threshold. Those questions require an evaluation procedure that repeats the relevant training and selection steps. A held-out interval also does not remove bias from test-set reuse or guarantee performance under a shifted population.

The usual bootstrap interpretation relies on a representative sample of independent units from the target population, a metric defined in advance, and enough units for the resampled distribution to approximate the sampling distribution. The unit-level bootstrap preserves all records belonging to a sampled unit. If the test design fixed the number of cases and controls, a stratified bootstrap may instead resample within outcome groups; its inference is conditional on that design. The resampling scheme should match the sampling process and the performance target.

Constructing a Unit-Level Bootstrap Interval

A 95% percentile interval is a procedure intended, under its assumptions, to cover the target performance in repeated samples about 95% of the time. It does not mean there is a 95% probability that a fixed population performance lies inside this particular interval. The empirical endpoints also have Monte Carlo error when \(B\) is finite, so use enough replicates for stable quantiles and check that repeated runs give similar results.

Worked Examples

Why Observation-Level Intervals Can Be Too Narrow

The repeated-record example has a general explanation. Let there be \(G\) independent clusters, each with \(m\) observations, and let \(Y_{gj}\) be a binary indicator that observation \(j\) in cluster \(g\) is correctly classified. Suppose each indicator has variance \(p(1-p)\), and any two observations within a cluster have correlation \(\rho\). The pooled accuracy is the mean of \(Gm\) indicators. Its variance is

The factor \(1+(m-1)\rho\) is the variance inflation relative to treating all \(Gm\) records as independent. If \(\rho>0\), the observation-level variance is too small; if \(\rho=1\), the inflation factor is \(m\), as in the repeated-record example. A unit-level bootstrap retains within-cluster dependence rather than silently setting it to zero.

Diagnostics, Alternatives, and Limitations

Before interpreting an interval, check whether the resampling unit matches the intended generalization. If the goal is performance on new subjects from the same sites, subject-level resampling may be appropriate, while keeping site membership intact if that is part of the sampling structure. If the goal is performance on new sites, resampling subjects within the observed sites does not represent site-to-site variation; the site or cohort is the relevant unit, and a small number of sites limits what can be learned. These choices are about the target population, not merely computational convenience.

Inspect the bootstrap distribution, not only its endpoints. Repeated values, skewness, or extreme replicates can occur, especially with small samples or bounded metrics. For AUC or other metrics requiring both outcome classes, a replicate that contains only one class cannot produce a defined value; frequent undefined replicates indicate that the test sample is too small for the chosen ordinary bootstrap to give a dependable interval. Do not silently discard such replicates without explaining how that changes the procedure. A stratified resampling design may be justified when class counts were fixed, but it answers a conditional question.

Alternatives include analytic intervals for simple metrics under suitable independent-sampling assumptions, and model-based variance estimates that explicitly represent clustering. These can be useful when their assumptions fit the design; they are not automatic repairs for dependence. The bootstrap is flexible for complex metrics, but it is not reliable simply because it can be run. In particular, few independent units, strong population shift, test-set reuse, or a metric that changes with threshold selection can undermine interpretation. Comparing two models on the same held-out subjects calls for accounting for their paired predictions; that is a separate question from estimating uncertainty for one model's score.

Common Pitfall