Why Quantify Uncertainty in a Validation Metric?
A validation metric is an estimate from a finite sample, not a fixed description of how a prediction method will perform in every population. For example, two cohorts drawn from the same target population could produce different Brier scores because they contain different subjects and event histories. A bootstrap confidence interval summarizes this sampling uncertainty under stated assumptions.
The target must be clear. If a fitted joint model is evaluated on an independent validation cohort, resampling that cohort estimates uncertainty in the metric for that cohort’s target population, conditional on the fitted model. It does not, by itself, account for variation from training the model. If the validation observations were also used to fit or select the model, resampling only those observations does not remove the resulting optimism.
The Resampling Procedure
Let \(n\) be the number of independent subjects in a validation cohort, and let \(\hat\theta\) denote a metric calculated from those subjects, such as a horizon-specific Brier score. In bootstrap replicate \(b\), sample \(n\) subject indices with replacement from \(1,\ldots,n\), and calculate the metric \(\hat\theta^{*(b)}\) on that resample. Repeat this \(B\) times. A percentile interval uses the empirical quantiles of the resulting \(B\) metric values.
Here \(Z_i\) denotes the complete record for subject \(i\), and \(T\) is the chosen metric estimator. Resampling records rather than individual visits is essential: measurements from the same person are dependent, and the event history and longitudinal trajectory belong together. If the sampling design has clusters, such as centers, the resampling unit may instead need to be the independently sampled cluster.
A \(100(1-\alpha)\%\) percentile interval is the interval between the \(\alpha/2\) and \(1-\alpha/2\) quantiles of the bootstrap estimates. The usual percentile method is straightforward, but it can perform poorly with small samples, strong skewness, boundary values, or unstable metric estimators. The interval is not a probability statement about a fixed parameter after the data have been observed.
Why Resampling Approximates Sampling Uncertainty
The bootstrap treats the observed cohort as an estimate of the population distribution and samples from its empirical distribution. The following variance argument illustrates the idea for a sample mean; for a complex prediction metric, the same resampling principle is applied to the whole estimator rather than relying on this mean-specific formula.
The asterisk indicates variation conditional on the observed data. The empirical distribution assigns probability \(1/n\) to each observed value, so a bootstrap sample varies because some observed subjects appear more than once and others are omitted. Its variability approximates the sampling variability that would arise from drawing a new cohort. For survival metrics with censoring weights or model-dependent predictions, the full estimator must be recalculated in each replicate.
Worked Example: A Percentile Interval for a Horizon-Specific Brier Score
Suppose a hypothetical independent validation cohort has \(n=120\) subjects, complete event-status ascertainment through a prespecified prediction horizon, and a fixed fitted joint model. At that horizon, the observed event indicator is \(Y_i\), and the model’s predicted event risk is \(\hat r_i\). The Brier score is the mean squared prediction error, \(n^{-1}\sum_i(Y_i-\hat r_i)^2\). The observed score is \(0.142\).
The analyst resamples 120 subjects with replacement, retaining each subject’s outcome and prediction together, and recalculates the score in each of \(B=2{,}000\) replicates. Suppose the sorted bootstrap scores give empirical 2.5th and 97.5th percentiles of \(0.119\) and \(0.169\). The percentile interval is therefore
The observed Brier score is \(0.142\), on the squared-probability scale, and the estimated 95% bootstrap interval is \(0.119\) to \(0.169\). Under the subject-sampling and fixed-model interpretation, the interval describes uncertainty in the population Brier score for comparable subjects. It does not mean that there is a 95% probability that the fixed population score lies in this particular interval.
Paired Bootstrap Differences Between Models
When two models are evaluated on the same validation subjects, their metric estimates are dependent. The natural bootstrap comparison resamples subjects once per replicate and calculates both model metrics on that same resample. The replicate difference is then the difference between those two values. This preserves the pairing and the within-subject relationship between the models’ predictions and outcomes.
The percentile limits of \(\hat\Delta^{*(1)},\ldots,\hat\Delta^{*(B)}\) give an interval for the difference. Its direction depends on the metric: for a loss such as the Brier score, a negative value of \(\hat\Delta\) favors model A; for a metric where larger is better, the interpretation reverses. This is an interval for the difference in the specified validation metric, not a general claim that one model is superior in all respects.
Worked Example: A Paired Difference in Brier Scores
In a hypothetical validation cohort, model A has an observed horizon-specific Brier score of \(0.142\), and model B has a score of \(0.151\). Define the difference as A minus B:
Thus, model A’s observed average squared prediction error is lower by \(0.009\) squared-probability units. The analyst draws 120 subjects with replacement in each replicate and calculates both models’ scores on the identical resampled subjects. From \(B=2{,}000\) paired replicate differences, suppose the 2.5th and 97.5th percentiles are \(-0.021\) and \(0.003\). The estimated 95% percentile interval is \([-0.021,0.003]\) squared-probability units.
Because the interval includes zero, these data do not establish a difference in either direction at the corresponding two-sided 5% level under this interval-based procedure. This does not show that the models are equivalent. The paired construction matters: drawing independent bootstrap samples for A and B would discard the shared-subject dependence and generally give the wrong uncertainty for their difference.
Censoring and Recalculation Within Replicates
For survival outcomes, a prediction metric at a horizon may require inverse-probability-of-censoring weights (IPCW). For example, an IPCW Brier score uses observed event and event-free contributions weighted according to the estimated probability of remaining uncensored. Such a metric depends not only on the sampled subjects and their predictions, but also on the estimated censoring distribution.
With independent subjects and independent censoring (possibly conditional on covariates when the weighting model accounts for them), bootstrap resampling should keep each subject’s event or censoring time, event indicator, longitudinal data, covariates, and predictions together. In each replicate, re-estimate the censoring distribution and recalculate the weights and metric. Holding weights estimated from the original cohort fixed omits one source of sampling variation. The censoring model and metric must also be defined so the required weights are estimable; for example, the probability of remaining uncensored at the horizon must not be effectively zero in relevant parts of the target population.
Worked Example: Recomputing an IPCW Score
Consider a hypothetical evaluation at \(t=2\) years. In one bootstrap replicate, suppose three sampled subject records contribute to an IPCW Brier-score numerator as follows: an observed event before \(t\) has squared prediction error \(0.04\) and censoring-survival estimate \(0.80\); a subject known to be event-free at \(t\) has squared error \(0.09\) and estimate \(0.75\); and a subject censored before \(t\) contributes zero under this estimator. The replicate score is
The arithmetic uses this replicate’s censoring-survival estimates. In an actual bootstrap analysis, those estimates would be recalculated from the full resampled cohort, not inferred only from these three illustrative contributions. This example shows why the metric calculation is repeated after resampling: the score depends on both prediction errors and censoring weights. If a replicate has no usable event information or produces extreme weights, it should be flagged and investigated rather than silently treated as an ordinary stable estimate.
Assumptions, Diagnostics, and Alternatives
A useful bootstrap interval requires the resampling scheme to reflect how the validation subjects were sampled. The basic subject bootstrap assumes subjects are independent and representative of the target population; it does not fix selection bias, distribution shift, or an incorrectly specified prediction target. The metric must be sufficiently estimable in resamples, and censoring assumptions required by an IPCW estimator still apply. A bootstrap cannot compensate for inadequate follow-up or a near absence of observed outcomes at the prediction horizon.
Inspect the bootstrap distribution, not only its endpoints. Check whether replicates fail, whether a small number of replicates have extreme values, whether interval limits change materially when \(B\) is increased, and whether the distribution is markedly skewed or piled up at a boundary. For weighted metrics, examine the weights and their effective contribution. When many replicates are undefined, report the problem and reconsider whether the metric and available cohort support the intended inference; discarding failed replicates without explanation can distort the interval.
The percentile interval is not the only option. A basic interval reflects bootstrap quantiles around the observed estimate, while bias-corrected and accelerated (BCa) intervals adjust for estimated bias and skewness under their regularity conditions. A bootstrap standard error can also be reported, but converting it to a symmetric interval may be unsuitable for bounded or highly skewed metrics. No interval method cures nonidentifiability or unstable censoring weights.
The bootstrap uncertainty question also differs from the bootstrap optimism correction in the earlier material. An optimism correction estimates and adjusts apparent performance after model fitting; a validation bootstrap interval quantifies sampling uncertainty for a specified metric and validation design. If uncertainty from model development is part of the target, the development process must be represented, typically by resampling development subjects and repeating the full fitting and selection procedure. A later cross-validation strategy addresses a different resampling design and is not interchangeable with this fixed-validation-cohort interval.
Common Pitfalls
- Resampling visits instead of subjects. This breaks within-subject dependence and can make intervals too narrow. Resample the independent sampling units and retain their complete records.
- Bootstrapping the two models separately. Separate resamples fail to preserve paired predictions on the same subjects. Use common subject indices for both metrics in each replicate.
- Calling an interval a model-development interval. Resampling only an independent validation cohort conditions on the fitted model. Include model fitting in the resampling procedure only when that uncertainty is part of the target.
- Keeping estimated censoring weights fixed. For an IPCW estimator, recompute the censoring model and weights within each replicate if their estimation uncertainty is to be represented.
- Interpreting inclusion of zero as proof of equality. An interval containing zero indicates insufficient evidence of a nonzero difference at that level under the stated procedure; it does not establish equivalence.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Why should a bootstrap sample retain all longitudinal measurements and the event record for a selected subject?
- Two models are evaluated on the same subjects. What should be resampled to form an interval for their metric difference?
- An IPCW metric uses an estimated censoring distribution. What should be done with that estimate in each bootstrap replicate?
- A paired 95% interval for model A minus model B’s Brier score is \([-0.018,0.004]\). What does its inclusion of zero imply, and what does it not imply?
- A validation-cohort bootstrap is performed after fitting a model on a separate development cohort. Which uncertainty does the interval represent?
Solutions. (1) The subject is the independent sampling unit, and the repeated measurements and event record are dependent components of that subject’s record. (2) Resample subjects once per replicate and calculate both models’ metrics on the same sampled subjects before taking their difference. (3) Re-estimate the censoring distribution and recalculate the weights and metric in each replicate, subject to the censoring assumptions. (4) The interval does not establish a nonzero difference at the corresponding two-sided level; it does not prove equality or equivalence. (5) It represents validation-cohort sampling uncertainty conditional on the fitted model, not uncertainty from fitting that model.