Why Prediction Metrics Need Uncertainty Intervals
A validation AUC, calibration estimate, or prediction score is computed from a finite set of subjects. A different sample from the same target population would generally produce a different value. Reporting only the observed metric can therefore suggest more precision than the data support. An uncertainty interval summarizes the sampling variability of the metric under a stated target and set of assumptions.
For joint-model predictions, a subject’s prediction must remain attached to that subject’s outcome and censoring history. Resampling a list of predictions separately from a list of outcomes destroys the observed pairing and no longer estimates the sampling variability of the metric calculated from paired data. The central operation here is to resample whole subjects.
The interval described here typically targets the performance of a prediction rule treated as fixed, evaluated in a population represented by the validation subjects. It captures uncertainty from having a finite validation sample. It does not, by itself, account for uncertainty from estimating the joint model or from choosing the prediction rule. If the intended target includes model-fitting uncertainty, that fitting procedure must be included in the resampling design. This is a different inferential target, not a reason to break subject-level pairing.
Resampling and Constructing an Interval
Write \(\hat\theta\) for the metric calculated from the original validation sample. Examples include a time-dependent AUC at a specified horizon, a calibration measure, or a Brier score. For each bootstrap replicate \(b=1,\ldots,B\), draw \(n\) subjects with replacement and calculate the same metric, using the same definition and evaluation horizon, to obtain \(\hat\theta^{*(b)}\). A basic percentile interval uses the empirical quantiles of these \(B\) values.
Here \(\hat q^{*}_{u}\) is the empirical \(u\)-quantile of the bootstrap estimates. For a nominal 95% interval, the endpoints are the 2.5th and 97.5th percentiles. The confidence level describes the long-run coverage of the interval procedure under its assumptions; it is not the probability that a fixed parameter lies in this particular computed interval.
Name the metric, the prediction horizon, the population or validation setting, and the estimator used. For censored outcomes, specify the censoring-aware version of the metric.
Draw \(n\) subject records with replacement. Keep each record’s prediction, outcome, censoring time, event indicator, and other required information together.
Apply the original estimator to that bootstrap sample. Re-estimate nuisance quantities, such as a censoring distribution, when they are part of the metric estimator.
Repeat for a sufficiently large \(B\), inspect the resulting distribution, and report the point estimate, interval, horizon, and resampling unit.
The justification is that the empirical distribution of the observed subject records approximates the population distribution. Drawing from that empirical distribution mimics repeated samples of subjects, and recalculating the metric in each sample approximates the metric’s sampling distribution. This approximation is most credible when subjects are independent and representative of the target sampling process, the metric is estimable over the relevant samples, and the sample is large enough for its sampling distribution to be adequately represented.
Worked Examples
Worked Example: A Bootstrap Replicate for a Brier Score
Suppose six hypothetical subjects have known event status at a fixed horizon. Their predicted risks and event indicators are shown below. Here \(Y_i=1\) means the event occurred by the horizon, and \(Y_i=0\) means it did not.
| Subject | Predicted risk | \(Y_i\) |
|---|---|---|
| 1 | 0.10 | 0 |
| 2 | 0.40 | 1 |
| 3 | 0.30 | 0 |
| 4 | 0.80 | 1 |
| 5 | 0.60 | 0 |
| 6 | 0.20 | 0 |
With fully observed horizon outcomes, the Brier score is the mean squared prediction error, \(n^{-1}\sum_i(Y_i-\hat p_i)^2\). The six squared errors are \(0.0100, 0.3600, 0.0900, 0.0400, 0.3600,\) and \(0.0400\). Thus the original score is
Consider the hypothetical bootstrap draw with subject indices \(1,2,2,4,5,5\). Subjects 2 and 5 appear twice, and subjects 3 and 6 do not appear. The corresponding squared errors are \(0.0100,0.3600,0.3600,0.0400,0.3600,0.3600\), so this replicate gives
This single replicate is not an interval. Repeating the draw many times produces a bootstrap distribution. The example assumes outcomes are known at the horizon; it would be incorrect to treat a subject censored before the horizon as a known non-event.
Worked Example: Preserving Pairs When Bootstrapping AUC
Consider six hypothetical subjects with fully observed binary horizon outcomes. Three experienced the event, with predicted risks \(0.80, 0.60,\) and \(0.40\); three did not, with risks \(0.70, 0.30,\) and \(0.20\). The empirical AUC is the proportion of event/non-event pairs for which the event subject has a higher risk, counting a tie as one-half. There are \(3\times3=9\) pairs. The event risk \(0.80\) exceeds all three non-event risks; \(0.60\) exceeds two; and \(0.40\) exceeds two. Therefore
Now take the hypothetical bootstrap draw with indices \(1,1,3,4,4,6\), where subjects 1 and 3 are event cases and subjects 4 and 6 are non-cases. The resample contains three case records with risks \(0.80,0.80,0.40\) and three control records with risks \(0.70,0.70,0.20\). The two \(0.80\) cases each beat two controls, and the \(0.40\) case beats one. Thus this replicate’s AUC is \(5/9=0.5556\). Repeated records count with their multiplicities.
The resampling unit is the subject record, not the event or non-event score alone. With censored survival outcomes, use the selected time-dependent AUC estimator, including its censoring adjustments, within each replicate; this simple pair-count calculation applies only when horizon status is fully observed.
Worked Example: An Interval for Calibration Error With Censoring
In a hypothetical validation sample of 240 subjects, suppose a censoring-aware estimator at \(t_0=2\) years gives an observed event-risk estimate of \(0.210\), while the mean predicted risk at two years is \(0.180\). Define calibration error here as estimated observed risk minus mean predicted risk. The point estimate is \(0.210-0.180=0.030\), or 3.0 percentage points: estimated event risk is higher than the average prediction by 3.0 points.
A subject-level bootstrap with \(B=2{,}000\) replicates, recalculating the censoring adjustment in each sample, hypothetically yields the 2.5th and 97.5th percentiles \(-0.015\) and \(0.075\). The percentile interval is therefore \([-0.015,0.075]\), or \(-1.5\) to \(7.5\) percentage points. Because zero lies in the interval, these data do not establish a nonzero calibration error at the 5% two-sided level using this interval procedure. This is not proof of perfect calibration; the interval also indicates appreciable uncertainty.
For censoring-aware metrics, a subject censored before the horizon has unknown event status at that horizon. The bootstrap must retain the censoring time and event indicator, and the metric must use an appropriate censoring adjustment. In every replicate, the calculation must follow the same estimator specified for the original sample.
Censoring, Assumptions, and Diagnostics
For a fixed horizon \(t_0\), let \(\tilde T_i=\min(T_i,C_i)\) be the observed time and \(\delta_i=I(T_i\leq C_i)\) the event indicator, where \(T_i\) is event time and \(C_i\) is censoring time. A censoring-aware estimator may use inverse-probability weights based on the censoring survival function \(G\). For example, an inverse-weighted Brier score can include contributions for observed events by \(t_0\) weighted by \(1/\hat G(\tilde T_i-)\), and for subjects known to be event-free through \(t_0\) weighted by \(1/\hat G(t_0)\). Subjects censored before \(t_0\) contribute no directly observed horizon status in this expression. The exact estimator and its assumptions depend on the metric; use the censoring treatment appropriate to that metric rather than substituting an uncensored formula.
Common requirements include independent sampling of subjects and adequate support for estimating the metric. Censoring assumptions used by the chosen estimator must also be plausible. In particular, inverse-weighted estimators need non-negligible probability of remaining uncensored over the times relevant to evaluation; very small estimated \(\hat G\) values can create extreme weights and unstable estimates. Conditional independent censoring may require modeling the censoring distribution conditional on covariates. The bootstrap does not repair a violated censoring assumption.
Inspect more than the two interval endpoints. Check how often a replicate cannot estimate the metric, whether AUC replicates contain both outcome groups, whether estimates pile up at boundaries, and whether a few extreme weighted observations dominate. A visibly skewed or multimodal distribution may make a symmetric estimate-plus-or-minus-standard-error interval misleading. A small number of distinct subjects or few events can also make percentile endpoints unstable. Increasing \(B\) reduces Monte Carlo error in the estimated quantiles, but does not add information to the validation sample or fix poor sampling support.
Choosing an Approach and Recognizing Its Limits
A subject bootstrap is attractive because it can be applied to many nonlinear metrics without deriving a metric-specific standard error. It also naturally preserves the dependence among records from the same subject, such as predictions at multiple horizons. Its validity still depends on matching the resampling scheme to the data design. If subjects are sampled in clusters, or repeated records within a subject are the independent sampling units, a simple independent-subject bootstrap may not be appropriate; the resampling unit must reflect the actual independent units.
An analytic standard error can be faster and more precise when a suitable derivation and regularity conditions are available. A normal-approximation interval, \(\hat\theta\pm z_{1-\alpha/2}SE\), can be poor for bounded, skewed, or highly weighted metrics, especially in small samples. The percentile interval avoids imposing that particular symmetric shape, but it is not automatically accurate in small samples or when the estimator is unstable. More elaborate bootstrap intervals exist, but the percentile method is a transparent starting point, not a universal remedy.
Finally, distinguish uncertainty in a validation metric from bias in how that metric was obtained. Resampling a fixed set of predictions estimates variability from sampling validation subjects under that setup. It does not automatically remove bias caused by evaluating predictions on the data used to fit or select the model, nor does it automatically represent uncertainty from the entire model-building process. The interval must be described in relation to the exact prediction and evaluation procedure it covers.
Common Pitfalls
- Breaking the subject-level pairing. Resampling predictions and outcomes separately creates combinations that were not observed and changes the metric’s sampling distribution. Resample complete subject records.
- Calling early censoring a non-event. Censoring before the horizon leaves horizon status unknown. Use the censoring-aware estimator rather than coding that subject as event-free.
- Interpreting the interval as a probability statement about the parameter. A 95% confidence procedure has a long-run coverage interpretation under its assumptions; it does not assign a 95% probability to the fixed parameter being inside the observed interval.
- Assuming more replicates fix inadequate data. A larger \(B\) makes bootstrap quantiles less noisy computationally; it cannot compensate for few events, extreme weights, or a nonrepresentative validation sample.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- A validation record contains a predicted risk, an observed event time, and a censoring indicator. What must remain together in a subject-level bootstrap draw?
- A bootstrap sample has 400 metric estimates. Which empirical order statistics approximately define a 95% percentile interval?
- At a two-year horizon, a subject is censored at one year without an event. Is that subject known to be event-free at two years? What should the analysis do?
- A reported 95% interval for calibration error includes zero. Does this establish that the predictions are perfectly calibrated?
Solutions. (1) Keep the subject’s prediction, event-time information, censoring indicator, and any other inputs needed by the estimator together; repeated selections repeat the whole record. (2) Use approximately the 2.5th and 97.5th percentiles of the 400 bootstrap estimates. (3) No. The subject’s two-year status is unknown, so use an estimator that handles censoring under its stated assumptions. (4) No. It indicates that the interval procedure does not establish a nonzero calibration error at the corresponding two-sided level; it does not prove exact calibration.