Why the Comparison Must Preserve Pairing
When two prediction models are evaluated on the same validation subjects, each subject contributes information to both models’ metrics. The comparison is therefore paired: the subjects, outcome histories, follow-up, and often censoring patterns are shared. Treating the two metric estimates as if they came from independent validation samples discards that pairing and can misstate uncertainty.
A useful comparison starts by fixing the validation target: the transition of interest, prediction horizon or integration range, metric definition, and any weighting scheme. Both models must be evaluated against that same target. For example, a transition-specific Brier score is a loss, for which smaller values are better; an AUC is a discrimination measure, for which larger values are better.
The target is usually the difference in performance in a population represented by the validation sample, estimated by the observed paired difference. A confidence interval describes uncertainty in that difference under the sampling and metric-estimation assumptions; it does not give the probability that one model is better.
Assumptions and a Paired-Variance Justification
For a simple additive metric, let \(L_{Ai}\) and \(L_{Bi}\) denote the subject-level contributions for subject \(i\), and set \(D_i=L_{Ai}-L_{Bi}\). Then the paired estimate is \(\bar{D}=n^{-1}\sum_{i=1}^n D_i\). This representation applies directly to an ordinary mean loss. For a censoring-adjusted or normalized metric, the metric may instead be a ratio or weighted average; in that case, the whole metric should be recomputed from the resampled data rather than assuming that a fixed subject-level contribution fully represents it.
The usual paired standard error for an additive mean difference is \(s_D/\sqrt{n}\), where \(s_D\) is the sample standard deviation of the \(D_i\). The benefit of pairing is visible by expanding the variance:
If the two models’ subject-level losses tend to move together, the covariance is positive, so the variance of their difference is smaller than it would be if that covariance were ignored. It is not guaranteed to be positive: pairing is the correct design, but it does not guarantee a narrower interval.
A paired standard-error calculation requires independent sampling units across subjects and an appropriate large-sample approximation for the mean difference. A subject bootstrap makes similar sampling-unit assumptions, while avoiding a normal approximation to the difference in some settings. If a subject contributes multiple records, transitions, or horizons, those records are not independent units: resample the subject and keep the records together.
For right-censored outcomes, a metric such as an inverse-probability-of-censoring-weighted (IPCW) score also relies on suitable censoring assumptions and adequate censoring-weight support. Those requirements apply to each model comparison, not just to the bootstrap. If censoring weights are estimated from the data, their estimation should generally be repeated within each bootstrap sample so that uncertainty from the weight estimation is reflected.
Worked Example: A Paired Difference in Brier Score
Suppose six hypothetical subjects have complete outcome ascertainment by a fixed horizon for a specified transition. Their Brier scores are:
| Subject | Model A | Model B | \(D_i=\text{Brier}_A-\text{Brier}_B\) |
|---|---|---|---|
| 1 | 0.12 | 0.15 | -0.03 |
| 2 | 0.20 | 0.18 | 0.02 |
| 3 | 0.08 | 0.10 | -0.02 |
| 4 | 0.30 | 0.22 | 0.08 |
| 5 | 0.16 | 0.14 | 0.02 |
| 6 | 0.24 | 0.20 | 0.04 |
The complete ascertainment means this illustration does not require censoring adjustment. The same subjects, transition, horizon, and Brier-score definition are used for both models. The mean scores and their difference are
Because Brier score is a loss, the positive difference means model A’s mean loss is 0.01833 higher than model B’s in this sample. It does not, by itself, establish a population difference. For illustration, the sample standard deviation of the six paired differences is
The paired standard error is \(0.04021/\sqrt{6}=0.01642\). This small-sample calculation is only an approximate summary of uncertainty: a normal-based interval would additionally require the distribution of the differences to be adequately represented by that approximation. In a validation analysis, subject-level resampling is a useful alternative, especially when the metric itself is not a simple mean of fixed contributions.
Subject-Level Bootstrap for the Difference
In the paired subject bootstrap, draw \(n\) subjects with replacement from the \(n\) validation subjects. Use the same resampled subject indices for both models. Recalculate both metrics on that resample and save their difference. Repeating this process \(B\) times gives a bootstrap distribution of differences. Its standard deviation estimates the bootstrap standard error; percentile quantiles can be used for a percentile confidence interval.
Fix the transition, horizon or integration range, metric, weighting, and sign convention. Confirm that both models are being compared on the same validation target.
Draw subject identifiers with replacement. Carry each selected subject’s full relevant outcome and censoring history, and both models’ predictions, into the resample.
Within the resample, recompute any estimated censoring weights and the full metric for each model. For AUC, recalculate the case-control comparisons; do not treat the original AUC as a fixed per-subject score.
Save \(\Delta^{*(b)}=M_A^{*(b)}-M_B^{*(b)}\) for each resample \(b\). Use the bootstrap distribution to report an uncertainty measure on the same metric scale as the difference.
For a percentile interval with confidence level \(1-\alpha\), take the \(\alpha/2\) and \(1-\alpha/2\) quantiles of the bootstrap differences. The interval describes sampling uncertainty under the subject-resampling scheme and the assumptions of the validation design. A bootstrap interval does not fix biased predictions, an inappropriate metric, or violations of the censoring assumptions.
Worked Example: Calculating Bootstrap Differences
For a transparent calculation, consider four hypothetical subjects with paired differences in a lower-is-better transition-specific loss of \(-0.02,\ 0.00,\ 0.04,\ 0.06\), where each difference is model A minus model B. The observed difference is
A positive value means model A has the higher, and therefore worse, loss in this sample. Consider these six illustrative subject bootstrap draws, shown by their sampled subject numbers: (1, 2, 3, 4), (1, 1, 2, 3), (2, 2, 3, 4), (1, 3, 4, 4), (1, 2, 2, 4), and (2, 3, 3, 4). Their paired differences are calculated as follows:
The mean of these six bootstrap differences is 0.02083, and their sample standard deviation is approximately 0.01393. These six draws demonstrate the arithmetic only; they are far too few to provide a reliable bootstrap standard error or confidence interval. In practice, use many resamples, check whether the resulting distribution is stable to increasing \(B\), and report the actual number of resamples and interval method.
Multiple Transitions and Non-Additive Metrics
A comparison may involve several transitions or horizons. If the goal is a separate difference for each transition, calculate and report each one with its own uncertainty. If the goal is a prespecified aggregate, define how transition-specific metrics are combined before looking at the results. Use the same subject resample for all components: this preserves dependence between the metrics and permits an uncertainty assessment for their aggregate.
For a non-additive metric such as AUC, the bootstrap still resamples subjects, but the metric must be rebuilt from the resampled cases and controls. A subject may appear more than once in a bootstrap sample; those repeated copies are part of the resampling calculation, not new independent people in the original cohort. If a resample contains too few cases or controls to define the metric, record and investigate that failure rather than silently replacing it with a convenient value.
Worked Example: Preserving Dependence Across Two Transitions
Suppose four hypothetical subjects have differences in lower-is-better Brier score for transitions 1 and 2, again defined as model A minus model B:
| Subject | Transition 1 difference | Transition 2 difference |
|---|---|---|
| 1 | -0.02 | 0.03 |
| 2 | 0.04 | 0.01 |
| 3 | 0.00 | 0.05 |
| 4 | 0.02 | -0.01 |
The observed mean differences are \(( -0.02+0.04+0.00+0.02)/4=0.010\) for transition 1 and \((0.03+0.01+0.05-0.01)/4=0.020\) for transition 2. Suppose the prespecified aggregate is the equal-weight average of those two transition differences. Its observed value is \((0.010+0.020)/2=0.015\).
Now resample subjects 1, 1, 3, and 4. For transition 1 the bootstrap difference is \((-0.02-0.02+0.00+0.02)/4=-0.005\); for transition 2 it is \((0.03+0.03+0.05-0.01)/4=0.025\). The aggregate in this same resample is \((-0.005+0.025)/2=0.010\). Using one common draw preserves the fact that each subject’s two transition-specific records belong together. Drawing different subjects separately for each transition would instead create a bootstrap sample that does not represent the joint validation data and could misstate uncertainty in the aggregate.
Diagnostics, Alternatives, and Limitations
A paired subject bootstrap is appropriate when subjects are the independent sampling units and the validation cohort represents the target population. Before relying on its interval, examine whether the metric can be computed consistently across resamples. Relevant warning signs include very few observed events for a transition, many resamples with no cases or controls, extreme or unstable censoring weights, and an interval that changes materially with the number of bootstrap draws. These symptoms often indicate limited information in the validation data, not a problem that can be solved by increasing the number of bootstrap replicates.
For an additive mean metric with a well-behaved distribution of paired differences, a paired standard-error or paired \(t\)-interval is a simpler alternative. It uses the subject-level differences and their variability. The subject bootstrap is more flexible for metrics whose estimates involve weighting, normalization, or pairwise comparisons, but it is not automatically superior in a very small sample or with very rare transitions. Independent-sample comparison methods are generally not the right alternative when the models were evaluated on the same subjects: they omit the covariance induced by the shared data.
The uncertainty estimated here is uncertainty in comparing fixed model predictions on sampled validation subjects. It does not, by itself, account for model-fitting variability if the models were trained using those same subjects, and it does not correct over-optimism from model development. The interpretation also remains specific to the chosen transition, horizon, metric, and target population. A narrow interval for one metric is not evidence that all clinically relevant aspects of prediction are satisfactory.
Check Your Understanding
Answer each question, then check the short solutions below.
- For a lower-is-better metric, what does a positive value of \(M_A-M_B\) indicate?
- Why must a bootstrap draw use the same subject identifiers for both models?
- In an IPCW comparison, why might censoring weights need to be re-estimated within each bootstrap sample?
- Why is it not enough to average fixed subject-level differences when the metric is AUC?
- Name two warning signs that a bootstrap interval for a rare-transition metric may be unreliable.
Solutions. (1) Model A has the larger loss in the observed comparison, so the difference favors model B. (2) Both models were evaluated on the same subjects; using the same draw preserves that pairing and its covariance. (3) The weights are estimated quantities, and re-estimation includes their sampling variability in the bootstrap distribution. (4) AUC is based on case-control comparisons, so it must be recomputed from the resampled cases and controls rather than treated as a simple mean of fixed individual contributions. (5) Examples include many resamples with too few cases or controls to calculate the metric, unstable censoring weights, and intervals that change substantially as the number of bootstrap draws increases.