Confidence Intervals For Transition-Specific Metrics

Why Uncertainty Must Follow the Subject

A transition-specific validation metric is calculated from predictions and observed outcomes for a particular transition, destination, or path at a specified horizon. Its value is itself uncertain because the validation cohort is only one sample from a population of interest. A confidence interval should reflect that sampling uncertainty without pretending that records from the same person are independent.

This distinction matters in multi-state data. A subject may contribute information to several transition-specific outcomes, or may have repeated records over time. Those records share the same person, follow-up history, and often the same censoring process. Resampling individual records breaks this structure and can make intervals too narrow. The basic resampling unit is therefore the subject, or the highest independent sampling unit if subjects are clustered within centres or families.

Definition: A subject-level bootstrap confidence interval is an interval formed by repeatedly sampling subjects with replacement, retaining each selected subject’s complete validation information, recalculating the metric in each resample, and using the resulting bootstrap distribution to quantify uncertainty.

Fix the transition-specific prediction target and horizon before resampling. Let \(Y_i(t)\) denote the defined binary outcome for subject \(i\) by horizon \(t\), and let \(\hat p_i(t)\) be that subject’s predicted probability for the same target. The outcome definition must treat competing transitions consistently with the estimand. For example, an event from a competing destination must not be recoded as ordinary censoring if the target defines it as a non-case.

With complete outcome ascertainment, the transition-specific calibration-in-the-large error, Brier score, and AUC can be written as follows. Calibration-in-the-large error is defined here as observed event proportion minus mean predicted probability, so zero is the target value. The Brier score is a mean squared prediction error, and lower values are better. The AUC is the probability that a randomly selected case receives a higher score than a randomly selected control, with ties contributing one half.

\widehat{\mathrm{CIL}}(t)=\frac{1}{n}\sum_{i=1}^n Y_i(t)-\frac{1}{n}\sum_{i=1}^n \hat p_i(t), \qquad \widehat{\mathrm{BS}}(t)=\frac{1}{n}\sum_{i=1}^n\{Y_i(t)-\hat p_i(t)\}^2.

\widehat{\mathrm{AUC}}(t)= \frac{1}{n_1n_0} \sum_{i:Y_i(t)=1}\sum_{j:Y_j(t)=0} \left[ I\{\hat p_i(t)>\hat p_j(t)\} +\frac{1}{2}I\{\hat p_i(t)=\hat p_j(t)\} \right],

where \(n_1\) and \(n_0\) are the numbers of cases and controls under the chosen transition-specific definition. When outcomes are subject to right censoring, replace these complete-outcome expressions with the appropriate censoring-adjusted estimators, such as the inverse-probability-of-censoring-weighted (IPCW) versions. The same resampling principle applies, but the weights and any estimated censoring distribution must be handled within each bootstrap replicate.

Constructing the Interval

Suppose the validation cohort consists of independent subjects sampled from the target population, and the prediction rule is fixed for this validation exercise. A bootstrap replicate draws \(n\) subject identities with replacement from the \(n\) observed subjects. Every selected copy carries that subject’s prediction, outcome or follow-up history, and all associated transition records. A subject selected twice contributes twice; all records belonging to each copy remain together.

Specify the estimand.
Fix the target transition, outcome definition, prediction horizon, and metric. Do not change the case or control definition from replicate to replicate.

Resample subjects.
Draw \(n\) subjects with replacement. Keep each selected subject’s complete trajectory and linked transition records together.

Recalculate the metric.
Recompute the calibration quantity, Brier score, or AUC. For censored outcomes, also re-estimate any censoring weights or nuisance quantities that are part of the estimator.

Repeat and form limits.
Repeat for \(B\) replicates, inspect invalid or unstable replicates, and use the chosen quantiles of the bootstrap estimates as interval limits.

For a percentile interval with nominal confidence level \(1-\alpha\), the lower and upper limits are the \(\alpha/2\) and \(1-\alpha/2\) quantiles of the bootstrap estimates. With a 95% interval, these are the 2.5th and 97.5th percentiles. This procedure does not mean that the fixed population metric has a 95% probability of being inside the realized interval. Under repeated sampling and suitable regularity conditions, the procedure is intended to produce intervals that cover the target metric about 95% of the time.

Justification: Let \(F\) be the population distribution of independent subjects and \(F_n\) the empirical distribution assigning mass \(1/n\) to each observed subject. The bootstrap draws from \(F_n\), so it approximates repeated samples from \(F\) by measuring how the statistic changes around its observed value. For a sufficiently regular metric \(T\), the distribution of \(\sqrt{n}\{T(F_n^*)-T(F_n)\}\), conditional on the data, approximates the sampling distribution of \(\sqrt{n}\{T(F_n)-T(F)\}\). Quantiles of the bootstrap estimates can then approximate sampling-error quantiles. This approximation may be poor when the metric is near a boundary, depends on very few cases, or is undefined in many resamples.

The bootstrap targets uncertainty for the procedure actually being evaluated. If predictions are fixed outputs from a model developed elsewhere, retain them in each resample. If the validation estimator includes fitted censoring weights, refit the censoring model within each replicate so its estimation uncertainty is represented. This is not, by itself, a correction for model-development optimism: refitting the prediction model to address optimism is a different task.

Worked Examples

Worked Example: Calibration-in-the-Large Error

Consider six hypothetical subjects with complete ascertainment of a specified transition by horizon \(t\). Their outcomes are \(Y=(1,0,0,1,0,0)\), and their corresponding predicted probabilities are \((0.60,0.40,0.20,0.70,0.30,0.10)\). The complete-outcome assumptions are satisfied for this example: each subject has a known binary outcome at \(t\), and the six subjects are treated as independent.

There are two cases, so the observed event proportion is \(2/6=0.3333\). The mean predicted probability is \((0.60+0.40+0.20+0.70+0.30+0.10)/6=2.30/6=0.3833\). Thus:

\widehat{\mathrm{CIL}}(t)=\frac{2}{6}-\frac{2.30}{6} =0.3333-0.3833=-0.0500.

The sample’s observed event proportion is 0.0500 lower than its mean predicted probability. This is an estimate of a calibration-in-the-large difference, not proof that predictions are systematically too high in the population.

For a bootstrap draw with subject indices \((1,2,2,4,5,6)\), the resampled outcomes sum to \(2\), and the resampled predictions sum to \(0.60+0.40+0.40+0.70+0.30+0.10=2.50\). The replicate error is therefore \(2/6-2.50/6=-0.0833\). Suppose 2,000 subject-level replicates produce a 2.5th percentile of \(-0.250\) and a 97.5th percentile of \(0.150\). The resulting illustrative 95% percentile interval is \([-0.250,0.150]\), in event-proportion units. It includes zero, but that alone does not establish adequate calibration; the interval also reflects substantial uncertainty in this small sample.

Worked Example: Brier Score and a Subject Bootstrap

For four hypothetical subjects, suppose the observed transition outcomes are \(0,1,1,0\), and the predicted probabilities are \(0.10,0.40,0.40,0.80\). Assume the outcomes at the horizon are fully known and the subjects are independent. The Brier score is:

\widehat{\mathrm{BS}}(t)= \frac{(0-0.10)^2+(1-0.40)^2+(1-0.40)^2+(0-0.80)^2}{4} =\frac{0.01+0.36+0.36+0.64}{4} =\frac{1.37}{4}=0.3425.

For a bootstrap sample with indices \((1,2,2,4)\), the four squared errors are \(0.01,0.36,0.36,0.64\), so this replicate’s Brier score is \(1.37/4=0.3425\). For indices \((1,1,2,3)\), the squared errors are \(0.01,0.01,0.36,0.36\), giving \((0.01+0.01+0.36+0.36)/4=0.1850\). These are two possible bootstrap outcomes, not two independent observations to pool with the original score.

Suppose 2,000 subject-level bootstrap replicates yield percentile limits of \(0.1225\) and \(0.5625\). The illustrative 95% interval is \([0.1225,0.5625]\) on the squared-probability scale. It quantifies uncertainty in the population Brier score under the sampling and outcome assumptions; it is not an interval for an individual subject’s prediction error. With only four subjects, these limits are highly unstable and should not be treated as a reliable precision estimate.

Worked Example: AUC and Case-Control Resampling

Consider six hypothetical subjects with known outcomes at a fixed horizon. Three are cases with scores \(0.80,0.60,0.40\), and three are controls with scores \(0.70,0.30,0.20\). Assume independent subjects and a common transition-specific case-control definition. There are \(3\times3=9\) case-control pairs. The case with score \(0.80\) outranks all three controls; the case with score \(0.60\) outranks two; and the case with score \(0.40\) outranks two. There are no tied scores, so:

\widehat{\mathrm{AUC}}(t)=\frac{3+2+2}{9} =\frac{7}{9}=0.7778.

In one subject-level bootstrap replicate, suppose the sampled cases have scores \(0.80,0.60,0.60\), while the sampled controls have scores \(0.70,0.30,0.20\). The first case wins three comparisons; each \(0.60\) case wins two. Thus the replicate AUC is \((3+2+2)/9=0.7778\). Duplicate selections count as repeated bootstrap copies, and the pair comparisons are formed within that replicate.

Suppose 2,000 replicates that contain at least one case and one control give percentile limits \(0.333\) and \(1.000\). These illustrative limits are bounded by the possible AUC range and show considerable uncertainty. Replicates with no cases or no controls have undefined AUCs; they cannot be assigned an AUC of zero or silently treated as ordinary valid replicates. Their frequency is itself a warning that the sample has too little information for a stable AUC interval.

Assumptions, Diagnostics, and Limitations

A subject bootstrap interval is most defensible when the observed subjects represent independent draws from the target validation population and the metric is estimable in most resamples. If sampling is clustered, sample independent clusters and retain all subjects and records within each selected cluster. If the validation design uses strata or unequal sampling probabilities, the resampling scheme must respect that design rather than treating all observed subjects as exchangeable.

For IPCW metrics, the censoring model and the positivity conditions required for the weights remain important. A subject bootstrap does not repair misspecified censoring assumptions or a lack of follow-up support. Re-estimate censoring weights within each replicate, and inspect their distribution and the fraction of replicates with extreme weights or undefined estimates. When a replicate fails, report how often and why; do not conceal instability by dropping failures without explanation.

Too few transition events: Many replicates may contain very few cases, or none. AUC may be undefined, while calibration and Brier estimates may be highly variable.
Incorrect resampling unit: Resampling transition rows independently breaks within-subject dependence and usually understates uncertainty. Keep each subject’s records together.
Unstable censoring weights: A few large weights can dominate a replicate. Check weight ranges and failed-replicate frequency, and revisit whether the estimator’s support assumptions are plausible.
Sparse calibration groups: If calibration is summarized in risk groups, bootstrap intervals for a group with few subjects can be erratic. Group membership and the grouping rule must be defined consistently.
Boundary estimates: AUCs near 0 or 1 and Brier scores near 0 are constrained by their ranges. A normal interval can extend beyond those ranges; percentile intervals respect the observed range but do not guarantee good coverage near a boundary.

An analytic standard-error or delta-method interval can be faster and useful when its approximation is appropriate, but it may be unreliable for sparse transition outcomes, complicated weighting, or statistics with awkward sampling distributions. A normal approximation based on bootstrap standard deviation is another option, but it can also produce impossible limits. Percentile intervals are straightforward and preserve the metric’s range when each replicate estimate is valid; they can nevertheless have poor coverage in small or highly discrete samples. No interval method creates information absent from the validation cohort.

Common Mistake

Common mistake: Treating each transition record as an independent bootstrap unit because the records appear as separate rows. This breaks the link between records from the same subject and can produce intervals that are too narrow. Resample the subject once, carry along all of that subject’s records, and recalculate the transition-specific metric from the resampled subject set. If a subject is selected more than once, include the complete record bundle once for each selected copy.

A second interpretive error is to treat an interval containing the ideal value as evidence that a metric is ideal. For calibration-in-the-large, an interval containing zero says that the data do not rule out zero at the stated confidence level under the method’s assumptions; it does not prove calibration. Likewise, an AUC interval describes uncertainty in discrimination for the defined horizon and target population, not uncertainty about whether an individual outcome will occur.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions.

A subject contributes records for three possible transitions. What should be sampled in a bootstrap replicate?
In a complete-outcome sample, the observed event proportion is \(0.24\) and the mean predicted probability is \(0.31\). What is the calibration-in-the-large error under the definition used here?
Why is an AUC undefined in a bootstrap replicate with no controls?
For IPCW validation, should estimated censoring weights automatically be held fixed across all bootstrap replicates?
What does a 95% bootstrap confidence interval claim about the population metric?

Solutions. (1) Sample the subject as the unit and retain all three transition records, along with the full relevant history; duplicated selections carry duplicated bundles. (2) The error is \(0.24-0.31=-0.07\), or 7 percentage points below zero, meaning the observed proportion is lower than the mean predicted probability in this sample. (3) AUC compares case scores with control scores; with no controls there are no case-control pairs and its denominator is zero. (4) Not if the weights are estimated as part of the validation estimator: re-estimate the censoring model and weights within each replicate, then diagnose instability. (5) Under repeated sampling and the method’s assumptions, the interval procedure is intended to cover the fixed population metric in about 95% of samples; it is not a 95% probability statement about the parameter after observing this particular interval.