Why Correct Apparent Performance?
A prediction model is usually fitted to development data, and its performance on those same data is often too favorable. This is especially important in multi-state prediction: a model may be developed to predict a particular transition by a particular horizon, while other transitions prevent that event from occurring and censoring may obscure outcomes. The validation measure must therefore remain tied to its stated transition, horizon, and evaluation procedure throughout the correction.
The bootstrap optimism correction estimates how much better a fitted model appears on data used to fit it than on data not used for that fit. It does not create a genuinely independent validation cohort. Rather, it uses repeated resampling of the development subjects to estimate and adjust the development sample’s apparent performance.
The Bootstrap Correction
For each replicate \(b=1,\ldots,B\), draw a bootstrap sample \(D_b\) by sampling the independent units in \(D\) with replacement. Fit the full prediction procedure to \(D_b\), obtaining \(\hat f_b\). Compute the chosen measure once on \(D_b\) and once on the original data \(D\), always using \(\hat f_b\). If higher values indicate better performance, define
The model is refitted for every bootstrap sample. The second calculation is not a refit: it evaluates that replicate’s fitted model on the original data. For measures where smaller values are better, such as a prediction loss, define \(O_b=L(D;\hat f_b)-L(D_b;\hat f_b)\). The bootstrap sample will often have a lower, more favorable loss, so positive loss optimism is added to the apparent loss:
Equivalently, one can change the sign of a loss and apply the higher-is-better formula. The key is to set the direction before calculating differences. A sign error can turn a correction that should reduce favorable performance into one that makes it look even better.
Name the transition, prediction horizon, target population, and validation measure. Keep the definition fixed across replicates.
For ordinary subject-level data, sample subjects with replacement and retain each selected subject’s complete observed multi-state history.
Repeat all data-dependent fitting and selection steps that were part of development, not just the final coefficient fit.
Evaluate the replicate-fitted model on its bootstrap sample and on the original data using the same prespecified metric procedure.
Subtract mean optimism for higher-is-better measures; add mean optimism for lower-is-better losses.
What Must Be Resampled and Repeated?
The resampling unit must match the independent sampling unit. If a person can contribute several transitions, visits, or landmark records, resample that person’s entire record, not each transition row separately. Splitting a person’s records across bootstrap and evaluation data would break within-person dependence and could make evaluation information leak into model development. If subjects are sampled in clusters, such as hospitals, and the target sampling variation is across hospitals, resample hospitals and carry along their subjects. Respect any design strata when the sampling design requires it.
Repeat the complete development pipeline that could benefit from the original data: imputation, variable screening, tuning, state- or transition-specific model fitting, and selection among candidate models. If a step is fixed in advance and uses no development-data information, it need not be re-estimated merely for form’s sake. But a data-selected step left outside the resampling will generally leave some optimism uncorrected.
The metric calculation also needs a consistent censoring procedure. For an inverse-probability-of-censoring-weighted measure, prespecify how censoring weights are estimated and apply that rule in the bootstrap and original-data evaluations. For example, estimate weights from the evaluation data each time if that is the intended metric estimator, or use externally specified weights if they are genuinely fixed. A change in weighting procedure between the two evaluations makes their difference difficult to interpret. The censoring assumptions required by the chosen metric, including adequate estimation of censoring probabilities over the relevant horizon, still apply; bootstrap resampling does not repair their failure.
Worked Example: Correcting a Transition-Specific AUC
Hypothetical data. A development cohort contains 300 people starting in state A. The intended prediction task is transition A to B within 2 years; a different transition, such as A to death, can occur first. Suppose the prespecified time-dependent AUC procedure handles observed competing transitions and censoring. Larger AUC indicates better discrimination. The model’s apparent AUC on all 300 development subjects is 0.790.
In four illustrative bootstrap replicates, the AUC on each bootstrap sample and the AUC on the original 300 subjects using that replicate’s fitted model are as follows. Four replicates are shown only to make the arithmetic visible; they are far too few for a stable applied correction.
| Replicate | Bootstrap-sample AUC | Original-data AUC using bootstrap fit | Optimism |
|---|---|---|---|
| 1 | 0.840 | 0.770 | 0.070 |
| 2 | 0.820 | 0.750 | 0.070 |
| 3 | 0.860 | 0.790 | 0.070 |
| 4 | 0.830 | 0.760 | 0.070 |
For replicate 1, the optimism is \(0.840-0.770=0.070\). The mean of the four differences is \((0.070+0.070+0.070+0.070)/4=0.070\). Thus the illustrative optimism-corrected AUC is \(0.790-0.070=0.720\). Under this procedure, estimated discrimination for predicting A-to-B transition within 2 years is lower after correcting for development-sample optimism. This is an internal correction, not an estimate from an independent cohort.
Derivation and Interpretation
The correction follows by comparing two quantities for each bootstrap-fitted model: how well it performs on the resampled data that helped fit it, and how well it performs on the original data. Their difference estimates the advantage associated with evaluating a model on data used in its development. Averaging those paired differences gives an estimated development-data advantage under the empirical resampling scheme. Removing that estimated advantage from the original apparent performance gives the corrected estimate.
The approximation relies on the bootstrap samples behaving like plausible samples from the population represented by the development data. It also relies on the resampling unit being appropriate, the model-development procedure being reproduced, and the chosen metric being estimable and meaningful in the relevant samples. In transition validation, that last condition can be demanding: a bootstrap replicate may contain very few occurrences of the target transition, or censoring weights may become extreme. The calculation is not a proof that the corrected measure equals future-cohort performance; it is an internal estimate that depends on these assumptions.
Worked Example: Correcting a Brier Score
Hypothetical data. For a specified transition and 1-year horizon, suppose a censoring-adjusted Brier-score procedure is used, with lower values indicating smaller prediction error. The apparent Brier score for the model fitted to the full development cohort is 0.130. In one bootstrap replicate, the fitted model has Brier score 0.112 on its bootstrap sample and 0.145 when evaluated on the original cohort using the prespecified evaluation procedure.
Because lower is better, the loss optimism for this replicate is original-data loss minus bootstrap-sample loss:
Suppose that across \(B=1{,}000\) successful replicates, the mean loss optimism is 0.021. The corrected Brier score is \(0.130+0.021=0.151\). The correction increases the estimated prediction error, as expected when in-sample performance is too favorable. In context, 0.151 is the internally optimism-corrected estimate of the specified 1-year transition prediction loss; it is not a probability of transition and does not describe an individual’s risk.
Diagnostics, Alternatives, and Limitations
A correction is only useful if its replicate calculations are credible. Inspect the distribution of \(O_b\), not only its mean. Large positive and negative differences may cancel to a modest average while revealing instability. Record failed fits, replicates with no or very few target transitions, extreme censoring weights, and metric values near their boundaries. Check that predicted transition probabilities remain coherent where the model requires competing destination probabilities to sum appropriately. If many replicates fail, or the metric is undefined in a substantial fraction, silently dropping those replicates can change the target of the calculation. Report the problem and reconsider whether the data support reliable estimation.
The number of replicates should be large enough that the estimated mean optimism is stable. A rough Monte Carlo standard error for the mean of the replicate optimism values is their sample standard deviation divided by \(\sqrt{B}\). This quantifies simulation noise from using a finite number of bootstrap replicates, not uncertainty from sampling a new study. Increase \(B\) if this Monte Carlo error is material relative to the correction being reported.
Apparent performance alone is easy to calculate but does not adjust for reuse of development data. A simple held-out split can evaluate predictions on data not used for fitting, but it uses less data for model development and its result can depend strongly on the particular split. Bootstrap optimism correction uses the development sample more efficiently for this internal adjustment, but its evaluation on the original data is not independent. Cross-validation and independent external validation are other approaches with different designs and interpretations; none should be described as interchangeable with this correction.
The method has important limits. Bootstrap resampling cannot correct a poorly defined transition outcome, inappropriate prediction horizon, invalid censoring assumptions, or a mismatch between the development cohort and the future target population. It can also be unstable for rare transitions, especially when replicates omit the target event or have too few events to fit the model. Finally, the corrected value is a point estimate of internal performance. It is not, by itself, a confidence interval, nor does it account for all uncertainty in model selection or population change.
Worked Example: Resampling the Correct Unit
Hypothetical data. A cohort has 120 people, each contributing an initial state and a sequence of follow-up records that may include an intermediate state and a later competing transition. A long-format analysis table therefore has 390 rows. The target is the A-to-B transition probability by 18 months, assessed with a transition-specific measure.
A subject-level bootstrap replicate draws 120 subject identifiers with replacement. If subject 17 is drawn twice, both copies of subject 17’s complete follow-up history are included as two bootstrap copies. Suppose subject 17 contributes four rows: all four rows travel together in each copy. The model is fitted to these resampled subject histories, and the metric is computed for the target transition and horizon on the bootstrap sample and on the original 120 subjects.
Resampling 390 rows independently would instead treat records from the same person as unrelated. Some of a person’s follow-up could enter model fitting while other records from that person enter evaluation. This changes the dependence structure and can leak subject-specific information across the comparison. The appropriate calculation resamples 120 people, not 390 records.
Common Pitfalls
- Reversing the sign. For AUC, positive optimism is subtracted. For Brier score, positive loss optimism is added. State whether larger or smaller values are better before calculating.
- Resampling transition rows. Repeated records from one subject are dependent. Resample complete subjects, or the appropriate independent clusters.
- Leaving model selection outside the bootstrap. If the development data selected predictors or tuning parameters, repeat that selection in each replicate to capture its contribution to optimism.
- Changing the validation task. Do not average across different transitions or horizons unless that aggregation is the prespecified metric. Keep the target and censoring procedure consistent.
- Calling the result external validation. The bootstrap correction estimates internal optimism; it does not replace evaluation in a new cohort.
Check Your Understanding
Answer each question, then check the concise solutions below.
- A higher-is-better transition-specific measure has apparent value 0.76 and estimated mean optimism 0.05. What is its corrected value?
- A lower-is-better loss has apparent value 0.18 and mean loss optimism 0.02. What is its corrected value, and why does the correction go in that direction?
- A person contributes records for three transitions. Should those records be sampled independently in a subject-level bootstrap? Explain.
- What does a large Monte Carlo standard error for mean optimism indicate, and what does it not indicate?
Solutions. (1) \(0.76-0.05=0.71\). (2) \(0.18+0.02=0.20\); positive loss optimism means the development-sample loss was too favorable, so it is added. (3) No. Sample the person and retain all their transition records together to preserve dependence and prevent information leakage. (4) It indicates that the estimated mean optimism is noisy because too few or too variable bootstrap replicates were used; it is simulation error, not a confidence interval for future-cohort performance or the full sampling uncertainty.