Bootstrap Optimism Correction

Why Correct Apparent Performance?

A prediction model is fitted to data, and its performance is often assessed on those same data. This apparent performance is usually optimistic: fitting has adapted the model to features of the development sample, including random variation that will not recur in new patients. The earlier material on apparent performance and optimism distinguishes this training-set advantage from the performance the model would achieve in new data.

Bootstrap optimism correction estimates that advantage by repeatedly drawing bootstrap samples, refitting the prediction procedure in each sample, and comparing its performance in the sample where it was fitted with its performance in the original development data. The difference estimates optimism; averaging those differences gives a correction to the apparent estimate. The procedure is an internal validation method, not an independent test of transportability to another population.

The sign convention matters. A concordance measure is generally better when larger, whereas a Brier score is generally better when smaller. Calibration measures require extra care: a slope near 1, a calibration-in-the-large near 0, and an observed-to-expected ratio near 1 are desirable, so a raw value does not have a single “better when larger” direction. For calibration, it is often clearer to define a loss measuring distance from the ideal value, or an overall calibration error, and correct that loss.

Assumptions and the Bootstrap Procedure

The ordinary case resampling procedure assumes that the sampled individuals are independent and representative of the target development population, and that the empirical distribution of those individuals is a useful approximation to that population. For longitudinal and time-to-event data, the resampling unit is ordinarily the patient: resample patients with replacement and carry along each selected patient’s complete longitudinal record, event or censoring time, and relevant baseline information. Resampling individual visits would break the within-patient structure.

The prediction target and evaluation protocol must be fixed before resampling. For example, specify the prediction horizon and landmark, the risk prediction being evaluated, the censoring-aware metric and its weighting method, and the model-fitting procedure. The same procedure is applied in every replicate. If the original workflow estimates a joint model, chooses among candidate predictors, tunes a penalty, imputes missing values, or selects a prediction rule, those steps must be repeated inside each bootstrap sample. Otherwise the bootstrap does not account for the optimism introduced by those steps.

In a survival setting, “same evaluation protocol” does not mean ignoring censoring. If the metric uses inverse censoring weights, for instance, those weights must be computed according to the prespecified evaluation procedure. Any estimation of the censoring distribution that is part of the metric estimator should be handled consistently in the bootstrap and original-data evaluations. The target estimand and horizon should not shift from replicate to replicate.

Why the Difference Estimates Optimism

A bootstrap sample plays two roles: it is the data used to fit a model, and it is the data used to evaluate that fitted model. Its performance is therefore subject to the same training-set advantage as apparent performance. Evaluating that fitted model on the original data provides a comparison that is less adapted to the particular bootstrap sample. The average gap between the two evaluations estimates the average advantage attributable to fitting and evaluating on the same sample.

More formally, imagine a training sample \(D\) from a population \(F\), and let \(A(D)\) denote the fitted prediction procedure. For a higher-is-better metric, the generalization optimism of the fit is the difference between expected performance on its training data and expected performance on new data:

The bootstrap substitutes the empirical distribution of the observed patients for \(F\). A bootstrap data set is used as the fitting sample, and the original data act as the comparison evaluation sample. Averaging the within-bootstrap-minus-original differences therefore estimates the training-set advantage under this empirical approximation. Subtracting that estimated advantage from apparent performance yields the correction. This is a justification for the estimator, not a guarantee that the empirical distribution perfectly represents the population.

Calibration Requires a Meaningful Error Measure

A calibration slope does not improve by simply increasing: values below and above 1 can both indicate miscalibration. One option is to use the absolute distance from the ideal slope, \(|\widehat{\gamma}-1|\), as a loss. Another is an integrated calibration error that summarizes disagreement between predicted and observed risks over a defined range. For these lower-is-better losses, the correction is apparent loss plus estimated optimism. If the corrected loss exceeds a practically meaningful threshold, that signals error; it is not itself a corrected calibration slope.

Diagnostics, Comparisons, and Limitations

An optimism correction should be accompanied by checks on the resampling and fitting process. Inspect the distribution of replicate differences, not only their average. Large variation, strong skew, or a few extreme values can make the mean correction unstable. Also monitor convergence failures, boundary estimates, and bootstrap samples with too few events to support the planned model fit. A high failure rate is evidence that the procedure is unstable; silently dropping difficult replicates can distort the correction. Report failures and investigate whether the fitting algorithm, model complexity, or resampling design is responsible.

• Preserve the analysis pipeline. Repeating only the final coefficient fit while leaving predictor selection, tuning, or preprocessing fixed misses optimism introduced by those steps.
• Preserve the data structure. Resample complete patients for longitudinal records. If observations are clustered by center and the target concerns new centers, patient-level resampling alone does not represent center-to-center variation; the resampling design must match the sampling structure and target.
• Check metric stability. At a fixed survival horizon, heavy censoring may leave little information for a censoring-adjusted score or calibration estimate. Bootstrap correction cannot create information absent from the data.
• Use enough resamples. A small \(B\) can make the estimated average noisy. Increase \(B\) until the correction is reasonably stable for the intended use, and report \(B\) and the fitting failure rate.

Bootstrap optimism correction is computationally demanding because the full procedure is refitted many times. Compared with reporting apparent performance alone, it explicitly estimates and adjusts for the training-set advantage, but it still relies on the development sample as the source of resampling information. Cross-validation uses a different split-based internal validation design; external validation evaluates predictions in an independent cohort and addresses performance in that cohort rather than estimating optimism from the development data. These approaches answer related but not identical questions.

A correction can be negative or can yield an implausible value, especially with a small sample, unstable fitting, or noisy performance metric. Do not truncate such a result automatically: first check the sign convention, code, resampling unit, evaluation protocol, and replicate distribution. Even a numerically plausible correction remains an estimate with uncertainty. It does not quantify that uncertainty by itself, and it cannot guarantee performance under a changed case mix, measurement process, or clinical setting.