Bootstrap Optimism Correction for Predictive Error

Why Apparent Prediction Error Can Be Optimistic

A population pharmacokinetic model is fitted to data, and its prediction error is often calculated on those same data. This is the apparent error. It describes how well the fitted model accounts for the observations used to fit it, but it generally understates the error expected when the model predicts data from new subjects. The model has been adapted to the original sample, including its sampling variation.

Bootstrap optimism correction estimates that gap using repeated refitting. In each replicate, fit the model to a bootstrap resample, then calculate that fitted model's error both on the resample and on the original dataset. The average difference between those two errors estimates optimism. Adding it to the original apparent error gives a corrected estimate.

Definition: Let $D$ be the original dataset of $n$ subjects, $D^{(b)}$ the $b$th bootstrap resample, and $\hat f^{(b)}$ the model fitted to that resample. For an error measure where larger values indicate worse prediction, let $E(D_{\rm eval};\hat f)$ be the error when model $\hat f$ is evaluated on dataset $D_{\rm eval}$. The optimism in replicate $b$ is $$ O_b = E(D;\hat f^{(b)}) - E(D^{(b)};\hat f^{(b)}). $$ With $B$ bootstrap replicates, the estimated optimism is $\bar O = \frac{1}{B}\sum_{b=1}^B O_b$. The optimism-corrected error is $E_{\rm corr} = E(D;\hat f) + \bar O$, where $\hat f$ is the model fitted to the original dataset.

This definition uses an error scale on which lower is better, such as mean squared error (MSE) or mean absolute error (MAE). If the evaluation measure is instead a score for which larger values are better, define an error such as the negative score, or reverse the difference consistently. This avoids interpreting an improvement in a score as an increase in error.

What Is Being Compared?

The evaluation rule must be defined before calculating the correction. In a PK application, it might be a subject-level loss based on observed concentrations and predictions at specified times. The prediction type, observations included, handling of below-quantification values, and weighting of subjects must be the same in both evaluations. For instance, if the intended target is prediction for a new subject, do not use that subject's outcome data to construct individual-specific predictions during evaluation. The target should reflect the intended use of the model.

A bootstrap sample draws $n$ subjects with replacement from the original $n$. A subject may appear more than once or not at all. When evaluating $E(D^{(b)};\hat f^{(b)})$, repeated appearances contribute according to their multiplicities: this is the resample's training error. When evaluating $E(D;\hat f^{(b)})$, evaluate the same fitted model on the original set of $n$ subjects, each once under ordinary subject-level weighting. This pairing is essential; comparing errors from different resamples or different metrics does not estimate the defined optimism.

The resampling unit must match the study design. If subjects are independently sampled, resample subjects, keeping each subject's records together. If the sampling design involves clusters or strata, the resampling scheme must preserve that structure. Treating repeated concentration records from one subject as independent bootstrap units would not reproduce subject-level sampling variation.

Why the Difference Estimates Optimism

Suppose the prediction target is expected error on new subjects sampled from the same population as the study. A model fitted to one sample usually has lower error when evaluated on that same sample than when evaluated on fresh subjects. In an ideal repeated-sampling argument, the optimism of a fitted model is the expected difference between its error on independent new data and its error on its training data. The original dataset cannot serve as truly independent new data for a model fitted to it, but bootstrap resampling approximates repeated samples from the empirical distribution represented by that dataset.

For each bootstrap sample, the fitted model has been adapted to that resample. Its error on the resample is therefore a training error, while its error on the original data provides a contrasting evaluation. Averaging their difference approximates the expected training-to-test gap. Adding this estimated gap to the original model's apparent error gives the correction.

\text{new-sample error} \approx \text{training error} + \text{optimism}, \qquad E_{\rm corr} = E(D;\hat f) + \frac{1}{B}\sum_{b=1}^{B} \left[E(D;\hat f^{(b)}) - E(D^{(b)};\hat f^{(b)})\right].

This is an approximation, not a guarantee of unbiased prediction error. The original dataset is reused to generate bootstrap samples and to evaluate bootstrap-fitted models; it is not an independent test set. The method relies on the empirical distribution being a useful stand-in for the population distribution.

Worked Example: Correcting Mean Squared Error

All values in this example are hypothetical. Suppose the original model has an apparent MSE of $12.0\;(\text{mg/L})^2$. Four bootstrap models are each evaluated on their own resample and on the original dataset. The paired errors are:

Replicate	Resample MSE $(\text{mg/L})^2$	Original-data MSE $(\text{mg/L})^2$	Optimism $(\text{mg/L})^2$
1	10.0	14.0	4.0
2	11.0	17.0	6.0
3	9.0	11.0	2.0
4	12.0	16.0	4.0

For replicate 1, $O_1=14.0-10.0=4.0\;(\text{mg/L})^2$. The four paired differences are $4.0, 6.0, 2.0,$ and $4.0\;(\text{mg/L})^2$, so

\bar O = \frac{4.0+6.0+2.0+4.0}{4} = \frac{16.0}{4} = 4.0\;(\text{mg/L})^2.

The corrected MSE is $12.0+4.0=16.0\;(\text{mg/L})^2$. The interpretation is that the bootstrap procedure estimates an average optimism of $4.0\;(\text{mg/L})^2$ for apparent MSE under these assumptions, not that every future dataset will have exactly this much additional error. Four replicates make the arithmetic transparent, but would ordinarily be too few for a stable estimate.

Worked Example: A Correction on the MAE Scale

Worked Example: Correcting Mean Absolute Error

All values are hypothetical. A model's apparent MAE on the original dataset is $1.80\;\text{mg/L}$. Three bootstrap fits yield resample MAEs of $1.50$, $1.70$, and $1.60\;\text{mg/L}$, and corresponding original-data MAEs of $2.10$, $2.00$, and $2.20\;\text{mg/L}$.

Calculate the paired optimism values:

O_1=2.10-1.50=0.60\;\text{mg/L}, \quad O_2=2.00-1.70=0.30\;\text{mg/L}, \quad O_3=2.20-1.60=0.60\;\text{mg/L}.

Their mean is

\bar O = \frac{0.60+0.30+0.60}{3} = \frac{1.50}{3} = 0.50\;\text{mg/L}.

Thus $E_{\rm corr}=1.80+0.50=2.30\;\text{mg/L}$. This is a correction to MAE; it should not be described as a correction to MSE or RMSE. The metric and its units remain part of the interpretation.

Model Fitting Must Be Repeated in Each Replicate

Each bootstrap replicate must repeat the steps that produced the model whose performance is being assessed. This includes parameter estimation and, when part of the analysis, model selection, covariate screening, transformations, or tuning choices. If a decision is made once using the full original dataset and then held fixed in all bootstrap fits, the bootstrap may omit the optimism introduced by that decision. The target is the performance of the modeling procedure as actually used, not merely the performance of a final equation treated as if it had been specified in advance.

Convergence and failed replicates also matter. A failed fit cannot simply be assigned zero optimism or silently removed if failures are associated with difficult resamples or poor performance. The fraction and pattern of failures should be examined, and the estimate should be qualified if successful replicates do not represent the intended fitting procedure. Bootstrap convergence and failed replicates are important to the validity of the result, not merely a computational footnote.

Worked Example: Resampling Subjects, Not Concentration Records

All values are hypothetical. Consider a study with four subjects, each contributing several concentration records. In one bootstrap resample, subjects A, A, C, and D are drawn. Subject A appears twice, subject B is absent, and the fitted model is $\hat f^{(b)}$. Suppose the subject-level losses for this fitted model are $1.0, 1.0, 3.0,$ and $4.0$ on A, B, C, and D, respectively, in squared concentration units.

The resample error weights A twice, so

E(D^{(b)};\hat f^{(b)}) = \frac{1.0+1.0+3.0+4.0}{4} = \frac{9.0}{4} = 2.25.

The original-data error evaluates each original subject once:

E(D;\hat f^{(b)}) = \frac{1.0+1.0+3.0+4.0}{4} = 2.25.

This particular replicate has $O_b=2.25-2.25=0$. Equal values can occur; the correction comes from averaging many replicates, not from assuming every replicate has positive optimism. If the losses had instead been calculated by resampling individual concentration records, records from a subject could be split between fitting and evaluation, changing the sampling unit and no longer representing the stated subject-level procedure.

Assumptions, Diagnostics, and Alternatives

A useful optimism correction requires several conditions to be plausible:

The resampling scheme represents how the original subjects or clusters were sampled, including relevant stratification or clustering.
The original dataset is informative enough that its empirical distribution approximates the target population for the intended prediction task.
The same outcome definition, prediction target, loss function, and weighting rule are used in all paired evaluations.
The full fitting procedure is repeated in each replicate, and enough replicates converge to estimate the average difference with reasonable Monte Carlo stability.

Inspect the distribution of $O_b$, not just its mean. Large variability, extreme values, or a mean that changes materially as replicates are added indicates instability. Negative values in individual replicates are possible: a bootstrap-fitted model can happen to perform worse on its resample than on the original dataset. Do not truncate negative differences to zero; truncation changes the estimator and tends to inflate the estimated average optimism. Also compare the original apparent error with the corrected value and check that both use identical evaluation rules.

The main alternatives answer related but distinct questions. Apparent error is simple but does not correct for reuse of the fitting data. Split-sample validation or cross-validation evaluates fitted models on held-out observations and can be useful when the sample supports it, but results may depend on the split or fold structure and the amount of data available for each fit. External validation evaluates a locked model on a separate dataset and is more direct evidence about performance in that external sample, provided the target population and measurements are relevant. Bootstrap optimism correction is an internal validation method: it uses the available sample to estimate and adjust apparent error, not to establish performance in a new setting.

The correction can remain unreliable when the dataset is small, the fitting procedure is unstable, the target population differs from the study sample, or the loss is dominated by a few influential subjects. It also does not repair a poorly chosen prediction target, inappropriate model assumptions, or data-quality problems. A corrected estimate is evidence about expected predictive error under stated conditions, not proof that the model will perform equally well across settings.

Common Mistake

Common pitfall: Subtracting the bootstrap-resample error from the original model's apparent error and calling the result optimism. Those errors come from different fitted models, so their difference does not isolate the training-to-evaluation gap for a bootstrap fit. For each replicate, evaluate the same $\hat f^{(b)}$ on both $D^{(b)}$ and $D$, calculate $E(D;\hat f^{(b)})-E(D^{(b)};\hat f^{(b)})$, average those paired differences, and add the average to the original model's apparent error.

Key takeaway: Bootstrap optimism correction is a paired comparison: each bootstrap-fitted model is evaluated on its own resample and on the original dataset. The mean difference estimates optimism in apparent error, which is added to the original model's apparent error. Its credibility depends on resampling the right units and repeating the full fitting procedure.

Check Your Understanding

Answer each question, then check the concise solutions below.

A model has apparent MSE $8.0\;(\text{mg/L})^2$. Across bootstrap replicates, the mean paired optimism is $2.5\;(\text{mg/L})^2$. What is the corrected MSE?
In one replicate, resample error is $5.0\;\text{mg/L}$ and original-data error for the same bootstrap fit is $4.0\;\text{mg/L}$. What is the replicate optimism, and should it be changed to zero?
Why must covariate selection be repeated within each replicate if selection was part of the original modeling procedure?
A study records ten concentration values per subject. If the intended resampling unit is the subject, what should remain together in a bootstrap draw?

Solutions. (1) $E_{\rm corr}=8.0+2.5=10.5\;(\text{mg/L})^2$. (2) $O_b=4.0-5.0=-1.0\;\text{mg/L}$; retain the negative value, because the correction averages the paired differences without truncation. (3) Repeating selection captures the variability and potential optimism introduced by that data-dependent step; selecting once would omit it from the resampled fitting procedure. (4) Draw whole subjects with replacement, keeping all ten records from each selected subject together, including repeated appearances of a subject when drawn more than once.