Presenting Bootstrap and External Validation Results

Why These Results Need Careful Reporting

Bootstrap results and external validation answer different questions. A bootstrap analysis repeatedly resamples the development data and refits the model to describe how estimates or performance might vary under repeated sampling from a similar population. External validation applies a model to a separate dataset to assess how well its predictions perform there. A narrow bootstrap interval is not evidence that a model predicts well in a new population, and good performance in one external dataset does not establish that the model will generalize everywhere.

A useful report makes the analysis reproducible: it states what was resampled, what was held fixed, how uncertainty was summarized, and which observations contributed to each performance metric. It also reports departures from expectations, including failed bootstrap fits and external-validation results that differ from a prespecified target.

What to Report for a Bootstrap Analysis

The resampling unit must match the dependence structure and sampling design. In population pharmacokinetics, several concentration records may belong to the same individual. Resampling individual records as if independent can break within-individual structure and understate uncertainty. If the target population is represented by individuals, resample individuals and retain all records belonging to each selected individual. If the design involved meaningful strata, the resampling scheme should preserve them as appropriate and should be described.

A report should name the bootstrap method, the number of attempted resamples, the resampling unit, and the statistic summarized. For a percentile interval, the lower and upper endpoints are selected from the corresponding quantiles of the bootstrap estimates. State the quantile levels, such as the 2.5th and 97.5th percentiles for a 95% interval, and report the number and proportion of successful refits. If estimates are summarized only among successful refits, say so: failures may be related to difficult data configurations, so excluding them can make the reported distribution unrepresentative.

A bootstrap interval describes uncertainty under the resampling scheme and the assumptions behind it. It is not a 95% probability statement about a fixed parameter. Its usefulness depends on the data representing the target sampling process, a suitable resampling unit, and a stable estimation procedure. Sparse data, boundary estimates, strong model nonlinearity, and frequent convergence problems can make bootstrap summaries unreliable.

External Validation: Define the Target and the Metrics

External validation evaluates predictions in data not used to estimate the model being assessed. The report should describe the validation population and data collection sufficiently to judge its relation to the development population, state whether the model parameters were fixed, and define each summary precisely. A model applied with its original estimates evaluates transport of that model. Re-estimating or adjusting parameters with validation data instead evaluates a modified model; report that separately rather than presenting the adjusted performance as validation of the untouched model.

For a simple prediction-error summary, define \(e_i=\hat{y}_i-y_i\), where \(\hat{y}_i\) is the model prediction and \(y_i\) is the observed concentration for the prespecified unit of analysis. Positive errors then mean overprediction. With \(m\) independent units, the mean error (bias under this sign convention), mean absolute error, and root mean squared error are:

These summaries answer different questions: mean error describes signed average discrepancy, MAE describes average absolute discrepancy, and RMSE gives relatively greater weight to large errors. State the scale and units. With repeated concentrations per individual, the report must clarify whether summaries are observation-weighted or first summarized per individual; treating all records as independent for uncertainty calculations can give individuals with many samples disproportionate influence and can misrepresent uncertainty.

Uncertainty and Departures from Expectations

Uncertainty for external-validation metrics can be estimated by resampling the validation units, provided the resampling reflects the design. For clustered concentration data, resample individuals with all their records, then recompute the full metric in each resample. Report the interval method and number of resamples, along with the point estimate. If the model is intended to remain fixed, keep its fitted parameters fixed during this validation bootstrap; refitting it on each validation resample answers a different question.

Performance expectations should be stated in context. Examples include a prespecified prediction-error threshold or the nominal coverage of model-based prediction intervals. If observed coverage is below its nominal level, report both the observed fraction and the denominator, and quantify uncertainty where appropriate. A discrepancy may indicate poor calibration, a changed population, data-processing differences, or sampling variation; it does not by itself identify the cause.

Justifying a Percentile Bootstrap Interval

The percentile interval is obtained directly from the bootstrap distribution. If \(\hat{\theta}^{*(1)},\ldots,\hat{\theta}^{*(B)}\) are the \(B\) successful bootstrap estimates, let \(q_{.025}\) and \(q_{.975}\) denote their 2.5th and 97.5th percentiles. The reported interval is \([q_{.025},q_{.975}]\). Its rationale is that, when the bootstrap distribution adequately approximates the sampling distribution of the estimator, its central 95% range approximates the estimator’s central 95% repeated-sampling range.

This is an approximation, not a guarantee. In particular, frequent failures, a skewed or boundary-constrained estimate, or resampling that does not reflect the study design can invalidate the approximation. A percentile interval need not be centered on the original estimate or on the bootstrap median. Report its actual endpoints rather than implying symmetry or relying only on a standard error.

Diagnostics, Alternatives, and Limits

Useful diagnostics include the proportion of failed refits, the distribution of successful estimates, extreme or boundary estimates, and whether failures cluster in particular bootstrap samples. For external validation, examine whether errors vary across the prediction range, time, or clinically relevant subgroups, and check whether the validation data and model implementation match the stated analysis. Such checks can reveal patterns hidden by a single average metric, but do not turn an exploratory pattern into a causal explanation.

Bootstrap intervals are one way to quantify uncertainty. Model-based standard errors and intervals can be more efficient when their distributional and estimation assumptions are suitable; profile-likelihood intervals can be useful when an estimate is asymmetric or near a boundary. Each approach has assumptions, so report the method and why it is appropriate rather than treating one interval as universally preferred. For external performance, a validation bootstrap quantifies uncertainty within the observed validation sample; it cannot correct for a validation population that poorly represents the intended future population.

Limits on generalization should be concrete. State relevant differences in age, disease severity, dosing, sampling schedules, assay or data handling, and other features that may affect predictions. Small samples make performance summaries uncertain; narrow observed ranges may conceal poor predictions elsewhere. If validation data were used to tune or recalibrate the model, say so, because the resulting performance is no longer an independent assessment of the original fixed model.

Common Reporting Pitfalls

• Omitting the resampling unit. “A bootstrap was performed” is not enough when observations are clustered. Name what was sampled and what records were retained together.
• Hiding unsuccessful fits. Report attempted and successful refits, and state whether intervals use successful fits only.
• Confusing interval centers. An interval symmetric around the original estimate may be asymmetric around the bootstrap median. Calculate distances from the center being discussed.
• Leaving error direction ambiguous. Define prediction minus observation or the reverse; otherwise, the sign of bias cannot be interpreted.
• Equating a nominal interval with observed coverage. A 90% prediction interval does not guarantee 90% coverage in every finite validation sample.
• Overgeneralizing validation. Performance in one external dataset supports conclusions about that setting, not automatic transport to every population or clinical context.