1. What Is Bootstrap Validation?
Bootstrap validation is an internal validation technique that evaluates how stable a population PK model is by repeatedly creating new datasets through resampling and then refitting the model to each resampled dataset.
In a typical population PK bootstrap, subjects are sampled with replacement from the original study population. Each bootstrap dataset therefore contains the same nominal number of subjects as the original dataset, but some subjects may appear more than once while others may be absent.
Bootstrap validation repeatedly resamples subjects, refits the population PK model, and uses the resulting parameter distributions to assess model stability and uncertainty.
2. Why Use a Bootstrap in Population PK?
Population PK models often contain many parameters, including typical population values, between-subject variability, covariate effects, and residual error parameters. Some estimates may be sensitive to the particular subjects included in the dataset.
A single model fit provides one set of parameter estimates. The bootstrap provides a distribution of estimates obtained from many plausible resamples of the observed population.
| Question | Bootstrap contribution |
|---|---|
| How stable is the parameter estimate? | Shows how much the estimate changes across resampled datasets. |
| How uncertain is the estimate? | Provides an empirical distribution from which confidence intervals can be derived. |
| Is there evidence of estimation bias? | Allows comparison of the bootstrap mean or median with the original estimate. |
| Are some parameters unstable? | Wide, skewed, or irregular bootstrap distributions can identify problematic parameters. |
| Does the model repeatedly converge? | Repeated fitting reveals convergence failures and numerical instability. |
Bootstrap validation is therefore useful not simply because it produces confidence intervals, but because the repeated model-fitting process can reveal how dependent the fitted model is on the particular composition of the original dataset.
3. Why Are Subjects Usually Resampled?
In population PK analysis, the independent sampling units are generally the subjects, not the individual concentration measurements.
A subject may contribute multiple observations over time. Those observations are correlated through the individual's underlying PK parameters and study-specific sampling structure. Resampling individual observations independently would disrupt this structure.
For example, if Subject 17 has eight concentration measurements, selecting Subject 17 twice means that the complete set of eight observations is represented twice in that bootstrap dataset.
This preserves the within-subject observation structure while allowing the composition of the population to vary from one bootstrap replicate to another.
4. How Does a Population PK Bootstrap Work?
A typical bootstrap can be described as a sequence of repeated steps.
- Fit the original model. Estimate the population PK model using the complete observed dataset.
- Resample subjects. Randomly select subjects with replacement until the bootstrap dataset contains the same number of subjects as the original dataset.
- Refit the model. Estimate the same population PK model using the bootstrap dataset.
- Store the estimates. Record the parameter estimates and relevant diagnostics.
- Repeat. Generate many bootstrap datasets and repeat the model fitting process.
- Summarize the distributions. Calculate quantities such as medians, means, standard deviations, percentiles, and confidence intervals.
- Compare with the original model. Evaluate whether the original estimates are consistent with the bootstrap distributions.
If \(B\) bootstrap replicates are generated, the result is a collection of parameter estimates:
These estimates form an empirical sampling distribution for the parameter under the bootstrap procedure.
5. What Does the Bootstrap Distribution Represent?
Suppose the original population PK model produces an estimate of clearance, \(\hat{CL}\). Each bootstrap replicate produces another estimate:
The collection of these values provides an empirical distribution of clearance estimates.
A narrow distribution suggests that the estimate is relatively stable under the bootstrap resampling procedure. A wide distribution indicates greater sensitivity to which subjects happen to be represented in the resampled populations.
| Bootstrap pattern | Possible interpretation |
|---|---|
| Approximately symmetric and narrow | Parameter is relatively stable and precisely estimated under the bootstrap. |
| Wide distribution | Parameter may have substantial sampling uncertainty. |
| Strong right or left skew | Normal-based summaries may be inappropriate; percentile-based summaries may be more informative. |
| Extreme outliers | Some bootstrap datasets may produce unusual parameter estimates or numerical problems. |
| Many failed fits | The model may be numerically or structurally unstable under resampling. |
6. Bootstrap Confidence Intervals
One common use of the bootstrap is to obtain empirical confidence intervals for model parameters.
For a simple percentile bootstrap interval, the lower and upper limits are selected from the corresponding percentiles of the bootstrap distribution.
For a nominal 95% percentile interval:
where \(Q_p\) denotes the \(p\)-th percentile of the bootstrap distribution.
For example, if the bootstrap distribution of clearance has a 2.5th percentile of 6.9 L/h and a 97.5th percentile of 10.4 L/h, the corresponding percentile interval is 6.9–10.4 L/h.
7. Assessing Bias With the Bootstrap
The original parameter estimate can be compared with the center of the bootstrap distribution.
For parameter \(\theta\), a simple bootstrap estimate of bias is:
where \(\overline{\hat{\theta}^{*}}\) is the mean of the bootstrap estimates and \(\hat{\theta}\) is the estimate from the original dataset.
A relative bias can be expressed as:
For example, suppose the original clearance estimate is 8.4 L/h and the mean of the bootstrap clearance estimates is 8.6 L/h:
This indicates that the bootstrap mean is approximately 2.4% above the original estimate. Such a comparison is descriptive; its interpretation depends on the parameter, model, and purpose of the analysis.
8. Bootstrap Precision and Stability
The spread of the bootstrap distribution provides information about the stability of an estimate.
For example, the empirical standard deviation of bootstrap estimates can be calculated as:
A useful relative measure is the bootstrap coefficient of variation:
These quantities should be interpreted alongside the actual parameter scale and scientific context. A parameter with a wide bootstrap distribution may deserve additional investigation, particularly when it is central to model interpretation or prediction.
9. Which Population PK Parameters Should Be Examined?
The bootstrap can be used to evaluate many of the parameters estimated by a population PK model.
| Parameter type | Examples | What to examine |
|---|---|---|
| Typical population PK | CL, V, \(k_a\), intercompartmental clearance | Center, spread, bias, confidence intervals |
| Between-subject variability | \(\omega^2\), CV% | Stability, skewness, boundary estimates |
| Covariate effects | Weight, age, renal function effects | Magnitude, direction, uncertainty, stability |
| Residual error | Proportional, additive, combined error | Stability and consistency across replicates |
| Correlation parameters | Random-effect correlations | Numerical stability and plausible ranges |
Not every parameter requires the same interpretation. Variance parameters are often constrained to be nonnegative and may have skewed sampling distributions, while covariate effects may be represented on logarithmic or other transformed scales.
10. Bootstrap Validation of Covariate Effects
Bootstrap analysis can be particularly useful for assessing the stability of covariate relationships in population PK models.
Suppose clearance is modeled as:
The bootstrap produces a distribution of estimates for \(\theta_{WT}\). This can help determine whether the estimated weight effect is stable across resampled populations.
For categorical covariates, a similar principle applies. A sex effect, treatment effect, or other categorical relationship can be re-estimated in every bootstrap replicate.
11. What If Some Bootstrap Models Fail to Converge?
Bootstrap validation requires repeated model fitting. Consequently, convergence failures can occur even when the original model converges successfully.
A failed bootstrap replicate is not merely an inconvenient missing value. The frequency and pattern of failed fits can provide information about model stability.
| Observation | Potential issue to investigate |
|---|---|
| Almost all replicates converge | Generally indicates good numerical reproducibility under the bootstrap procedure. |
| A small number of failures | Investigate whether failures are random numerical events or associated with particular parameter configurations. |
| Many failures | May indicate model instability, weak identifiability, sparse information, or numerical problems. |
| Repeated boundary estimates | May indicate weakly supported variance or correlation parameters. |
| Extreme parameter estimates | May indicate unstable estimation or highly influential resampled subjects. |
Bootstrap summaries should therefore report how many replicates were successfully fitted and, when relevant, describe the reasons for unsuccessful runs.
12. How Many Bootstrap Replicates Are Needed?
The number of bootstrap replicates affects the stability of the empirical bootstrap distribution and its reported percentiles.
There is no universal number that is optimal for every population PK model. The appropriate number depends on the purpose of the bootstrap, the complexity of the model, the desired precision of percentile estimates, and the computational resources available.
In practice, population PK bootstrap analyses may use hundreds or thousands of replicates. More replicates generally provide a more stable representation of the tails of the bootstrap distribution, although additional replicates do not compensate for a poorly specified model or uninformative dataset.
13. Should the Bootstrap Preserve Study Structure?
Sometimes a simple unrestricted subject-level bootstrap is not sufficient to represent the study design.
For example, a dataset may contain treatment groups, dose cohorts, study centers, age strata, or other important design features. If those features are important to the model or scientific question, the resampling strategy may need to preserve relevant structure.
| Situation | Possible bootstrap consideration |
|---|---|
| Multiple treatment groups | Consider whether each group should be represented appropriately in every replicate. |
| Multiple studies pooled together | Consider stratification by study when study-level structure is important. |
| Different dose cohorts | Consider preserving the relevant dose structure. |
| Very small subgroups | Unrestricted resampling may produce replicates with poor representation of important subgroups. |
The resampling strategy should therefore be connected to the sampling structure of the original analysis rather than treated as a purely mechanical procedure.
14. Worked Example: Bootstrap Validation of a Population PK Model
Consider a hypothetical population PK analysis of an IV drug in 120 subjects. A one-compartment model with first-order elimination is fitted using a proportional residual error model.
The original model produces the following estimates:
| Parameter | Original estimate |
|---|---|
| CL | 8.4 L/h |
| V | 42 L |
| IIV on CL | 35% |
| IIV on V | 28% |
| Proportional residual error | 20% |
Step 1: Generate bootstrap populations
For each replicate, 120 subjects are sampled with replacement from the original 120 subjects. Complete subject records are retained whenever a subject is selected.
Step 2: Refit the model
The same structural and statistical model is fitted to every bootstrap dataset.
Step 3: Summarize the bootstrap estimates
Suppose 1,000 successful bootstrap fits produce the following hypothetical results for clearance:
| Statistic | Bootstrap CL |
|---|---|
| Mean | 8.6 L/h |
| Median | 8.5 L/h |
| Standard deviation | 0.9 L/h |
| 2.5th percentile | 6.9 L/h |
| 97.5th percentile | 10.4 L/h |
Step 4: Calculate bootstrap bias
The relative bias is:
Step 5: Calculate a percentile interval
Step 6: Interpret the result
The bootstrap distribution is centered close to the original clearance estimate, with the hypothetical bootstrap mean approximately 2.4% higher than the original estimate. The distribution has a finite spread, providing an empirical indication of sampling uncertainty.
15. What Should You Plot?
Bootstrap validation is much more informative when the parameter distributions are examined visually.
Useful displays include:
- Histograms or density plots of bootstrap parameter estimates.
- Box plots comparing bootstrap distributions with the original estimates.
- Scatter plots comparing bootstrap estimates with the original parameter values.
- Percentile plots showing empirical confidence intervals.
- Convergence summaries showing successful and unsuccessful replicates.
For a parameter such as clearance, the original estimate can be superimposed on the bootstrap distribution. A useful visual question is whether the original estimate lies in a plausible and well-populated portion of the bootstrap distribution.
A bootstrap parameter distribution can reveal spread, skewness, outliers, and the location of the original estimate.
16. Bootstrap as a Test of Model Stability
One of the most useful aspects of bootstrap validation is that it evaluates the complete modeling procedure repeatedly rather than evaluating only the original parameter estimates.
A model may converge on the original dataset but behave poorly when a modestly different set of subjects is analyzed. Bootstrap resampling can expose this sensitivity.
Particular attention should be paid to:
- parameters that frequently approach boundaries;
- very wide or highly skewed parameter distributions;
- extreme parameter estimates in individual replicates;
- repeated changes in the relative importance of covariate effects;
- frequent minimization or convergence failures;
- substantial changes in objective-function behavior; and
- bootstrap estimates that are systematically different from the original estimates.
17. What Bootstrap Validation Cannot Tell You
Bootstrap validation is powerful, but it does not answer every model-validation question.
- It is an internal validation method. The bootstrap repeatedly resamples from the observed dataset and therefore does not provide an independent external validation population.
- It cannot rescue a misspecified model. A systematically incorrect structural model can still produce stable bootstrap estimates.
- It depends on the original dataset. If the dataset poorly represents the target population, bootstrap results inherit that limitation.
- It does not prove biological correctness. Parameter stability is not equivalent to mechanistic truth.
- It can be computationally expensive. Every replicate requires another model fit.
- Failed fits require investigation. Simply deleting unsuccessful replicates without understanding why they failed can obscure important information.
Bootstrap validation should therefore be considered one component of a broader population PK model evaluation strategy.
18. Bootstrap Versus Other Population PK Evaluation Methods
Different model-evaluation methods answer different questions.
| Method | Primary purpose |
|---|---|
| Bootstrap | Assess parameter stability, empirical uncertainty, bias, and reproducibility through resampling. |
| Visual predictive check | Compare observed data with prediction distributions generated from the model. |
| Prediction-corrected VPC | Assess predictive performance while accounting for changes in typical predictions across covariate or design space. |
| Goodness-of-fit diagnostics | Evaluate relationships between observations, predictions, residuals, and individual predictions. |
| External validation | Evaluate predictive performance in an independent dataset. |
These methods are complementary. A model can have stable bootstrap parameter estimates while still showing systematic predictive deficiencies in a VPC or external validation dataset.
19. A Practical Bootstrap Validation Workflow
- Finalize the candidate model. The bootstrap should generally be performed after the structural and statistical model has been specified.
- Fit the original model. Record parameter estimates, uncertainty measures, diagnostics, and convergence information.
- Define the resampling unit. For conventional population PK bootstrapping, this is typically the subject.
- Decide whether stratification is needed. Consider treatment, study, dose, center, or other important design structures.
- Generate bootstrap datasets. Sample subjects with replacement.
- Refit the identical model. Use the same structural model, covariate model, and residual-error model.
- Record every result. Save parameter estimates, convergence status, and relevant estimation diagnostics.
- Investigate failed or unusual fits. Do not treat them as automatically ignorable.
- Summarize parameter distributions. Examine medians, means, standard deviations, percentiles, and confidence intervals.
- Compare bootstrap results with the original model. Evaluate bias, precision, skewness, and stability.
- Integrate the findings with other diagnostics. Consider GOF plots, VPCs, prediction-corrected VPCs, and external validation where appropriate.
20. How Should Bootstrap Results Be Interpreted?
A useful bootstrap interpretation considers several features simultaneously rather than focusing on a single confidence interval.
| Finding | What it may suggest |
|---|---|
| Original estimate near bootstrap center | The original estimate is reasonably consistent with the bootstrap distribution. |
| Bootstrap distribution is narrow | The parameter is relatively stable under resampling. |
| Bootstrap distribution is very wide | The parameter has substantial empirical sampling uncertainty. |
| Strong skewness | The parameter distribution may not be well summarized by symmetric measures. |
| Large bootstrap bias | The original estimate differs materially from the center of the bootstrap distribution and warrants investigation. |
| Frequent convergence failures | The model may be sensitive to the composition of the dataset or numerically unstable. |
Importantly, none of these findings should be interpreted in isolation. A bootstrap result is most informative when considered together with the model's scientific purpose, parameter identifiability, diagnostic plots, and predictive performance.
21. Key Takeaways
- Bootstrap validation is an internal validation technique that repeatedly resamples subjects and refits a population PK model.
- Subjects are generally resampled as complete units so that within-subject observation structure is preserved.
- Each bootstrap replicate produces a new set of population PK parameter estimates.
- The resulting empirical distributions provide information about parameter stability and sampling uncertainty.
- Bootstrap percentile intervals can be constructed from empirical parameter quantiles.
- Bootstrap bias can be assessed by comparing the mean or median bootstrap estimate with the original estimate.
- Bootstrap distributions can reveal skewness, extreme estimates, unstable parameters, and numerical problems.
- Convergence failures should be investigated rather than automatically discarded without explanation.
- Covariate effects can also be evaluated for stability across bootstrap populations.
- The resampling strategy should respect important features of the study design when appropriate.
- Bootstrap validation is complementary to goodness-of-fit diagnostics, VPCs, prediction-corrected VPCs, and external validation.
- A stable bootstrap does not prove that the structural model is biologically correct or predictive outside the observed population.
- Bootstrap results should be interpreted in the context of the scientific question, data quality, model identifiability, and other model-evaluation evidence.
Where to Go Next
A natural progression is to study Visual Predictive Checks for Population PK, followed by Prediction-Corrected Visual Predictive Checks. These methods complement bootstrap validation by focusing more directly on the model's ability to reproduce the observed distribution of concentrations over time.
It is also useful to study parameter uncertainty, shrinkage, residual diagnostics, covariate stability, and external validation as components of a broader population PK model evaluation strategy.