Bootstrapping and Resampling Methods in Clinical Trials

Introduction

Clinical-trial statisticians frequently need to estimate uncertainty for statistics whose sampling distributions are difficult to derive analytically.

Examples include:

• Median treatment differences
• Risk differences
• Ratios of medians or means
• Restricted mean survival time differences
• Quantiles
• Model-derived treatment effects
• Prediction or classification performance measures
• Complex composite estimands

For some statistics, classical formulas provide an exact or highly accurate standard error. For others, the mathematics becomes complicated quickly.

Bootstrapping provides a general computational approach. Instead of deriving the sampling distribution of a statistic analytically, we repeatedly resample the observed data and calculate the statistic again.

What Is Resampling?

Resampling refers broadly to statistical methods that repeatedly construct alternative datasets from observed data.

Several different techniques fall under the resampling umbrella.

These methods answer related but different questions.

This tutorial focuses primarily on the bootstrap, while also explaining how other resampling methods fit into clinical-trial analysis.

Why Bootstrap Methods Are Useful

Suppose a clinical trial compares two treatment groups. The investigator wants to estimate the difference in median change from baseline.

A mean difference has a familiar standard-error formula. A difference between medians does not have an equally simple general formula, particularly when the distributions are skewed or contain substantial outliers.

The bootstrap provides an alternative:

The Empirical Distribution

Suppose the observed data are:

The empirical distribution assigns probability:

to each observed patient.

The bootstrap samples from this empirical distribution rather than directly from the unknown population distribution.

For a dataset containing 100 patients, each patient therefore has probability 1/100 of being selected on each bootstrap draw.

What Does "With Replacement" Mean?

Suppose the original dataset contains five observations:

A possible bootstrap sample is:

Another might be:

Both samples contain five observations, but the observations are not required to be unique.

This process approximates repeated sampling from the population.

The Bootstrap Algorithm

Let:

• \(n\) = original sample size
• \(\hat{\theta}\) = statistic calculated from the original sample
• \(B\) = number of bootstrap replicates
• \(\hat{\theta}^{*}_b\) = statistic from bootstrap sample \(b\)

The algorithm is:

1. Draw a bootstrap sample of size \(n\) with replacement.
2. Calculate the statistic.
3. Store the result.
4. Repeat \(B\) times.
5. Analyze the resulting \(B\) bootstrap statistics.

The bootstrap distribution is therefore:

How Many Bootstrap Replicates?

The number of bootstrap replicates is denoted by \(B\).

Common choices include:

Increasing \(B\) does not create additional clinical information. It only reduces the Monte Carlo error associated with approximating the bootstrap distribution.

A Simple Clinical Example

Suppose a Phase II study measures change from baseline in a continuous biomarker for 20 patients. Assume the observed treatment-group changes are:

Suppose the investigator is particularly interested in the median change.

The observed median is calculated from the original dataset. Then thousands of bootstrap samples are generated and the median is calculated for each one.

The Bootstrap Distribution of the Median

After \(B\) bootstrap replicates, we obtain:

The spread of this distribution reflects the estimated sampling variability of the median.

A bootstrap standard error can be calculated as:

where:

is the mean of the bootstrap estimates.

Bootstrap Bias

The bootstrap can also be used to estimate bias.

The bootstrap estimate of bias is:

If the average bootstrap estimate is close to the original estimate, estimated bias is small.

If the bootstrap distribution is systematically shifted relative to the original estimate, the statistic may have noticeable finite-sample bias.

Bootstrap Confidence Intervals

One of the most common applications is constructing a confidence interval.

Several bootstrap confidence-interval methods are available.

Percentile Bootstrap Interval

The percentile interval is especially easy to understand. For a 95% confidence interval, take the 2.5th and 97.5th percentiles of the bootstrap distribution.

For example, suppose the bootstrap distribution of a treatment-effect estimate has:

and:

The percentile bootstrap 95% confidence interval is therefore:

Interpreting the Percentile Interval

The interval does not mean that 95% of future patient-level observations will fall between 1.8 and 7.4.

It is an interval estimate for the underlying statistical parameter.

For a treatment effect, interpretation should therefore be phrased in terms of the population treatment effect, not individual patient outcomes.

The Basic Bootstrap Interval

The basic bootstrap interval reflects the bootstrap quantiles around the observed estimate.

If:

and the bootstrap quantiles are:

the basic interval is:

giving:

The basic and percentile methods can therefore produce different intervals when the bootstrap distribution is asymmetric.

The Normal Bootstrap Interval

The normal bootstrap interval uses the bootstrap standard error:

For a 95% interval:

This approach is simple but assumes that the sampling distribution is sufficiently well approximated by a normal distribution.

BCa Confidence Intervals

BCa stands for bias-corrected and accelerated.

The BCa method adjusts the bootstrap percentile cutoffs to account for two features:

• Bias: asymmetry between the bootstrap distribution and the observed estimate.
• Acceleration: changes in the standard error of the statistic across different regions of the data.

The acceleration parameter is commonly estimated using jackknife calculations.

BCa intervals can be particularly useful when the bootstrap distribution is skewed or the statistic behaves nonlinearly.

Bootstrap Treatment Effects

Bootstrap methods can be applied to treatment effects directly. Suppose a randomized trial compares treatment \(T\) with control \(C\). Let:

A bootstrap confidence interval can be obtained by repeatedly resampling patients within treatment groups and recalculating the treatment difference.

Stratified Bootstrap for Randomized Trials

Suppose randomization was stratified by region. If the bootstrap ignores the treatment and stratification structure, the resampling scheme may not reflect the design that generated the data.

A common approach is to resample within appropriate strata and treatment groups.

The exact resampling scheme should correspond to the estimand and analysis model.

Bootstrap and the Randomization Structure

Randomized clinical trials have an important feature that observational datasets do not: treatment assignment was generated by a prespecified randomization mechanism.

Therefore, bootstrap resampling should not automatically be interpreted as a replacement for randomization-based inference.

For example:

• Bootstrap methods can estimate uncertainty for an estimator.
• Permutation or randomization methods can evaluate treatment assignments under a null hypothesis.
• Model-based methods can incorporate covariates and repeated measurements.

These approaches solve different statistical problems.

Bootstrap vs. Permutation Testing

A Key Clinical-Trial Example: Risk Difference

Suppose a randomized trial observes:

The observed risk difference is:

The point estimate suggests a 6-percentage-point reduction in risk.

We can bootstrap the risk difference by resampling patients within each treatment group and recalculating:

After thousands of repetitions, the empirical distribution of \(RD^{*}\) can be used to construct a confidence interval.

Bootstrap Odds Ratios

Odds ratios can also be bootstrapped. Suppose:

Because odds ratios are often skewed, the bootstrap distribution may be noticeably asymmetric.

An important practical consideration is whether to bootstrap the odds ratio itself or a transformed parameter such as:

The appropriate choice depends on the inferential method and analysis plan.

Bootstrap Hazard Ratios

Time-to-event endpoints introduce additional complications. Suppose the treatment effect is represented by a Cox proportional-hazards model:

The hazard ratio is:

A bootstrap analysis can repeatedly resample patients, fit the Cox model, and store the estimated hazard ratio.

However, the resampling procedure must preserve the fact that each patient contributes a complete time-to-event record.

Bootstrap for Restricted Mean Survival Time

Restricted mean survival time, or RMST, is another setting in which bootstrap methods can be useful.

Suppose the estimand is:

where \(\tau\) is a prespecified restriction time.

The bootstrap procedure can repeatedly resample patients within treatment groups, estimate RMST in each sample, and calculate:

The resulting bootstrap distribution can then be used to estimate the uncertainty of the RMST treatment effect.

Bootstrap for Median Survival

The median survival time can be difficult to estimate reliably when there are few events or substantial censoring.

Bootstrap methods can sometimes be used to quantify uncertainty, but the resampling distribution can behave poorly when the median is poorly defined.

Bootstrap and Clustered Data

Many clinical datasets contain correlated observations. Examples include:

• Repeated measurements within patients
• Multiple lesions within a patient
• Patients nested within centers
• Paired organs or anatomical sites
• Longitudinal laboratory measurements

The observations within a cluster are not independent.

A naive bootstrap that samples individual observations independently can destroy this dependence structure.

Cluster Bootstrap

The solution is often to resample the cluster rather than the individual observation.

If the patient is the independent sampling unit, sample patients with replacement and retain all observations belonging to each selected patient.

For most patient-level clinical-trial analyses, the patient is the natural resampling unit.

Longitudinal Clinical-Trial Data

Consider a study with measurements at:

A patient may therefore contribute four rows.

A naive row-level bootstrap could select Week 4 from one copy of a patient and Week 12 from another copy while omitting the other visits.

That would create artificial datasets that do not resemble the original sampling process.

Instead, bootstrap the patient as a unit and carry the patient's complete trajectory into each replicate.

Bootstrap for Mixed Models

Mixed-effects models are commonly used for repeated clinical measurements. A model might be written as:

where \(b_i\) is a patient-specific random effect.

A bootstrap can repeatedly resample patients, refit the complete model, and store the treatment-by-time estimate:

This can provide an empirical distribution for the treatment-by-time effect.

Bootstrap and Missing Data

Missing data require particular care. Suppose a longitudinal trial has missing Week 12 measurements. Simply resampling the observed records does not automatically account for the missing-data mechanism.

The analyst must distinguish between:

• Resampling uncertainty
• Missing-data uncertainty
• Model uncertainty
• Treatment-assignment uncertainty

A bootstrap procedure can address one component of uncertainty without automatically addressing the others.

Bootstrap After Multiple Imputation

When multiple imputation is used, the analysis can become more complex.

A principled procedure needs to account for both:

• Variation between imputed datasets
• Sampling variation represented by the bootstrap

There are several possible bootstrap-plus-imputation strategies, and their validity depends on the missing-data assumptions and estimand.

Bootstrap and Small Samples

The bootstrap is often described as a distribution-free or nonparametric method. That description requires qualification.

The bootstrap does not magically eliminate small-sample problems.

If the original dataset is tiny, the empirical distribution contains very limited information about the population.

For example, if only six patients are observed, bootstrap samples repeatedly reuse those same six observations.

The resulting distribution may provide a useful sensitivity analysis, but it cannot manufacture information that the study did not collect.

Bootstrap Failure Modes

Bootstrap methods can perform poorly in several situations.

Rare Events and Bootstrap Instability

Suppose a safety endpoint occurs in only two patients. A bootstrap sample can easily contain zero events.

For an odds ratio or relative risk, this can result in:

• Zero cells
• Infinite estimates
• Undefined estimates
• Extremely skewed bootstrap distributions

This does not necessarily mean the bootstrap code is wrong. It may indicate that the estimator itself is poorly behaved with the available data.

Bootstrap and Zero Cells

Consider a \(2\times2\) table in which no treatment patients experience an event.

A bootstrap sample may reproduce the zero-event structure in many replicates.

For a ratio measure such as:

The appropriate response may involve an alternative estimator, exact methods, penalized methods, or a prespecified continuity correction—not simply deleting failed bootstrap replicates.

Bootstrap Standard Error vs. Standard Error

A bootstrap standard error estimates the variability of an estimator from the bootstrap distribution.

A model-based standard error may instead be derived from an assumed probability model.

Bootstrap Does Not Mean Assumption-Free

One of the most common misconceptions is:

"The bootstrap requires no assumptions."

That is too strong.

The bootstrap assumes that the observed sample provides a reasonable representation of the population or sampling mechanism being approximated.

The validity of a bootstrap therefore depends on:

• The sampling unit being correctly identified
• The estimator being appropriately defined
• The resampling scheme matching the data structure
• The sample containing sufficient information
• The treatment and design structure being respected

Bootstrap for Diagnostic Accuracy

Bootstrap methods are frequently useful when estimating diagnostic performance measures. Examples include:

• Area under the ROC curve
• Sensitivity
• Specificity
• Positive predictive value
• Negative predictive value
• Youden's index

For example, Youden's index is:

The sampling distribution of \(J\) can be complicated. A bootstrap can repeatedly resample subjects, recalculate sensitivity and specificity, and obtain:

Bootstrap ROC Analysis

For an ROC analysis, each bootstrap replicate can recalculate the entire ROC curve and its area under the curve.

The bootstrap distribution is then:

A percentile or BCa confidence interval can be constructed from this distribution.

Bootstrap and Nonlinear Estimators

The bootstrap is especially attractive for nonlinear statistics. Suppose the estimator is:

or:

Deriving the exact sampling variance can become complicated. The bootstrap simply recalculates the entire estimator for each resampled dataset.

Bootstrap the Estimand, Not Just a Component

A crucial principle in modern clinical-trial analysis is: the resampling procedure should reproduce the complete estimator of the prespecified estimand.

Suppose the treatment effect is derived through:

1. Covariate adjustment
2. Model fitting
3. Marginalization over a target population
4. Calculation of a treatment contrast

The bootstrap replicate should repeat these steps.

It is generally not sufficient to bootstrap a model coefficient if the reported estimand is a nonlinear transformation of that coefficient.

Bootstrap G-Computation Example

Suppose a regression model estimates a conditional treatment effect, but the reported estimand is a standardized population-average risk difference. The bootstrap procedure should:

The resulting bootstrap distribution reflects the variability of the actual reported estimand rather than merely one intermediate model parameter.

Jackknife Resampling

The jackknife is another resampling technique. For a dataset of \(n\) patients, the jackknife creates \(n\) datasets by omitting one patient at a time.

The \(i\)-th jackknife estimate is:

The jackknife mean is:

Jackknife calculations are particularly relevant to the estimation of the acceleration parameter used by BCa confidence intervals.

Bootstrap vs. Jackknife

R Implementation: Basic Bootstrap

The following example bootstraps the mean change from baseline.

set.seed(2026)

x <- c(
  -2, 1, 3, 4, 5,
   5, 6, 7, 8, 8,
   9, 10, 10, 11, 12,
   12, 13, 14, 15, 18
)

B <- 5000

boot_mean <- numeric(B)

for (b in seq_len(B)) {

  sample_b <- sample(
    x,
    size = length(x),
    replace = TRUE
  )

  boot_mean[b] <- mean(sample_b)
}

mean(x)
sd(boot_mean)

The vector boot_mean contains the bootstrap distribution of the mean.

Bootstrap the Median

set.seed(2026)

boot_median <- numeric(B)

for (b in seq_len(B)) {

  sample_b <- sample(
    x,
    size = length(x),
    replace = TRUE
  )

  boot_median[b] <- median(sample_b)
}

median(x)
sd(boot_median)

The bootstrap standard deviation provides an estimate of the standard error of the median.

Percentile Confidence Interval in R

quantile(
  boot_median,
  probs = c(0.025, 0.975)
)

This returns the 95% percentile bootstrap confidence interval.

Bootstrap With the boot Package

The R package boot provides a widely used framework for bootstrap calculations.

library(boot)

statistic_median <- function(data, indices) {

  sample_data <- data[indices]

  median(sample_data)
}

set.seed(2026)

boot_result <- boot(
  data = x,
  statistic = statistic_median,
  R = 5000
)

boot_result

Percentile and BCa Intervals in R

boot.ci(
  boot_result,
  type = c("perc", "bca")
)

The output provides percentile and BCa intervals.

The BCa interval is particularly useful when the bootstrap distribution is asymmetric or the statistic has nontrivial bias or curvature.

Bootstrap a Two-Group Treatment Difference

Suppose the treatment and control groups are:

treatment <- c(
  8, 9, 10, 12, 13,
  14, 15, 16, 18, 20
)

control <- c(
  4, 5, 6, 8, 9,
  10, 11, 12, 13, 15
)

The treatment effect can be defined as a difference in medians:

median(treatment) -
median(control)

A group-specific bootstrap can then be constructed:

set.seed(2026)

B <- 5000

boot_difference <- numeric(B)

for (b in seq_len(B)) {

  t_b <- sample(
    treatment,
    size = length(treatment),
    replace = TRUE
  )

  c_b <- sample(
    control,
    size = length(control),
    replace = TRUE
  )

  boot_difference[b] <-
    median(t_b) - median(c_b)
}

quantile(
  boot_difference,
  c(0.025, 0.975)
)

Why Resample Treatment Groups Separately?

In a simple two-arm bootstrap for a treatment-effect estimator, resampling within treatment groups preserves the observed group sizes and treatment labels.

This is appropriate when the target is the sampling distribution conditional on the observed treatment-group structure.

It is different from a permutation test, where treatment labels may be reassigned to construct a null distribution.

Bootstrap a Risk Difference in R

set.seed(2026)

event_t <- 18
n_t <- 200

event_c <- 30
n_c <- 200

B <- 5000

boot_rd <- numeric(B)

for (b in seq_len(B)) {

  y_t <- c(
    rep(1, event_t),
    rep(0, n_t - event_t)
  )

  y_c <- c(
    rep(1, event_c),
    rep(0, n_c - event_c)
  )

  t_b <- sample(
    y_t,
    size = n_t,
    replace = TRUE
  )

  c_b <- sample(
    y_c,
    size = n_c,
    replace = TRUE
  )

  boot_rd[b] <-
    mean(t_b) - mean(c_b)
}

quantile(
  boot_rd,
  c(0.025, 0.975)
)

Cluster Bootstrap in R

Suppose each patient has repeated observations. The bootstrap should sample patients rather than individual rows.

set.seed(2026)

patient_ids <- unique(dat$USUBJID)

B <- 5000

boot_estimates <- numeric(B)

for (b in seq_len(B)) {

  sampled_ids <- sample(
    patient_ids,
    size = length(patient_ids),
    replace = TRUE
  )

  boot_dat <- do.call(
    rbind,
    lapply(
      seq_along(sampled_ids),
      function(i) {

        id <- sampled_ids[i]

        d <- dat[
          dat$USUBJID == id,
        ]

        d$BOOT_ID <- i

        d
      }
    )
  )

  # Fit the complete analysis model here.
  # boot_estimates[b] <- ...
}

The key concept is that each selected patient brings all of that patient's measurements into the bootstrap replicate.

Bootstrap Survival Analysis in R

For survival analysis, the patient-level dataset might contain:

• Time-to-event
• Event indicator
• Treatment assignment
• Baseline covariates

A bootstrap replicate should resample complete patient records.

library(survival)

boot_cox <- function(data, indices) {

  d <- data[indices, ]

  fit <- coxph(
    Surv(time, status) ~ treatment + age,
    data = d
  )

  exp(coef(fit)["treatment"])
}

The function can then be passed to boot().

set.seed(2026)

cox_boot <- boot(
  data = survival_data,
  statistic = boot_cox,
  R = 5000
)

boot.ci(
  cox_boot,
  type = c("perc", "bca")
)

Bootstrap and the Cox Model

When bootstrapping a Cox model, the complete model-fitting process should be repeated for every replicate.

That includes:

• Risk-set construction
• Coefficient estimation
• Covariate adjustment
• Transformation to the hazard ratio

If the analysis uses a stratified Cox model, robust variance, time-varying covariates, or other special structures, the bootstrap procedure must preserve those features.

Bootstrap Model Selection

Bootstrap methods can also be used to evaluate model-selection stability. For example, an investigator might repeatedly bootstrap the dataset and record which covariates are selected by a variable-selection procedure.

This can answer: How stable is the selected model under perturbations of the observed sample?

However, bootstrap model selection should not automatically be interpreted as confirmatory evidence.

Bootstrap and Prediction

Resampling methods can quantify uncertainty in predictive performance. Examples include:

• AUC
• Brier score
• Calibration slope
• Prediction error
• Mean absolute error

A bootstrap can repeatedly fit the predictive model and evaluate its performance.

This is often more informative than calculating performance only once on the development dataset.

Optimism in Predictive Modeling

A model evaluated on the same data used to fit it often appears more accurate than it will perform in new patients.

Bootstrap methods can estimate this optimism. A simplified procedure is:

Bootstrap vs. Cross-Validation

These approaches are related but should not be treated as interchangeable.

Bootstrap Hypothesis Testing

Bootstrap methods can be used in hypothesis testing, but ordinary bootstrap resampling does not automatically impose a null hypothesis.

For a null hypothesis:

the bootstrap distribution generated directly from the observed data may not represent the null distribution.

This is an important distinction from permutation or randomization methods.

Bootstrap Confidence Intervals vs. P-Values

A bootstrap confidence interval and a hypothesis-test p-value are related but are not automatically interchangeable.

For a clinical-trial report, the statistical analysis plan should specify:

• The estimand
• The point estimator
• The confidence-interval method
• The confidence level
• The hypothesis-testing method, if applicable
• The handling of multiplicity

Bootstrap and Multiplicity

Suppose a trial has several endpoints:

• Overall survival
• Progression-free survival
• Objective response
• Quality of life

Constructing a separate 95% bootstrap confidence interval for each endpoint does not automatically solve multiplicity.

Multiplicity concerns the collection of statistical inferences being made.

Bootstrap methodology and multiplicity adjustment are therefore separate issues.

Bootstrap and Subgroup Analyses

Subgroup analyses can produce unstable bootstrap distributions when subgroup sample sizes are small.

For example, if a treatment-by-sex interaction is estimated from a small subgroup, many bootstrap samples may contain very few observations in one subgroup.

Before using a bootstrap interval, inspect:

• Subgroup sample size
• Number of events
• Bootstrap failures
• Distributional asymmetry
• Extreme estimates

Bootstrap Diagnostics

A professional bootstrap analysis should not end with a confidence interval. The bootstrap distribution itself should be inspected.

Useful diagnostics include:

• Histogram or density of bootstrap estimates
• Observed estimate marked on the distribution
• Percentile endpoints
• Extreme bootstrap estimates
• Number of failed replicates
• Skewness
• Convergence problems

Failed Bootstrap Replicates

Complex models can fail to converge for some bootstrap samples.

For example:

• A logistic model may experience separation.
• A Cox model may have too few events.
• A mixed model may fail to estimate a variance component.
• A nonlinear model may fail to converge.

The analyst should not silently discard failed replicates.

Bootstrap Convergence

Suppose a bootstrap analysis uses 5,000 replicates. If the resulting 95% interval changes substantially when the analysis is rerun with 10,000 replicates and a different random seed, the Monte Carlo error may be important.

A useful sensitivity analysis is therefore:

The goal is not to find a magical value of \(B\), but to ensure that the reported inference is sufficiently stable for its intended use.

Clinical-Trial Reporting

If a bootstrap method is used in a clinical-trial analysis, the statistical report should explain the resampling procedure.

At minimum, document:

• Sampling unit
• Resampling method
• Number of bootstrap replicates
• Random-number seed when reproducibility is relevant
• Point estimator
• Confidence-interval method
• Confidence level
• Handling of failed replicates
• Handling of missing data
• Handling of clustering or stratification

Protocol and SAP Language

The statistical analysis plan should ideally define the bootstrap method before the data are unblinded for the relevant analysis.

For example, a generic SAP statement might specify that the confidence interval for a prespecified estimand will be obtained using a nonparametric patient-level bootstrap with a prespecified number of replicates and a BCa confidence interval.

The exact wording should be adapted to the estimand and study design.

A Complete Worked Treatment-Effect Example

Consider a randomized two-arm trial evaluating change from baseline in a continuous clinical outcome. Suppose:

The estimated treatment effect is:

Suppose a bootstrap analysis produces a 95% BCa interval:

The interval excludes zero.

The treatment effect is therefore estimated to be negative, with the direction defined by the prespecified treatment-minus-control estimand.

The clinical interpretation still depends on the outcome definition. If lower values represent improvement, a negative difference favors treatment.

Bootstrap Workflow for Clinical Trials

Common Mistakes

1. Sampling individual rows when the patient is the sampling unit. This can destroy within-patient correlation.
2. Assuming bootstrap means assumption-free. The method still depends on the observed sample and an appropriate resampling scheme.
3. Using the bootstrap to fix an undefined estimator. A statistic that cannot be estimated reliably from the original data does not necessarily become valid after resampling.
4. Ignoring failed model fits. Failed bootstrap replicates can be evidence of estimator instability.
5. Reporting only the final confidence interval. The bootstrap distribution and diagnostics can be important for evaluating whether the interval is trustworthy.
6. Confusing bootstrap and permutation testing. The two methods have different inferential foundations.
7. Changing the estimator inside bootstrap replicates. The complete prespecified analysis should generally be repeated.
8. Using an arbitrary confidence-interval method. Percentile, basic, normal, and BCa intervals can differ materially.
9. Assuming more bootstrap replicates solve a small-sample problem. More computation does not create more information.
10. Ignoring the randomization or stratification structure. The resampling mechanism should be compatible with the trial design and estimand.

Bootstrap vs. Parametric Modeling

Bootstrap methods and parametric methods are not necessarily competitors. They can be complementary.

Nonparametric vs. Parametric Bootstrap

The standard bootstrap described in this tutorial is generally nonparametric. It resamples directly from the observed data.

A parametric bootstrap instead:

1. Fits a probability model.
2. Uses estimated parameters to generate simulated datasets.
3. Refits the model or recalculates the estimator.
4. Repeats the process.

For example, if a model assumes:

a parametric bootstrap can generate new observations from the fitted normal distribution.

Which Bootstrap Should Be Used?

There is no universal best bootstrap method.

The appropriate method depends on:

• Study design
• Estimand
• Endpoint type
• Sampling unit
• Cluster structure
• Missing-data strategy
• Model specification
• Sample size
• Degree of censoring

A patient-level nonparametric bootstrap may be appropriate for one analysis and completely inappropriate for another.

When Bootstrap Methods Are Especially Attractive

• The estimator has no convenient closed-form standard error.
• The statistic is nonlinear.
• The sampling distribution is noticeably asymmetric.
• The endpoint is a quantile or median.
• The treatment effect is a complex function of model parameters.
• The analysis involves complicated prediction or standardization.
• A direct empirical approximation to sampling variability is desirable.

When Bootstrap Methods May Be Less Attractive

• The sample size is extremely small.
• The estimator frequently fails to converge.
• There are very few events.
• The statistic is undefined in many bootstrap samples.
• The data structure is complicated but cannot be represented correctly by a simple resampling scheme.
• The primary inferential question is fundamentally about the randomization mechanism.

The Most Important Question: What Is the Sampling Unit?

Before writing any bootstrap code, ask:

For many clinical trials, the answer is: the patient.

But other study designs may have different sampling structures.

Examples include:

The correct unit follows the sampling and randomization structure, not simply the number of rows in the analysis dataset.

Cluster-Randomized Trials

In a cluster-randomized trial, treatment may be assigned to centers, practices, schools, hospitals, or other clusters.

If the cluster is the randomization unit, a bootstrap that samples individual patients independently may fail to reproduce the study's dependence structure.

A cluster-level bootstrap may instead resample entire randomized clusters.

Bootstrap in Regulatory Clinical-Trial Analysis

Bootstrap methods can be appropriate in confirmatory clinical-trial analyses, but they should be treated as formal statistical methodology rather than an informal computational trick.

The method should be:

• Scientifically justified
• Technically validated
• Prespecified when used for primary inference
• Reproducible
• Compatible with the estimand
• Appropriately documented

For a regulated submission, reproducibility of the programming and analysis environment is especially important.

Validation of Bootstrap Programs

Bootstrap programs can be validated using several approaches.

• Independent programming
• Known simulated examples
• Seed-controlled reproducibility checks
• Comparison with established software
• Extreme-case testing
• Small hand-calculated examples
• Inspection of the number of failed replicates

Because bootstrap procedures involve random-number generation, the analysis should also document the random seed when reproducibility of a particular simulation is important.

A Reproducible Bootstrap Template in R

library(boot)

set.seed(2026)

statistic <- function(data, indices) {

  d <- data[indices, ]

  # Reproduce the complete prespecified estimator.
  estimate <- ...

  estimate
}

B <- 5000

result <- boot(
  data = analysis_data,
  statistic = statistic,
  R = B
)

print(result)

boot.ci(
  result,
  type = c("perc", "bca")
)

The most important part of the template is not the call to boot(). It is the definition of the statistic.

That function should implement the estimator that the study actually intends to report.

A Practical Checklist

Key Formulas

The central bootstrap quantities can be summarized as follows.

Bootstrap Mean

Bootstrap Standard Error

Bootstrap Bias Estimate

Percentile 95% Confidence Interval

Normal Bootstrap Interval

What Bootstrap Is Really Doing

The deepest idea behind the bootstrap is not "sampling the data many times."

It is this: use the observed sample to approximate the unknown population, then use repeated resampling to approximate what repeated clinical trials might have produced.

The original clinical trial provides the information. The bootstrap provides a computational approximation to repeated sampling from the population represented by that information.

Final Clinical-Trial Perspective

Bootstrap methods are powerful because they allow statisticians to estimate uncertainty for complicated estimators without deriving a new mathematical sampling distribution for every problem.

But the flexibility of the bootstrap creates a responsibility: the resampling scheme must reproduce the statistical structure of the study.

For a simple independent-patient analysis, this may be straightforward. For a modern clinical trial involving repeated measurements, stratification, survival outcomes, missing data, covariate adjustment, multiple imputation, or complex estimands, the procedure requires considerably more thought.

References

Efron, B. (1979). Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics, 7(1), 1–26.
Efron, B. & Tibshirani, R.J. (1993). An Introduction to the Bootstrap. Chapman & Hall/CRC.
Davison, A.C. & Hinkley, D.V. (1997). Bootstrap Methods and Their Application. Cambridge University Press.
Carpenter, J. & Bithell, J. (2000). Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. Statistics in Medicine, 19, 1141–1164.
Chernick, M.R. & LaBudde, R.A. (2011). An Introduction to Bootstrap Methods with Applications to R. Wiley.
Hastie, T., Tibshirani, R. & Friedman, J. (2009). The Elements of Statistical Learning. Springer.
Harrell, F.E. (2015). Regression Modeling Strategies. Springer.