Bootstrap Intervals That Respect Adaptation

Why the Replicate Must Include the Decision Rule

In an adaptive trial, the final estimator depends not only on outcomes but also on decisions made while the trial is running. An interim statistic may determine whether recruitment stops, whether another stage begins, or how many participants are enrolled. To bootstrap the estimator generated by that trial, each replicate must reproduce both the data-generation process and the rule that maps interim data to later trial conduct. Resampling only the final observed data while holding the observed path fixed answers a different question.

This tutorial develops a parametric bootstrap for a two-stage trial with a prespecified interim rule. The same principle applies to other adaptation rules: specify how data are generated under the fitted model, apply the original rule to each replicate’s interim data, and compute the estimator that the rule calls for. The interval developed below is an approximate, equal-tail interval based on bootstrap estimation errors. It is not an exact interval.

A Model and a Two-Stage Rule

For a concrete setting, suppose the endpoint is a continuous outcome and the treatment effect \(\theta\) is the difference between treatment and control means, in outcome units. Outcomes are independent and normally distributed within each arm, with a common, known variance \(\sigma^2\). Patients are randomized independently within each stage, and stage 1 and stage 2 observations are independent. The known variance is used here to make the simulation steps transparent; a real application must specify how variance and other nuisance parameters are estimated or generated.

Suppose stage 1 enrolls \(n_1\) participants per arm. Let \(D_1=\bar{Y}_{T1}-\bar{Y}_{C1}\) be the stage-1 treatment-minus-control difference. Under the model, its standard error is \(SE_1=\sqrt{2\sigma^2/n_1}\). The trial continues to stage 2 if the interim statistic \(Z_1=D_1/SE_1\) is at least a prespecified threshold \(c\); otherwise it stops and reports \(D_1\). If the trial continues, it enrolls \(n_2\) more participants per arm and reports the pooled difference \(\hat{\theta}=(n_1D_1+n_2D_2)/(n_1+n_2)\), where \(D_2\) is the stage-2 difference.

This rule is part of the estimator’s definition. A bootstrap replicate that continues when its own \(Z_1^*\) is below \(c\), or stops when \(Z_1^*\) is above \(c\), is not a replicate of this trial. The original rule must be applied exactly, including its threshold, equality convention, sample sizes, and estimator on each path.

Constructing the Bootstrap Distribution

Fit the model using the observed trial and obtain \(\hat{\theta}\). For each bootstrap replicate, generate stage-1 data under the fitted model, calculate \(Z_1^*\), and apply the trial’s interim rule. If the replicate stops, its estimate is the stage-1 difference \(D_1^*\). If it continues, generate independent stage-2 data and calculate the pooled estimate using the prescribed stage sizes. Record \(\hat{\theta}^*-\hat{\theta}\). Repeating this process \(B\) times gives an empirical distribution of bootstrap errors.

For the model above, one convenient implementation generates independent stage differences directly. Under the fitted model, each stage difference follows \(N(\hat{\theta},2\sigma^2/n_j)\), for stage \(j\). The simulated stage differences are sufficient for applying this particular rule and calculating this particular estimator. If the actual adaptation uses other interim data, such as separate arm summaries, subgroup outcomes, or safety outcomes, the replicate must generate the data needed to apply that rule—not merely a summary that omits relevant decision information.

From Bootstrap Errors to an Interval

Let \(q^*_{.025}\) and \(q^*_{.975}\) denote the empirical 2.5th and 97.5th percentiles of the \(B\) bootstrap errors. The corresponding basic bootstrap interval reverses the error quantiles around the observed estimate:

The justification is an approximation to the sampling distribution of the estimation error. If the bootstrap distribution of \(E^*=\hat{\theta}^*-\hat{\theta}\) adequately approximates the sampling distribution of \(E=\hat{\theta}-\theta\), then its quantiles approximate those of \(E\). Under that approximation, an equal-tail 95% interval is obtained by solving the inequalities that put \(E\) between its 2.5th and 97.5th percentiles:

Thus, if the bootstrap error quantiles are good approximations to the sampling-error quantiles, the interval has approximately 95% repeated-trial coverage under the assumed model and adaptation rule. This is a justification for an approximation, not a coverage theorem for every adaptive design. Correct model specification, suitable regularity and approximation conditions, and adequate simulation accuracy matter. In particular, a threshold rule can produce a non-smooth, mixture-shaped estimator distribution, so the adequacy of the approximation should not be taken for granted.

This construction uses error quantiles and their inversion. It is not the same calculation as simply taking the 2.5th and 97.5th percentiles of the bootstrap estimates. The distinction in the sign and order of the quantiles follows directly from the event \(\hat{\theta}-\theta=E\).

Assumptions, Diagnostics, and Alternatives

The bootstrap is only as credible as the model and the reconstruction of the trial. For the worked setting, the key assumptions are independent normally distributed outcomes with the stated common known variance, correct stage sizes and randomization, and an accurately coded interim rule and final estimator. In other settings, the resampling model must also represent relevant features such as overdispersion, clustering, censoring, missing outcomes, or nuisance-parameter estimation. If the actual interim rule depends on a feature absent from the simulated data, the procedure cannot reproduce that rule.

Several practical checks help detect implementation problems. Verify the decision at the threshold, including what happens when a statistic equals \(c\). Check that each simulated path has the appropriate sample size and that stage-2 outcomes are generated only when required. Summarize the simulated proportions stopping and continuing, and inspect the error distribution for skewness, multiple modes, or extreme tails. Compare those path frequencies and distributional features with expectations under the fitted model. Repeat the simulation with a larger \(B\) to see whether the estimated tail quantiles and interval endpoints are stable; more replicates reduce Monte Carlo error but do not repair model misspecification.

The procedure as described simulates over possible adaptation paths under the fitted model, so it targets an unconditional repeated-trial distribution. It does not condition each replicate on the observed trial having continued. A conditional interval instead targets inference given a specified selection event and requires a procedure built for that conditional target. Analytic or small-sample exact methods may be preferable when their assumptions fit and they provide a more reliable calibration. The bootstrap is useful when the full adaptive estimator can be simulated but its sampling distribution is difficult to derive; it does not automatically outperform those alternatives.

What the Interval Can—and Cannot—Guarantee

The interval’s nominal 95% level is an approximate repeated-sampling target, not a guarantee for the particular trial. Its accuracy depends on whether the fitted model generates realistic replicate data and whether the bootstrap error distribution approximates the actual error distribution despite the adaptation. Small samples, highly influential nuisance-parameter estimates, rare adaptation paths, or a rule that creates sharp changes in the estimator can make that approximation weak. A diagnostic plot or stable Monte Carlo quantile is useful evidence about computation, but neither establishes model adequacy or coverage.

When reporting the result, state the estimand, the model used to generate replicates, the adaptation rule replayed, the error-quantile interval method, and the number of replicates. Also identify whether the target is unconditional over adaptation paths or conditional on a specified path. These details make clear what the interval is intended to quantify.