Why Naive Bootstrap Intervals Fail

What a Naive Bootstrap Leaves Out

A bootstrap interval is only as informative as the resampling process used to construct it. In a fixed-design analysis, resampling observations within the observed groups can approximate repeated sampling under that design. In an adaptive trial, however, the observed groups, sample size, or stopping time may themselves be consequences of interim data. If resampling holds those decisions fixed, it varies the final observations but not the path that led to them.

This distinction matters because selection and stopping can change the distribution of the reported estimator. For example, an arm may be selected precisely because its interim estimate was unusually high. Repeatedly resampling the selected arm as though it had been specified in advance does not recreate occasions on which a different arm would have been selected. Likewise, resampling the data from a trial that stopped early does not recreate trials that would have continued.

Here, a naive bootstrap interval means an interval formed by resampling the observed final data while treating the realized adaptive decisions as fixed. Such an interval is not automatically invalid in every setting: it may approximate uncertainty conditional on a fixed path under suitable assumptions. The concern is using it as though it gave the intended coverage over repeated trials in which adaptation can produce different paths.

Selection Changes the Distribution Being Covered

Consider a simplified model with \(K\) treatment arms. Each arm has the same true effect \(\theta\), and its estimate has a known standard error \(\sigma\). Suppose the standardized errors are independent and normally distributed:

The trial selects the arm with the largest estimate and reports the usual two-sided interval for that arm, \(\hat{\theta}_{\max}\pm c\sigma\), where \(\hat{\theta}_{\max}=\max_j\hat{\theta}_j\) and \(c\) is the nominal normal critical value. The model assumes independent normal estimates with the same standard error; these assumptions make the calculation transparent, rather than describing every real adaptive trial.

The derivation subtracts the probability that every estimate lies below \(-c\) from the probability that every estimate lies below \(c\). That difference accounts for the selected estimate being the maximum. A bootstrap that resamples only the winning arm and fixes its identity instead approximates the variation of one prespecified estimate; it does not reproduce the maximum operation.

The selected estimate is also systematically high in this equal-effects example: taking the maximum favors positive errors. This is often called selection bias or a winner's curse. An interval's coverage and an estimator's bias are related but distinct properties: bias alone does not determine coverage, and the calculation above establishes the coverage result for the particular interval and model.

Stopping Can Change Coverage Too

A similar issue arises when the trial's stopping time depends on interim evidence. To isolate the effect, suppose there are two independent estimates of the same parameter, each with standard error \(\sigma\), and each standardized error is \(N(0,1)\). The design examines the first estimate and stops if its usual two-sided 95% interval excludes \(\theta\). If it does not stop, the design reports the second estimate and its interval. The independence assumption is deliberately simple; actual sequential data often have correlated cumulative estimates.

A practical interpretation depends on the target guarantee. A conditional interval asks about uncertainty given a specified path or decision; an unconditional interval asks about performance across repetitions in which the path may change. These are different targets. Fixing the selected arm or stopping time can be compatible with a clearly stated conditional target, but it cannot be assumed to deliver an unconditional guarantee merely because the bootstrap interval has a familiar nominal level.

What the Resampling Scheme Actually Repeats

To evaluate a bootstrap, trace one replicate from beginning to end. Ask what is resampled, what is recalculated, and which rules are replayed. If the replicate resamples only the observed winning arm, the resampling distribution cannot include another arm becoming the winner. If it resamples final data at the observed sample size, it cannot include a different stopping time. Those omissions are structural; increasing the number of bootstrap replicates reduces Monte Carlo noise but does not fix them.

For a selection design, a procedure that aims to reproduce unconditional uncertainty would generally need to represent the candidate arms and rerun the selection rule in each replicate. For a stopping design, it would need to represent the sequence of interim data and rerun the stopping rule. The earlier material on bootstrap intervals that respect adaptation addresses this design-reproducing principle. Even then, validity depends on whether the resampling model appropriately represents the data-generating process and the adaptive rule; rerunning a rule cannot repair a poor model or inadequate data.

Other approaches answer related but distinct questions. A fixed-path bootstrap describes variation after treating the observed decisions as given. Analytic adjustment can model selection or stopping directly when the relevant distribution is tractable. A design-based or randomization-based analysis uses the trial's assignment mechanism as its basis. None is automatically preferable for every estimand: the choice must match the target and the assumptions that can be defended. The key diagnostic is not the method's label, but whether its repetitions reproduce the source of uncertainty the interval claims to cover.

Diagnostics, Failure Modes, and Limits

• Check whether decisions vary across replicates. If every bootstrap replicate retains the same winning arm or stopping time by construction, the procedure is not estimating variation across those decisions.
• Check the target of inference. State whether coverage is intended conditional on the observed adaptation path or unconditional over possible paths. Do not report one as though it were the other.
• Look for near-ties and decision boundaries. If two arms have similar interim estimates, modest data perturbations may change the winner. A fixed-winner bootstrap hides this instability.
• Distinguish sampling variation from adaptation variation. More resamples can make a conditional bootstrap calculation numerically stable while leaving its omitted adaptation uncertainty untouched.
• Inspect model and data limitations. A design-reproducing bootstrap can still perform poorly with sparse data, misspecified outcome distributions, or an inaccurately encoded adaptation rule. Its interval is not automatically a coverage guarantee.

A common mistake is to argue, “The bootstrap resampled the final observations many times, so it included all relevant uncertainty.” The correction is to identify which decisions were recomputed. Resampling many observations from a fixed selected arm can represent within-arm sampling variability, but it cannot represent the chance that another arm would have been selected. Similarly, resampling a fixed-size final data set cannot represent continuation or early stopping if the stopping rule is never replayed.