Why resampling the observed data can fail
A bootstrap interval can be carefully computed and still answer the wrong sampling question. In a group sequential trial, the data available for analysis are selected by a rule: monitoring continues or stops depending on interim results. A bootstrap that resamples only the outcomes observed at the stopping time usually treats that selected data set as though it had arisen from an ordinary fixed-sample design. It does not recreate the chance that a different trial path would have crossed a boundary at a different time, or would have continued instead.
The issue is not that bootstrap resampling is inherently unsuitable. The issue is what is resampled and what is repeated. To estimate uncertainty under the sequential design, resampling must account for the rule that determined the observed stopping stage. Earlier material on bootstrap intervals that reproduce the stopping rule develops that approach. Here the focus is the failure mode when the rule is ignored.
A normal model makes the selection visible
Consider a hypothetical trial with independent participant outcomes \(X_i\sim N(\theta,\sigma^2)\), where \(\theta\) is the treatment effect in the outcome’s units and \(\sigma\) is known. At the first look, \(n_1\) participants have been observed, and the estimate is \(\bar{x}_1\). Under the null value \(\theta_0\), define
Suppose the design stops for efficacy when \(Z_1\ge c\), for a prespecified boundary \(c\). For this illustration, assume the model is correct, outcomes are independent and normally distributed, \(\sigma\) is known, and the boundary and analysis schedule were set before outcomes were examined. These assumptions give \(Z_1\sim N(0,1)\) when \(H_0:\theta=\theta_0\) is true. The selection event is therefore an explicit part of the sampling process, not an incidental feature of the observed sample.
If we condition on an efficacy stop, the distribution of \(Z_1\) is not the unselected standard normal distribution. Its density is the standard normal density restricted to values at least \(c\), and renormalized by the probability of that event. Consequently, its conditional mean is positive under the null:
Here \(\phi\) and \(\Phi\) are the standard normal density and cumulative distribution function. The second equality follows because \(\phi'(z)=-z\phi(z)\), so \(\int_c^\infty z\phi(z)\,dz=\phi(c)\). Since the conditional mean is positive, the treatment-effect estimate is, on average, above \(\theta_0\) among trials that stop at this efficacy boundary.
Worked Example: Bias Among Early Efficacy Stops
Suppose a hypothetical trial has \(\sigma=1\) outcome unit, \(n_1=25\), and an efficacy boundary \(c=1.96\). Under \(H_0:\theta=0\), the standard error of the first-look mean is
For a standard normal variable, \(\phi(1.96)\approx0.05844\) and \(1-\Phi(1.96)\approx0.02500\). Thus,
Because \(\bar{X}_1-\theta_0=SE\,Z_1\), the conditional mean of the estimate’s error is
So even when the true effect is zero, the average estimate among trials crossing this boundary is about 0.468 outcome units above zero. This is a statement about the selected group of trials, not a claim that every estimate is biased by exactly that amount.
What the naive bootstrap does—and does not—repeat
At an early stop, an ordinary participant-level bootstrap draws \(n_1\) outcomes with replacement from the observed \(n_1\) outcomes and recalculates the mean. This procedure can represent resampling variation around the empirical distribution of those outcomes. But the observed outcomes were collected on a path that crossed the boundary. The bootstrap does not generate a fresh trial path from a specified treatment effect, allow interim results to vary under that model, and apply the stopping rule to those results. In particular, it does not restore the unobserved paths that would have continued or stopped at another look.
The distinction can be expressed as a difference in the target distribution. A fixed-sample bootstrap approximates variation conditional on the empirical data. Inference after stopping instead concerns a sampling distribution shaped by the model and by the rule mapping interim data to a stopping stage. Conditioning on the realized stage may be appropriate for a conditional analysis, but even then the relevant distribution must account for the event defining that stage. Simply holding the observed outcomes and stage fixed does not do so.
Worked Example: A Naive Interval After a Selected Stop
Suppose a hypothetical trial observes \(n_1=25\) outcomes with \(\sigma=1\) outcome unit and stops at the first look. The observed mean is \(\bar{x}_1=0.44\) outcome units, and the null value is \(\theta_0=0\). The observed statistic is
This exceeds the hypothetical efficacy boundary \(1.96\), so the early stop is consistent with the stated rule. A simple normal-based 95% fixed-sample interval, which a naive bootstrap might approximate in this setting, is
The interval excludes zero. That calculation is correct as a fixed-sample calculation under its assumptions, but it does not establish that the interval has 95% coverage under the sequential design. At the null, the estimate was observed specifically because its standardized value crossed the efficacy boundary. The interval has not accounted for that selection. A bootstrap percentile interval is not guaranteed to equal this normal interval exactly; the example illustrates the problem with treating a resampling distribution centred on the selected data as if it automatically represented the stopped-trial sampling distribution.
Selection can damage coverage, not just the centre
A confidence interval has coverage \(1-\alpha\) only relative to a specified repeated-sampling procedure and target parameter. If the procedure is meant to cover \(\theta\) across repetitions of the sequential design, its coverage calculation must include the possible stopping stages and the data-dependent decision to stop. A fixed-sample interval used after stopping generally has no such guarantee. Resampling the observed, selected outcomes does not add the missing guarantee.
The direction and size of the error are design- and method-dependent. It is therefore too strong to claim that every naive bootstrap interval is always narrower than the correct interval, or that selection always increases the conditional variance. For example, conditioning on an upper-tail crossing can reduce the conditional spread of the statistic while shifting its centre away from the null. The important failure is that the bootstrap distribution may have the wrong location, shape, or relationship to the stopping stage. Those errors can produce poor coverage, including intervals that are overconfident about an effect.
Worked Example: False Boundary Crossings and Nominal Intervals
Consider the same one-look null model \(Z_1\sim N(0,1)\), and suppose the design stops for efficacy at \(c=2.00\). Under \(H_0:\theta=0\), the probability of crossing this boundary is
A two-sided 95% fixed-sample normal interval excludes zero whenever \(|Z_1|>1.96\). Every efficacy stop in this example has \(Z_1\ge2.00>1.96\), so the fixed-sample interval excludes the true null value on every such early-stop path. The probability of these early-stop exclusions under the null is about 0.0228. This calculation does not by itself determine the overall coverage of a particular multi-look interval; later looks and their rules also matter. It does show why a nominal fixed-sample confidence level cannot simply be carried over to a stopped trial.
How to diagnose the mismatch and choose an approach
Before interpreting a post-stopping bootstrap interval, ask what was reproduced in each bootstrap replicate. If the observed stopping stage is held fixed and only the observed outcomes are resampled, the procedure has not simulated the full sequential design. Also check whether resampling respects the randomized treatment structure, the sampling units, and any nuisance-parameter estimation. A bootstrap that ignores clustering or resamples treatment groups incorrectly has additional problems, independent of stopping.
The main alternatives answer related but distinct questions:
| Approach | What it accounts for | Important qualification |
|---|---|---|
| Naive post-stopping bootstrap | Resampling variability around the observed data at the realized stage | Does not generally reproduce selection by the stopping rule; nominal coverage need not hold. |
| Stopping-aware bootstrap | Generates or resamples trial paths and reapplies the prespecified stopping rule | Requires a credible generating or resampling model and correct implementation of the design. |
| Stopping-aware likelihood or test inversion | Uses the stopped sampling law, including its stopping events, to form inference | Can require more modelling and computation; validity depends on the specified model and design. |
A practical diagnostic is to compare the analysis procedure with the actual trial protocol: do bootstrap replicates have the same planned looks, boundaries, information schedule, and stopping decisions? If not, they are not reproducing the design. Further evaluation can use repeated-trial simulation under plausible parameter values, but a good-looking bootstrap histogram from the observed data alone is not evidence of correct coverage.
The limitation is fundamental rather than computational. Resampling cannot recover aspects of the sampling process that have been omitted from the resampling scheme. A stopping-aware bootstrap can still be inaccurate if its outcome model is wrong, if it mishandles nuisance parameters, or if the design is not encoded faithfully. Conversely, a naive bootstrap may be a useful descriptive summary of local variation around the observed data; it should not be presented as a design-valid confidence interval without a justification of its coverage target.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Under \(H_0:\theta=\theta_0\), why is the mean estimate among trials with \(Z_1\ge c\) generally above \(\theta_0\)?
- A bootstrap resamples the observed early-stop outcomes but never reapplies the efficacy boundary. What part of the trial procedure is missing?
- For \(Z_1\sim N(0,1)\) and a boundary \(c=2.00\), what is the null probability of an early efficacy stop, approximately?
- Is it always correct to say that a naive bootstrap interval is too narrow? Explain.
Solutions. (1) Selection retains only upper-tail values, and \(E(Z_1\mid Z_1\ge c)=\phi(c)/[1-\Phi(c)]>0\); multiplying by the positive standard error shifts the conditional mean estimate above \(\theta_0\). (2) It omits the generation of new interim results and the rule-based decision about whether and when to stop. (3) \(1-\Phi(2.00)\approx0.0228\), or about 2.28%. (4) No. Selection can change the location and shape as well as the spread, and conditional spread can even be smaller. The concern is that the interval may have the wrong sampling distribution and poor coverage, not that its width must always be smaller.