Why the stopping rule belongs in the bootstrap
An estimate reported after an interim analysis is selected by the trial’s stopping process. Its sampling distribution is therefore not generally the same as the distribution of an estimate from a fixed sample of the same size. To calibrate an interval for that stopped estimate, a parametric bootstrap must reproduce the design: generate a complete trial under a candidate treatment effect, apply the planned boundaries, and calculate the estimate at the simulated stopping time.
The method below targets unconditional coverage across all possible stopping stages. That means the coverage probability averages over trials that stop early and trials that continue. A conditional interval, calibrated only among trials that stop at a particular stage, answers a different question and is described separately.
Model and stopping-aware calibration
Suppose the treatment effect is \(\theta\), measured in the outcome’s units. At look \(k\), let \(S_k\) be the accumulated information-weighted score and \(I_k\) the cumulative information, with \(I_1<I_2<\cdots<I_K\). Use the normal independent-increments model
where increments from different looks are independent and \(S_0=I_0=0\). The estimate at look \(k\) is \(\hat\theta_k=S_k/I_k\), with standard error \(1/\sqrt{I_k}\). The standardized statistic is \(Z_k=S_k/\sqrt{I_k}\). The model assumes the information schedule is fixed in advance, the effect is constant over the trial, the increments follow the stated normal model, and the planned boundaries are applied as specified.
For each candidate value \(\theta\), simulate \(B\) complete trials. In every replicate, generate increments under that candidate, calculate each look’s statistic, stop at the first boundary crossing, and save the estimate at that look. The empirical distribution function is
Here the asterisk marks a simulated quantity. For an observed stopped estimate \(y=\hat\theta_T\), an equal-tailed \(100(1-\alpha)\%\) interval is obtained by finding the candidate effects satisfying
In a continuous model, \(F_\theta(\hat\theta_T)\) is uniform on \((0,1)\) when the model is correct. Thus, at the true effect, the probability of falling outside these two central quantiles is \(\alpha\). Replacing \(F_\theta\) with its simulated estimate approximates this calibration. The lower endpoint solves \(F_\theta(y)=1-\alpha/2\); the upper endpoint solves \(F_\theta(y)=\alpha/2\), when these roots are ordered and unique.
Why the calibration has the stated coverage
The key is that the candidate-specific reference distribution includes the stopping process. Under a fixed true value \(\theta\), the observed statistic \(Y=\hat\theta_T\) has CDF \(F_\theta\), not the fixed-look CDF at whichever sample size happened to be observed. If \(F_\theta\) is continuous, the probability integral transform gives \(F_\theta(Y)\sim\mathrm{Uniform}(0,1)\). Therefore,
This is a model-based justification: it depends on generating trials from the correct sampling model and applying the actual rule. With finite \(B\), the empirical CDF has simulation error, so the computed limits only approximate the ideal limits. If the acceptance set over candidate effects is not a single interval, retain the set or report its interval hull; the hull is conservative because it contains the acceptance set.
Worked Example: Inverting a two-look trial simulation
Hypothetical data. A two-look normal-outcome trial has \(I_1=1.5625\), \(I_2=3.000\), interim efficacy boundary \(Z_1\ge2.00\), and a final analysis at look 2. At the first look, the observed estimate is \(y=1.80\) outcome units. Its standard error is \(1/\sqrt{1.5625}=0.8000\) units, so \(Z_1=1.80/0.8000=2.25\). The trial therefore stops for efficacy at look 1.
For each candidate effect, suppose \(B=100{,}000\) complete trials are simulated using the two independent normal increments with the stated information schedule and the same interim rule. The following hypothetical simulation proportions are the fractions with stopped estimate at most \(1.80\):
| Candidate \(\theta\) (outcome units) | \(\hat F_\theta(1.80)\) |
|---|---|
| 0.50 | 0.982 |
| 1.00 | 0.946 |
| 2.50 | 0.105 |
| 3.00 | 0.018 |
For a 95% interval, the lower endpoint solves \(\hat F_\theta(1.80)=0.975\). Linear interpolation between 0.50 and 1.00 gives
The upper endpoint solves \(\hat F_\theta(1.80)=0.025\). Interpolating between 2.50 and 3.00 gives
The resulting approximate unconditional interval is \((0.597,\,2.96)\) outcome units. It is wide because the observed estimate was obtained at an early efficacy stop, and the calibration accounts for the variability and selection induced by that stopping rule. These endpoints are illustrative Monte Carlo estimates, not values from a real trial.
Unconditional and conditional targets
The unconditional distribution \(F_\theta\) mixes over every possible stopping stage and stopping reason. It is appropriate when the desired coverage statement averages over the trial’s stopping outcomes. If the inferential target is instead coverage conditional on the observed stage, simulate under \(\theta\) and retain only replicates with the specified stopping event. For example, define \(A=\{T=1\}\), or more narrowly define \(A\) as stopping at look 1 for efficacy. The conditional calibration uses
The condition must be stated precisely: conditioning only on the look can mix different boundary causes, whereas conditioning on the stage and reason does not. The resulting interval has a conditional coverage target and need not agree with the unconditional interval. This distinction parallels the difference between unconditional and conditional confidence coverage.
Worked Example: Separating a stage-conditional distribution
Hypothetical simulation output. At candidate effect \(\theta=1.50\) outcome units, generate 20,000 complete trials using the planned boundaries. Suppose 13,600 stop at look 1, and among these, 10,800 have stopped estimates at or below the observed \(y=1.80\). Of the 6,400 trials that proceed to look 2, suppose 5,200 have final estimates at or below 1.80.
The unconditional empirical CDF at 1.80 is based on all 20,000 trials:
Conditioning on stopping at look 1 instead gives
The two values are close here, but they are not interchangeable: the second excludes every trial that continues to look 2. To form a stage-conditional interval, this conditional CDF would be evaluated over a range of candidate effects and inverted at the chosen tail probabilities. The numbers above illustrate the CDF calculation; they are not themselves interval endpoints.
Implementation, diagnostics, and limitations
A practical implementation separates the trial simulator from the calibration step. That makes it easier to verify that the boundaries, information schedule, and estimator used in simulation match the protocol.
Specify the information at every look, the upper and lower boundaries, the maximum number of looks, and the statistic used to stop.
At each candidate effect, generate all planned increments. Apply the boundary at each look and save only the estimate at the first stopping look.
Estimate \(F_\theta(y)\) for candidate effects around each tail-probability crossing. Refine the grid or use a root finder where the estimated CDF is monotone.
Repeat with more simulations or independent random seeds, and check whether the resulting endpoint estimates are stable to simulation noise and grid resolution.
At any fixed candidate and observed value, \(\hat F_\theta(y)\) is a binomial proportion with approximate Monte Carlo standard error \(\sqrt{\hat F_\theta(y)\{1-\hat F_\theta(y)\}/B}\). Tail estimates are especially sensitive: too few replicates can make an endpoint appear precise while the corresponding tail probability is poorly estimated. Common diagnostics include checking that every replicate stops exactly once, verifying that the simulated stopping-stage frequencies are plausible, and plotting \(\hat F_\theta(y)\) against \(\theta\). Unexpected jumps or nonmonotonicity can indicate a coding error, an overly coarse simulation, or a model for which a simple interval inversion is not appropriate.
This approach differs from a fixed-sample interval, which uses a reference distribution for a prespecified sample size, and from a likelihood-based interval, which calibrates a likelihood criterion rather than the stopped-estimate CDF. A conventional bootstrap that does not regenerate the planned monitoring process also does not directly estimate \(F_\theta\). The stopping-aware parametric method is useful when the trial model and design can be simulated, but its results can be inaccurate if the assumed outcome distribution, effect model, variance, or information schedule is wrong. If nuisance parameters such as variance are estimated, their treatment must be specified; simply inserting one estimate does not automatically guarantee nominal coverage.
Worked Example: Quantifying Monte Carlo error near an endpoint
In the first example, the upper endpoint is near a candidate effect where the estimated tail probability is 0.025. With \(B=100{,}000\), the approximate Monte Carlo standard error of that estimated CDF is
The local slope from the two reported candidates is approximately \((0.105-0.018)/(3.00-2.50)=0.174\) per outcome unit. Treating this slope as locally linear, the endpoint’s Monte Carlo standard error from CDF noise alone is approximately
This is only a local approximation: it does not include interpolation error, uncertainty in the estimated slope, or model misspecification. It does show how simulation error in a tail probability can translate into endpoint uncertainty. Increasing \(B\) reduces Monte Carlo standard error at the usual rate \(1/\sqrt{B}\).
Common mistake
Wrong reasoning: “The trial stopped at look 1, so I will simulate only look-1 data under each candidate effect and use that distribution.” This does not generally reproduce the unconditional stopped-estimate distribution. It omits paths that would continue, and it may also fail to condition correctly on the event that caused the observed stop.
Correction: For unconditional coverage, simulate the complete trial and apply the rule at every look, retaining the estimate at each simulated stopping time. For conditional coverage, simulate complete trials too, then retain only replicates satisfying the explicitly defined conditioning event. In both cases, the stopping rule determines which estimates belong in the reference distribution.
Check Your Understanding
Answer each question, then compare with the solutions.
- For unconditional calibration, should simulated paths that continue to the final look be discarded when the observed trial stopped early?
- At a candidate effect, 2,000 of 10,000 complete simulations have stopped estimates no greater than \(y\). What is the unconditional empirical CDF at \(y\)?
- In a continuous stopped-estimate model, why do the equal-tailed cutoffs use \(\alpha/2\) and \(1-\alpha/2\)?
- What event must be used to estimate a conditional CDF when the target is coverage given an early efficacy stop?
Solutions. (1) No. Unconditional calibration includes every simulated path, whether it stops early or reaches the final look; each contributes its estimate at its own stopping time. (2) The estimate is \(2{,}000/10{,}000=0.2000\). (3) Under the true effect, the continuous probability integral transform is uniform, so the two excluded tails have probabilities \(\alpha/2\) each and the central acceptance probability is \(1-\alpha\). (4) Retain complete simulated trials that stop at the early look for efficacy, not merely all trials that happen to stop at that look if the stage can have another stopping cause.