Why Naive Bootstrap Corrections Fail at Selection Boundaries

Why Selection Boundaries Challenge the Bootstrap

In a multi-arm multi-stage trial, an arm may continue only if its interim statistic crosses a prespecified threshold. The reported estimate can therefore depend on both the outcome data and a discrete decision: which arm passed, which arm was retained, or which arm ranked highest. Near a threshold, a small change in the interim data can change that decision. A bootstrap that treats the observed decision as fixed does not reproduce this part of the sampling process.

This is not a general objection to bootstrap methods. Resampling can be useful when the data-generating model and the full selection procedure are represented adequately. The concern is that ordinary resampling intuition relies on a relatively stable relationship between the data and the quantity being estimated. Threshold selection creates a boundary at which that relationship changes abruptly. Bias corrections and intervals can then be inaccurate even when the resampled outcomes look plausible.

To isolate the issue, suppose an interim estimate \(\hat\theta\) satisfies the model \(\hat\theta\sim N(\theta,s^2)\), where \(\theta\) is the arm’s true treatment effect, \(s\) is a known standard error, and larger effects are desirable. The arm is selected when \(\hat\theta>c\), for a fixed threshold \(c\). This simple model assumes normal sampling, a correctly specified standard error, and a fixed rule. Real MAMS trials may have correlated estimates, multiple stages, and more complex paths; those features must also be represented when relevant.

What Threshold Selection Does to an Estimate

Selection changes the distribution of the estimate being reported. Define \(a=(c-\theta)/s\). The probability of selection is \(P(\hat\theta>c)=1-\Phi(a)\), where \(\Phi\) is the standard normal cumulative distribution function. Conditional on selection, the estimate tends to exceed \(\theta\), particularly when the true effect is near or below the threshold.

Here \(\phi\) is the standard normal density. To see the result, write \(Z=(\hat\theta-\theta)/s\), so \(Z\sim N(0,1)\), and selection is equivalent to \(Z>a\). Then

because \(\phi'(z)=-z\phi(z)\), giving \(\int_a^\infty z\phi(z)\,dz=\phi(a)\). This is a conditional bias: it describes the average excess among repetitions in which the arm is selected. It is not a claim that every selected estimate is too large.

Two Ways a Naive Bootstrap Can Miss the Boundary

A bootstrap replicate is generated from the observed data or from a fitted model. A common failure is to reproduce outcome variation while omitting the decision that made the estimate eligible to be reported. For instance, resampling only the retained arm and calculating its estimate in every replicate conditions on the observed selection path. The resulting distribution answers a different question from the distribution across repeated trials in which arms can be retained or dropped.

A second problem remains even if the rule is replayed: a plug-in bootstrap uses fitted parameters, and those fitted parameters may lie on the opposite side of the boundary from the true parameters. Near the boundary, this can produce a substantially different selection frequency and conditional bias. The following example makes the distinction concrete.

The instability can be stated precisely for the selection probability. Let the true mean approach the threshold along \(\theta_n=c+h s_n\), where \(s_n\) is the standard error at sample size \(n\) and \(h\) is fixed. The true selection probability is \(1-\Phi(-h)=\Phi(h)\). If the bootstrap plugs in \(\hat\theta\), its estimated selection probability is \(\Phi((\hat\theta-c)/s_n)\). Since \((\hat\theta-c)/s_n\) has distribution \(N(h,1)\), this bootstrap estimate does not settle to the fixed value \(\Phi(h)\) as \(n\) increases: its randomness remains on the boundary scale. At \(h=0\), the true probability is \(0.500\), while the plug-in probability is \(\Phi(Z)\) for \(Z\sim N(0,1)\), a random quantity rather than a value converging to \(0.500\). This is why merely reapplying the rule does not guarantee reliable bootstrap calibration at a boundary.

What a Selection-Aware Bootstrap Must Represent

For a MAMS procedure, the selection event may involve several arms and interim looks. A bootstrap intended to approximate repeated-trial behavior needs to preserve the joint variation that drives the decisions, including dependence between arm estimates when present. Each replicate must apply the prespecified thresholds and ranking rules at the correct stages, follow the resulting path, and compute the reported estimator using the same data-use rule as the analysis. If a stage stops an arm, later-stage outcomes for that arm cannot be treated as though they had been observed under the trial’s actual procedure.

This is a necessary design principle, not a guarantee of accuracy. In a parametric bootstrap, the assumed outcome model and its fitted parameters must be credible. In a nonparametric bootstrap, the resampling unit and dependence structure must be appropriate; resampling patients independently when the relevant data are clustered or jointly constrained can distort the selection statistics. Most importantly here, even a carefully replayed plug-in bootstrap may be unreliable when the fitted model is close to a selection boundary.

Diagnostics, Alternatives, and Limitations

Warning signs include a bootstrap that always reports the observed arm, a selection frequency that is effectively one despite an estimate close to a threshold, or a bias correction that changes sharply under small changes to fitted means. Also check whether replicates follow the actual stage structure and whether uncertainty in the interim statistics is represented. A narrow interval is not reassuring if the resampling procedure excluded the event that generated the estimate.

The main alternative is a selection-aware analysis that directly conditions on the observed path or uses a selection-adjusted likelihood or test inversion. These approaches address the selection event explicitly, although their validity also depends on the model and target being appropriate. Sample splitting or a validation stage can reduce the direct reuse of selection data for estimation, at the cost of precision and with its own design requirements. The introductory material on selection-aware bootstrap estimation and the parametric bootstrap for multi-stage selection provides the broader resampling context; the present concern is the special instability at thresholds.

No bootstrap correction can remove uncertainty about a misspecified outcome model, an incorrectly implemented rule, or a target that has not been clearly defined. Nor does a correction that performs well away from a boundary establish reliable behavior near one. In practice, report the selection rule and the bootstrap scheme, distinguish conditional from marginal performance, and treat estimates sensitive to plausible boundary perturbations cautiously.

Common Mistake