Why the Selection Rule Must Be Repeated
In a multi-arm multi-stage (MAMS) trial, interim data determine which treatment arms continue. An estimate reported at the end may therefore come from an arm that was favored by a noisy interim result. If the same data also contribute to the reported effect estimate, the estimate can be optimistic: sampling variation helped choose the arm and can also make its estimated effect look large. A bootstrap that holds the observed arm fixed does not reproduce that selection step.
A selection-aware bootstrap resamples trial data, reapplies the specified selection rule in every resample, and calculates the selected treatment-effect estimate each time. The distribution of those estimates can be used to assess the average bias of the selection procedure under a bootstrap approximation to the data-generating process. This is a point-estimation tool; it does not, by itself, provide a selection-adjusted confidence interval or guarantee unbiasedness for the particular arm selected in the observed trial.
This definition concerns the performance of a procedure over repeated trials: both the selected arm and its estimate can vary. It is not the same as the bias conditional on the particular selection event observed in one trial. The introductory material on false coverage rates and post-selection estimation motivates why the distinction between an overall procedure and a selected result matters; here the focus is how to implement and interpret a bootstrap bias estimate.
Bootstrap Construction and Bias Correction
The bootstrap must mimic the trial features that created the estimate. For a nonparametric bootstrap, resample observational units within the relevant randomized arm and stage, preserving the design. If interim and later-stage data come from different participants, resample separately within each stage and arm. If the same participant contributes linked measurements, resample that participant's complete record rather than breaking the link. For a contrast using a shared control group, resample the control observations once per replicate and use that same resample for all arm-control contrasts.
In replicate \(b\), apply the original eligibility criteria, stopping and continuation rules relevant to the estimate, selection statistic, tie-breaking rule, and estimator. Denote the selected arm by \(\hat{J}^{*(b)}\), and the resulting selected estimate by \(T^{*(b)}=\hat{\theta}^{*(b)}_{\hat{J}^{*(b)}}\). The bootstrap plug-in population assigns arm \(j\) its original estimate \(\hat{\theta}_j\). Thus the replicate-specific plug-in target is \(\hat{\theta}_{\hat{J}^{*(b)}}\), not necessarily the estimate for the arm selected in the observed data.
Here \(B\) is the number of bootstrap replicates, \(\hat{b}_{\mathrm{boot}}\) estimates the average selection bias in the bootstrap approximation, and \(\hat{\theta}_{\mathrm{corr}}\) is the corrected estimate for the arm selected in the observed trial. A positive estimated bias is subtracted; a negative one is added. The correction is a plug-in approximation, not a mathematical guarantee that the corrected estimate is unbiased.
Calculate each arm's effect estimate, apply the actual selection rule, and record the selected estimate. Preserve the complete rule, including thresholds, stage eligibility, and tie handling.
Resample the observational units according to the trial's design, keeping the arm and stage structure and any within-unit dependence intact.
Recompute interim statistics, apply the selection rule, and calculate the same selected-effect estimator that the trial would report.
Subtract the original plug-in estimate for the arm selected in that replicate from its selected bootstrap estimate. Average these differences, then subtract the average from the observed selected estimate.
The arm-specific plug-in target is important. Simply comparing the average bootstrap selected estimate with the observed selected estimate can mix together two things: selection bias and genuine differences in the effects of the arms selected across replicates. Referencing each replicate's selected arm to its own plug-in effect makes the target explicit. It still relies on the original arm estimates as approximations to the bootstrap population effects.
Worked Calculations
Worked Example: Correcting an Estimate From Bootstrap Replicates
Suppose a hypothetical two-arm trial selects the arm with the larger interim statistic. In the observed data, arm A has an estimated effect of \(0.8\) points and arm B has an estimated effect of \(1.1\) points; B is selected, so the reported estimate is \(1.1\) points. Eight bootstrap replicates, each using the same selection rule, produce the following selected-arm estimates. The plug-in target is \(0.8\) points when A is selected and \(1.1\) points when B is selected.
| Replicate | Selected arm | Bootstrap estimate (points) | Plug-in target (points) | Difference (points) |
|---|---|---|---|---|
| 1 | A | 1.0 | 0.8 | 0.2 |
| 2 | B | 1.5 | 1.1 | 0.4 |
| 3 | B | 1.2 | 1.1 | 0.1 |
| 4 | A | 0.7 | 0.8 | -0.1 |
| 5 | B | 1.4 | 1.1 | 0.3 |
| 6 | B | 1.3 | 1.1 | 0.2 |
| 7 | A | 1.1 | 0.8 | 0.3 |
| 8 | B | 1.6 | 1.1 | 0.5 |
The replicate differences sum to \(0.2+0.4+0.1-0.1+0.3+0.2+0.3+0.5=1.9\) points. Therefore,
The corrected estimate is \(0.8625\) points for the arm selected in the observed trial. The positive bootstrap bias estimate indicates that, in these replicates, the selected estimates averaged \(0.2375\) points above their replicate-specific plug-in targets. With only eight replicates this is an illustration of the arithmetic, not a reliable precision assessment.
Worked Example: Applying the Selection Rule in One Resample
Consider hypothetical interim treatment-effect contrasts, in points, for arms A and B against a shared control. The rule is to continue the arm with the larger interim contrast. In one bootstrap replicate, resampled interim data give \(\hat{\theta}^{*}_{A,I}=0.6\) and \(\hat{\theta}^{*}_{B,I}=0.9\). Resampled later-stage data give estimates of \(0.7\) for A and \(1.0\) for B. Assume the reporting estimator combines the interim and later-stage estimates with equal weights.
The rule selects B because \(0.9>0.6\). The bootstrap selected estimate is therefore
Suppose the original-data plug-in estimates are \(\hat{\theta}_A=0.5\) and \(\hat{\theta}_B=0.8\) points. Since B was selected in this replicate, its replicate-specific plug-in target is \(0.8\), and the contribution to the bootstrap bias sum is \(0.95-0.8=0.15\) points. This is one contribution, not a bias estimate by itself: the average over many replicates is required.
This calculation assumes that the specified resampling preserved the shared-control structure and reproduced the same interim and later-stage estimator. If the trial instead used a different weighting rule, the bootstrap must use that rule; changing it would estimate the bias of a different procedure.
Worked Example: Summarizing Bias When Arms Are Selected at Different Rates
Suppose 1,000 hypothetical bootstrap replicates select A 280 times and B 720 times. Among replicates selecting A, the selected estimates average \(0.64\) points, compared with A's original plug-in estimate of \(0.50\). Among replicates selecting B, the selected estimates average \(0.94\), compared with B's plug-in estimate of \(0.80\). The within-selection-group average differences are \(0.14\) and \(0.14\) points. The overall bias estimate is the selection-frequency-weighted average:
If the observed trial selected B and reported \(0.90\) points, the corresponding corrected estimate is \(0.90-0.14=0.76\) points. The calculation uses the selection frequencies as well as the within-arm differences. In general those differences need not be equal, so omitting the weighting would give the wrong average for the selection procedure.
Assumptions, Diagnostics, and Interpretation
The method is most defensible when the resampling scheme reflects how the trial generated its data and when the empirical arm-specific distributions provide reasonable approximations to the underlying distributions. It also assumes that the original selection rule and estimator are known and can be reproduced exactly. The relevant sampling units must be independent under the resampling design; when observations are clustered or repeated within participants, the resampling unit must preserve that dependence.
Useful checks include confirming that every replicate actually reapplies the selection rule, tabulating how often each arm is selected, and checking that the bootstrap estimates and bias contributions are not driven by a small number of extreme replicates. Compare the observed selected estimate with the replicate-specific targets and inspect bias summaries by selected arm. A large difference across selected arms, or a selection frequency concentrated almost entirely on one arm, can reveal that a single overall correction obscures important behavior. Increasing \(B\) can reduce Monte Carlo noise in the bootstrap average, but it cannot repair an unsuitable resampling scheme or an unrepresentative empirical distribution.
A key assumption for the correction's interpretation is that the bootstrap's plug-in effects are useful stand-ins for the true arm effects. Those estimates can themselves be noisy, particularly with few participants or substantial selection. The correction also targets average bias over bootstrap repetitions, not the conditional bias given the observed selection event. It should be reported as an approximate bias correction, with the resampling unit, number of replicates, selection rule, and resulting correction stated clearly.
How This Bootstrap Differs From Alternatives
| Approach | What it does | Important distinction |
|---|---|---|
| Naive bootstrap with the observed arm held fixed | Resamples the selected arm's data but does not redo arm selection. | Does not reproduce the selection step and can miss the bias targeted here. |
| Selection-aware bootstrap | Resamples trial data, repeats selection, and compares each selected estimate with its arm-specific plug-in target. | Estimates procedure-level bias under the bootstrap approximation; it is not a conditional adjustment or an interval procedure. |
| Sample splitting or validation-stage estimation | Uses data not involved in selection to estimate an effect. | Separates selection and estimation by design, often at a cost in precision; it does not require bootstrap bias subtraction. |
| Selective likelihood or test inversion | Conditions on a selection event or inverts selection-aware tests to form inferential statements. | Addresses selection-conditioned inference rather than estimating average procedure-level bias by resampling. |
A selection-aware bootstrap is attractive when the full selection algorithm is reproducible and a useful resampling model is available. It is less compelling if the sample is too small for the empirical distribution to represent plausible trial outcomes, or if selection depends on features the resampling scheme fails to preserve. A corrected point estimate should therefore be presented alongside the original selected estimate and the estimated bias, rather than as a replacement that removes all uncertainty or selection effects.
Common Mistake
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Why is resampling only the arm selected in the observed trial inadequate for estimating bias from adaptive arm selection?
- In a replicate, arm A is selected and its bootstrap estimate is \(0.73\) points. The original plug-in estimate for A is \(0.61\) points. What is this replicate's contribution to the bias sum?
- Suppose the average bootstrap bias is \(-0.08\) points and the observed selected estimate is \(0.55\) points. What is the corrected estimate, and what does the negative bias estimate mean?
- Why should the same resampled control observations be used across arm-control contrasts when the trial has a shared control group?
- Does this correction estimate the bias conditional on the observed arm being selected? Explain.
Solutions. (1) It omits the random selection step, so it does not reproduce the procedure whose bias is being assessed. Resample the relevant trial data and reapply the selection rule in every replicate. (2) The contribution is \(0.73-0.61=0.12\) points. (3) The corrected estimate is \(0.55-(-0.08)=0.63\) points. The negative bias estimate means the bootstrap selected estimates averaged \(0.08\) points below their replicate-specific plug-in targets. (4) A shared control induces dependence among the arm-control contrasts. Reusing the same resampled control observations preserves that dependence; independently resampling controls for each arm would change the design's joint structure. (5) No. It estimates average procedure-level bias under the bootstrap approximation, where the selected arm can vary across replicates. Conditional bias for the observed selection event is a different target.