Randomization-Based Estimation After Selection

Why Selection Must Be Part of the Randomization

In a multi-arm multi-stage trial, the arm whose effect is ultimately reported may have been chosen because its interim results looked promising. An estimate computed for that arm as if it had been fixed in advance ignores a key feature of the design: the same random assignment that generated the outcomes also helped determine which arm was selected. A randomization-based analysis addresses this by reproducing both the trial’s assignment mechanism and its selection rule.

The result is not automatically a correction for every form of selection bias. Exact randomization calculations require a clearly specified randomization mechanism and, for direct imputation of outcomes under candidate effects, a sharp potential-outcome model. They are valuable because their assumptions are explicit and because the selection step can be represented directly rather than approximated by resampling observed records.

Let \(Z_i\) denote the treatment assigned to participant \(i\), with \(0\) denoting control and \(j=1,\ldots,J\) denoting active arms. A complete assignment path includes assignments at every stage and any randomization-dependent stopping or continuation decisions. Let \(\mathcal{A}\) be the set of paths that could have occurred under the protocol, and let \(\pi(a)\) be the probability of path \(a\). Equal-probability paths are common in small examples, but the general calculation must use the actual probabilities \(\pi(a)\).

The selection rule is a function of the data available at the relevant decision point. It may choose an active arm for continuation, drop an arm for futility, or stop an arm for efficacy. For each candidate assignment path, the analysis must generate the outcomes that would have been observed on that path, apply the rule at the same point in the trial, and calculate the statistic for the arm selected on that path. Reassigning treatment labels while leaving the observed selected arm fixed does not reproduce this procedure.

Sharp Effects and Outcome Imputation

A transparent starting point is a sharp additive treatment-effect model. For each participant \(i\), let \(Y_i(j)\) be the outcome that would be observed under treatment \(j\). For candidate effects \(\boldsymbol{\theta}=(\theta_1,\ldots,\theta_J)\), the sharp model states that \(Y_i(j)=Y_i(0)+\theta_j\) for every participant and every active arm. Thus the treatment effect is constant across participants for each arm. This assumption makes it possible to recover each participant’s control outcome from the observed outcome and the assigned treatment.

The imputation reproduces the observed outcomes under the actual assignment: when \(a_i=Z_i\), the imputed value is \(Y_i^{\mathrm{obs}}\). For other assignments it supplies the outcomes implied by the candidate sharp model. The randomization distribution is then formed over the paths in \(\mathcal{A}\), weighted by \(\pi(a)\). If a path contains stage-specific assignments, the imputation and selection rule must follow the protocol’s timing: only outcomes available at an interim decision may influence that decision.

This construction differs from a bootstrap. A bootstrap resamples observed data according to a sampling scheme; a randomization calculation varies assignments according to the design. It also differs from a permutation that freely shuffles labels: only assignments the trial could actually have generated belong in \(\mathcal{A}\).

A Small Exact Example

Testing Candidate Effects and Inverting the Test

A sharp null need not set every active effect to zero. To test a candidate effect vector, impute the control outcomes under that vector, enumerate or sample valid assignment paths, and replay the selection rule. For a scalar selected-effect analysis, a prespecified statistic might be the selected contrast minus its candidate effect. A two-sided test can compare the absolute observed deviation with the absolute deviations across the randomization distribution. The statistic and tail convention must be specified before examining the resulting p-value; different choices can give different tests.

The logic of exactness is short but important. Under a true sharp candidate model, all potential outcomes needed for each allowed assignment are known after imputation. The actual assignment path is drawn according to the known design probabilities. Therefore the statistic computed from the actual path has the distribution generated by those same probabilities and the same selection rule. Comparing it with that distribution controls the randomization-test error rate for the sharp null, apart from any discreteness or Monte Carlo approximation.

Here the probability is over the trial’s random assignment when the sharp candidate \(\boldsymbol{\theta}\) is true; the inequality allows for conservative tests caused by discrete assignment distributions. Test inversion turns this collection of tests into a confidence set. It does not make a sharp constant-effect model appropriate when effects vary across participants.

Selection Conditioning, Alternatives, and Diagnostics

There are two distinct questions an analysis may ask. An unconditional randomization distribution replays the entire design, including which arm is selected, and provides inference for the procedure as a whole. A conditional analysis may restrict attention to assignment paths that produce the same selection event as observed. That restriction changes the reference distribution and can leave very few paths, or none under some candidate values. Conditioning can be appropriate for a stated conditional target, but it is not achieved merely by holding the observed selected arm fixed while permuting outcomes.

Randomization-based methods are especially attractive when the assignment probabilities and decision algorithm are fully documented. Model-based selective likelihood methods can use outcome-distribution assumptions to obtain more detailed inference; bootstrap approaches can be useful for more complex estimators but must reproduce the design and selection adequately. These methods answer related but not identical questions. Randomization inference is exact for its sharp null under the actual design, whereas it does not by itself provide exact inference for a general average treatment effect under arbitrary effect heterogeneity.

Diagnostics begin with the trial record, not a histogram of the resampled statistics. Verify that each generated path obeys stage sizes, allocation probabilities, stratification, stopping boundaries, and the information available at each decision. Check that selection is reapplied identically on every path and that ties are handled as specified in the protocol. If Monte Carlo sampling replaces full enumeration, report the number of sampled paths, use the correct path probabilities, and assess simulation error in the estimated tail probability.

A frequent failure is to permute the observed outcomes among all treatment labels and then calculate the contrast for the arm selected in the actual trial. This breaks the link between assignments and selection: under most permuted datasets, a different arm would have been selected. The correction is to generate outcomes under each allowable assignment and rerun the selection rule, including any interim stopping or continuation decision. A second limitation is lack of support: when few paths can lead to the observed selection event, conditional results may be unstable or uninformative. No computational refinement can create assignment information that the design did not provide.