Why the Selection Rule Is Part of the Analysis
In a multi-arm multi-stage trial, which treatments continue may depend on interim outcomes. An estimate for an arm retained at an interim analysis is therefore tied not only to the outcome model but also to the rule that determined retention. If the analysis assumes a different rule from the one actually used, the selection adjustment and the interval can both change.
Here, sensitivity analysis means repeating the post-selection estimation under a set of plausible selection rules and comparing the results. It does not establish which rule was actually followed. The primary analysis should use the documented rule; alternatives are useful for showing how dependent the conclusion is on that rule.
A rule specifies more than a numerical cutoff. It may describe which interim statistic is used, whether an arm must exceed a threshold, whether a fixed number of arms continue, how ties are resolved, and how decisions at successive stages combine. Two rules that use the same threshold can still define different selection events.
Setup and Assumptions for a Threshold Illustration
To make the effect of a threshold mismatch explicit, consider one treatment-versus-control contrast. Let \(Y\) be an interim statistic used for retention and \(Z\) the treatment-effect estimate whose selection bias is being adjusted. For this illustration, assume
- \((Y,Z)\) has a joint normal distribution with means \((\theta,\theta)\);
- \(Y\) and \(Z\) have standard deviations \(\sigma_Y\) and \(\sigma_Z\), and correlation \(\rho\);
- the arm is retained exactly when \(Y>c\); and
- the model parameters used in the correction are treated as known.
These assumptions isolate the effect of changing the threshold. They are not a general description of every MAMS design: actual designs can use multiple correlated arms, several stages, or different interim and final estimators. In those settings, the selection event and its adjustment must reflect the full design.
The result follows from conditioning in a bivariate normal distribution. Given \(Y=y\), the conditional mean of \(Z\) is \( \theta+\rho(\sigma_Z/\sigma_Y)(y-\theta) \). For a standard normal variable \(U\), its mean above \(a\) is \(E(U\mid U>a)=\phi(a)/[1-\Phi(a)]=\lambda(a)\). Because \(Y\) has mean \(\theta\) and standard deviation \(\sigma_Y\), retention implies \(E(Y-\theta\mid Y>c)=\sigma_Y\lambda(a)\). Taking the conditional expectation of \(Z\) given \(Y>c\) therefore gives
If the assumed cutoff is \(c_a\), the corresponding model-based adjustment subtracts \( \rho\sigma_Z\lambda((c_a-\theta)/\sigma_Y) \) from \(Z\). In practice, parameters are estimated and selective intervals may be obtained by inverting a selection-aware test; the known-parameter calculation here gives a transparent benchmark for how the rule itself affects the adjustment.
Worked Example: Recalculating the Correction Across Thresholds
Suppose a hypothetical trial has \(\theta=0.50\) treatment-effect units, \(\sigma_Y=1.00\) interim-statistic units, \(\sigma_Z=0.80\) treatment-effect units, and \(\rho=0.60\). An arm is observed with \(Y=1.80\) and \(Z=1.70\). Compare assumed retention thresholds \(c_a=0.50,\ 1.00,\) and \(1.50\). Assume the actual rule retained the arm at \(c=1.00\). These values are hypothetical and the known \(\theta\) is used only to illustrate the adjustment.
The multiplier is \( \rho\sigma_Z=0.60(0.80)=0.48 \). At \(c_a=0.50\), \(a=(0.50-0.50)/1.00=0\), and \(\lambda(0)=0.7979\). The adjustment is \(0.48(0.7979)=0.3830\), so the adjusted estimate is \(1.70-0.3830=1.317\) treatment-effect units.
At \(c_a=1.00\), \(a=0.50\) and \(\lambda(0.50)=1.141\). The adjustment is \(0.48(1.141)=0.5477\), giving \(1.70-0.5477=1.152\) treatment-effect units.
At \(c_a=1.50\), \(a=1.00\) and \(\lambda(1.00)=1.525\). The adjustment is \(0.48(1.525)=0.7320\), giving \(1.70-0.7320=0.9680\) treatment-effect units. The observed \(Y=1.80\) exceeds all three assumed thresholds, so each assumed rule is compatible with this particular retention observation.
The adjustment increases as the assumed cutoff rises: a higher cutoff implies stronger selection on the interim statistic. The corrected point estimate consequently falls across these assumptions. This comparison is conditional on the model and known parameters stated above; it does not show that any one corrected value is the uniquely correct estimate in a real trial.
What a Threshold Mismatch Does to Bias
Under the illustration's assumptions, the expected residual bias after applying the correction for \(c_a\), when the actual cutoff is \(c_0\), is
This follows by subtracting the assumed-rule correction from the conditional expectation under the actual retention event. Thus, when \(\rho>0\), assuming a lower cutoff than was actually used generally subtracts too little selection bias in this example; assuming a higher cutoff generally subtracts too much. The direction need not carry over unchanged to more complex rules, where several arms and selection events interact.
Worked Example: Quantifying Residual Bias Under a Misspecified Cutoff
Use the hypothetical model from the preceding example, with actual cutoff \(c_0=1.00\). Suppose the analysis instead assumes \(c_a=0.50\). The actual-rule bias is \(0.48\lambda(0.50)=0.48(1.141)=0.5477\) treatment-effect units. The assumed-rule adjustment is \(0.48\lambda(0)=0.48(0.7979)=0.3830\) treatment-effect units.
The expected residual bias among arms retained by the actual rule is therefore \(0.5477-0.3830=0.1647\) treatment-effect units. Equivalently, the formula gives \(0.48[1.141-0.7979]=0.1647\). This is an expectation conditional on retention, not a prediction that the realized estimate is exactly 0.1647 units too high.
If instead the analysis assumes \(c_a=1.50\), the residual is \(0.48[1.141-1.525]=-0.1843\) treatment-effect units. The negative sign indicates overcorrection in expectation under this model. Both calculations depend on the stated Gaussian assumptions and known parameter values.
Intervals Must Be Recomputed, Not Merely Relabelled
A selection-adjusted interval is tied to its assumed event. For example, an interval obtained by inverting a test conditional on \(S_a\) is calibrated for that assumed selection event; changing the cutoff or selection rule changes the conditional distribution used in the inversion. A standard error for the corrected estimate is also not automatically an adequate substitute for selection-aware interval construction. The earlier material on selective confidence intervals and standard errors for corrected treatment estimates distinguishes these inferential steps.
The following calculation gives a deliberately limited illustration of how an approximate interval summary can move across assumed thresholds. Continue with the known-parameter Gaussian setup. The conditional variance of \(Y\) given \(Y>c\) is \(\sigma_Y^2[1+a\lambda(a)-\lambda(a)^2]\). Combining this with the conditional normal distribution of \(Z\) given \(Y\) gives
Since the adjustment in the illustration is a constant when model parameters are known, subtracting it does not change this conditional variance. A normal approximation using this conditional standard deviation is only a descriptive Wald summary here: the selected distribution need not be normal, and this calculation does not establish nominal conditional coverage. A selective interval obtained by test inversion is the appropriate choice when conditional coverage is required.
Worked Example: Comparing Approximate Intervals
For the three assumed thresholds in the first example, the values of \(a\) are \(0,\ 0.50,\) and \(1.00\). The conditional variance of \(Z\) at \(c_a=0.50\) is \(0.80^2\{1-0.60^2+0.60^2[1+0(0.7979)-0.7979^2]\} =0.4933\), giving a conditional standard deviation of \(0.7024\) treatment-effect units.
At \(c_a=1.00\), the variance is \(0.80^2\{0.64+0.36[1+0.50(1.141)-1.141^2]\}=0.4715\), so the standard deviation is \(0.6867\) units. At \(c_a=1.50\), it is \(0.80^2\{0.64+0.36[1+1.00(1.525)-1.525^2]\}=0.4555\), so the standard deviation is \(0.6749\) units.
For a transparent comparison only, form approximate intervals as the corrected estimate plus or minus \(1.96\) times this conditional standard deviation. The resulting summaries are \(1.317\pm1.96(0.7024)=[-0.060,\,2.694]\) units for \(c_a=0.50\), \(1.152\pm1.96(0.6867)=[-0.194,\,2.498]\) units for \(c_a=1.00\), and \(0.9680\pm1.96(0.6749)=[-0.355,\,2.291]\) units for \(c_a=1.50\). These are not claimed to be exact 95% selective confidence intervals. Their purpose is to show that both the center and the uncertainty summary depend on the assumed threshold.
When the Selection Rule Changes, Not Just the Cutoff
With several arms, a cutoff rule such as “retain every arm whose interim statistic exceeds \(c\)” differs from a ranking rule such as “retain the two arms with the largest interim statistics.” The selection event for a particular arm under the ranking rule depends on the other arms’ statistics. Shared control can also make those statistics dependent. Consequently, a single-arm truncated-normal adjustment cannot generally be reused for a ranking rule by substituting a different cutoff.
Worked Example: Threshold Retention Versus Ranking
Suppose a hypothetical three-arm trial has interim scores \(1.80,\ 1.40,\) and \(0.70\) for arms A, B, and C. Under the actual rule, retain every arm with score greater than \(1.00\); arms A and B continue. An analyst instead assumes that exactly one arm, the arm with the largest score, continues. Under that assumed rule, only A continues.
The mismatch is visible without any distributional calculation: B was retained under the actual rule but would not be retained under the assumed rule. An analysis that treats B as selected by the top-one rule conditions on the wrong event. Its correction and interval for B therefore do not describe the actual selection pathway. The analysis should use the documented threshold rule, or explicitly report results under the ranking rule as a separate sensitivity scenario; it should not combine the two rules as though they were equivalent.
A Practical Sensitivity Analysis
For each plausible rule, preserve the same outcome model and observed treatment data, then recompute the selection-adjusted estimate and interval using that rule's selection event. This keeps the question focused: how much do conclusions change because of selection-rule uncertainty? The following sequence helps make the comparison interpretable.
Record the interim statistic, threshold or ranking criterion, number of arms retained, stage-by-stage decisions, and tie handling from the trial protocol and conduct records.
Vary one feature at a time where possible, such as a nearby cutoff or retaining the top two rather than all arms exceeding a threshold. Keep the outcome model fixed for this comparison.
Confirm that each assumed rule could produce the observed retained and dropped arms from the recorded interim information. An incompatible rule is a diagnostic scenario, not a credible reconstruction of the trial.
For each compatible scenario, recompute the correction, standard error, and interval using that scenario's selection event. Report changes in treatment-effect units and whether the substantive conclusion changes.
Diagnostics, Alternatives, and Limitations
A large shift in an estimate or interval across plausible rules signals that inference depends strongly on accurate documentation of selection. A small shift is reassuring about this particular sensitivity, but it does not verify other assumptions or guarantee coverage. Check for impossible retention paths, cutoffs applied to the wrong stage statistic, and unrecorded changes to the protocol. For multi-arm rules, verify that the analysis accounts for dependence among interim statistics rather than treating arms as independently screened.
Sensitivity analysis complements, rather than replaces, the main approaches:
| Approach | What it does | Key limitation |
|---|---|---|
| Use the documented rule | Targets the selection event the trial actually used. | Cannot resolve uncertainty if records are incomplete or the rule changed in practice. |
| Selection-aware interval under one assumed rule | Accounts for selection conditional on that specified event. | Its calibration depends on the assumed event being appropriate. |
| Sensitivity analysis across rules | Shows how estimates and intervals vary over plausible selection events. | Does not identify which rule is true or repair an implausible outcome model. |
The threshold formulas above rely on a joint normal model, a one-sided threshold, and known parameters. In real analyses, nuisance parameters are estimated, multiple stages create more complicated selection events, and the conditional distribution may require computation or test inversion. The next subject, outcome-model assumptions, is a separate source of sensitivity; changing those assumptions at the same time would make it difficult to attribute observed changes to the selection rule.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- In the Gaussian threshold illustration, what does \(a=(c-\theta)/\sigma_Y\) measure?
- If the actual cutoff is higher than the assumed cutoff and \(\rho>0\), what is the expected direction of residual bias in the illustration?
- Why can a top-two ranking rule not generally be handled by applying separate single-arm threshold corrections?
- In the numerical example, why are the three approximate intervals not asserted to have exact 95% conditional coverage?
- What should be held fixed when the purpose is specifically to assess sensitivity to the selection rule?
Solutions. (1) It is the assumed cutoff's distance above the interim statistic's mean, expressed in interim-statistic standard-deviation units. (2) The assumed correction is too small under the stated model, so the expected residual bias is positive. (3) Whether an arm ranks in the top two depends on the other arms' scores and their joint dependence; its event is not a separate one-arm cutoff event. (4) The selected distribution is generally not normal, and the displayed calculation is a Wald-style approximation rather than a test-inverted selective interval. (5) Hold the outcome model and observed treatment data fixed while changing the assumed selection event, so changes can be attributed to the rule.