Why Single-Arm Boundaries Fail After Selection

Why selection changes the final test

A boundary for a single prespecified arm is calibrated for that arm’s statistic, not necessarily for the statistic of an arm chosen because its interim result looked favorable. When the same trial data contribute to both selection and final testing, the selected arm tends to have an unusually large interim statistic even if every treatment is ineffective. That favorable interim contribution remains in its final cumulative statistic.

This tutorial isolates that mechanism in a two-stage, two-arm trial with a shared control. The point is not that every selection rule must inflate error in every design. The result below applies under stated assumptions to a specific rule—retain the arm with the larger interim statistic—and shows why a boundary calibrated for a fixed arm cannot simply be assumed valid after selection.

Two-stage model with a shared control

Assume two experimental arms, A and B, and one shared control. Outcomes are normally distributed with a common known variance, allocation is balanced, and the treatment effect is zero for both arms. At each stage, form a one-sided standardized treatment-versus-control statistic \(Z_{1A}\) or \(Z_{1B}\) at the interim, and use independent stage-two data to form the corresponding increment statistic. Under the global null, each interim statistic has a standard normal distribution.

Because both interim comparisons use the same control observations, their statistics are positively correlated. In the balanced equal-variance setting used here, take their correlation to be \(1/2\). Assume that stage-two data are independent of all interim data, while allowing the stage-two increments for A and B to be correlated with each other through their shared control. These assumptions are important: the selection argument uses independence between stages, and the numerical covariance describes this particular shared-control example.

Let \(t\) be the fraction of the final information available at the interim, where \(0<t<1\). For arm \(j\), with \(j\in\{A,B\}\), write its final cumulative statistic as

Here \(Z_{2j}^{\mathrm{inc}}\) is the standardized stage-two increment for arm \(j\). Under the null, it is marginally standard normal and independent of the interim statistics. The weights combine independent information increments so that a prespecified arm’s final statistic has mean zero and variance \(t+(1-t)=1\).

At the interim, retain exactly one arm: the arm with the larger statistic. Define \(M=\max(Z_{1A},Z_{1B})\), and let \(I\) denote the selected arm. The final test rejects if \(Z_{\mathrm{final},I}>c\), where \(c\) is the single-arm critical value. For a one-sided test with level \(\alpha=0.025\), use the exact critical value \(c=\Phi^{-1}(0.975)\), approximately 1.9600. Using this quantile, rather than a rounded value, makes the prespecified-arm null rejection probability exactly 0.025 under the standard normal model.

Deriving the selection effect

Under the global null, the final statistic for a prespecified arm has a standard normal distribution. For example, for arm A, \(Z_{\mathrm{final},A}\) is a weighted sum of independent standard normal variables, with weights whose squared values sum to one. Therefore

After selection, condition on the interim statistics. The selected index \(I\) is then fixed, and the stage-two increment for that arm is independent of the interim data and standard normal. Consequently, conditional on the interim values, the chance of crossing the boundary is

The right-hand side increases as \(M\) increases: a larger interim statistic makes it easier for the cumulative statistic to cross \(c\). In every realization \(M\geq Z_{1A}\), and when arm B has the larger statistic, \(M>Z_{1A}\). Since the bivariate normal interim distribution with correlation \(1/2\) is not degenerate, \(P(Z_{1B}>Z_{1A})>0\).

This argument does not require the interim statistics to be independent of each other. Shared control creates dependence between the arm-specific statistics, but it does not remove the ordering \(M\geq Z_{1A}\). Nor does the proof require a numerical evaluation of the selected rejection probability: it establishes that the probability is strictly greater than the exact prespecified-arm rate \(\alpha\). Its size depends on \(t\), the dependence between interim statistics, and the selection rule.

What the result does and does not say

The result is about a specific procedure: choose the larger of two interim statistics, retain one arm, and test that arm’s cumulative statistic against a boundary calibrated for one prespecified arm. Under the model, the probability of at least one false rejection under the global null is the selected-arm rejection probability, and that probability exceeds \(\alpha\). This is a weak familywise-error failure: it occurs when all tested treatment effects are null.

Selection does not automatically invalidate every later test. In particular, if the confirmatory statistic uses only new data that are independent of the selection data, then conditional on selecting one arm, its null statistic remains standard normal. Testing that single selected arm at level \(\alpha\) then has conditional and unconditional rejection probability \(\alpha\), provided the null applies to the selected arm and the confirmatory analysis is otherwise valid. This is a different data structure from the cumulative analysis above.

If more than one arm is retained and each is tested at level \(\alpha\), independent confirmation alone does not generally make the probability of any false rejection equal to \(\alpha\). Testing multiple null hypotheses creates a familywise-error question even when each individual test is valid. Likewise, if stage two is not independent of interim data, or if the stopping and selection rules differ from those specified, the derivation above may not apply as written.

Diagnostics and common failure modes

Before applying a single-arm boundary after selection, check whether the test statistic contains information used to choose the arm. If it does, assess the null rejection probability of the entire rule—selection followed by testing—not merely the marginal null distribution for a fixed arm. Useful checks include simulation under the global null using the planned shared-control covariance, information fractions, selection rule, and stopping rule. The simulated quantity must be the probability of any false rejection by the procedure, not just the rejection rate for an arm named in advance.

The derivation’s assumptions should also be checked against the design. In particular, confirm that the stage-two increment is independent of interim data, that the standardized statistic has the stated null distribution, and that the information-weighted combination correctly represents the planned analysis. A different selection criterion, such as a threshold that can retain both arms, changes the event being controlled and requires its own evaluation.

• Wrong reasoning: “Each arm’s final statistic is standard normal under its null, so the usual single-arm boundary controls error after selection.”
• Correction: A standard normal marginal distribution for each fixed arm does not imply a standard normal distribution for the arm selected using interim data. The selection rule changes which statistic is carried forward.
• Another pitfall: Treating shared-control statistics as independent when assessing the design. Their dependence changes the distribution of the interim maximum and therefore affects the size of the selection effect, even though the inequality proving an increase does not require independence.

The principal limitation of the simple result is its scope: it establishes inflation for the stated two-arm, one-retained-arm, same-data procedure under the specified model, but it does not provide a generally valid adjusted boundary. Exact control for a particular MAMS design requires evaluating its complete selection and testing procedure under the assumptions and hypotheses relevant to that design.