Error-Rate Derivation For A Two-Arm Selection Rule

Why the Interim Selection Rule Matters

In a two-arm adaptive design, an interim analysis may select the experimental arm with the larger statistic and send only that arm to independent confirmatory testing. The selection step is data-dependent, but that fact alone does not determine the familywise error rate (FWER). The key question is whether the confirmatory test remains valid after selection. When the confirmatory data are independent of the interim data and the selected arm is tested at level \(\alpha\), the FWER is controlled at \(\alpha\), even though the arm was chosen for looking favorable at interim.

This result is different from testing both treatment hypotheses on the same confirmatory data. In the selection design, only one hypothesis is tested at confirmation. Independence makes the conditional rejection probability for each true null no larger than \(\alpha\), regardless of how often that null arm is selected. The derivation below makes those conditions explicit and also quantifies how often a null arm is selected under a normal interim model.

Design, Hypotheses, and Assumptions

The basic error-control argument requires that the interim selection rule be fixed in advance and use only interim information. It also requires that the confirmatory data be independent of the interim information used for selection. For each arm whose null hypothesis is true, its confirmatory test must have size at most \(\alpha\): under \(H_{0j}\), the probability of rejection is no greater than \(\alpha\). The test, endpoint, analysis population, and direction of testing must be specified so that this size statement applies to the actual confirmatory analysis.

For the closed-form selection probabilities below, impose an additional model: the interim statistics are jointly normal with unit variances, \((Z_1,Z_2)\sim N_2((\delta_1,\delta_2),\begin{pmatrix}1&\rho\\\rho&1\end{pmatrix})\), where \(-1<\rho<1\), and \(\delta_j\) is the interim mean on the standardized statistic scale. This model is useful for calculating selection probabilities; it is not needed for the general FWER bound. The confirmatory normal test, when used, has a standard normal statistic under its null and rejects for \(Z^C_j>c_\alpha\), where \(c_\alpha=\Phi^{-1}(1-\alpha)\). Here \(\Phi\) is the standard normal cumulative distribution function.

Deriving the FWER

Let \(S\) represent all interim information and let \(J\) be the selected arm. For a selected arm \(j\) whose null is true, independence means that learning \(S\) and \(J=j\) does not change the null distribution of its confirmatory statistic. Consequently, \(P(\text{reject }H_{0j}\mid S,J=j)\leq\alpha\). A false rejection can occur only if the selected arm has a true null and its confirmatory test rejects. Partitioning by the selected arm gives the result.

Here \(\mathcal{H}_0\) is the set of true null hypotheses and \(\mathbf{1}\{J=j\}\) equals one when arm \(j\) is selected and zero otherwise. The events \(J=j\) are mutually exclusive, so the sum of their probabilities over true-null arms is at most one. If the null confirmatory tests have exact size \(\alpha\), the first inequality is an equality. Thus \(\mathrm{FWER}=\alpha\sum_{j\in\mathcal{H}_0}P(J=j)\). This gives strong control: the bound holds whether both, one, or neither of the arm-specific null hypotheses is true. If neither is true, the FWER is zero by definition.

Selection Probabilities Under the Normal Model

Under the stated bivariate normal model, arm 1 is selected when \(Z_1-Z_2>0\). The difference has mean \(\delta_1-\delta_2\) and variance \(1+1-2\rho=2(1-\rho)\). Therefore:

Because the joint distribution is continuous, a tie has probability zero in this model. Under the global null, \(\delta_1=\delta_2=0\), so each arm is selected with probability one half. These selection probabilities describe which arm advances; they do not by themselves describe confirmatory error. The independent confirmation test contributes its own size-\(\alpha\) rejection probability after selection.

Comparison With Testing Both Arms

The procedure above tests only the selected arm in a separate confirmatory sample. It is not the same as using one data set to test both arm hypotheses after choosing the larger interim statistic. If both hypotheses are tested in the confirmatory sample, the event of at least one false rejection includes the possibility of rejecting either true null. A multiplicity procedure such as Bonferroni, Holm, Dunnett, or a suitable maxT method may then be required, depending on the design and the dependence structure. The selection rule alone does not justify treating two confirmatory tests as one.

A conservative alternative is to allocate \(\alpha/2\) to each of two confirmatory tests, whether or not both arms are ultimately tested. It controls the FWER but can spend more error probability than necessary when exactly one arm is tested and confirmation is independent. The next example makes that distinction numerical.

Diagnostics, Limitations, and a Common Mistake

The error-rate result is exact only when the conditional size argument matches the implemented design. Before relying on it, check that the confirmatory sample is genuinely independent of the interim information driving selection; that the test for each true null is calibrated at the stated level; and that the protocol tests only the selected arm. Also verify that the selection rule, test direction, endpoint, missing-data handling, and confirmatory analysis were specified so selection has not silently changed the test being applied.

If confirmatory data are reused from the interim analysis, independence no longer supplies the conditional-size guarantee. A nominally level-\(\alpha\) test can then have a rejection probability above \(\alpha\) among arms selected for favorable interim results. Likewise, testing both arms, selecting according to a different statistic than the prespecified one, or making additional data-dependent choices at confirmation changes the argument. Such designs need an error-control method justified for their actual information structure. The bound also says nothing about power, bias in the selected interim estimate, or the probability of selecting the clinically best arm.

Common mistake: “There are two possible arms, so the selected-arm confirmation test must use \(\alpha/2\).” This incorrectly treats the procedure as if it always tested both hypotheses. When exactly one arm is selected and only that arm is tested on independent data, the FWER bound follows from the single conditional size-\(\alpha\) test. Conversely, it is also wrong to conclude that selection never matters: if confirmatory data are dependent on selection or both hypotheses are tested, the single-test derivation may not apply.