Why Dropping an Arm Can Affect Error Control
An interim review may show that one treatment arm is less promising than another. Dropping the weaker arm can simplify the trial, focus resources, or avoid continuing a treatment with little prospect of benefit. But the decision is based on data, and that matters: the arm that continues may be the one whose interim results look unusually favorable by chance.
The key question is not simply whether an arm was dropped. It is whether the final rejection rule has a greater chance of rejecting a true null hypothesis because the trial selected an arm using data that also contribute to its rejection. If selection and testing use overlapping evidence, treating the chosen arm as if it had been specified in advance can miss that extra opportunity.
An arm-dropping rule does not automatically inflate the Type I error rate. For example, selecting one arm using interim data and testing it using independent future data can be valid at the test's nominal level under suitable assumptions. The danger is assuming that any test of the selected arm is valid merely because only one arm remains. The data used, the selection rule, and the rejection rule must be considered together.
A Selection Rule Can Preserve Two Rejection Chances
Consider two treatment arms, each compared with control. Suppose, under the global null hypothesis that neither treatment is effective, the final one-sided test statistics \(U_1\) and \(U_2\) are standard normal with correlation \(0.5\). The positive correlation could arise because both comparisons use a shared control group. Assume the trial selects the arm with the larger statistic and declares success if that selected statistic exceeds the one-sided 5% cutoff \(c=1.645\).
Worked Example: Selecting the Larger of Two Correlated Statistics
Under the global null, take \((U_1,U_2)\) to have a bivariate normal distribution with means zero, variances one, and correlation \(\rho=0.5\). The decision rule rejects if the selected statistic exceeds \(1.645\). Because the selected statistic is \(\max(U_1,U_2)\), rejection occurs if either statistic exceeds the cutoff:
For a bivariate standard normal pair, the joint upper-tail probability is
where \(\phi\) and \(\Phi\) are the standard normal density and cumulative distribution functions. With \(c=1.645\) and \(\rho=0.5\), numerical integration gives a joint upper-tail probability of approximately \(0.01217\). Each marginal upper-tail probability is \(P(Z>1.645)\approx0.049985\). Inclusion-exclusion therefore gives
The chance of at least one false rejection is about 8.8%, not 5%. Correlation reduces the inflation compared with independent statistics, but does not remove it. The rule effectively gives the global null two chances to produce a statistic above the nominal single-test cutoff.
This example isolates the selection issue: choosing the larger statistic and testing it at the ordinary single-arm cutoff is equivalent to rejecting when either statistic crosses that cutoff. It is not equivalent to testing one arm that was fixed before any data were observed. If the rule instead selected an arm using an interim statistic and tested only an independent future statistic, the calculation would be different.
When a Single Selected-Arm Test Can Be Valid
Suppose interim data \(D\) determine a single arm \(J(D)\) to continue, and the final test uses only future data that are independent of \(D\). Assume that, under each true arm-specific null hypothesis, the future test statistic for that arm has its stated null distribution conditional on \(D\). If the test is one-sided at level \(\alpha\), then for any selected arm whose null is true, its conditional rejection probability is at most \(\alpha\).
Worked Example: Dropping One Arm and Testing on Independent Future Data
Suppose two arms are compared with control. An interim rule selects one arm using only interim data, and the other arm is dropped. Under either arm's null hypothesis, assume its future-only statistic \(Z_{F,i}\) is standard normal conditional on all interim data and the selection decision. The selected arm is tested at the one-sided 5% cutoff \(1.645\).
Conditional on the interim data, the selected index \(J\) is fixed. If \(H_J\) is true, the assumed conditional null distribution gives
Thus the conditional chance of falsely rejecting the selected arm's null is at most 0.05. Averaging over interim data preserves that bound:
This argument depends on using independent future evidence and testing only one arm. It does not justify combining interim and future evidence in an ordinary fixed-sample statistic after choosing the arm. Nor does it justify testing multiple retained arms at 5% each and treating their tests as one 5% family.
A future-only test may sacrifice power because it discards interim information. It is one valid strategy, not a universal requirement. If the design wants to reuse interim evidence, or wants a continuation rule that preserves the original design's error guarantee, conditional error provides a general way to constrain that rule.
Conditional Error as a Constraint on Continuation
Let \(I\) denote an intersection hypothesis: all component null hypotheses in \(I\) are true. Suppose the original design has a valid test of \(I\) at level \(\alpha\). After interim data \(D\), define its conditional error as
The preservation argument follows from the law of total probability. The original test is level \(\alpha\), so its conditional error averages to at most \(\alpha\) under \(I\). If the adapted rule's conditional rejection probability is bounded by that conditional error, its overall rejection probability is also at most \(\alpha\):
For strong familywise error control, this reasoning must apply to every intersection hypothesis required by the closed testing procedure, not only to the global intersection. Closed testing rejects an individual null only when the tests of all intersections containing it reject. Consequently, a continuation rule must preserve valid local tests for the relevant intersections; a bound established only for the global null does not by itself establish strong control.
Worked Example: Conditional Error After an Interim Review
Consider two treatment arms. Under each arm's null, suppose its interim statistic \(Z_{I,i}\) and future statistic \(Z_{F,i}\) are independent standard normal variables; also suppose the future statistics for the two arms are independent. The original design combines increments as \(Z_i=(Z_{I,i}+Z_{F,i})/\sqrt{2}\) and tests each arm at the one-sided Bonferroni cutoff \(1.96\). This allocates at most 0.025 to each arm, so the familywise error rate is at most 0.05.
Suppose the observed interim statistics are \(Z_{I,1}=1.00\) and \(Z_{I,2}=0\), and the rule drops arm 2. Under the original design, conditional on these interim values, arm 1 would reject if
Its conditional rejection probability is therefore about \(P(Z>1.772)=0.0382\). Arm 2 would reject if \(Z_{F,2}>\sqrt{2}(1.96)-0\approx2.772\), with conditional probability about \(0.00279\). Since the future statistics are independent, the original conditional probability of at least one rejection is
For this observed interim result, a continuation procedure that claims to preserve the original global-null conditional error should not have a conditional rejection probability above approximately 0.0409. Testing arm 1 using only \(Z_{F,1}\) at the ordinary 5% level would have conditional rejection probability 0.05, which exceeds that particular conditional-error bound. This comparison does not mean the future-only strategy must always inflate overall Type I error; the preceding example gives a separate argument for its validity. It means that it does not preserve this original procedure's conditional-error allocation at these interim data.
Derivation: Why a Conditional Bound Preserves the Error Rate
The conditional-error argument requires a valid original local test, a continuation rule whose conditional rejection probability is controlled for every interim outcome, and valid probability calculations under the null being tested. The interim decision must be based on observed data, so that conditioning on \(D\) makes the selected arms and continuation rule fixed. The continuation test may be designed in many ways, but it must respect the bound for each relevant intersection.
For each intersection \(I\), its original rejection probability is at most \(\alpha\) whenever \(I\) is true.
At each observed \(D\), calculate the original design's probability \(A_I(D)\) of rejecting \(I\) using the remaining data.
Require the continuation procedure's conditional probability of rejecting \(I\) to be no greater than \(A_I(D)\), including after an arm is selected or dropped.
The law of total probability bounds the adapted rejection probability by the original test's probability, so it remains at most \(\alpha\).
This is a sufficient preservation strategy: it carries forward the error allowance of a specified original procedure. It may be conservative. Another continuation strategy, such as a future-only test, can be valid by a different argument even if it does not satisfy that pointwise conditional bound. The earlier material on conditional error and closure provides the general error-control framework; arm dropping makes explicit why the bound must follow the data-dependent decision.
Diagnostics, Alternatives, and Limitations
Before approving an arm-dropping rule, identify exactly which data determine selection and which data enter the rejection statistic. Then examine the null rejection probability under the actual rule, not under a hypothetical rule that chose the retained arm in advance. Useful checks include:
- Selection and test reuse: Does the final statistic include interim observations that helped select the arm? If so, its nominal fixed-design distribution may not apply after selection.
- Number of rejection opportunities: Can success be declared for any of several arms or claims? If so, the probability of at least one false rejection must be controlled across that rule.
- Conditional validity: Is the future-data test valid conditional on interim data and on the selected arm? Independence or an appropriate conditional null distribution must be justified, not assumed from the fact that the data were collected later.
- Intersection hypotheses: For strong control through closure, are the local tests still valid for every intersection required to reject an individual hypothesis?
- Operational changes: Does dropping an arm change the control group, randomization, follow-up, or available comparisons in a way that changes the assumed null distribution?
The main alternatives have different advantages and requirements:
| Approach | How it handles selection | Main limitation |
|---|---|---|
| Test each arm at the ordinary level using overlapping selected evidence | Does not, by itself, adjust for the selection opportunity | Can exceed the target error rate, as in the correlated-statistic example |
| Select one arm, then use independent future-only data | Uses conditional validity of the future test after selection | Can discard interim evidence and requires the stated conditional null distribution |
| Conditional-error continuation | Limits conditional rejection probability to the original design's allowance | Requires specifying and evaluating the original conditional error for each relevant intersection |
| Fixed multiplicity adjustment | Controls a prespecified set of rejection opportunities, for example by Bonferroni allocation | May be conservative and may not reflect the interim rule's conditional error allocation |
A common mistake is to reason, “Only one arm remains, so testing it at 5% must control error at 5%.” That conclusion ignores how the arm was chosen. If selection used the same noisy statistics that are tested, the selected arm is more likely to have favorable evidence by chance; a nominal single-arm cutoff can therefore yield an inflated rejection probability. The correction is to evaluate the complete selection-and-testing rule. Depending on the design, a valid solution may use independent future evidence, a multiplicity adjustment, or a conditional-error bound together with closure.
Conditional error is not a universal recipe for recovering any desired amount of power. Its usefulness depends on having a well-defined original test and correct conditional distributions under the relevant nulls. If the interim model is misspecified, the selected arm's future data are not conditionally valid, or some required intersection is omitted, the claimed guarantee may fail. Arm dropping also does not establish that an arm is ineffective; it is a design decision, not a statistical proof of no treatment effect.
Check Your Understanding
Use the selection and conditional-error principles above. Give each answer in the context of the stated decision rule.
- Two standard normal statistics each have an upper-tail rejection probability of about 0.05 under the global null. If the trial rejects when either exceeds its cutoff, why is the error probability not automatically 0.05?
- An interim rule selects one arm, and its final test uses only independent future data. State the condition that makes a nominal 5% test valid for a true selected-arm null.
- In the conditional-error example, explain why the original probability of at least one rejection at interim values \((1,0)\) is not the sum \(0.0382+0.00279\).
- Why is controlling the conditional error only for the global intersection insufficient to claim strong familywise error control through closure?
Solutions. (1) The two rejection events overlap, but their union includes an opportunity to reject through either statistic; its probability is generally greater than either marginal probability alone. The amount depends on their joint distribution. (2) Conditional on all interim data and the selected index, the future statistic for a true selected-arm null must still have its stated null distribution, so its conditional rejection probability is at most 0.05. (3) The two rejection events are not mutually exclusive. With independent future statistics, the union probability is \(1-(1-0.0382)(1-0.00279)\approx0.0409\), not their unadjusted sum. (4) Closure requires valid tests of all relevant intersections containing a rejected individual hypothesis. A guarantee for the global intersection alone does not establish that the other required local tests control their errors.