Arm Dropping Under Futility Without Losing Strong Control

Why Futility Dropping Needs an Error-Control Argument

An interim analysis may indicate that one experimental treatment is unlikely to meet its objective. Dropping that arm can spare participants and resources, while allowing promising arms to continue. But a rule that changes which hypotheses are tested also changes the testing procedure. To claim confirmatory familywise error control, it is not enough to say that the rule is “only for futility.” The complete procedure—including the interim rule, the final tests, and any change to the significance thresholds—must preserve the error guarantee.

The key distinction is between removing an opportunity to reject and changing the rejection thresholds for the arms that remain. A fixed allocation of error to each arm can make the first change safe: a true null hypothesis can be rejected only if its own valid test crosses its pre-specified threshold. By contrast, increasing thresholds after dropping arms may make a retained true null easier to reject.

Definition: Suppose a trial has \(m\) experimental arms, with null hypotheses \(H_{01},\ldots,H_{0m}\). The familywise error rate is the probability of rejecting at least one true null hypothesis. Strong familywise error control at level \(\alpha\) means that this probability is at most \(\alpha\) for every possible configuration of true and false null hypotheses.

For example, the configuration in which all three arms are ineffective is different from one in which only arm 2 is ineffective and the other two treatments truly work. Strong control requires the error bound in both configurations, and in every configuration between them. It is not enough to control the probability of any false rejection only when all null hypotheses are true.

This tutorial considers a pre-specified futility rule: at an interim analysis, an arm meeting a stated futility criterion is discontinued and cannot later be declared successful. The rule is not itself evidence that an arm is ineffective, and stopping it does not make its null hypothesis true or false. The confirmatory question is whether any true null among the arms that continue can still be rejected too often.

Conditions for a Simple Strong-Control Guarantee

A direct guarantee uses valid arm-specific tests and error allocations fixed before the interim decision. Let \(R\) be the set of arms retained after applying the futility rule. For each arm \(j\), let \(p_j\) be its final p-value and let \(\alpha_j\) be its pre-specified testing threshold. The test rejects \(H_{0j}\) only if \(j\in R\) and \(p_j\leq \alpha_j\).

Theorem: Suppose that (1) the interim futility rule is pre-specified and an arm that is dropped cannot subsequently be rejected; (2) for every true \(H_{0j}\), its final p-value is super-uniform under that null, meaning \(P(p_j\leq t\mid H_{0j})\leq t\) for every \(0\leq t\leq1\), under the trial’s actual design; and (3) the fixed allocations satisfy \(\sum_{j=1}^m \alpha_j\leq\alpha\). Then the procedure has strong familywise error rate at most \(\alpha\). The result does not require independence among arms or between the interim decision and the final p-values.

The phrase “under the trial’s actual design” matters. If an interim decision changes the sample size, testing time, or analysis, the final p-value must still be calibrated for that design. A p-value that would be valid for a fixed sample size is not automatically valid after unaccounted optional stopping or adaptive sample-size changes. One sufficient setting is that each continuing arm follows its pre-specified sampling plan and uses a final test valid for that plan; another is a properly calibrated sequential or adaptive test.

The theorem’s other important condition is that the \(\alpha_j\) values are not increased merely because some arms were dropped. They may differ across arms, but their total must be no larger than the familywise level. A trial may use another multiplicity procedure instead of fixed allocations, but then the procedure must itself provide strong control under the adaptive rule.

Why the Bound Holds

Let \(T\) denote the set of true null hypotheses in a particular configuration. A familywise error occurs only if at least one true null is rejected. Since a dropped arm cannot be rejected, the event of a false rejection is contained in the union of the events that each true arm’s p-value crosses its fixed threshold. This containment holds even though the retained set depends on interim data.

\begin{aligned} P(\text{at least one true null is rejected}) &\leq P\left(\bigcup_{j\in T}\{p_j\leq\alpha_j\}\right)\\ &\leq \sum_{j\in T}P(p_j\leq\alpha_j)\\ &\leq \sum_{j\in T}\alpha_j\\ &\leq \sum_{j=1}^{m}\alpha_j\\ &\leq \alpha. \end{aligned}

The second inequality is the union bound, which does not require independent tests. The third follows from super-uniformity for each true null. Crucially, the argument does not condition on an arm having survived the interim analysis. Conditioning on survival can change a p-value’s distribution; instead, the proof bounds the actual false-rejection event by events involving the valid marginal p-values.

Worked Example: Three Arms With a Pre-Specified Futility Rule

Consider a hypothetical trial with three experimental arms and one-sided tests of whether each treatment improves the primary outcome over control. Set the familywise level to \(\alpha=0.025\), and assign \(\alpha_j=0.025/3=0.008333\) to each arm. At the interim analysis, drop any arm whose pre-specified futility statistic is below 0.5. Suppose arms A and C continue, while arm B is dropped. The calibrated final p-values for A and C are 0.0060 and 0.0120, respectively.

Arm A crosses its fixed threshold because \(0.0060\leq0.008333\), so its null hypothesis is rejected. Arm C does not, because \(0.0120>0.008333\). Arm B cannot be rejected because it was dropped. These decisions do not depend on whether any particular null is true; the error guarantee concerns the probability of such decisions over repeated trials under each null configuration.

If only A is a true null, a false rejection can occur only through A’s test, with probability at most 0.008333. If A and C are both true, the union bound gives a familywise error probability of at most \(0.008333+0.008333=0.01667\). If all three are true, the bound is \(3(0.008333)=0.0250\). Thus the same allocation controls the familywise error rate across these configurations, provided the final p-values remain valid under the actual design.

Worked Example: Checking the Bound With Two True Nulls

Suppose a hypothetical four-arm trial has \(\alpha=0.04\) and allocates \(\alpha_j=0.01\) to each arm. In one configuration, arms 1 and 4 are true nulls, while arms 2 and 3 are false nulls. Assume the pre-specified futility rule drops arm 4, and the test for arm 1 remains valid under the trial design.

A false rejection is possible only through arm 1: the dropped arm 4 cannot be rejected, and arms 2 and 3 are not true nulls in this configuration. Therefore, \(P(\text{familywise error})\leq P(p_1\leq0.01)\leq0.01\). If instead both true-null arms continue, the union bound gives \(P(\text{familywise error})\leq0.01+0.01=0.02\). Both bounds are at most the planned familywise level of 0.04. This check illustrates why the number of true nulls—not merely the total number of arms—appears in the proof, while the fixed allocation across all arms ensures the bound holds without knowing which nulls are true.

When Dropping an Arm Can Break the Guarantee

The theorem does not say that any procedure labelled “futility stopping” controls error. Two design features need particular scrutiny. First, the interim decision may change the data used for the final test. If the final analysis uses only observations collected after an arm survives, or if its sample size is changed, the original fixed-sample p-value may no longer be valid. Use a test calibrated for the actual sequential or adaptive design; do not assume that a conventional final p-value remains valid just because the rule was pre-specified.

Second, a trial might redistribute the error allocation after dropping arms—for example, by giving each survivor the full familywise level or by dividing \(\alpha\) only among the survivors. Such a change is not covered by the fixed-allocation proof. A new allocation can be valid if it is part of a formally justified multiplicity procedure, but the fact that the decision is described as futility does not establish that validity.

Worked Example: Why Giving Survivors the Full Alpha Can Fail

Consider a hypothetical two-arm all-null setting with \(\alpha=0.025\). To isolate the threshold issue, let \(U_1\) and \(U_2\) be independent, valid Uniform\((0,1)\) p-values under the two true nulls. Use a futility rule that drops an arm if its p-value exceeds 0.5. Suppose the trial then tests each surviving arm at the full level 0.025 rather than at a fixed per-arm allocation.

Any p-value at most 0.025 is below 0.5, so that arm survives and is rejected. The familywise error probability is therefore \(P(U_1\leq0.025\text{ or }U_2\leq0.025)\). By independence, the probability that neither is rejected is \(0.975^2=0.950625\), giving \(1-0.950625=0.049375\), or about 0.0494. This exceeds 0.025, despite the p-values each being valid marginally.

This calculation is a simple illustration of the risk of expanding thresholds after selection; it is not a recommendation to use interim p-values as final tests. With a fixed Bonferroni allocation of \(0.025/2=0.0125\) per arm, the union bound instead gives a familywise error probability no larger than \(2(0.0125)=0.025\). In a real trial, the validity of the p-values must also account for how interim data and later observations enter the final analysis.

Practical Checks and Alternative Approaches

Before implementation, write down the complete decision and testing procedure, not just the interim futility boundary. For each arm, specify what data determine futility, whether dropping is irreversible, what data enter its final test if retained, and what significance threshold applies. Then verify that the test’s null distribution accounts for the design actually used. If a survival rule changes sample sizes or analyses, validity cannot be inferred from the fixed-sample test alone.

Fixed Bonferroni allocations are transparent and robust to dependence, but can be conservative. Other multiplicity methods, including Holm or a suitably calibrated closed-testing or maxT procedure, may use dependence or ordering information to improve power. Their strong-control guarantees must still apply after the interim adaptation; one cannot simply remove dropped hypotheses and assume the remaining procedure retains its original guarantee. If the rule changes allocations or testing opportunities, justify that complete adaptive procedure rather than relying only on the original method’s name.

Useful diagnostics include checking whether the interim rule and all thresholds were specified before unblinded outcome review; verifying that dropped arms are never tested or declared successful later; and confirming that the final test remains calibrated for the stopping and sampling plan. Simulation across all-null and partial-null configurations can reveal implementation errors or poor calibration, but simulation alone does not replace a mathematical guarantee unless the claim is explicitly limited to the scenarios and precision evaluated.

Common mistake: “The arm was dropped only for futility, so removing it cannot increase the chance of a false positive.” This is incomplete reasoning. Dropping an arm removes its own rejection opportunity, but reallocating its unused alpha can increase the chance of rejecting a different true null. The correction is to keep fixed per-arm thresholds whose total is at most \(\alpha\), or to prove strong control for the alternative adaptive multiplicity procedure. Also verify that the final p-values are valid under the actual interim design.

Check Your Understanding

Answer each question, then compare with the solutions below.

A three-arm trial uses fixed thresholds of 0.01 per arm. Two null hypotheses are true. What familywise error bound follows from the union bound?
Why does the strong-control argument not require independence between the arms’ test statistics?
What must be checked if the interim futility decision changes the sample size for a retained arm?
In a two-arm all-null trial, two independent Uniform\((0,1)\) p-values are each tested at 0.02. What is the probability of at least one rejection?
Does pre-specifying a futility rule alone establish strong familywise error control? State the additional requirements.

Solutions. (1) At most \(0.01+0.01=0.02\), because only the two true nulls contribute to the false-rejection event. (2) The union bound applies under arbitrary dependence, and each true-null p-value is bounded by its own marginal threshold probability. (3) The final test must be valid under the changed sample-size and stopping design; a fixed-sample p-value cannot be presumed valid. (4) Independence gives \(1-(1-0.02)^2=1-0.9604=0.0396\). (5) No. The final tests must be valid under the actual design, dropped arms must not later be rejected, and the thresholds or adaptive multiplicity procedure must preserve strong control across all null configurations.