Multiplicity Control When Arms Are Dropped At Interim

Why the Family Must Be Tracked Through Arm Dropping

An interim review may remove treatment arms that appear unlikely to be useful, leaving fewer comparisons for the final analysis. It is tempting to treat only the surviving arms as the family and divide the significance level among them. But the surviving set may itself depend on interim data. When selection and testing are connected, using a smaller, data-dependent family can make false-positive risk exceed its intended level.

A robust approach is to define the confirmatory family before the trial begins and preserve its multiplicity allocation when arms are dropped. For example, if four treatment-versus-control hypotheses are planned, a fixed Bonferroni procedure can retain a threshold of \(\alpha/4\) for each hypothesis that remains testable. The threshold does not increase merely because some arms were removed.

Definition: Let \(H_1,\ldots,H_M\) be the pre-specified treatment hypotheses in the confirmatory family, where \(M\) is the number of hypotheses at the start of the procedure. Let \(S\subseteq\{1,\ldots,M\}\) denote the arms retained after interim selection. The familywise error rate (FWER) is the probability of rejecting at least one true null hypothesis in the family. Strong FWER control at level \(\alpha\) means that this probability is at most \(\alpha\) for every configuration of true and false null hypotheses.

The original family is a set of hypotheses, not just the list of arms still under consideration. An arm dropped at interim is no longer tested for efficacy in the planned final analysis, but its hypothesis remains part of the family used to determine the multiplicity adjustment. This distinction matters especially when selection was informed by trial data.

Fixed Bonferroni Control After Selection

Suppose each final p-value \(p_j\) is valid for its null hypothesis: under \(H_j\), it is super-uniform, meaning \(P(p_j\leq t)\leq t\) for every \(t\) between 0 and 1. The fixed Bonferroni rule rejects a surviving \(H_j\) only if \(p_j\leq\alpha/M\). The denominator is the number of hypotheses in the initial family, not the number \(K=|S|\) that happen to survive.

\text{Reject } H_j \text{ if } j\in S \text{ and } p_j\leq\frac{\alpha}{M}.

This rule does not require independent tests or independent selection and testing. It does require valid p-values for the hypotheses being tested. In particular, the interim adaptation must not make a nominal final p-value invalid through an unaccounted change in analysis, follow-up, or sampling.

Justification: Write \(I_0\) for the set of indices of true null hypotheses. A false rejection after selection can occur only if at least one true-null p-value is at most \(\alpha/M\). Therefore,

\begin{aligned} \mathrm{FWER} &=P\left(\bigcup_{j\in I_0}\{j\in S,\ p_j\leq\alpha/M\}\right)\\ &\leq P\left(\bigcup_{j\in I_0}\{p_j\leq\alpha/M\}\right)\\ &\leq \sum_{j\in I_0}P(p_j\leq\alpha/M)\\ &\leq |I_0|\frac{\alpha}{M}\\ &\leq \alpha. \end{aligned}

The first inequality uses the fact that requiring an arm to survive can only make the event smaller. The second inequality is the union bound; it does not require independence. The last inequality follows from \(|I_0|\leq M\). Thus the rule provides strong FWER control even if the selection rule is data-dependent, provided the marginal p-values remain valid.

Worked Example: Four Initial Arms, Two Survivors

Consider a hypothetical trial with four planned treatment-versus-control hypotheses, \(H_1,\ldots,H_4\), and familywise level \(\alpha=0.05\). At interim, a pre-specified rule drops arms 2 and 3. Suppose the final p-values for the two survivors are \(p_1=0.009\) and \(p_4=0.018\). Assume these p-values are valid under their respective null hypotheses.

The initial family size is \(M=4\), so the fixed Bonferroni threshold is

\frac{\alpha}{M}=\frac{0.05}{4}=0.0125.

For arm 1, \(0.009\leq0.0125\), so reject \(H_1\). For arm 4, \(0.018>0.0125\), so do not reject \(H_4\). The conclusion is that the final analysis rejects the null for arm 1, but not for arm 4, at the specified familywise level. This comparison does not claim that arm 1 is clinically beneficial; that interpretation also depends on the estimated effect, its direction, and the clinical context.

The number of surviving arms is two, but it does not replace the pre-specified denominator of four. Using \(0.05/2=0.025\) would reject both p-values, but that survivor-only calculation does not, by itself, establish FWER control when the surviving set was chosen using trial data.

Following the Family and Its Allocation

A practical way to maintain control is to record the complete hypothesis family and its testing rule before interim selection. After selection, the trial team should be able to identify which hypotheses remain testable and the threshold assigned to each. The rule must specify whether the allocation attached to a dropped arm is retained, forfeited, or redistributed under a separately justified procedure. It should not be decided after seeing which arms survived.

The simplest fixed allocation leaves a dropped arm’s share unused. A pre-specified weighted Bonferroni rule can assign different shares to hypotheses in advance. Let \(w_j\geq0\) be the weight for \(H_j\), with \(\sum_{j=1}^{M}w_j\leq1\). Test a surviving \(H_j\) only if \(p_j\leq\alpha w_j\). Since each true-null p-value is valid, the union bound gives

\mathrm{FWER} \leq \sum_{j\in I_0}P(p_j\leq\alpha w_j) \leq \alpha\sum_{j\in I_0}w_j \leq \alpha.

The weights can reflect pre-specified priorities, but they must be set before the selection outcome is known. A dropped arm’s weight is not automatically transferred to a survivor. Such recycling changes the threshold and requires a procedure whose error control has been established for that redistribution rule.

Worked Example: Pre-Specified Weighted Allocation

Suppose a hypothetical trial has three planned treatment hypotheses and familywise level \(\alpha=0.05\). Before enrollment, the protocol assigns weights \(w_1=0.50\), \(w_2=0.30\), and \(w_3=0.20\). These weights sum to 1. At interim, arm 2 is dropped; the final p-values are \(p_1=0.021\) and \(p_3=0.009\). Assume valid p-values for the surviving hypotheses.

The thresholds remain the pre-specified weighted thresholds:

\alpha w_1=0.05(0.50)=0.025,\qquad \alpha w_3=0.05(0.20)=0.010.

For arm 1, \(0.021\leq0.025\), so reject \(H_1\). For arm 3, \(0.009\leq0.010\), so reject \(H_3\). The unspent allocation for arm 2 has not been added to either survivor. Strong control follows because, for any true-null subset, the sum of its weights cannot exceed the sum of all weights, which is 1.

The result depends on the weights being specified in advance. If the protocol instead says to redistribute a dropped arm’s weight, it must define exactly how and when redistribution occurs and use a method valid for that adaptive rule. The calculation above does not justify ad hoc redistribution.

Worked Example: A Larger Initial Family Than the Survivor Set

In a hypothetical study, five treatment hypotheses are planned at \(\alpha=0.05\). At interim, only arms A and C continue. Their final p-values are \(p_A=0.007\) and \(p_C=0.013\), and both are valid under their respective nulls.

The fixed threshold is \(\alpha/M=0.05/5=0.010\). Since \(0.007\leq0.010\), reject the null for A. Since \(0.013>0.010\), do not reject the null for C. If all five null hypotheses were true, the union-bound guarantee would be at most \(5(0.010)=0.050\). If only three were true, the same bound would be \(3(0.010)=0.030\). These are upper bounds on error probability, not probabilities inferred from the observed p-values.

Assumptions, Diagnostics, and Alternatives

Multiplicity control and p-value validity are separate requirements. Bonferroni controls the probability of one or more false rejections when each p-value is valid under its own null. It does not repair a p-value made anti-conservative by an unaccounted adaptive analysis. For instance, an analysis that changes the endpoint, follow-up, or final sample size in response to interim results may need an analysis method that accounts for that adaptation.

Useful checks include confirming that the initial family and its size are recorded, that the dropping rule and testing thresholds were specified, and that every tested hypothesis has a valid p-value under the actual design. Verify that a removed arm cannot re-enter the confirmatory family under an undocumented rule. Also check that the reported denominator is the initial \(M\), unless a different valid procedure was explicitly planned.

The fixed Bonferroni rule is transparent and robust, but can be conservative, particularly when tests are strongly correlated. Pre-specified weighted Bonferroni changes the allocation, not the basic union-bound guarantee. Other established approaches include Holm step-down testing and Dunnett or MaxT procedures that use the joint distribution of treatment-versus-control statistics. Such approaches may gain power, but they need to be applied to a family and selection scheme for which their error-control conditions hold. Applying a procedure only to the selected subset does not automatically provide that justification.

When the scientific goal specifically concerns inference conditional on the selected arms, selection-adjusted p-values or other selection-aware methods may be appropriate; the earlier material on adjusted p-values for adaptively selected treatments addresses that distinct inferential target. The fixed-family procedure here targets control of false rejections across the original confirmatory family. It does not correct bias in effect estimates caused by selection, nor does it guarantee that a surviving arm is clinically important.

Common Mistake

Wrong reasoning: “Only two arms survived, so I should divide 0.05 by two and test each at 0.025.” This treats a data-selected set as though it had been fixed in advance and can understate the multiplicity involved in the selection process.

Correction: Keep the original family size in the pre-specified fixed Bonferroni threshold: if four hypotheses were planned, use \(0.05/4=0.0125\) for each surviving hypothesis. Alternatively, use another method whose control has been justified for the selection rule. Do not assume that deleting hypotheses from the analysis deletes their role in the original error-control plan.

Key takeaway: Track the initial family through interim arm dropping. A fixed threshold based on that family, or another explicitly justified pre-specified allocation, can control FWER among surviving arms when the tested p-values remain valid under the adaptive design.

Check Your Understanding

Answer each question, then compare with the solutions below.

A trial begins with six hypotheses at \(\alpha=0.05\). Two arms survive. What is the fixed Bonferroni threshold, and why is it not \(0.05/2\)?
For four hypotheses at \(\alpha=0.04\), the valid p-values for two surviving arms are 0.008 and 0.012. Under fixed Bonferroni, which are rejected?
State the role of the union bound in the FWER argument and the reason the final bound is at most \(\alpha\).
Three hypotheses receive pre-specified weights 0.50, 0.30, and 0.20 at \(\alpha=0.05\). The second arm is dropped. What thresholds apply to the first and third arms, and may the dropped arm’s weight automatically be redistributed?

Solutions. (1) The threshold is \(0.05/6=0.00833\) (to three significant figures). The original family has six hypotheses; survival of two does not erase the selection from the multiplicity plan. (2) The threshold is \(0.04/4=0.010\). Reject the null with p-value 0.008, but not the null with p-value 0.012. (3) The union bound gives \(P(\bigcup_{j\in I_0}\{p_j\leq\alpha/M\})\leq\sum_{j\in I_0}P(p_j\leq\alpha/M)\). Validity bounds this sum by \(|I_0|\alpha/M\), and \(|I_0|\leq M\) makes the final bound at most \(\alpha\). (4) The thresholds are \(0.05(0.50)=0.025\) and \(0.05(0.20)=0.010\). No; redistribution requires a pre-specified procedure with a valid error-control justification.