Why Arm Dropping Does Not Automatically Recalibrate a Design
At an interim analysis, a MAMS trial may drop arms whose results look unpromising and continue with the others. The number of hypotheses still under consideration is then smaller, but that fact alone does not establish that the remaining tests may use less stringent boundaries. The surviving arms were selected using trial data, and earlier tests may already have used part of the familywise error budget. Calibration must account for both.
The central diagnostic question is not simply “How many arms remain?” It is “Under the stated model and the actual selection rule, what is the probability of rejecting at least one true null hypothesis?” A calculation based only on the number of survivors can miss dependence across arms, selection conditioning, earlier boundary crossings, and the rule used to reassign error.
A calibration argument must specify the hypotheses, the stopping and dropping rules, the stagewise boundaries, and the joint distribution or other validity conditions for the test statistics. For example, statistics for two treatment-versus-control comparisons may be correlated because they share a control group. Treating them as independent without justification can give an incorrect familywise probability.
A convenient sufficient argument is a union bound. If, for every true \(H_j\), the probability of rejecting it at any stage is no greater than an assigned amount \(\alpha_j\), then \(P(\text{reject at least one true }H_j)\leq \sum_{j\text{ true}}\alpha_j\). The bound is valid whether or not the tests are independent, but the assigned amounts must cover the entire testing path, including any adaptation. A bound above the target level is not a guarantee at that target.
Three Pitfalls to Check After an Interim Drop
The following errors are related but distinct. “Recalibrating for fewer arms” changes the multiplicity adjustment; “ignoring selection” treats selected data as if the selection had not happened; and “reusing a critical value” may discard the original design’s multiplicity or stagewise protection. A valid procedure may adapt after dropping arms, but it must preserve a stated error-control argument.
Worked Example: A Union Bound That Exceeds the Target
Suppose a design has two treatment hypotheses and a target FWER of \(\alpha=0.05\). A proposed calculation assigns each arm an error allowance of 0.01 at the first stage and 0.02 at the second stage. Assume, for this calculation, that each allowance is a valid bound on that arm’s probability of a false rejection at the specified stage, and that the bounds apply under every relevant null configuration.
For one arm, the total bound across its two stages is \(0.01+0.02=0.03\). Summing across both arms gives \(2(0.01+0.02)=0.06\).
Conclusion: No. Under the stated assumptions, this union-bound argument guarantees at most 0.06, not at most 0.05. The calculation does not prove that the actual FWER equals 0.06; it shows only that this bound is too large to establish the requested control. One remedy is to use smaller allocations. For example, assigning 0.01 per arm at each of the two stages gives the bound \(2(0.01+0.01)=0.04\), which is below 0.05, provided those stagewise bounds are valid.
Worked Example: Surviving Arms and Reused Unadjusted Values
Suppose two arms remain at a final look and their one-sided test statistics are independent standard normal variables under the global null. If each is tested at the unadjusted 0.05 level, the critical value is approximately 1.645. The probability of at least one false rejection is \(1-P(Z_1\leq1.645,\ Z_2\leq1.645)\).
The resulting FWER is 0.0975, nearly twice the target 0.05. If instead each test uses a one-sided level of 0.025, the critical value is approximately 1.960, and under this independent model the familywise probability is \(1-(1-0.025)^2=0.049375\), which is below 0.05. This is a Bonferroni allocation for two tests. It is not automatically the most efficient adjustment, and the arithmetic depends on the stated independence model; under dependence, the union bound still guarantees at most \(2(0.025)=0.05\).
The example diagnoses the danger of giving every surviving arm the original unadjusted 0.05 value merely because the number of arms has changed. If one arm remains, a single-test 0.05 threshold might be appropriate under a valid design, but a data-dependent decision to treat the survivor as a fresh, isolated test is not itself that validity argument.
Selection Conditioning Changes the Relevant Error
An arm retained for further study is generally not a randomly chosen arm. It survived a rule based on interim data. Consequently, the distribution of its interim statistic conditional on retention differs from its unconditional null distribution. A calculation that uses an unconditional reference distribution after selecting on that same statistic may give a misleading conditional error probability.
The distinction matters even when the unconditional test statistic has a standard normal distribution. The next example isolates the selection effect in a simple Gaussian model. It is a diagnostic calculation, not a proposed MAMS boundary rule.
Worked Example: Conditional Error After Favorable Selection
For one arm under its null hypothesis, let \(X\) be the standardized interim statistic and \(Y\) an independent standardized increment from the next stage, with \(X,Y\sim N(0,1)\). Define the standardized cumulative statistic at the next look as \(S=(X+Y)/\sqrt{2}\). Before selection, \(S\sim N(0,1)\). Suppose the arm is retained only if \(X>1.645\), an event with probability approximately 0.05. Consider the conventional one-sided final critical value 1.960.
Conditional on \(X=x\), \(S\) is normal with mean \(x/\sqrt{2}\) and variance \(1/2\). Therefore, \(P(S>1.960\mid X=x)=\Phi(x-1.960\sqrt{2})=\Phi(x-2.772)\), where \(\Phi\) is the standard normal cumulative distribution function. Averaging over the selected interim values gives
Here \(\phi\) is the standard normal density; numerical integration gives a numerator of about 0.0125. Thus, among arms selected by this favorable interim rule, the chance of crossing the ordinary final boundary is about 0.25 under the null, not 0.025. The unconditional joint probability of selection and crossing is about 0.0125. These are different probabilities and answer different questions.
This calculation does not imply that every design using interim selection has inflated unconditional FWER. It shows why an unconditional reference distribution cannot be substituted for a conditional calculation without checking the design. The retention event, the statistic used at the final look, and the dependence between looks all matter.
What a Valid Post-Drop Argument Must Preserve
An arm drop changes the future testing path; it does not retroactively erase prior opportunities to reject a true hypothesis. Nor does dropping a hypothesis automatically transfer its unused error allowance to the survivors. Whether such transfer is allowed depends on the pre-specified procedure and on whether its error-control proof remains valid after the data-dependent selection.
A valid adaptation can be justified in several ways. The approach must match the design and assumptions; simply counting the remaining arms is not a method.
| Approach | What it requires | Common diagnostic question |
|---|---|---|
| Bonferroni allocation | Assigned error amounts whose sum over relevant true hypotheses and stages is no greater than \(\alpha\). | Do the allocations cover every possible stagewise rejection, including after a drop? |
| Joint multivariate calibration | A valid joint null distribution for the test statistics and a boundary calibrated for the specified correlation structure. | Does the model account for shared control, unequal information, and the actual looks? |
| Conditional-error method | A valid conditional error under the original design and an adapted procedure whose conditional rejection probability does not exceed it. | Was the conditional error computed given the observed interim information and selection? |
| Closed testing | Valid tests of the required intersection hypotheses, combined according to the closed-testing rule. | Are all intersection tests needed for a claim still valid under the adaptation? |
These approaches are not interchangeable labels for “adjusting for fewer arms.” Bonferroni is often transparent but can be conservative. Joint calibration can use dependence information but relies on a correct joint model. Conditional-error adaptation can accommodate changes after an interim analysis, but its condition is precisely what a naïve survivor-only recalibration may omit. Closed testing can provide strong control when its intersection tests remain valid; it does not license arbitrary replacement of those tests.
Practical Diagnostics and Failure Modes
Before accepting a post-drop calibration, check the following points. A “yes” to one item does not compensate for a failure on another.
- Selection rule: Is retention determined by interim statistics, and is that rule included in the probability calculation? If not, conditional claims about survivors may be wrong.
- Stagewise accounting: Are error allocations bounds on the whole probability of rejecting each true null, or only on one stage? If they are stage-specific, add them using a valid argument and check that the resulting total is within \(\alpha\).
- Dependence: Are arm statistics correlated through a shared control or other common data? An independence calculation is appropriate only when independence is part of the model.
- Null configurations: Does the justification cover all configurations of true and false hypotheses, as strong control requires, rather than only the global null?
- Information and timing: Do the assumed information fractions and look times match the actual design? Changing when outcomes are available can change the joint distribution of stagewise statistics.
- Error transfer: Is any error amount assigned to a dropped arm explicitly allowed to be reused by the procedure? Do not treat an unspent allowance as automatically available.
A frequent mistake is to say, “One arm was dropped, so the remaining arm is now a single test and can use the ordinary single-test cutoff.” The correction is to ask whether the remaining test’s full path—including the selection that made it the survivor—has a valid error bound. Another frequent mistake is to add nominal stage levels and call the result the exact FWER. Addition usually supplies an upper bound, not an equality; the bound can be conservative, but it must still be no larger than the target if it is being used as the guarantee.
These diagnostics establish whether a proposed calculation is justified; they do not replace design-specific checks of the model or boundary implementation. In particular, one can have a valid unconditional FWER argument while conditional error among selected survivors is large, or a small conditional error for one decision while the overall familywise probability across multiple decisions is too large. State which probability is being controlled.
List the treatment hypotheses, stages, stopping rules, drop rules, and all possible rejection events.
Give the null distributions and dependence structure, and say whether the calculation is unconditional or conditional on interim information.
Verify that stagewise allocations, a joint calibration, a conditional-error argument, or valid intersection tests cover the actual adapted path and all required null configurations.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Two independent one-sided tests are each conducted at level 0.04. What is the familywise error under the global null?
- Two arms each receive 0.015 at the first stage and 0.01 at the second. What does the union bound give for their combined error, and does it guarantee FWER at most 0.05?
- Why can the conditional error among retained arms differ from the unconditional type I error?
- Name two pieces of information, beyond the number of survivors, needed to justify a new post-drop boundary.
Solutions. (1) Under the stated independence assumption, the probability of at least one false rejection is \(1-(1-0.04)^2=1-0.9216=0.0784\), or 7.84%. (2) Each arm’s stagewise total is \(0.015+0.01=0.025\); across two arms the union bound is \(2(0.025)=0.05\). It guarantees FWER at most 0.05 only if the stagewise amounts are valid bounds under the relevant configurations and cover all possible rejections. (3) Retention is determined by interim data, so the distribution of those data conditional on retention is generally different from their unconditional null distribution. (4) Suitable answers include the selection rule, the joint distribution or dependence of the statistics, the error already spent, and the null configurations for which control is required.