Why the Choice of Strategy Matters
A multi-arm multi-stage (MAMS) design makes decisions across both treatment arms and interim stages. A strategy that controls error for one comparison at one analysis may not control the chance of any false treatment claim across the whole design. The choice of strong-control method therefore affects both the validity and the efficiency of the trial: it determines which dependence information can be used, which adaptations are permissible, and what claims can be supported.
The aim here is to choose among four related approaches: closed testing, Bonferroni, Dunnett-type calibration, and conditional-error methods. They are not interchangeable labels for the same calculation. Closed testing organizes the hypotheses that must be rejected; Bonferroni controls error by allocating it across tests; Dunnett-type methods use the joint distribution of treatment-versus-control statistics; and conditional error provides a way to preserve error control after certain adaptations.
For \(m\) treatment hypotheses \(H_1,\ldots,H_m\), a MAMS procedure must account for the planned analyses, stopping rules, and any multiplicity adjustments when defining its rejection rule. The relevant error rate is the probability of at least one false rejection over the complete decision process. A nominal level at an individual stage does not, by itself, establish this whole-design control.
The first design question is therefore not “Which method is most powerful?” It is “What is the inferential task, and what features of the design are known well enough to support a valid calibration?” In particular, consider whether the goal is to make any valid treatment claim or a specified collection of claims; whether treatment statistics share a control group; whether their joint distribution is trustworthy; and whether the design may change after interim data are observed.
What Each Strategy Provides
Bonferroni allocates the familywise level across tests. For \(m\) hypotheses, testing each at level \(\alpha/m\) controls FWER at no more than \(\alpha\), regardless of dependence, provided each individual test is valid at its allocated level. More general allocations \(\alpha_1,\ldots,\alpha_m\) are allowed when \(\sum_{i=1}^m\alpha_i\leq\alpha\). In a MAMS design, the test or allocation must also account for repeated looks; dividing \(\alpha\) across arms alone does not handle stagewise multiplicity unless the sequential procedure is itself calibrated.
Dunnett-type calibration uses the joint distribution of treatment-versus-control statistics. With a shared control, those statistics are correlated. If the assumed joint distribution is accurate, a maximum-statistic cutoff can be less conservative than Bonferroni while controlling the probability of any false treatment rejection. In a sequential design, the joint distribution must cover the relevant statistics across arms and stages, or the stagewise boundaries must otherwise be calibrated to the complete design. A cutoff based on a single-stage distribution cannot simply be reused at every interim analysis.
Closed testing translates multiple treatment claims into a family of intersection hypotheses. For any nonempty subset \(I\) of treatment hypotheses, define \(H_I=\bigcap_{i\in I}H_i\). To reject \(H_i\), the procedure must reject every intersection hypothesis \(H_I\) for which \(i\in I\), using a valid level-\(\alpha\) test for each intersection. The tests may use different statistics or calibrations, but each must be valid for its intersection under the relevant null.
Conditional-error methods address adaptation. At an interim time, the conditional error for a null hypothesis (or intersection null) is the probability, under that null and given the interim data, that the original design would reject it in the remaining stages. A revised design can change future tests or allocations while preserving control if its conditional probability of rejection is no greater than the original conditional error for every relevant intersection, with the required unconditional validity of the original procedure.
A Short Justification for Closed Testing
The reason closed testing can provide strong control follows directly from the intersection structure. Let \(T\) be the set of indices of the true null hypotheses in a particular configuration, with \(T\) nonempty. If any true \(H_i\), \(i\in T\), is rejected by the closed procedure, then every intersection containing \(H_i\) must have been rejected. In particular, the intersection \(H_T=\bigcap_{i\in T}H_i\) must have been rejected.
The last inequality holds because the test of \(H_T\) is a valid level-\(\alpha\) test when all hypotheses in \(T\) are true. This argument applies to any configuration of true hypotheses, which is why the control is strong. In a MAMS setting, validity of each intersection test must cover the actual repeated-look rule and any allowed adaptation; closure does not repair an invalid component test.
Worked Example: Shared-Control Dependence Favors Dunnett-Type Calibration
Worked Example: Shared-Control Dependence Favors Dunnett-Type Calibration
Suppose a hypothetical fixed-analysis comparison has three treatment-versus-control statistics. Under the complete null, assume their joint distribution is standard trivariate normal with pairwise correlation \(0.5\), as might occur with equal-variance contrasts sharing a control. The one-sided familywise level is \(\alpha=0.025\). This example isolates the arm multiplicity calculation; a MAMS trial with multiple looks would require calibration for the whole sequential rule.
Bonferroni calculation. Each of the three one-sided tests uses level \(0.025/3=0.008333\). The corresponding standard-normal upper-tail critical value is approximately \(z_{1-0.008333}=2.394\). Thus a statistic must exceed \(2.394\) for its individual Bonferroni test to reject.
Dunnett-type calculation. Let \(Z_1,Z_2,Z_3\) denote the correlated statistics. Choose \(c\) to satisfy \(P(\max(Z_1,Z_2,Z_3)>c)=0.025\) under the specified joint distribution. For this trivariate normal distribution, the calibrated cutoff is approximately \(c=2.348\), or \(2.35\) to two decimal places. The smaller cutoff than the Bonferroni cutoff reflects the positive dependence; it is valid only insofar as the joint distribution and calibration are appropriate.
Now suppose the hypothetical observed statistics are \(2.37,\ 2.05,\) and \(1.60\). The largest statistic is \(2.37\). Since \(2.37<2.394\), none of the tests rejects under Bonferroni. Since \(2.37>2.348\), the first treatment comparison rejects under the stated Dunnett-type calibration. This difference is not evidence that Dunnett is universally preferable: it illustrates the gain available when the dependence model is credible. If the covariance or tail distribution is misspecified, the nominal cutoff may not give the stated FWER.
Worked Example: Closed Testing Protects a Set of Claims
Worked Example: Closed Testing Protects a Set of Claims
A hypothetical trial has three treatment hypotheses, and investigators want the option to make a claim for any treatment or subset of treatments. They choose closed testing because they need strong control across these claims and do not want to rely on a single all-arm maximum-statistic calibration. Suppose all eight intersection tests are valid at \(\alpha=0.025\) for the complete MAMS analysis, and their hypothetical p-values include \(p_{\{1\}}=0.009\), \(p_{\{1,2\}}=0.031\), and \(p_{\{1,2,3\}}=0.012\).
Although the singleton test for \(H_1\) has \(0.009<0.025\), closed testing does not reject \(H_1\). The required intersection \(H_{\{1,2\}}\) has \(0.031>0.025\), so it is not rejected. A rejection of \(H_1\) would require rejection of every intersection containing it, including \(H_{\{1,2\}}\) and \(H_{\{1,2,3\}}\). The latter has \(0.012<0.025\), but that does not compensate for the failed pairwise intersection test.
This example shows why the procedure can be more demanding than checking each treatment’s singleton test alone. Closed testing is attractive when the claims must remain valid under arbitrary patterns of treatment activity, but the chosen intersection tests still need to be appropriate and calibrated for the stages and dependence in the design.
Worked Example: Conditional Error After an Adaptation
Worked Example: Conditional Error After an Adaptation
Consider a hypothetical two-stage MAMS design with a prespecified level-\(0.025\) test for an intersection null. After observing stage-one data, the investigators propose dropping one treatment arm and reallocating future participants to the remaining arms. Under the intersection null and the observed interim data, suppose calculation under the original design gives a conditional error of \(0.080\). This conditional value can exceed \(0.025\); the level restriction applies unconditionally across interim outcomes, not separately to every possible interim result.
Suppose the revised stage-two rule is calibrated so that, conditional on these same interim data and under the intersection null, its probability of rejecting that intersection is \(0.060\). The revised rule respects the available conditional error at this interim result because \(0.060\leq0.080\). For validity, the analogous inequality must hold for every relevant intersection and for the interim data values where the adaptation is allowed. The revised conditional rejection probabilities must also be evaluated under the appropriate conditional null distribution.
The underlying argument is that the original design’s conditional errors average to no more than its unconditional type I error. If the revised design’s conditional rejection probability never exceeds the original conditional error, then averaging over interim data cannot increase the rejection probability beyond the original bound. Thus the adaptation can preserve strong control when this condition is enforced for all relevant intersection hypotheses. Checking only the complete-null intersection is not enough to establish strong control.
A Practical Selection Framework
The following sequence helps match the method to the design. It is a selection aid, not a substitute for calibrating the final rejection rule.
List the treatment hypotheses and decide whether the goal is any treatment claim, claims for specified groups, or a flexible set of treatment-specific claims. If claims must remain valid for arbitrary subsets, ensure the procedure provides strong control.
Record which statistics share a control group, which stages reuse accumulated data, and whether unequal allocation or other design features alter their joint distribution. Use a Dunnett-type method only when the relevant joint model can be justified and calibrated.
If the design may change in response to interim data, specify the permitted changes and how error control will be retained. Conditional-error methods can support adaptations, but the constraints must cover every intersection needed for strong control.
Use Bonferroni when dependence is uncertain and robustness is more important than efficiency. Use a joint-distribution approach when dependence is credible and its calibration covers the complete analysis. Use closed testing when the family of treatment claims needs explicit intersection protection.
Check the stated error target across stages, stopping rules, arm dropping, and relevant null configurations. A method name alone does not establish strong control.
| Design feature or goal | Strategy to consider | Key condition to check |
|---|---|---|
| Dependence is uncertain; a robust simple allocation is desired | Bonferroni, possibly within a broader closed procedure | Each component or sequential test is valid at its allocated level, and allocations sum to at most \(\alpha\) |
| Treatment statistics share control and their joint distribution is well supported | Dunnett-type calibration | The joint calibration covers all relevant arms and stages and yields strong control under the null patterns of interest |
| Flexible, treatment-specific claims must be protected | Closed testing | Every required intersection test is valid for the complete MAMS decision process |
| Interim-data-driven changes are allowed | Conditional-error constraints, often combined with closure | Revised conditional rejection probabilities do not exceed original conditional errors for every required intersection |
Diagnostics, Pitfalls, and Limitations
A strategy is going wrong when its stated assumptions do not match the implemented design. For a Dunnett-type method, inspect the covariance structure, stagewise dependence, allocation pattern, and any changes to the set of arms. If the cutoff is computed for a different joint distribution than the one generated by the trial, nominal familywise control is not guaranteed. For Bonferroni, check that the allocation covers both the arms and all relevant looks; a per-stage division that ignores repeated analyses can inflate the overall error. For closed testing, verify that every intersection needed for each treatment claim is actually tested. For conditional error, verify the constraint intersection by intersection, not only for the global null.
A common mistake is to select a method because it gives the smallest critical value in a simple calculation. The correction is to compare procedures only after they have been calibrated for the same estimand, family of claims, stages, stopping rules, and adaptation plan. A lower cutoff under one assumed dependence model is not a valid advantage if that model is unsupported. Likewise, Bonferroni’s validity under arbitrary dependence does not excuse an incorrect sequential allocation.
No method is cost-free. Bonferroni may lose power when tests are strongly dependent. Dunnett-type methods rely on distributional and covariance assumptions that may be difficult to defend under complex adaptation. Closed testing can require many intersection tests as the number of arms grows, and its protection depends on the validity of every required test. Conditional-error methods can accommodate certain design changes, but they do not license unrestricted adaptation: the revised procedure must respect the conditional error available for each relevant null.
Check Your Understanding
Answer each question, then compare with the solutions.
- Three tests each use level \(0.01\). What familywise upper bound does the Bonferroni inequality provide?
- In a hypothetical three-arm trial, the treatment statistics share a control and have a reliably specified joint distribution. Which approach can use this dependence, and what must its calibration cover in a MAMS design?
- Under closed testing, a singleton test for \(H_2\) rejects, but one intersection containing \(H_2\) does not. Can \(H_2\) be rejected?
- At an interim look, an original design has conditional error \(0.045\) for an intersection null. A revised rule has conditional rejection probability \(0.050\) for that null and interim data. Does it meet the conditional-error constraint?
- Why does checking FWER only when every treatment null is true fail to establish strong control?
Solutions. (1) The bound is \(3(0.01)=0.03\), so FWER is at most \(0.03\), assuming each test is valid at level \(0.01\). (2) Dunnett-type calibration can use the shared-control dependence; it must represent the relevant joint distribution across arms and stages, including the repeated-look decision rule. (3) No. Closed testing requires rejection of every intersection containing \(H_2\), so failure of any one required intersection prevents rejection. (4) No. The revised conditional rejection probability \(0.050\) exceeds the available conditional error \(0.045\). (5) Strong control requires the FWER bound under every configuration of true and false nulls; the complete-null configuration checks only one such configuration.