Why Procedure Selection Starts With the Questions
A multi-arm trial can generate several treatment hypotheses, but the number of arms alone does not determine the right multiplicity procedure. The relevant questions are which claims the trial will make, which claims form one family, whether the tests are dependent, and whether some hypotheses have prespecified priority over others. A procedure that is valid for one arrangement may be unnecessarily conservative or invalid for another.
The introductory material on testing three treatments against control illustrates a particular family of comparisons. Here the emphasis is on choosing among procedures for a stated family, rather than selecting a procedure after seeing which results are most appealing.
Strong control matters because a trial may have no effective treatments, one effective treatment, or several. A procedure justified only when every null hypothesis is true does not automatically control the chance of a false rejection when some treatments truly differ from control. Before choosing a method, the family must also be fixed from the trial objectives and analysis plan, not narrowed or rearranged in response to observed results.
A Practical Selection Framework
Start with the planned claims, not with a menu of available procedures. The testing family might consist of all treatment-versus-control hypotheses, all pairwise treatment comparisons, or a prespecified set of scientifically meaningful contrasts. These are different families. Once the hypotheses and directions of testing are specified, consider whether the claims have a priority structure and what dependence follows from the design.
List the hypotheses whose rejections will support the claims being protected by one FWER requirement. Fix the family before examining the test results.
If the protocol says a later objective can be tested only after an earlier objective succeeds, consider a prespecified sequence or gatekeeping plan. If claims have equal standing, do not impose an artificial order based on their observed p-values.
Determine whether tests share a control group, share participants, or otherwise have correlated statistics. Do not assume independence merely because the hypotheses are distinct.
Use a procedure valid under the dependence and priority structure that actually applies. If a procedure requires assumptions that cannot be justified, choose a robust alternative.
Confirm that the method addresses all multiplicity relevant to the claims. Controlling across arms does not by itself handle repeated interim looks or additional families of endpoints.
What the Main Procedure Choices Require
For a family of \(m\) valid p-values, Bonferroni rejects \(H_{0i}\) when \(p_i\leq\alpha/m\). It controls the FWER strongly under arbitrary dependence, but can be conservative because it applies the same stringent cutoff to every hypothesis. Holm’s step-down procedure also controls strongly under arbitrary dependence and is never less powerful than the single-step Bonferroni procedure for the same family.
To apply Holm, order the p-values as \(p_{(1)}\leq\cdots\leq p_{(m)}\). Compare \(p_{(1)}\) with \(\alpha/m\), then \(p_{(2)}\) with \(\alpha/(m-1)\), continuing until a comparison fails. Reject hypotheses corresponding to the p-values passed before that failure; do not reject the failing hypothesis or any later one. Hochberg’s step-up method can be more powerful in suitable settings, but requires conditions such as independence or an applicable positive-dependence condition. The fact that a trial has multiple arms does not establish those conditions.
When every comparison is against a common control, Dunnett’s procedure can use the joint distribution of the treatment-control statistics. This often improves efficiency over a method that ignores their dependence. Its validity relies on an appropriate model for the statistics and their joint dependence; the analysis should account for such features as the endpoint distribution, variance assumptions, and degrees of freedom. A Šidák adjustment similarly gains over Bonferroni under independence, but should not be selected on that basis if independence is not justified.
A fixed-sequence procedure is appropriate when the order of hypotheses is specified in advance and later testing is allowed only after earlier hypotheses are rejected. It tests each hypothesis at level \(\alpha\), stopping at the first non-rejection. It is not a general substitute for a simultaneous procedure when all hypotheses have equal priority: it can leave a later, compelling result untested simply because an earlier test failed.
Worked Decisions for Different Trial Structures
Worked Example: Equal-Priority Hypotheses With Unknown Dependence
Suppose a hypothetical trial has three prespecified treatment hypotheses with two-sided p-values \(0.008\), \(0.021\), and \(0.047\). The trial will make any of these three claims if supported, and the dependence among the tests is not adequately characterized for a specialized procedure. Use \(\alpha=0.05\).
The hypotheses have equal priority, so a fixed sequence would not reflect the objective. Holm is a suitable choice because it gives strong FWER control under arbitrary dependence. The ordered values are already \(p_{(1)}=0.008\), \(p_{(2)}=0.021\), and \(p_{(3)}=0.047\).
At the first step, the cutoff is \(0.05/3=0.0167\); since \(0.008\leq0.0167\), reject the first null. At the second step, the cutoff is \(0.05/2=0.025\); since \(0.021\leq0.025\), reject the second. At the third step, the cutoff is \(0.05\); since \(0.047\leq0.05\), reject the third. Holm therefore rejects all three hypotheses while controlling the probability of one or more false rejections at no more than 0.05.
For comparison, single-step Bonferroni uses \(0.05/3=0.0167\) for every test and would reject only the hypothesis with p-value \(0.008\). The difference is not a contradiction: Holm uses the ordered evidence to relax later cutoffs while retaining strong control.
Worked Example: Shared-Control Comparisons
Consider a hypothetical, normally distributed continuous endpoint measured in milligrams per deciliter. A trial randomizes 80 participants to a control group and 80 participants to each of three treatment groups. Assume independent participants across groups, a common within-group variance, and a prespecified family of the three two-sided treatment-versus-control hypotheses. The shared control makes the contrast statistics dependent.
For two different treatment-control contrasts, the shared control contributes to both estimates. Under the stated equal-size, common-variance model, the correlation between any two contrast estimates is \( \sigma^2/80 \div [\sigma^2(1/80+1/80)]=0.5 \). Thus, treating these tests as independent would not match the design. Suppose a suitable joint multivariate-\(t\) calibration for the model and its degrees of freedom gives a two-sided Dunnett critical value of 2.36. The hypothetical observed absolute test statistics are 2.51, 2.20, and 1.82.
Compare each statistic with 2.36. The first exceeds the critical value, \(2.51>2.36\), so reject that treatment’s null hypothesis. The other two do not: \(2.20<2.36\) and \(1.82<2.36\). The Dunnett procedure therefore supports a treatment-control difference for the first treatment only, with strong FWER control for the three prespecified comparisons under the model used for calibration.
This choice uses the known shared-control dependence rather than ignoring it. If the outcome model or variance assumptions are implausible, the joint calibration may not be appropriate; a valid method robust to the actual dependence and model uncertainty should be considered instead.
Worked Example: A Prespecified Priority Order
Suppose a hypothetical trial specifies in its protocol that a primary treatment hypothesis must be tested first, a second treatment hypothesis may be tested only if the first succeeds, and a third may be tested only if the second succeeds. Their p-values are \(0.060\), \(0.001\), and \(0.003\), respectively, and the procedure uses \(\alpha=0.05\).
This is a priority structure, so a fixed-sequence procedure matches the planned claims. Test the first hypothesis at 0.05. Because \(0.060>0.05\), it is not rejected and the sequence stops. The second and third hypotheses are not rejected under this procedure, regardless of their smaller p-values. This preserves the prespecified rule that neither later objective can support a claim unless the earlier objective succeeds.
A different procedure might reject the second and third hypotheses, but that would answer a different testing plan. Procedure selection should reflect the trial’s planned claims, not be changed after noticing that later p-values are smaller.
Comparing Options and Checking for Failure
| Procedure | Useful when | Key condition or limitation |
|---|---|---|
| Bonferroni | A simple robust adjustment is needed for a fixed family. | Strong control under arbitrary dependence; may be conservative. |
| Holm | Equal-priority hypotheses need strong control without a dependence model. | Strong control under arbitrary dependence; requires valid p-values and a prespecified family. |
| Dunnett | The family consists of treatment-versus-common-control comparisons. | Requires an appropriate joint model for the contrast statistics. |
| Hochberg or Šidák | Conditions allowing their less conservative calibration are justified. | Do not use the independence or positive-dependence rationale without supporting assumptions. |
| Fixed sequence | Hypotheses have a genuine, prespecified priority order. | Later hypotheses are not tested after the first failure. |
Several checks help reveal a poor choice. If an analysis describes shared-control tests as independent without justification, its calibration may be wrong. If it uses Hochberg or Šidák solely because those methods appear more powerful, the required dependence condition may have been overlooked. If a hierarchy or sequence is created after seeing p-values, the testing plan no longer reflects prespecified priorities. Finally, an arm-level FWER procedure does not automatically control multiplicity from additional endpoints or repeated interim looks; those sources must be addressed in the broader analysis plan.
No procedure is uniformly best. Holm is a dependable default when the family is fixed but dependence is uncertain; a joint method may be more efficient when the design supplies a defensible dependence model; and a sequence is meaningful when priority is part of the scientific objective. The choice should be documented together with the family, error-rate target, assumptions, and rules for what conclusions the rejections permit.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A trial has four equal-priority hypotheses and no justified dependence model. Name a suitable procedure and state its dependence requirement.
- Why might Dunnett’s procedure be appropriate for several treatment-versus-control comparisons?
- A fixed sequence has p-values \(0.02\), \(0.07\), and \(0.001\), tested at \(\alpha=0.05\). Which hypotheses are rejected?
- Why is it not valid to select a procedure after seeing which method rejects the most hypotheses?
Solutions. (1) Holm is suitable and does not require independence; it provides strong control under arbitrary dependence, assuming valid p-values. (2) The comparisons share a control and so are dependent; Dunnett uses their joint distribution when that dependence model is appropriate. (3) Reject the first hypothesis, then stop at the second because \(0.07>0.05\); the third is not rejected under the sequence even though its p-value is \(0.001\). (4) Choosing after seeing results makes the analysis data-dependent and can invalidate the promised FWER control. The family and procedure should be set from the planned claims and justified assumptions before examining outcomes.