Why Order Treatment Tests?
A multi-arm trial may include treatments with different clinical roles. One treatment might be the preferred option for routine use, while another is a promising alternative or a treatment for a narrower group. If the confirmatory question is whether the preferred treatment works, it may be important to prevent a lower-priority claim from becoming confirmatory when that main clinical objective has not been met. Gatekeeping formalizes this rule: a higher-priority hypothesis must pass its test before testing can proceed to the next one.
The order must reflect clinical and regulatory priorities set before examining the trial results. It is not a ranking by observed treatment effect, p-value, or which arm looks most promising after data collection. This tutorial focuses on a fixed sequence of hypotheses, one of the clearest forms of gatekeeping. It uses the familywise significance level available to the confirmatory tests; any allocation or spending of alpha at interim analyses must already be reflected in that available level.
Specify the Hypotheses and the Gate
For example, let \(\Delta_i\) be the treatment effect for arm \(i\) relative to a common control, measured in a clinically interpretable unit such as points on a symptom scale. If a benefit of at least \(\delta_i\) points is required, the hypotheses for arm \(i\) can be written \(H_i:\Delta_i\leq\delta_i\) versus \(H_{ai}:\Delta_i>\delta_i\). The null includes effects that fail to reach the prespecified clinically important benefit. Other estimands and one-sided or two-sided hypotheses are possible; the test and its p-value must match the protocol’s stated hypothesis.
The design specifies (1) the family of treatment hypotheses, (2) their clinical priority order, (3) the significance level \(\alpha\) available to this testing procedure, and (4) a valid test for each hypothesis. A p-value is valid here if, whenever its null hypothesis is true, its probability of being at most \(u\) is no greater than \(u\) for every \(0\leq u\leq1\). This condition includes correct handling of the outcome model, randomization, and any interim looks. The procedure below does not make an invalid p-value valid.
With p-values \(p_1,\ldots,p_m\), the decision rule is simple:
- Reject \(H_1\) if \(p_1\leq\alpha\); otherwise stop.
- Only after rejecting \(H_1\), reject \(H_2\) if \(p_2\leq\alpha\); otherwise stop.
- Continue in the same way. A hypothesis is rejected only if its own p-value is at most \(\alpha\) and all hypotheses earlier in the sequence were rejected.
Equality is included: a p-value exactly equal to \(\alpha\) passes its gate under this convention. Testing a hypothesis can mean calculating its p-value for analysis, but the confirmatory decision is still gated: a lower-priority result cannot be declared a confirmatory rejection if an earlier gate has closed.
Worked Example: A Promising Lower-Priority Result Is Blocked
Suppose three experimental treatments have been ordered before unblinding: a preferred regimen, an alternative regimen, and a lower-priority regimen. The familywise level for this sequence is \(\alpha=0.025\). For illustration, assume the prespecified one-sided tests produce standard normal statistics \(z_1=2.20\), \(z_2=1.70\), and \(z_3=2.80\), with larger values favoring treatment.
For a one-sided standard normal test, \(p_i=P(Z\geq z_i)\), where \(Z\sim N(0,1)\). Thus the p-values are approximately \(p_1=0.0139\), \(p_2=0.0446\), and \(p_3=0.00256\).
First, \(0.0139\leq0.025\), so reject \(H_1\) and open the gate to \(H_2\). Next, \(0.0446>0.025\), so do not reject \(H_2\). The procedure stops. Although \(p_3=0.00256\) is small, \(H_3\) cannot be rejected confirmatorily because its gate was never opened. The conclusion is that the preferred regimen met its test criterion, but the alternative did not; the lower-priority regimen has no confirmatory rejection under this procedure.
This conclusion assumes the stated normal-reference tests produce valid p-values for the prespecified hypotheses. In an actual trial, the analysis plan must justify that model and account for any interim monitoring.
Why the Procedure Controls Familywise Error
The familywise error rate is the probability of rejecting at least one true null hypothesis in the family. Strong control means that this probability is at most \(\alpha\) for every configuration of true and false null hypotheses. Fixed-sequence testing achieves this without dividing \(\alpha\) among all the treatments.
To see why, consider any configuration in which at least one null hypothesis is true. Let \(k\) be the index of the first true null in the sequence. Every hypothesis before \(H_k\) is false. A false rejection anywhere in the sequence requires the procedure to reject \(H_k\): if \(H_k\) is not rejected, the gate closes and no later hypothesis can be rejected. Therefore,
The last inequality follows from p-value validity under the true null \(H_k\). If all hypotheses are false, there is no type I error to make. The argument does not use independence, so correlation among comparisons, including comparisons sharing a control group, does not invalidate this error bound. It does require that the ordered testing plan and the tests themselves are not changed in response to observed outcomes.
Worked Example: All Gates Open
A trial has three ordered treatment hypotheses and \(\alpha=0.025\). Suppose their valid one-sided p-values are \(0.0080\), \(0.0190\), and \(0.0210\).
Test \(H_1\): \(0.0080\leq0.025\), so reject it. Test \(H_2\): \(0.0190\leq0.025\), so reject it. Test \(H_3\): \(0.0210\leq0.025\), so reject it as well. Each hypothesis passes its own level-\(0.025\) test after all previous gates have opened.
The procedure rejects all three hypotheses, while the theorem still bounds the probability of one or more false rejections by \(0.025\) under any configuration of true nulls. This is a repeated use of the same threshold in a gated sequence, not three independent opportunities to make a type I error.
Choosing Priorities and Checking the Procedure
A clinically defensible order is part of the design, not a statistical result. It can reflect a primary treatment objective, the intended treatment strategy, or the consequences of making different claims. Document the rationale and the exact hypotheses before unblinding. If clinical priorities do not support a strict ranking, forcing an arbitrary order can make the procedure inefficient or poorly aligned with the intended claims.
Before interpreting results, check that:
- The family and priority order were specified before results were examined.
- Each p-value tests the stated hypothesis, with the correct direction and clinical margin.
- Any interim looks, alpha spending, or previous alpha allocation have been accounted for in the level available for the sequence.
- The conclusion for each treatment respects the gate: no lower-priority confirmatory rejection follows a failed earlier test.
- Any changes in the treatment set or the order are handled by a prespecified valid procedure, rather than improvised after seeing results.
A common failure is to select the most favorable arm after examining the data, place it first, and then apply the fixed-sequence rule as though the order had been prespecified. That is not the procedure justified here: the first true null in the proof is defined relative to a fixed sequence, and data-dependent ordering can alter the probability of reaching and rejecting true hypotheses. Another failure is to treat a nominal p-value from repeated interim looks as valid without an appropriate sequential test. In either case, the gatekeeping label alone does not establish error control.
Worked Example: A Fixed Sequence Versus Alpha Splitting
Suppose three prespecified hypotheses have p-values \(0.018\), \(0.021\), and \(0.023\), and the familywise level is \(\alpha=0.025\). Under fixed-sequence testing, \(0.018\leq0.025\), \(0.021\leq0.025\), and \(0.023\leq0.025\). All three gates open, so all three hypotheses are rejected.
For comparison, an equal Bonferroni allocation would test each hypothesis at \(\alpha/3=0.025/3=0.00833\). Since \(0.018>0.00833\), the first hypothesis would not be rejected by that rule, and neither would the others. These different decisions do not mean either rule is automatically wrong: fixed sequence uses the clinical priority structure and allows later tests the full level only after earlier success, while Bonferroni distributes protection across tests without requiring that success sequence.
The choice should follow the scientific claims the trial must support. Fixed sequence can be efficient when the ordered claims are genuinely hierarchical, but it can leave later treatments without confirmatory conclusions if an early, high-priority test fails.
How Gatekeeping Differs From Other Procedures
| Procedure | How testing proceeds | When it may fit |
|---|---|---|
| Fixed-sequence gatekeeping | Tests one hypothesis at a time at \(\alpha\); a failure stops the sequence. | Claims have a defensible, prespecified clinical priority order. |
| Bonferroni | Tests all hypotheses, commonly at \(\alpha/m\) for \(m\) tests. | Claims need not be ordered; a simple conservative adjustment is desired. |
| Holm step-down | Orders observed p-values and compares them with successively adjusted thresholds. | There is no required clinical testing sequence and several hypotheses may be tested through an adjusted step-down rule. |
| Alpha splitting | Assigns portions of the available alpha to different tests or groups in advance. | Different claims need simultaneous access to specified portions of the error budget. |
These methods answer different design needs. Gatekeeping is not a general substitute for multiplicity adjustment whenever many arms are present; its defining feature is that access to lower-priority tests depends on success at earlier gates. It can also be conservative for lower-priority claims in the practical sense that a failed early test prevents those claims from being confirmed, even if their own evidence is strong. It does not establish treatment effects for hypotheses that were not rejected, and it does not by itself provide selection-adjusted estimates or confidence intervals.
Check Your Understanding
Answer each question using the fixed-sequence rule. Solutions are included below the questions.
- At \(\alpha=0.05\), the ordered p-values are \(0.03\), \(0.051\), and \(0.001\). Which hypotheses are rejected?
- Why does a fixed-sequence procedure not require dividing \(\alpha\) by the number of hypotheses?
- What must be true about the order for the stated error-control argument to apply?
- At \(\alpha=0.025\), three ordered p-values are \(0.010\), \(0.020\), and \(0.030\). Which hypotheses are rejected?
- Does correlation between treatment-control test statistics alone invalidate the fixed-sequence error bound? Explain.
Solutions. (1) Reject \(H_1\), since \(0.03\leq0.05\). Do not reject \(H_2\), since \(0.051>0.05\), and stop; \(H_3\) is not a confirmatory rejection despite its p-value of \(0.001\). (2) Any false rejection requires rejection of the first true null in the fixed order, whose valid test has rejection probability at most \(\alpha\); the familywise error is therefore bounded by \(\alpha\). (3) The order must be specified independently of the test results, and the tests must yield valid p-values for their stated hypotheses. (4) Reject \(H_1\) and \(H_2\), because both p-values are at most \(0.025\); stop at \(H_3\), since \(0.030>0.025\). (5) No. The bound uses validity of the p-value for the first true null and the gate structure, not independence among test statistics.