Why Order Matters in Sequential Testing
A MAMS trial may ask whether several treatments improve an outcome relative to a shared control. If the hypotheses have a prespecified priority order, a gatekeeping procedure can require stronger evidence for a lower-priority treatment: its hypothesis is tested only after the preceding hypothesis has been rejected. This can use the familywise error budget more efficiently than testing every treatment in parallel, but only if the order and the sequential testing rules are specified and followed.
Here the focus is a fixed sequence of treatment hypotheses examined at multiple stages. At each stage, the investigator tests in priority order and stops testing down the sequence at the first non-rejection. The boundary for each hypothesis must account for repeated looks over time. The gatekeeping rule then controls the probability of making one or more false treatment claims across both arms and stages.
For example, one-sided hypotheses might be \(H_j:\theta_j\leq 0\) versus \(H_{a,j}:\theta_j>0\), where \(\theta_j\) is the treatment effect for arm \(j\) relative to control, measured in the outcome's units. The procedure applies more generally, provided each hypothesis has a valid sequential test under its null.
Stagewise Boundaries and Error Budgets
Suppose there are \(K\) planned analyses and the desired overall FWER is \(\alpha\). Choose nonnegative stagewise error budgets \(\delta_1,\ldots,\delta_K\) such that their cumulative totals \( \gamma_k=\sum_{\ell=1}^{k}\delta_\ell \) satisfy \(\gamma_K\leq\alpha\). For each treatment hypothesis, choose stage boundaries so that under that hypothesis's null, the probability of a first boundary crossing at stage \(k\) is at most \(\delta_k\). Consequently, the probability of crossing by stage \(k\) is at most \(\gamma_k\).
A simple conservative construction uses a valid one-sided p-value \(p_{j,k}\) at each stage for each \(H_j\), and rejects \(H_j\) at stage \(k\) if \(p_{j,k}\leq\delta_k\). Each stagewise p-value must be valid under its null: for every \(u\) between 0 and 1, \(P(p_{j,k}\leq u\mid H_j)\leq u\). Because the test may be repeated, the probability of any crossing by stage \(k\) is bounded by the sum of its stagewise crossing probabilities. This construction does not require independence between stages. More efficient sequential boundaries can use the joint distribution of repeated statistics, but must still provide the stated cumulative bound.
For a one-sided standard normal statistic \(Z_{j,k}\), a conservative stagewise boundary is \(Z_{j,k}\geq c_k\), where \(c_k=\Phi^{-1}(1-\delta_k)\). Under the null, if \(Z_{j,k}\) has a standard normal distribution, its stagewise crossing probability is \(\delta_k\). This per-look construction can be conservative because it does not exploit the correlation between repeated statistics. In practice, a design may instead use boundaries calibrated to the sequential distribution, as long as the cumulative crossing bound is verified.
Worked Example: Setting Two-Stage Boundaries
Suppose three treatment hypotheses are tested in a fixed priority order at two planned looks. The target is one-sided \(\alpha=0.025\), and the design assigns \(\delta_1=0.010\) to the first look and \(\delta_2=0.015\) to the second. Assume that each stage's p-value is valid under its null.
The cumulative error budgets are \(\gamma_1=0.010\) and \(\gamma_2=0.010+0.015=0.025\). Using the conservative normal construction, the stage 1 boundary is \(c_1=\Phi^{-1}(0.990)=2.326\), and the stage 2 boundary is \(c_2=\Phi^{-1}(0.985)=2.170\). Under a true null, the probability of crossing at either look is at most \(0.010+0.015=0.025\).
Consider hypothetical stagewise p-values at look 1 of \(0.006\), \(0.012\), and \(0.004\) for \(H_1,H_2,H_3\), respectively. The procedure rejects \(H_1\), since \(0.006\leq0.010\). It does not reject \(H_2\), since \(0.012>0.010\), so it stops. It does not test \(H_3\), even though its p-value is small. At look 2, the sequence cannot resume for \(H_2\) or \(H_3\), because \(H_2\) was not rejected at the earlier look. This example treats rejection as absorbing: once a hypothesis fails its boundary at a planned look, later evidence cannot reverse that decision. A design that permits later testing after an interim non-rejection needs a separately specified sequential rule.
Why the Gatekeeping Rule Controls FWER
The central argument is that any false rejection must include a boundary crossing for the first true hypothesis in the priority order. Earlier hypotheses may be false and can be rejected without error. But the procedure cannot reach a later true hypothesis without rejecting that first true one. Thus the probability of a false rejection is bounded by the sequential type I error probability for one hypothesis, not by a sum over all treatment hypotheses.
Let \(r\) be the smallest index for which \(H_r\) is true in the configuration under consideration. If there are no true hypotheses, the probability of a false rejection is zero. Otherwise, any false rejection requires rejecting \(H_r\): if \(H_r\) is not rejected, gatekeeping prevents tests of all hypotheses with index greater than \(r\), and hypotheses before \(r\) are false. Therefore, by stage \(k\),
This is a strong-control argument: it holds for every pattern of true and false hypotheses. It does not assume that treatment statistics are independent. The key conditions are that the priority order is fixed in advance, testing follows the gatekeeping rule, and each true hypothesis has a valid sequential test with crossing probability at most \(\gamma_k\) by stage \(k\). In particular, the FWER is at most \(\gamma_k\) at each planned stage and at most \(\alpha\) at the final stage.
Worked Example: Checking FWER When the First True Hypothesis Is Second
Suppose \(H_1\) and \(H_2\) are tested in that order, \(H_1\) is false, and \(H_2\) is true. Use two looks with budgets \(\delta_1=0.010\) and \(\delta_2=0.015\). A false rejection can occur only if \(H_2\) is rejected, and \(H_2\) is reached only if \(H_1\) was rejected.
Let \(A\) be the event that the procedure falsely rejects \(H_2\) by look 2, and \(B\) the event that the sequential test for \(H_2\) crosses a boundary by look 2. Gatekeeping gives \(A\subseteq B\). The stagewise error budgets give \(P(B\mid H_2)\leq0.010+0.015=0.025\). Therefore, \(P(A)\leq P(B\mid H_2)\leq0.025\).
This bound remains valid whether the test statistics for \(H_1\) and \(H_2\) are positively correlated, negatively correlated, or otherwise dependent: restricting the event to cases where \(H_1\) was rejected cannot make it more likely than the event \(B\). The hypothesis \(H_1\) being false does not add a type I error contribution.
Applying the Procedure and Checking Its Operation
List the hypotheses in priority order and choose stagewise budgets whose cumulative total does not exceed the desired \(\alpha\).
For each hypothesis, define the statistics or p-values and the boundaries at each planned look. Verify that the probability of crossing by stage \(k\) under its null is at most \(\gamma_k\).
At each stage, test the highest-priority hypothesis not yet rejected. Stop when a hypothesis fails its boundary; do not test lower-priority hypotheses.
Report which hypotheses were reached and rejected, the boundaries used, and the cumulative FWER bound \(\gamma_k\) for the stage of each analysis.
Worked Example: Unequal Spending Across Looks
A two-look design again targets \(\alpha=0.025\), but assigns \(\delta_1=0.005\) and \(\delta_2=0.020\). Assume valid one-sided normal p-values at both looks. The cumulative budgets are \(\gamma_1=0.005\) and \(\gamma_2=0.025\); the conservative normal boundaries are \(c_1=\Phi^{-1}(0.995)=2.576\) and \(c_2=\Phi^{-1}(0.980)=2.054\).
For a hypothetical ordered pair \(H_1,H_2\), suppose the stage 1 p-value for \(H_1\) is \(0.004\). It passes the \(0.005\) threshold, so \(H_2\) may be tested at that look. Suppose \(H_2\)'s stage 1 p-value is \(0.006\): it does not pass, and testing stops. At stage 2, \(H_1\)'s p-value is \(0.018\), which is below \(0.020\), so it crosses its stage 2 boundary. This later crossing does not undo the earlier failure of \(H_2\); \(H_2\) and lower-priority hypotheses remain closed to testing under this rule.
Under any true hypothesis, the probability of crossing at the first look is at most \(0.005\), and the probability of crossing at the second look is at most \(0.020\). Their sum is \(0.025\), so the sequential test respects the final error budget. This allocation spends less error early and more later; it is a design choice, not a consequence of the priority order.
Alternatives, Diagnostics, and Limitations
Fixed-sequence gatekeeping differs from parallel testing with Bonferroni adjustment. A parallel procedure tests all hypotheses regardless of earlier results and typically divides or otherwise allocates the familywise budget across them. Fixed-sequence testing instead uses the priority order: a later hypothesis is inaccessible unless all earlier ones are rejected. It can offer more power for high-priority hypotheses, but a non-rejection early in the sequence blocks all lower-priority treatment claims, even if their effects appear promising.
It also differs from graphical alpha recycling, which can transfer testing weight among hypotheses according to specified transitions after rejections. In a strict fixed sequence, there is no transfer rule to calculate: the next hypothesis is tested only after the preceding one is rejected. Choosing between these strategies depends on whether the scientific priority order is compelling and whether flexibility after a non-rejection is important.
The main diagnostics are procedural and statistical. Check that the order was prespecified rather than chosen after viewing results; that each p-value or boundary is valid under the relevant null; that repeated looks are included in the error calculation; and that no lower-priority hypothesis was tested after a gate closed. If the analysis schedule, test statistic distribution, or stopping rules change, the original boundary justification may no longer apply. The simple per-look construction is valid but may be conservative, particularly when stage statistics are highly correlated. A calibrated sequential boundary can improve efficiency, but its crossing probability must be verified under the actual design assumptions.
A common mistake is to give each hypothesis a separate \(\alpha\) budget at every look and assume the gatekeeping order alone handles repeated testing. Gatekeeping protects across hypotheses; it does not by itself control repeated looks within a hypothesis. For instance, testing one true hypothesis at \(0.025\) at each of two looks can have a probability of at least one false rejection greater than \(0.025\). The correction is to use a valid sequential test for each hypothesis, such as a spending schedule with cumulative budget no greater than \(\alpha\), and then apply the fixed-sequence gatekeeping rule.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For three ordered hypotheses, the stage 1 budgets are \(0.01\) and \(0.015\) at stage 2. What is the cumulative FWER bound after each stage?
- If \(H_1\) is false and \(H_2\) is the first true hypothesis, which event must occur for any false rejection, and why?
- At stage 1, \(H_1\) is not rejected but the observed p-value for \(H_2\) is \(0.001\). May \(H_2\) be rejected by the fixed-sequence procedure?
- Why is it not enough to use a level-\(\alpha\) test for the same hypothesis at each of several looks?
Solutions. (1) The cumulative bounds are \(\gamma_1=0.01\) and \(\gamma_2=0.01+0.015=0.025\). (2) The procedure must reject \(H_2\), the first true hypothesis; a later true hypothesis cannot be reached without rejecting it. (3) No. Failure to reject \(H_1\) closes the gate, so \(H_2\) is not tested regardless of its p-value. (4) Repeated level-\(\alpha\) tests can have a probability of at least one false rejection greater than \(\alpha\); the within-hypothesis sequential test needs its own cumulative error control across looks.