Why Order Matters
A multi-arm trial may have several treatment-versus-control hypotheses, but the scientific objectives do not always have equal priority. A trial might specify that evidence for one treatment is essential, while a second and third treatment are considered only if the higher-priority claims succeed. Fixed-sequence testing gives this priority order a precise inferential consequence: test the first hypothesis, proceed only if it is rejected, and stop at the first non-rejection.
This is a multiplicity procedure for a prespecified ordered family of hypotheses. It is not a rule for sorting the observed p-values. The order must come from the trial objectives or another defensible plan established independently of the results.
A valid p-value has the property that, whenever its null hypothesis is true, \(P(p_i\leq u)\leq u\) for every \(u\) between 0 and 1. The tests may be dependent. The familywise error rate (FWER) is the probability of rejecting at least one true null hypothesis in the family. Strong FWER control means that this probability is at most \(\alpha\) for every configuration of true and false null hypotheses.
Applying the Rule
The same threshold is used at every position in the sequence. There is no division of \(\alpha\) by the number of hypotheses. The protection comes from requiring every preceding hypothesis to be rejected before a later one can be rejected.
Specify the hypotheses and their priority before examining the trial results.
Reject \(H_1\) if \(p_1\leq\alpha\); otherwise stop, with no rejections in the sequence.
After rejecting \(H_i\), test \(H_{i+1}\) at \(\alpha\). Stop at the first p-value greater than \(\alpha\), or finish if all are rejected.
Worked Example: Three Consecutive Rejections
Suppose a hypothetical trial prespecifies the order \(H_1,H_2,H_3,H_4\), with \(\alpha=0.05\). Its valid p-values, in that order, are \(0.008\), \(0.031\), \(0.049\), and \(0.12\). Apply the fixed-sequence rule.
For \(H_1\), \(0.008\leq0.05\), so reject \(H_1\) and proceed. For \(H_2\), \(0.031\leq0.05\), so reject \(H_2\) and proceed. For \(H_3\), \(0.049\leq0.05\), so reject \(H_3\) and proceed to \(H_4\). Since \(0.12>0.05\), do not reject \(H_4\) and stop.
The procedure rejects the first three hypotheses, not just the first two: a p-value equal to or below the threshold passes its test, including \(0.049\). The conclusion concerns the stated hypotheses under their respective tests; it does not mean that all treatments have identical effects or that a non-rejected null has been proved true.
Why the Procedure Controls FWER
The key argument is about the first true null in the prespecified order. If there are no true null hypotheses, the probability of rejecting any true null is zero. Otherwise, let \(k\) be the smallest index for which \(H_k\) is true. A true null can be rejected only if the procedure reaches and rejects \(H_k\): later hypotheses cannot be reached unless \(H_k\) itself has been rejected. Thus the event of at least one false rejection is contained in the event \(p_k\leq\alpha\).
To see the bound explicitly, when \(H_k\) is the first true null, any rejection of a true hypothesis requires rejecting \(H_k\). The probability of that event is bounded by the marginal validity of \(p_k\):
The first inequality holds because rejecting any true null requires passing the test at position \(k\). The second follows because \(H_k\) is true and \(p_k\) is valid. No independence assumption appears in the argument. The procedure may be conservative: it can fail to reject a later true null even when its own p-value is small, because an earlier test stopped the sequence.
Worked Example: The First True Null Determines the Error Bound
Consider three prespecified hypotheses with \(\alpha=0.05\). Suppose \(H_1\) is false, \(H_2\) is true, and \(H_3\) is false. The p-values may be dependent. A familywise error occurs only if the procedure rejects the true \(H_2\).
To reach \(H_2\), the procedure must first reject \(H_1\). It then rejects \(H_2\) only if \(p_2\leq0.05\). Therefore, \[ P(\text{familywise error})=P(p_1\leq0.05,\ p_2\leq0.05) \leq P(p_2\leq0.05)\leq0.05. \] The first inequality uses the fact that the joint event is a subset of \(p_2\leq0.05\); the last uses validity under the true null \(H_2\). This reasoning does not require knowing the dependence between \(p_1\) and \(p_2\).
A Treatment-Arm Example With Calculated P-Values
The multiplicity rule is applied to p-values from the chosen treatment comparisons. The comparisons still need appropriate statistical models, standard errors, and test assumptions. Fixed-sequence testing does not repair invalid individual tests.
Worked Example: Ordered Comparisons With a Normal Approximation
Suppose a hypothetical three-arm trial compares each treatment with a shared control on a continuous outcome measured in points. Positive differences favour treatment. Before unblinding, the investigators order the two-sided hypotheses as \(H_1\): Treatment A versus control, then \(H_2\): Treatment B versus control. Assume the estimated differences are approximately normally distributed under their null hypotheses, and the reported standard errors are appropriate. The resulting test statistics and p-values use the standard normal reference distribution.
For Treatment A, the estimated difference is \(4.2\) points with \(SE=1.8\) points. Testing a zero difference gives \[ z_1=\frac{4.2-0}{1.8}=2.333. \] The two-sided normal p-value is approximately \(p_1=2\{1-\Phi(2.333)\}=0.0196\). Since \(0.0196\leq0.05\), reject \(H_1\) and proceed.
For Treatment B, the estimated difference is \(3.1\) points with \(SE=1.5\) points: \[ z_2=\frac{3.1-0}{1.5}=2.067. \] The two-sided p-value is approximately \(p_2=2\{1-\Phi(2.067)\}=0.0387\). Because \(0.0387\leq0.05\), reject \(H_2\) as well. The fixed-sequence procedure therefore rejects both hypotheses at familywise level \(0.05\), conditional on the stated model and valid p-values. The estimates are in outcome points; the z-statistics and p-values are unitless.
Stopping, Diagnostics, and Limitations
If the first p-value exceeds \(\alpha\), the procedure stops immediately, even if later p-values are very small. Those later results do not become formal rejections under this testing plan. Non-rejection is not proof of no treatment effect; it means the evidence did not satisfy the prespecified decision rule.
Worked Example: A Small Later P-Value Does Not Restart the Sequence
A hypothetical trial orders \(H_1,H_2,H_3\) and uses \(\alpha=0.05\). The p-values are \(0.064\), \(0.001\), and \(0.020\). For \(H_1\), \(0.064>0.05\), so do not reject \(H_1\) and stop. The procedure makes no rejection of \(H_2\) or \(H_3\), even though their reported p-values are below \(0.05\). Reporting those values descriptively may be useful, but they are not rejections made by this fixed-sequence procedure.
The validity and interpretation of a fixed sequence depend on several conditions and choices:
- Prespecified order: the sequence should reflect scientific or clinical priority, not a ranking chosen after looking at the p-values. If the order is selected to put the smallest observed p-value first, the fixed-order error argument no longer applies as stated.
- Valid component tests: each p-value must be calibrated under its own null, accounting for the outcome, design, and analysis assumptions. Shared controls may make tests dependent, but dependence itself does not invalidate the FWER proof.
- Defined family: the hypotheses included in the family and their order need to be clear. Changing the family or order after seeing results changes the testing plan.
- Limited reach for later claims: an early non-rejection blocks all downstream formal rejections. This is the price of giving the earliest hypothesis priority.
A useful diagnostic is to trace each decision from the first test onward and verify that no later hypothesis was rejected after an earlier p-value exceeded \(\alpha\). Also check that the order in the analysis matches the documented order and that the p-values correspond to the intended hypotheses and valid tests.
Fixed-sequence testing differs from Holm’s step-down procedure. Holm orders hypotheses by their observed p-values and compares them with progressively adjusted thresholds; it does not require a scientific priority order. Fixed-sequence testing keeps the prespecified hypothesis order and uses the same threshold at each position, stopping at the first failure. Holm is generally preferable when all hypotheses have comparable priority and the order should not determine which claims can be made. Fixed-sequence testing can be attractive when the ordering has a clear substantive rationale, but it may leave strong later results unclaimed after an earlier non-rejection. Neither procedure makes a non-rejection equivalent to evidence that a null is true.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- At \(\alpha=0.05\), the ordered p-values are \(0.02, 0.05, 0.004\). Which hypotheses are rejected?
- Why does fixed-sequence testing control FWER when the first true null is \(H_k\)? State the event inclusion used in the argument.
- At \(\alpha=0.05\), the first p-value is \(0.08\) and the second is \(0.001\). What does the procedure conclude about each hypothesis?
- Does the FWER proof require independent p-values? Explain briefly.
- Why is it problematic to select the hypothesis order after inspecting the p-values?
Solutions. (1) Reject \(H_1\), because \(0.02\leq0.05\), and then reject \(H_2\), because equality passes the rule: \(0.05\leq0.05\). Proceed to \(H_3\); since \(0.004\leq0.05\), reject \(H_3\) as well. (2) If \(H_k\) is the first true null, any rejection of a true null requires reaching and rejecting \(H_k\). Thus the familywise-error event is contained in \(\{p_k\leq\alpha\}\), whose probability is at most \(\alpha\) by validity of \(p_k\). (3) Do not reject \(H_1\), since \(0.08>0.05\), and stop. The procedure makes no rejection of \(H_2\); its small p-value does not override the stopping rule. (4) No. The argument bounds the error event by the marginal event \(\{p_k\leq\alpha\}\), so arbitrary dependence is allowed. (5) Data-dependent ordering can favour small p-values and invalidates the fixed, prespecified-order argument unless a suitable procedure accounts for that selection.