Why Put Treatment Hypotheses in a Clinical Order?
A multiarm trial may compare several treatments with a shared control, yet the clinical value of the possible conclusions need not be equal. A treatment that could materially change practice may be more important to establish than a treatment expected to offer a modest improvement. A prespecified hierarchy lets the trial test the most important claim first and makes later confirmatory claims conditional on success at earlier steps.
This is a testing order, not an estimate of which treatment works best. It should be justified by clinical considerations before outcomes are examined. If investigators instead order treatments after seeing which had the most favorable results, the resulting p-values cannot automatically be treated as if the order had been fixed in advance.
Defining the Hierarchy
Suppose a trial compares \(m\) experimental treatments with a common control, and a larger value of the endpoint is better. Let \(\theta_j\) denote the treatment-minus-control effect for treatment \(j\), under the prespecified estimand and population. For a superiority claim, define \(H_j\) as the null hypothesis that treatment \(j\) does not improve the endpoint, and \(H_{a,j}\) as the alternative that it does. The priority order is written \(H_1,H_2,\ldots,H_m\), from highest to lowest clinical priority.
For superiority, the hypotheses can be written \(H_j:\theta_j\leq 0\) versus \(H_{a,j}:\theta_j>0\). A clinically relevant margin can replace zero when the desired claim is that the improvement exceeds a prespecified threshold. The direction, endpoint, estimand, analysis method, order, significance level, and stopping rule should all be specified before the trial results are examined.
The hierarchy answers a precise question for each treatment: are the gatekeeping conditions satisfied so that this hypothesis may be tested confirmatorily? It does not say that a later treatment is ineffective when its test is blocked. It says that the trial's allocated confirmatory evidence does not extend to that hypothesis under this testing plan.
Testing Rule and Its Justification
Let \(p_j\) be a valid p-value for \(H_j\): under \(H_j\), its probability of being at most \(u\) is no greater than \(u\), for any \(u\) between zero and one. The fixed-sequence rule is:
Thus, for the first treatment, the condition is simply \(p_1\leq\alpha\). For the second, both \(p_1\leq\alpha\) and \(p_2\leq\alpha\) are required. For the \(j\)th treatment, all \(j\) p-values through \(p_j\) must be at most \(\alpha\). A p-value for a later hypothesis may be calculated for exploratory purposes even when its gate is closed, but it does not then support a confirmatory rejection under this hierarchy.
The key justification is short. Consider the first true null hypothesis in the prespecified order, say \(H_k\). All hypotheses before \(H_k\) are false. To reject any true null later in the sequence, the procedure must first reject \(H_k\). To reject \(H_k\) itself, its valid p-value must be at most \(\alpha\). Therefore, the event of rejecting at least one true null is contained in the event \(p_k\leq\alpha\), whose probability is at most \(\alpha\). This explains why the procedure controls the familywise error rate, the probability of at least one false rejection, at level \(\alpha\), without requiring independence among the treatment statistics.
This justification depends on the stated testing rule and on valid local tests. In particular, every p-value must be valid for its own null under the actual design and analysis, including any relevant selection or reuse of data. The sequence does not repair an invalid p-value.
Worked Example: Applying a Three-Treatment Gate
Suppose three treatments were ranked clinically before enrollment, in the order A, B, C. The trial uses a one-sided significance level \(\alpha=0.025\), and valid p-values for the three superiority hypotheses are 0.012, 0.031, and 0.004. Which hypotheses may be rejected confirmatorily?
For A, compare \(p_1=0.012\) with \(\alpha=0.025\). Since \(0.012<0.025\), reject \(H_1\), so the gate for B opens.
For B, compare \(p_2=0.031\) with \(0.025\). Since \(0.031>0.025\), do not reject \(H_2\). The sequence stops here. Although C has \(p_3=0.004\), that result cannot be used for a confirmatory rejection because \(H_2\) was not rejected.
The confirmatory conclusion is evidence of superiority for A, but not for B or C under this procedure. The small p-value for C may be described as exploratory if appropriate, but it does not bypass the prespecified gate.
Worked Example: Calculating the First Local Test
In a hypothetical trial, the highest-priority treatment has an estimated improvement of 5.0 points over control on a scale where higher scores are better. Suppose the standard error is 1.8 points. Use a one-sided normal test of \(H_1:\theta_1\leq0\), with \(\alpha=0.025\), and assume the prespecified large-sample model makes the test statistic approximately standard normal at the null boundary.
The test statistic is the estimated effect divided by its standard error:
The one-sided p-value is \(P(Z\geq2.778)\), where \(Z\) has a standard normal distribution. This is approximately 0.0027. Because \(0.0027<0.025\), reject \(H_1\). The units cancel in the statistic, as required. This rejection opens the gate for the next hypothesis; it does not by itself determine whether that next hypothesis will pass its own test.
The calculation relies on the stated normal approximation and on the validity of the estimated standard error for the treatment-control contrast. If the trial's planned analysis instead uses a different model or reference distribution, its corresponding valid p-value must be used in the hierarchy.
What the Rule Means for Clinical Claims
A fixed sequence can be attractive when the trial has a clear clinical priority order and a limited set of claims that should be supported in that order. It spends the available type I error opportunity efficiently on the first claim and allows each subsequent claim at the same nominal level after prior success. It does not divide \(\alpha\) across the hypotheses as a Bonferroni procedure would.
The price is that a failed early test blocks every later confirmatory test, even if a later treatment's observed result is more compelling. That is a consequence of the claim hierarchy, not a conclusion that later treatments are clinically inferior. When no defensible clinical ranking exists, or when investigators need the option to claim success for any treatment that works, a procedure designed for simultaneous or multiple claims may be more appropriate. Procedures such as Dunnett testing for treatment-control comparisons or Holm testing address a different goal: they do not impose this same all-predecessors-must-pass gate.
The clinical order must be meaningful and fixed independently of the results being tested. It can reflect considerations such as unmet need, safety profile, feasibility, or the importance of a particular treatment strategy. It must not be silently changed because one treatment has the smallest observed p-value. If an adaptive selection rule determines which arms or hypotheses proceed, the analysis needs to account for that selection; a clinically ranked fixed sequence alone does not make outcome-driven selection harmless.
Worked Example: Clinical Priority Is Not Observed Ranking
A hypothetical study prespecifies treatment A as the top priority because its intended clinical use addresses the greatest unmet need, followed by B and then C. The valid p-values are 0.018 for A, 0.006 for B, and 0.040 for C, with \(\alpha=0.025\).
The first test passes because \(0.018<0.025\), so A is rejected and B may be tested. B also passes because \(0.006<0.025\), so C may be tested. C fails because \(0.040>0.025\). The confirmatory claims are superiority for A and B, but not C.
It would be incorrect to reorder the hypotheses after seeing the p-values so that B is first. The analysis follows the clinical order fixed in advance; B's smaller p-value does not change the gatekeeping rule. The result also does not establish that B has a larger true effect than A: comparing effect sizes requires an appropriate contrast and uncertainty assessment, not a comparison of p-values.
Operational Conditions and Failure Modes
A usable hierarchy requires more than an ordered list. The protocol and analysis plan should make clear what each hypothesis concerns and what event opens or closes each gate. In particular:
- Define the claims. Specify each treatment comparison, endpoint, estimand, analysis population, direction, and any clinical margin.
- Fix the order in advance. Explain the clinical basis for the ranking and do not change it after examining efficacy results.
- Use valid local tests. The statistical test for each treatment must account for features of the design that affect its null distribution, including any data reuse or selection that applies.
- Make the stopping rule explicit. If a hypothesis fails, later hypotheses in the fixed sequence are not confirmatorily rejected under this plan.
- Handle unavailable arms explicitly. If an arm is dropped or its endpoint cannot be analyzed, the plan must state how that affects the sequence. It is not automatically valid to skip a failed or unavailable gate and resume testing later.
A practical diagnostic is to trace the decision path from the first hypothesis to each proposed claim. If a report calls a later treatment statistically significant after an earlier hypothesis failed, check whether a separate prespecified procedure supports that claim. Also inspect whether the ranking, endpoint, or analysis changed after results were observed. Such changes may be scientifically explainable, but they do not preserve the original confirmatory interpretation automatically.
A frequent mistake is to say, “C has \(p=0.004\), so C is significant at 0.025,” while ignoring that B failed its gate. The correction is: C's p-value is below the nominal threshold, but the fixed-sequence condition for testing C was not met, so C cannot be rejected confirmatorily through this hierarchy. Another mistake is to assume the fixed sequence is valid regardless of how its p-values were generated. The gatekeeping rule controls multiplicity only when the local tests themselves are valid under the design.
Check Your Understanding
For each question, apply the gate rule using the stated order and significance level. Solutions follow the questions.
- For order A, B, C and \(\alpha=0.05\), the p-values are 0.03, 0.08, and 0.01. Which hypotheses may be rejected?
- For order A, B, C and \(\alpha=0.025\), the p-values are 0.004, 0.020, and 0.024. Which hypotheses may be rejected?
- Why does the fixed-sequence procedure not require independence among the treatment statistics for its familywise error argument?
- A team ranks treatments after inspecting their p-values, then applies a fixed-sequence rule to that new order. What is the problem?
Solutions. (1) Reject A because \(0.03<0.05\). B fails because \(0.08>0.05\), so testing stops; C is not confirmatorily rejected despite \(0.01<0.05\). (2) Reject A, then B, then C: each p-value is at most 0.025, so every gate opens. (3) The error event is contained in rejection of the first true null in the fixed order, whose valid local test has rejection probability at most \(\alpha\). This containment argument does not use independence. (4) The order is now data-dependent, so the procedure is not the prespecified fixed sequence used in the justification. The apparent p-values and claims need a method that accounts for the selection or a genuinely prespecified order.