Why Organize Hypotheses by Trial Objective?
A multi-arm trial can address several distinct objectives: establishing efficacy on a primary endpoint, evaluating an additional endpoint, or supporting a secondary clinical claim. Each objective may involve several hypotheses. Treating all these tests as one undifferentiated list can obscure which conclusions are confirmatory and which results are allowed to support later claims. A hierarchical plan makes those relationships explicit before outcomes are examined.
The key design decision is not merely how many tests there are. It is which claims belong together, and what evidence is required before a later objective may be tested confirmatorily. A family groups hypotheses whose rejections jointly contribute to a specified objective or set of claims. A hierarchy specifies which earlier testing outcomes permit testing a later family.
Family membership should follow the claims the trial intends to make, not simply the endpoint names or the order in which results appear in a report. For example, two dose comparisons on one primary endpoint may belong to the same primary family if either rejection supports the primary efficacy objective. A secondary endpoint may form a later family if its confirmatory interpretation is permitted only after the primary objective succeeds.
Specify the Hierarchy Before Testing
A usable plan identifies each family, its testing procedure, the significance level, and the event that opens each gate. In a simple serial plan, all hypotheses in the primary family must be rejected by the prespecified familywise procedure before the secondary family is tested confirmatorily. The secondary family is then tested at the trial’s nominal level \(\alpha\), using its own prespecified familywise procedure.
The requirement to reject all primary hypotheses is one possible gate rule, not a universal definition of primary success. If the objective instead succeeds when at least one primary hypothesis is rejected, or when a particular subset is rejected, that rule must be stated explicitly and the resulting error-control procedure must be justified. A gate cannot be chosen after seeing which rule gives the more favorable result.
For example, a Bonferroni procedure for a family of \(m\) hypotheses rejects a null hypothesis when its valid p-value is at most \(\alpha/m\). Holm’s procedure orders the p-values from smallest to largest and compares them successively with \(\alpha/m,\alpha/(m-1),\ldots,\alpha\), stopping at the first failed comparison. Both procedures control the familywise error rate under arbitrary dependence among valid p-values. The hierarchy determines when a family is tested; the chosen procedure determines how its hypotheses are tested.
Worked Example: Defining Primary and Secondary Families
Consider a hypothetical three-arm trial of a new treatment at two doses and a control. The protocol names improvement on a prespecified symptom score at week 12 as the primary objective, with a confirmatory claim for each dose comparison against control. It names a quality-of-life endpoint as secondary. State an appropriate family structure and gate.
Let \(H_{P1}\) and \(H_{P2}\) be the null hypotheses of no treatment effect versus control on the primary endpoint for the two doses. These form the primary family \(F_P=\{H_{P1},H_{P2}\}\), because either rejection would support a primary efficacy claim. Let \(H_{S1}\) and \(H_{S2}\) be the corresponding null hypotheses for quality of life; these form the secondary family \(F_S=\{H_{S1},H_{S2}\}\).
Suppose the protocol specifies Bonferroni testing at \(\alpha=0.05\) in \(F_P\), and requires rejection of both primary hypotheses to open the secondary gate. The primary cutoff is \(0.05/2=0.025\). This structure makes clear both the claims in each family and the evidence needed before the secondary analyses can support confirmatory conclusions. If the gate is not met, secondary results may still be summarized descriptively, but they do not become confirmatory results under this plan.
Why Serial Gatekeeping Controls Familywise Error
The error-control argument for a simple serial hierarchy is a case analysis. Assume that every p-value is valid under its null hypothesis, each within-family procedure strongly controls familywise error at level \(\alpha\), and the family membership and gate rule are fixed in advance. Strong control means control at level \(\alpha\) for every configuration of true and false null hypotheses in that family. No independence assumption between families is required for the argument below.
If any later family is tested confirmatorily, the gate requires rejection of every primary hypothesis. Therefore at least one true primary null must have been rejected. Any erroneous rejection anywhere in the hierarchy is then contained in the event of a primary-family error, whose probability is at most \(\alpha\).
There is no primary-family type I error. Any confirmatory error must occur in the secondary family. Its procedure strongly controls familywise error at level \(\alpha\); restricting that procedure to cases when the gate opens cannot increase the probability of a false rejection beyond the probability of a false rejection when the procedure is considered without that restriction.
In either case, the probability of one or more false rejections across the hierarchy is at most \(\alpha\). The same reasoning extends down a serial chain: at the first family containing a true null, any erroneous rejection in a later family requires an erroneous rejection in that family to pass its gate.
This justification depends on the specified gate. For instance, if a later family can be opened after rejecting only one of several primary hypotheses, the first case no longer guarantees that passing the gate entails a primary-family error when a true primary null remains. A more general gate may still be used, but it needs a procedure with a separate error-control justification; it does not inherit the simple all-primary-rejected argument automatically.
Worked Example: Applying a Two-Stage Gate
Suppose the two primary p-values are 0.018 and 0.021. The protocol uses Bonferroni at \(\alpha=0.05\) for the two primary hypotheses and requires both to be rejected before testing the secondary family. The secondary family has three p-values: 0.006, 0.027, and 0.041, and uses Holm’s procedure at 0.05.
The primary Bonferroni cutoff is \(0.05/2=0.025\). Since \(0.018<0.025\) and \(0.021<0.025\), both primary null hypotheses are rejected and the secondary gate opens.
Order the secondary p-values: \(0.006\leq0.027\leq0.041\). Holm’s first comparison is \(0.006\leq0.05/3\approx0.0167\), so the first secondary null is rejected. The second comparison is \(0.027\leq0.05/2=0.025\), which is false. Holm’s procedure stops, so the second and third secondary nulls are not rejected. The plan therefore supports both primary rejections and one secondary rejection; it does not support confirmatory rejection of the other two secondary hypotheses.
Worked Example: A Failed Gate Stops Confirmatory Testing
In a different hypothetical trial, the primary family again contains two hypotheses, tested by Bonferroni at \(\alpha=0.05\), with both rejections required to open the secondary gate. The primary p-values are 0.011 and 0.034. A secondary p-value is 0.001.
The primary cutoff is \(0.05/2=0.025\). The first primary p-value passes because \(0.011<0.025\), but the second does not because \(0.034>0.025\). Thus the gate fails. The secondary p-value of 0.001 does not change that outcome: under this plan it cannot produce a confirmatory secondary rejection. Reporting it as confirmatory would contradict the prespecified hierarchy, even though its numerical value is small.
Diagnostics, Alternatives, and Limitations
Before analysis, check that the plan can be followed without interpreting the results. Every hypothesis should have a family assignment; each confirmatory objective should have a clear success rule; and the rule for opening each later family should be unambiguous. During analysis, record the familywise procedure and gate decision at each stage. These checks help detect common failures, such as silently dropping an inconvenient hypothesis, treating an unopened family as confirmatory, or using different gate criteria in the report than in the protocol.
A serial all-rejections gate is transparent and can reserve confirmatory claims for later objectives until the primary objective is established. Its cost is that failure to reject even one required primary hypothesis blocks the next family, including when the remaining primary tests have weak evidence of their own. That may be appropriate when the secondary claim is scientifically meaningful only after the full primary objective succeeds; it may be unnecessarily restrictive otherwise.
An alternative is to place all claims in one family and apply a single familywise procedure, such as Holm’s method. This can allow testing claims without a gate, but it does not encode the same priority between objectives. Other gatekeeping or graphical procedures can permit partial progress or transfer testing weight, but their control depends on their specific rules. The earlier material on gatekeeping and graphical procedures provides those frameworks; a hierarchy across trial objectives still requires the protocol to define what each family represents and which claims each rejection supports.
A further limitation is that familywise control does not guarantee adequate power for every objective. A plan can control the probability of any false rejection and still have a low probability of establishing the intended claims. Objectives, family membership, and gates should therefore be chosen alongside sample-size planning, not treated as an analysis-stage repair. Exploratory analyses outside the confirmatory hierarchy may be useful, but they should be identified as exploratory rather than presented as if they passed a gate.
Check Your Understanding
Answer each question using the stated hierarchy. Short solutions follow.
- A primary family has three hypotheses tested by Bonferroni at \(\alpha=0.05\). What is its cutoff, and which p-values, 0.012, 0.016, and 0.018, pass an all-rejections gate?
- A secondary family is tested only after all primary nulls are rejected. If at least one primary null is true, why does an erroneous secondary rejection require a primary-family error?
- A protocol says the secondary gate opens if “the primary objective succeeds,” but does not define success. What should be clarified before testing?
- All primary nulls are false, the gate opens, and the secondary procedure strongly controls its familywise error at 0.05. What bound does the serial plan provide for the probability of any false rejection across the hierarchy?
Solutions. (1) The cutoff is \(0.05/3\approx0.0167\). The p-values 0.012 and 0.016 pass, but 0.018 does not, so the all-rejections gate fails. (2) The gate requires rejection of every primary hypothesis, including at least one true primary null. Passing it therefore entails a false primary rejection, and any secondary error after passage is contained in that primary-family error event. (3) The protocol must state the precise rejection event, such as whether all primary hypotheses or a specified subset must be rejected; the rule must be fixed before results are examined. (4) There are no primary type I errors because all primary nulls are false. Any false rejection is in the secondary family, so its probability is at most 0.05, giving strong familywise control at the stated level.