Why Access to Secondary Tests May Need a Gate
A multi-arm trial may have a small set of primary hypotheses that determine its main confirmatory conclusions, alongside secondary hypotheses about additional outcomes or objectives. If investigators test every hypothesis at the same nominal level and then emphasize whichever results are favorable, the overall chance of at least one false rejection can exceed the intended level. A gatekeeping rule addresses this by making access to secondary testing depend on prespecified results in the primary family.
This tutorial considers a simple serial rule: test the primary family first, and allow confirmatory testing of the secondary family only if every specified primary hypothesis is rejected. If the gate opens, test the secondary family using a procedure that controls its familywise error rate (FWER) at the same level. The gate is a decision rule for access; it does not make an unrejected secondary result more favorable or change its p-value.
Strong FWER control means that, for every configuration of true and false null hypotheses in a family, the probability of rejecting at least one true null in that family is no greater than \(\alpha\). The component procedures may be Bonferroni, Holm, or another method with the required strong control. The p-values must be valid under their corresponding null hypotheses, the families and gate must be specified before examining results, and the component procedures must actually control strong FWER. No independence assumption between primary and secondary test statistics is needed for the rule and proof below.
In this basic rule, the primary family is the complete family of hypotheses that must all be rejected to open the gate. If other primary hypotheses are tested but are omitted from that gate, the simple argument below does not automatically control FWER across all primary and secondary claims. Those additional hypotheses require an explicit allocation or a broader error-control design.
How the Rule Controls Familywise Error
The key fact is that a false secondary rejection can occur only after the gate opens. When at least one primary null is true, opening the gate necessarily includes rejecting that true primary null. Thus, in this case, any false secondary rejection is contained within the event of a primary-family error. When all primary nulls are false, the primary family contributes no false rejection, and the secondary procedure’s own strong FWER control bounds false secondary rejections.
To verify the claim, consider any fixed configuration of true and false nulls. Write \(E_P\) for the event that at least one true primary null is rejected, and \(E_S\) for the event that at least one true secondary null is rejected. The overall familywise error event is \(E_P\cup E_S\).
If at least one primary null is true, then \(E_S\) can occur only if the gate opens. Opening the gate requires rejection of every primary hypothesis, including at least one true primary null. Therefore \(E_S\subseteq E_P\), and \(E_P\cup E_S=E_P\). The primary procedure’s strong control gives \(P(E_P)\leq\alpha\).
If all primary nulls are false, then \(E_P\) is impossible. A false secondary rejection under the gatekeeping rule is also a false rejection by the secondary procedure that would be applied to the secondary family. The event of such a rejection is contained in that procedure’s rejection event, so its probability is at most \(\alpha\) by strong control. These cases cover every configuration, establishing strong FWER control. This argument uses event containment, not an assumption that primary and secondary p-values are independent.
Worked Example: A Primary Hypothesis Fails the Gate
In a hypothetical trial, investigators prespecify two primary treatment-versus-control hypotheses and three secondary outcome hypotheses. They use Bonferroni testing at \(\alpha=0.05\) within each family. The primary p-values are \(0.018\) and \(0.029\); the secondary p-values are \(0.001\), \(0.004\), and \(0.032\). Determine which hypotheses can be rejected confirmatorily.
For two primary hypotheses, the Bonferroni cutoff is \(0.05/2=0.025\). The first primary p-value satisfies \(0.018\leq0.025\), so its null is rejected. The second does not: \(0.029>0.025\). Therefore not every primary hypothesis is rejected, and the gate remains closed.
Although the secondary p-values are small, the rule gives no confirmatory rejection in the secondary family. They may be reported descriptively or as exploratory findings, with the gate failure made clear. Testing them at the Bonferroni cutoff \(0.05/3\approx0.01667\) and declaring rejections would not follow the prespecified gatekeeping rule.
Worked Example: The Gate Opens and Bonferroni Testing Continues
Consider another hypothetical trial with two primary and three secondary hypotheses. At \(\alpha=0.05\), Bonferroni testing is used in each family. The primary p-values are \(0.006\) and \(0.014\). The secondary p-values are \(0.009\), \(0.018\), and \(0.044\). Work through the gate and the secondary tests.
The primary cutoff is \(0.05/2=0.025\). Both primary p-values are at or below \(0.025\), so both primary nulls are rejected and the gate opens.
The secondary cutoff is \(0.05/3\approx0.01667\). The first secondary p-value, \(0.009\), is below the cutoff and its null is rejected. The second, \(0.018\), and third, \(0.044\), exceed it and are not rejected. Thus the rule supports confirmatory rejection of both primary hypotheses and the first secondary hypothesis, but not the other two secondary hypotheses.
The secondary cutoff is not increased merely because some secondary hypotheses are not rejected. The procedure allocated the familywise error budget across all three secondary hypotheses, as prespecified.
Using Different Valid Procedures in the Two Families
The primary and secondary families need not use the same procedure. Each can use a method suited to its size and structure, provided it has strong FWER control at the stated level. For example, Holm’s step-down procedure may be used for the primary family and Bonferroni for the secondary family. The gate decision is based on whether the primary procedure rejects every primary hypothesis, not merely whether it rejects at least one.
Worked Example: Holm for the Primary Family, Bonferroni for the Secondary Family
Suppose a hypothetical trial has three primary hypotheses and two secondary hypotheses. Holm’s procedure is applied to the primary p-values \(0.004\), \(0.013\), and \(0.041\), at \(\alpha=0.05\). If the gate opens, the two secondary hypotheses are tested by Bonferroni at the same level, with p-values \(0.021\) and \(0.032\).
Order the primary p-values from smallest to largest: \(0.004\), \(0.013\), \(0.041\). Holm compares them in order with \(\alpha/3=0.01667\), \(\alpha/2=0.025\), and \(\alpha=0.05\):
Each comparison passes, so Holm rejects all three primary nulls and the gate opens. For the secondary family, Bonferroni’s cutoff is \(0.05/2=0.025\). The p-value \(0.021\) is below the cutoff, so that secondary null is rejected; \(0.032\) is above it, so the other secondary null is not rejected. The combination is valid because Holm and Bonferroni each provide strong FWER control at \(0.05\) within their respective families.
List the primary hypotheses whose rejection is required, the secondary family, the significance level, and the procedure for each family before examining results.
Apply the prespecified strong-FWER procedure at level \(\alpha\). Record exactly which primary nulls it rejects.
If every specified primary null is rejected, proceed to secondary testing. If even one is not rejected, make no confirmatory rejection in the secondary family under this rule.
Use its prespecified strong-FWER procedure at \(\alpha\), then report secondary rejections separately from non-rejections.
What the Gate Does—and Does Not—Establish
The gate is a prespecified logical condition for making secondary hypotheses eligible for confirmatory testing. It does not assert that a secondary effect is scientifically important, nor does it prove that all primary effects are clinically meaningful. Those judgments require interpretation of the estimated effects, their units, uncertainty, and the trial’s scientific objectives.
A gate that requires all primary hypotheses to be rejected is deliberately stringent: one unsuccessful primary test blocks every secondary test. This differs from a rule that opens after only a specified subset of primary hypotheses is rejected. Such a subset gate may be appropriate for some trial objectives, but its error control cannot be inferred by simply applying the proof above if other primary hypotheses can be falsely rejected outside the gate. The full family and its error allocation must be designed to protect all confirmatory claims.
The rule also differs in purpose from fixed-sequence testing across treatment arms. Fixed-sequence testing specifies an order in which individual hypotheses are tested, often stopping at the first failure. Gatekeeping instead separates hypotheses into primary and secondary families and makes secondary access depend on a family-level primary condition. The distinction is useful when the hierarchy reflects trial objectives rather than a ranking of treatment arms.
Compared with testing both families under a prespecified split of the error level, this all-primary gate can permit testing the secondary family at the full \(\alpha\) after success on the primary family. The price is that no secondary confirmatory test is available when the gate fails. Unlike a method that transfers or recycles portions of error across hypotheses, this basic rule does not calculate partial transfers; the gate either opens for the whole secondary family or remains closed.
Diagnostics and Common Pitfalls
Before interpreting results, check that the family definitions and gate were fixed before the results were reviewed, that the primary procedure rejected every hypothesis required by the gate, and that the secondary method controls strong FWER for its family. Also confirm that every hypothesis whose rejection can affect the gate is included in the prespecified error-control structure.
- Testing secondary outcomes anyway. A small secondary p-value does not override a closed gate. Under this rule, it cannot support a confirmatory rejection.
- Opening the gate after “enough” primary successes. Requiring all specified primary rejections is not the same as requiring most of them or a favorable average. Any alternative condition must be specified and justified in its own error-control design.
- Using an unadjusted primary test. If the primary family has multiple hypotheses, testing each at \(\alpha\) does not generally provide strong FWER control. The gate’s proof depends on a valid primary-family procedure.
- Assuming independence is needed. The event-containment proof requires strong control in each family, not independence between families. Dependence may matter for the validity of a chosen component procedure, but it is not an extra gatekeeping assumption here.
A particularly consequential mistake is to test some primary hypotheses at level \(\alpha\), omit them from the gate, and then test secondary hypotheses at level \(\alpha\) as if the theorem still applied. If a primary null outside the gate is true, its false rejection and a false secondary rejection need not be the same event. Their union can exceed \(\alpha\). Include all relevant primary hypotheses in the gate or use a separately justified allocation that covers the complete family of claims.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- There are two primary hypotheses tested by Bonferroni at \(\alpha=0.05\), with p-values \(0.020\) and \(0.026\). Does a secondary p-value of \(0.001\) permit a confirmatory rejection under the all-primary gate?
- Why does the gatekeeping proof not require independence between primary and secondary p-values?
- Three primary hypotheses have ordered p-values \(0.003\), \(0.018\), and \(0.030\). Under Holm at \(\alpha=0.05\), does the gate open?
- Suppose one primary hypothesis is omitted from the gate but is still tested and reported confirmatorily. Why does the basic proof no longer guarantee FWER control across all claims?
Solutions. (1) No. The primary Bonferroni cutoff is \(0.05/2=0.025\). The p-value \(0.026\) exceeds it, so not every primary null is rejected; the secondary p-value cannot open the gate. (2) If any primary null is true, a false secondary rejection requires the gate to open, which entails a false rejection in the primary family; if all primary nulls are false, the secondary procedure’s own FWER control bounds secondary errors. These event bounds do not use independence. (3) Yes. Holm’s ordered cutoffs are \(0.05/3\approx0.01667\), \(0.05/2=0.025\), and \(0.05\). The comparisons are \(0.003\leq0.01667\), \(0.018\leq0.025\), and \(0.030\leq0.05\), so all three are rejected. (4) A false rejection of the omitted true primary null need not imply that the gate failed to open, and it need not coincide with a false secondary rejection. The two error events are not contained in the same primary-family error event, so their combined probability is not bounded by the basic argument.