Why test intersections?
In a multi-arm trial, a conclusion about one treatment arm can be affected by the other arms tested alongside it. Repeated interim looks and data-dependent arm selection add further opportunities for a false rejection. Closed testing handles this multiplicity by requiring evidence not only against an arm’s own null hypothesis, but also against every intersection hypothesis that contains it. The result is a general route from valid sequential tests of intersections to strong familywise error rate (FWER) control.
The construction does not prescribe how to design each sequential test. Its key requirement is that every local test be valid for its intersection null under the trial’s complete analysis and stopping protocol. The closure rule then combines those tests into decisions about individual arms.
For three arms, the full collection of intersections is \(H_1,H_2,H_3,H_{12},H_{13},H_{23},H_{123}\), where, for example, \(H_{13}=H_1\cap H_3\). A rejection of \(H_1\) therefore requires rejection of \(H_1\), \(H_{12}\), \(H_{13}\), and \(H_{123}\). It is not enough that the single-arm test for arm 1 rejects.
Sequential validity and the closure rule
A local test for \(H_I\) may use data from several planned stages and may stop early for efficacy or futility. Its level is a statement about the probability of rejecting \(H_I\) across the entire procedure, not just at one look. In particular, the test must account for the repeated looks and any selection or dropping rules that affect which data are used. For every parameter value \(\theta\) satisfying \(H_I\), validity means
This condition must hold under the planned design, including its stopping rules, randomization, and permitted adaptations. A test valid for a fixed sample at the final stage is not automatically valid when applied repeatedly or only to an arm retained after looking at the data.
If each local test supplies a valid sequential p-value \(p_I\), the closure rule has a convenient equivalent form: define the closed-testing adjusted p-value for arm \(i\) by
Then reject \(H_i\) at level \(\alpha\) precisely when \(p_i^{\mathrm{closed}}\leq\alpha\). The maximum appears because every intersection containing \(i\) must pass. This calculation does not make the local tests valid; it combines p-values that must already be valid for their respective intersection nulls.
Worked Example: Applying Closure to Three Arms
Consider three hypothetical treatment arms and a sequential family of local tests conducted at \(\alpha=0.025\). Suppose the tests return the following valid, across-the-planned-looks p-values: \(p_{123}=0.012\), \(p_{12}=0.018\), \(p_{13}=0.014\), \(p_{23}=0.044\), \(p_1=0.008\), \(p_2=0.017\), and \(p_3=0.006\). Each value is assumed to come from a procedure that accounts for its intersection null, repeated looks, and any planned selection.
For arm 1, the relevant intersections are \(H_1,H_{12},H_{13},H_{123}\). Thus \(p_1^{\mathrm{closed}}=\max(0.008,0.018,0.014,0.012)=0.018\), which is below \(0.025\). Reject \(H_1\).
For arm 2, the relevant values are \(0.017,0.018,0.044,0.012\), so \(p_2^{\mathrm{closed}}=\max(0.017,0.018,0.044,0.012)=0.044\). Since \(0.044>0.025\), do not reject \(H_2\).
For arm 3, the relevant values are \(0.006,0.014,0.044,0.012\), so \(p_3^{\mathrm{closed}}=0.044>0.025\). Do not reject \(H_3\). The small single-arm p-values for arms 2 and 3 do not suffice: the local test of \(H_{23}\) does not reject at the chosen level.
Why the procedure controls strong FWER
Strong FWER control means that the probability of rejecting one or more true elementary null hypotheses is at most \(\alpha\), for every configuration of true and false nulls. The key argument uses the intersection of the true elementary hypotheses. It does not require every null to be true.
Here is the justification. Let \(T\) be the set of indices of the true elementary hypotheses for a particular parameter value. If \(T\) is empty, there is no true null to reject, so the FWER is zero. Otherwise, define \(H_T=\bigcap_{i\in T}H_i\), which is true at that parameter value. If the procedure falsely rejects any true \(H_i\), with \(i\in T\), the closure rule requires it to reject every intersection containing \(i\), including \(H_T\). Therefore, the event of at least one false elementary rejection is contained in the event that the local test of \(H_T\) rejects. Taking probabilities gives
The last inequality follows from the assumed level-\(\alpha\) validity of the local test for \(H_T\). This argument works for any dependence among the arm statistics, provided each local test is valid under its own null. It also explains why sequential validity matters: the event on the right includes rejection at any allowed look under the actual protocol.
Worked Example: Bounding Error When Two Arms Are Truly Null
Suppose a hypothetical three-arm trial uses closed testing with \(\alpha=0.05\). At the true parameter value, \(H_1\) is false but \(H_2\) and \(H_3\) are true. The set of true null indices is \(T=\{2,3\}\), so \(H_T=H_{23}\) is true.
A false elementary rejection can only be a rejection of \(H_2\), \(H_3\), or both. Rejection of either requires rejection of \(H_{23}\). Thus \(P_{\theta}(\text{false elementary rejection})\leq P_{\theta}(\text{local test rejects }H_{23})\leq0.05\). The strong FWER bound is 0.05, regardless of whether the local test of \(H_{23}\) uses shared-control data or repeated interim looks, as long as its stated level is valid for this sequential design.
Closure along a sequential trial
The closure rule is a final decision principle; it does not mean that every intersection test must necessarily run to the maximum sample size. A local sequential test can stop at an earlier stage according to its pre-specified rules. The p-value or rejection decision supplied to closure must summarize that test over its full sequential path. One should not treat a collection of unadjusted stage-specific p-values as if each were a valid sequential p-value.
Worked Example: A Crossing at One Look Is Not Enough
Consider two hypothetical arms, with planned looks at stages 1 and 2 and \(\alpha=0.025\). Assume the local sequential tests have been calibrated to control their rejection probabilities across both looks. At the end of the trial, suppose their sequential p-values are \(p_1=0.010\), \(p_2=0.018\), and \(p_{12}=0.031\). These are p-values for the full sequential tests, not raw p-values selected from a particular look.
The closed adjusted values are \(p_1^{\mathrm{closed}}=\max(0.010,0.031)=0.031\) and \(p_2^{\mathrm{closed}}=\max(0.018,0.031)=0.031\). Both exceed \(0.025\), so neither elementary null is rejected. Even if an arm-specific statistic crossed its own boundary at stage 2, that crossing would not establish a closed-testing rejection when the local test of \(H_{12}\) fails to reject.
If instead the valid sequential p-value for \(H_{12}\) were \(0.020\), the adjusted values would be \(0.020\) and \(0.020\). Both arms would then meet the closure criterion, because each singleton p-value and the intersection p-value would be at most \(0.025\).
The same logic applies when arms are dropped or selected at interim analyses. Dropping an arm can change which statistics are observed or used later, but it does not remove the requirement to test every relevant intersection. The local procedures must be valid for the data-generating and selection protocol actually used. Treating an arm as though it had been selected independently of its interim results, when selection depended on those results, can invalidate a nominal local test and therefore undermine the theorem’s premise.
How closure compares with alternatives
| Approach | What it does | What to check |
|---|---|---|
| Closed testing | Rejects an arm only when all intersection tests containing its null reject. | Every sequential local test must be valid for its intersection null. |
| Bonferroni testing | Controls familywise error by testing each arm at a reduced level, such as \(\alpha/m\). | The allocation must remain valid for repeated looks and any selection. |
| One global test | Tests the intersection of all arm nulls, often asking whether any evidence against the global null exists. | A global rejection alone does not identify which individual arm nulls can be rejected with strong control. |
Closure is a framework rather than a single numerical boundary. It can use different valid local tests for different intersections, and it can exploit dependence when those local tests do so validly. Its cost is that a family of \(m\) hypotheses has \(2^m-1\) nonempty intersections, which can become computationally demanding. In some settings, symmetry or other structure allows an equivalent computation without separately listing every test, but the closure criterion remains the governing principle.
Diagnostics and common pitfalls
- Check the null for each local test. The test labelled \(H_I\) must be valid whenever every null in \(I\) is true; a test valid only under the complete global null is not automatically valid for a partial intersection.
- Check the whole sequential path. Repeated opportunities to reject must be included in the local test’s type I error calculation. A fixed-look test does not by itself justify repeated use.
- Check selection and dropping rules. If the observed data determine which arms continue or which comparisons are made, the local test must remain valid under that rule.
- Check all supersets of an arm index. Omitting even one intersection can break the closure argument, because a false rejection need no longer imply rejection of the intersection of all true nulls.
A common mistake is to reject an arm because its own sequential p-value is below \(\alpha\), then argue that the procedure is protected because the global intersection test also rejected. That reasoning is incomplete: the arm must pass every intersection containing it, not just the singleton and the all-arms intersection. The correction is to compute the maximum of the relevant valid local p-values, or to check every required local rejection directly.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For four treatment arms, how many nonempty intersection hypotheses are in the closure, and which intersections must reject to reject \(H_2\)?
- Suppose the true elementary nulls are \(H_1\) and \(H_3\). Which local test bounds the probability of a false elementary rejection under closure?
- With \(\alpha=0.05\), an arm has valid local p-values \(0.02,0.04,0.03\) across the intersections containing it. Is its closed-testing adjusted p-value \(0.03\), and is the arm rejected?
- Why does a local test calibrated at a single final look not automatically qualify as a valid local sequential test when interim looks are added?
- Does rejection of the all-arms intersection alone authorize rejection of a particular arm null? Explain.
Solutions. (1) There are \(2^4-1=15\) nonempty intersections. To reject \(H_2\), every intersection whose index set contains 2 must reject: \(H_2\), \(H_{12}\), \(H_{23}\), \(H_{24}\), \(H_{123}\), \(H_{124}\), \(H_{234}\), and \(H_{1234}\). (2) The relevant true intersection is \(H_{13}=H_1\cap H_3\); any false rejection of either true elementary null requires rejection of its local test. (3) Yes. The maximum is \(0.04\), so the arm is rejected at \(0.05\). (4) Interim looks add opportunities to cross a rejection boundary, so the probability of rejection over the full path can exceed the single-look level unless the test accounts for those looks. (5) No. The arm must also pass every other intersection containing its null, including the singleton test.