Why Test Intersections?
A multi-arm trial may support several treatment claims, while the scientific question may concern a particular collection of arms: for example, whether any of three candidate treatments is effective, or whether at least one of two priority arms is effective. Testing each arm separately at level \(\alpha\) does not, by itself, control the probability of making one or more false claims. The intersection hypotheses used in closed testing provide a way to make each claim while controlling that familywise error probability.
The central task is to give every relevant intersection a valid local sequential test. A local test evaluates the null that all arms in a specified subset are ineffective, using the statistics collected over the planned stages. Closure then requires a rejection of every intersection containing an arm before declaring that arm effective. This may appear to require testing many subsets, but some families of local tests allow an equivalent, much shorter calculation.
Assumptions and Local Sequential Tests
For clarity, consider one-sided efficacy tests with larger values of an arm’s statistic favouring treatment. At each planned look \(\ell=1,\ldots,L\), let \(Z_{i\ell}\) be the standardized statistic for arm \(i\). The usual canonical model has standard normal marginal distributions at a look under the boundary null, with a specified joint covariance across arms and looks. The covariance can reflect shared control patients and repeated observations over time. The local test must account for the planned looks; treating each look as though it were the only look would generally fail to control the sequential type I error.
For a composite null, a local test is valid at level \(\alpha\) if its probability of rejection is at most \(\alpha\) for every parameter value in \(H_I\). A common sufficient condition for calibration at the boundary is that the rejection rule is monotone in the statistics and that the joint distribution is stochastically increasing as treatment effects increase. Then the largest rejection probability over the null occurs when each effect in \(I\) is at its null boundary. The calibration must still reflect the actual design assumptions and the planned stopping and futility rules.
One transparent construction uses a boundary-crossing probability. Choose critical values so that, under the intersection boundary, the probability of crossing any boundary for any arm in \(I\) at any planned look is at most \(\alpha\). If the covariance model is known, this probability can be calculated or numerically evaluated under that joint model. A simpler, generally more conservative construction combines arm-level sequential p-values.
To verify this result, if \(H_I\) is true then every \(H_i\), \(i\in I\), is true. Rejection at level \(\alpha\) requires at least one \(p_i\leq\alpha/|I|\). The union bound gives \(P(p_I\leq\alpha)\leq\sum_{i\in I}P(p_i\leq\alpha/|I|)\leq |I|(\alpha/|I|)=\alpha\). The argument uses the sequential p-values, so the looks are already accounted for; it does not require treating looks as independent.
Worked Example: A Bonferroni Boundary for Two Arms and Two Looks
Suppose a prespecified intersection contains arms A and B, with two planned efficacy looks. For this hypothetical calculation, assume each \(Z_{i\ell}\) has a standard normal marginal distribution under the intersection boundary, and use the rule that the local test rejects if any of the four statistics exceeds the same upper boundary \(c\). Dependence among the four statistics is allowed.
There are \(|I|L=2(2)=4\) opportunities to cross. By the union bound, each marginal crossing probability can be limited to \(\alpha/4=0.05/4=0.0125\). Thus choose \(c=\Phi^{-1}(1-0.0125)=\Phi^{-1}(0.9875)\approx2.241\), where \(\Phi\) is the standard normal cumulative distribution function. The probability of any crossing under the intersection boundary is at most \(4(0.0125)=0.05\).
Suppose the observed statistics are \(Z_{A1}=1.80\), \(Z_{A2}=2.30\), \(Z_{B1}=2.00\), and \(Z_{B2}=2.10\). Since \(2.30>2.241\), arm A crosses at the second look, so the local intersection test rejects. The other three values do not cross. This conclusion relies on the stated marginal null distributions and monotone upper-tail rule; no independence assumption is needed for the bound.
This construction is easy to audit but may be conservative: the union bound does not use the actual dependence among looks or arms. If a joint sequential boundary is calibrated directly under a justified joint model, it can use that dependence and may reject more often while still satisfying the local level requirement.
Closure and Strong Familywise Error Control
For a family of \(K\) elementary hypotheses, the complete closure contains the intersection \(H_I\) for each nonempty subset \(I\subseteq\{1,\ldots,K\}\). A closed-testing decision rejects \(H_i\) only when every local test for an intersection containing \(i\) rejects at its specified level. Under the complete closure, this definition gives strong familywise error control: it applies whether none, some, or all of the elementary nulls are true.
The key argument concerns the set \(T\) of true elementary nulls under the actual configuration. If any true elementary null is rejected, choose a rejected \(H_i\) with \(i\in T\). Closure requires rejection of every intersection containing \(i\), including \(H_T=\bigcap_{j\in T}H_j\), since \(i\in T\). Therefore the event of at least one false elementary rejection is contained in the event that the local test of the true intersection \(H_T\) rejects. If every local test has size at most \(\alpha\), then \(P(\text{at least one false elementary rejection})\leq P(\text{local test of }H_T\text{ rejects})\leq\alpha\). This containment, rather than separate level-\(\alpha\) control of each elementary test, is what establishes strong familywise control.
Worked Example: Closed Testing With Three Sequential p-Values
Consider three arms with hypothetical, already sequentially adjusted p-values \(p_1=0.006\), \(p_2=0.018\), and \(p_3=0.041\). Assume each is valid for its elementary null. Use the Bonferroni local p-value \(p_I=\min\{1,|I|\min_{i\in I}p_i\}\) for every intersection, and set \(\alpha=0.05\).
For arm 1, the local p-values for the intersections containing it are \(p_{\{1\}}=0.006\), \(p_{\{1,2\}}=2(0.006)=0.012\), \(p_{\{1,3\}}=2(0.006)=0.012\), and \(p_{\{1,2,3\}}=3(0.006)=0.018\). All are at most 0.05, so closure rejects \(H_1\).
For arm 2, the relevant values are \(p_{\{2\}}=0.018\), \(p_{\{1,2\}}=0.012\), \(p_{\{2,3\}}=2(0.018)=0.036\), and \(p_{\{1,2,3\}}=0.018\). All are at most 0.05, so closure rejects \(H_2\).
For arm 3, they are \(p_{\{3\}}=0.041\), \(p_{\{1,3\}}=0.012\), \(p_{\{2,3\}}=0.036\), and \(p_{\{1,2,3\}}=0.018\). These also are all at most 0.05, so closure rejects \(H_3\). These are strong familywise-controlled rejections, not three unrelated level-0.05 tests: each decision has passed every required intersection test.
A Shortcut: Holm’s Step-Down Calculation
For the Bonferroni local tests just defined, one need not calculate and record all \(2^K-1\) intersection p-values. Sort the sequential p-values as \(p_{(1)}\leq\cdots\leq p_{(K)}\). Compare \(p_{(r)}\) with \(\alpha/(K-r+1)\), starting at \(r=1\). Reject in order while each comparison passes; stop at the first failure and retain that hypothesis and all hypotheses with larger p-values. This is Holm’s step-down procedure, and it is exactly equivalent to closed testing with Bonferroni local tests.
A short justification explains the computational saving. For a hypothesis at rank \(r\), any intersection containing it may also contain hypotheses with smaller p-values. The most restrictive Bonferroni comparison among the relevant intersections is captured by the rank-based cutoff \(\alpha/(K-r+1)\). Consequently, checking the sorted sequence against these cutoffs accounts for the required intersections without listing each subset separately. This shortcut depends on the specified Bonferroni local tests; a different collection of local tests does not automatically have the same step-down representation.
Worked Example: Testing Two Priority Arms Without Listing Intersections
Suppose there are four arms and the sequential p-values are \(0.004\), \(0.009\), \(0.031\), and \(0.200\), in that order. The prespecified claims of primary interest are for the first two arms, and \(\alpha=0.05\). Apply the step-down calculation corresponding to Bonferroni closed testing.
At rank 1, the cutoff is \(0.05/4=0.0125\), and \(0.004\leq0.0125\), so the first hypothesis passes. At rank 2, the cutoff is \(0.05/3\approx0.01667\), and \(0.009\leq0.01667\), so the second passes. At rank 3, the cutoff is \(0.05/2=0.025\), but \(0.031>0.025\); stop. The first two hypotheses are rejected, and the third and fourth are not rejected by this procedure.
There are \(2^4-1=15\) nonempty intersections in the complete closure, but this calculation required only the ordered p-values and the first three comparisons. It remains valid for claims about the first two arms because their step-down comparisons incorporate the relevant closure requirements, including intersections involving other arms.
When Subset-Specific Calibration Helps
Bonferroni is not the only way to test an intersection. A local test can instead use a joint boundary-crossing rule calibrated for the specified subset \(I\), using a justified joint distribution for the statistics in \(I\) across the planned looks. For instance, a rule may reject \(H_I\) when a specified combination of the arm statistics crosses a boundary. The calibration target remains the same: the probability of local rejection under \(H_I\) must be no more than \(\alpha\), uniformly over the intersection null or at its boundary under the stated monotonicity conditions.
Subset-specific calibration can gain power by using the dependence structure and the size of the subset. It also creates more work: the covariance model, stages, stopping rules, and subset definition must match the calibration. If the local tests do not admit a shortcut such as Holm’s, the closure decision may require more intersection calculations. It is legitimate to omit an intersection calculation only when a mathematical equivalence or logical implication guarantees the same closed-testing decision—not merely because that intersection appears unimportant.
Diagnostics and Common Failure Modes
- Ignoring the sequential looks. A marginal p-value from the most favourable look is not automatically a valid sequential p-value. Use a boundary-crossing calibration or a p-value that already accounts for all planned looks and stopping rules.
- Using the wrong subset size. The Bonferroni local test for \(I\) uses \(|I|\), not the total number of arms unless \(I\) contains every arm. Check the subset used in each comparison.
- Assuming independence without justification. The Bonferroni argument allows dependence. A dependence-aware joint calibration, by contrast, must use a defensible covariance model for the arms and looks.
- Checking each elementary p-value at \(\alpha\) and stopping there. Separate level-\(\alpha\) tests do not generally control the probability of at least one false rejection. Closure or another valid multiplicity procedure is still needed.
- Applying Holm to incompatible local tests. Holm’s shortcut follows from Bonferroni local tests and valid sequential p-values. It is not a generic shortcut for arbitrary subset-specific boundaries.
- Leaving gaps in the closure. If a claim relies on an intersection test, that test must be valid and either performed or logically covered by an equivalent procedure. Testing only a hand-picked set of intersections, with no such justification, does not establish closure.
A practical audit should record the tested family, the definition of each intersection, the sequential calibration used for its local test, and the rule used to translate local results into elementary decisions. For a shortcut, the audit should also identify the equivalence that makes the omitted intersection calculations unnecessary.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For a subset of three arms and two planned looks, a union-bound rule allocates equal marginal crossing probabilities. What is the per-statistic allocation when \(\alpha=0.06\)?
- Why does closure provide strong familywise error control, while separately testing each elementary null at level \(\alpha\) does not establish that control?
- With four valid sequential p-values ordered as \(0.003, 0.020, 0.040, 0.60\) and \(\alpha=0.05\), how many hypotheses does Holm’s procedure reject?
- Does the Bonferroni local test require independence across arms and looks? State the reason.
Solutions. (1) There are \(3(2)=6\) arm-look opportunities, so the allocation is \(0.06/6=0.01\) per statistic; the corresponding standard normal upper boundary is \(\Phi^{-1}(0.99)\approx2.326\). (2) If any true elementary null is rejected, closure requires rejection of the intersection of all true elementary nulls. The familywise error event is therefore contained in rejection of that valid level-\(\alpha\) intersection test. Separate level-\(\alpha\) bounds do not bound the probability of their union by \(\alpha\). (3) The cutoffs are \(0.05/4=0.0125\), \(0.05/3\approx0.01667\), and \(0.05/2=0.025\). The first p-value passes, but \(0.020>0.01667\) at rank 2, so the procedure stops and rejects one hypothesis. (4) No. The union bound gives validity from the marginal sequential p-value bounds, regardless of dependence; the sequential calibration is needed so each arm-level p-value already accounts for its looks.)