Why Test Intersections?
A clinical trial may support several related claims: that a treatment improves one endpoint, improves another endpoint, or benefits a specified subgroup. Testing each claim at level \(\alpha\) on its own does not generally keep the probability of making one or more false rejections within \(\alpha\). Closed testing organizes those claims around tests of their intersections. The decision for any single claim depends not only on its own test, but also on the tests of every intersection that includes it.
In an adaptive trial, the local tests must also be valid for the design and data-dependent changes that were used. Closed testing supplies a rule for combining those local tests; it does not make an invalid local test valid. The relevant groundwork is the earlier material on strong familywise error control, intersection hypotheses, and local tests for intersection nulls.
For example, with three elementary hypotheses, the intersections are \(H_1,H_2,H_3,H_{12}=H_1\cap H_2,H_{13}=H_1\cap H_3,H_{23}=H_2\cap H_3\), and \(H_{123}=H_1\cap H_2\cap H_3\). To reject \(H_1\), the local tests must reject \(H_1,H_{12},H_{13}\), and \(H_{123}\). A rejection of \(H_1\)'s local test alone is not enough.
Assumptions and the Decision Rule
The rule is defined for the whole family of hypotheses under consideration, not just for the claims that look promising after seeing the data. For every nonempty \(I\), the local test must be a level-\(\alpha\) test of \(H_I\): when \(H_I\) is true, its probability of rejection is no greater than \(\alpha\). In an adaptive design, this validity must apply to the actual testing procedure, including any permitted data-dependent changes. The local tests can use different statistics or distributions for different intersections, provided each is valid for its own intersection null.
The same family and the same prespecified local testing rules are used to make the final decisions. The tests need not be independent of one another. Their dependence does not alter the closed testing rule; it may, however, matter when constructing or calibrating a particular local test. The procedure requires a local test for every nonempty intersection, including singleton hypotheses and the intersection of the entire family.
If \(p_I\) is the p-value from the local test of \(H_I\), the rule can be written as a maximum. For each elementary hypothesis \(H_i\), let \(\mathcal{C}_i\) be the collection of all nonempty subsets \(I\) that contain \(i\). Its closed-testing adjusted p-value is the largest local p-value in that collection:
This expression assumes the local p-values are calibrated so that \(p_I\leq\alpha\) corresponds to rejection by the local level-\(\alpha\) test. If a local procedure is specified directly as a rejection decision rather than through a p-value, use that decision for each intersection.
Worked Decisions with Two Hypotheses
Worked Example: Both Claims Pass Their Intersection Test
Suppose a hypothetical trial evaluates two treatment claims, \(H_1\) for improvement in endpoint 1 and \(H_2\) for improvement in endpoint 2. The local p-values are \(p_1=0.012\), \(p_2=0.026\), and \(p_{12}=0.041\). Assume each is a valid local p-value for its stated null, and use \(\alpha=0.05\).
The intersections containing \(H_1\) are \(H_1\) and \(H_{12}\). Thus \(p_{1,\mathrm{closed}}=\max(0.012,0.041)=0.041\), which is at most \(0.05\), so reject \(H_1\).
The intersections containing \(H_2\) are \(H_2\) and \(H_{12}\). Thus \(p_{2,\mathrm{closed}}=\max(0.026,0.041)=0.041\), also at most \(0.05\), so reject \(H_2\). The conclusions concern the trial's endpoint claims; they do not by themselves establish the size or clinical importance of either treatment effect.
Three Hypotheses: One Local Test Can Block a Claim
Worked Example: A Failed Intersection Test Blocks Rejection
Consider a hypothetical trial with elementary nulls \(H_1,H_2,H_3\) for three endpoint claims. Suppose the following local p-values are obtained, and assume each is valid for its corresponding intersection. The prespecified level is \(\alpha=0.05\).
| Intersection null | Local p-value |
|---|---|
| \(H_1\) | 0.008 |
| \(H_2\) | 0.021 |
| \(H_3\) | 0.033 |
| \(H_{12}\) | 0.024 |
| \(H_{13}\) | 0.054 |
| \(H_{23}\) | 0.042 |
| \(H_{123}\) | 0.043 |
For \(H_1\), the containing intersections are \(H_1,H_{12},H_{13},H_{123}\). Therefore, \(p_{1,\mathrm{closed}}=\max(0.008,0.024,0.054,0.043)=0.054\). Since \(0.054>0.05\), do not reject \(H_1\).
For \(H_2\), the relevant values are \(0.021,0.024,0.042,0.043\), giving \(p_{2,\mathrm{closed}}=0.043\). Since \(0.043\leq0.05\), reject \(H_2\).
For \(H_3\), the relevant values are \(0.033,0.054,0.042,0.043\), giving \(p_{3,\mathrm{closed}}=0.054\). Since \(0.054>0.05\), do not reject \(H_3\). The local result for \(H_{13}\) blocks rejection of both \(H_1\) and \(H_3\), although their singleton local p-values are below 0.05. Only the claim represented by \(H_2\) passes every required local test.
Why the Maximum Is the Adjusted P-Value
The maximum formula follows directly from the all-intersections decision rule. For a given \(H_i\), closed testing rejects precisely when every \(p_I\) in \(\mathcal{C}_i\) is at most \(\alpha\). A finite collection of numbers is entirely at most \(\alpha\) exactly when its maximum is at most \(\alpha\). Therefore,
The maximum is the smallest common test level at which all the local tests required for \(H_i\) would reject. It is not an average of p-values, and the p-value for the largest intersection is not automatically the adjusted p-value: the maximum must include every intersection containing the particular elementary null.
Worked Example: Applying Closure to Valid Adaptive Local Tests
Suppose a hypothetical adaptive trial assesses three elementary treatment claims. Its design permits a data-dependent change, and the local testing procedures have already been calibrated to remain valid for each intersection under that design. Assume the resulting p-values are \(p_A=0.008\), \(p_B=0.021\), \(p_C=0.030\), \(p_{AB}=0.017\), \(p_{AC}=0.026\), \(p_{BC}=0.022\), and \(p_{ABC}=0.019\). Use \(\alpha=0.025\).
For claim \(A\), the required values are \(0.008,0.017,0.026,0.019\). The maximum is \(0.026\), so \(H_A\) is not rejected because \(0.026>0.025\).
For claim \(B\), the required values are \(0.021,0.017,0.022,0.019\). The maximum is \(0.022\), so \(H_B\) is rejected because \(0.022\leq0.025\).
For claim \(C\), the required values are \(0.030,0.026,0.022,0.019\). The maximum is \(0.030\), so \(H_C\) is not rejected. The adaptive setting changes what must be checked about local-test validity; it does not change which intersections contain each elementary null.
Diagnostics, Alternatives, and Limitations
Before reporting closed-testing decisions, verify the test family, the intersection list, the local test for each intersection, and the common threshold \(\alpha\). For a family of \(m\) elementary hypotheses, there are \(2^m-1\) nonempty intersections. A missing intersection can make an apparent rejection too permissive. A local p-value copied from the wrong endpoint combination, or a test calibrated for a different null, likewise cannot support the required decision.
Pay particular attention to local-test validity when the design or analysis is adaptive. If a local test ignores a data-dependent change that affects its null distribution or type I error, the closed-testing calculation does not repair that problem. Likewise, seeing small singleton p-values is not evidence that an elementary null passes closure: inspect every intersection containing it. In a large family, calculation software can help, but the mapping from each elementary null to its required intersections should still be checked.
Closed testing differs from running unadjusted tests because it imposes additional local-test requirements before an elementary claim can be rejected. A simpler single-step Bonferroni procedure controls multiplicity by using a conservative threshold for individual tests; it can be convenient when there are many hypotheses. Closed testing can use more informative, problem-specific tests for intersections, but may require substantially more calculations. Its advantages depend on the quality and validity of those local tests, not just on the number of hypotheses.
The procedure is defined for the specified family and local tests. It does not decide which outcomes belong in the family, demonstrate that an effect is clinically meaningful, or compensate for biased measurements. Nor is it computationally trivial for a large family: testing every nonempty intersection can be demanding. The key practical discipline is to keep the elementary claims, their intersections, and the local tests aligned.
Check Your Understanding
For each question, apply the closed testing rule and then compare your result with the solution.
- For two hypotheses, suppose \(p_1=0.018\), \(p_2=0.047\), and \(p_{12}=0.052\). At \(\alpha=0.05\), which elementary hypotheses are rejected?
- With three hypotheses, list all intersections that must be rejected to reject \(H_2\).
- For \(H_1\), the local p-values of its containing intersections are \(0.012,0.031,0.024,\) and \(0.028\). Find its closed-testing adjusted p-value and decision at \(\alpha=0.03\).
- A trial’s local test for one intersection is not valid after a data-dependent design change. Does applying the closed-testing rule make that local test valid? Explain.
Solutions. 1. \(p_{1,\mathrm{closed}}=\max(0.018,0.052)=0.052\) and \(p_{2,\mathrm{closed}}=\max(0.047,0.052)=0.052\). Neither is rejected at 0.05. 2. Reject \(H_2\), \(H_{12}\), and \(H_{23}\), and the full intersection \(H_{123}\); these are exactly the nonempty intersections containing index 2. 3. The adjusted p-value is the maximum, \(0.031\). Since \(0.031>0.03\), do not reject \(H_1\). 4. No. The closure rule combines local decisions, but each local test must first be valid for its intersection under the design actually used.