Why Every Intersection Matters
In an adaptive trial, an analysis may produce evidence about several treatment claims, and the data or analysis plan may have changed after an interim review. A small p-value for one claim is not, by itself, enough to establish that the claim can be rejected while controlling the chance of any false rejection in the family. Closed testing makes the decision depend on tests of the claim and of every intersection that includes it.
For three claims, this means checking seven local tests. The task in this tutorial is to turn their results into the set of elementary hypotheses that may be rejected. The local results are inputs to the decision rule; they must already come from tests valid for the adaptive procedure being used.
For example, \(H_1\) is eligible for rejection only if the local tests reject \(H_1\), \(H_{12}=H_1\cap H_2\), \(H_{13}=H_1\cap H_3\), and \(H_{123}=H_1\cap H_2\cap H_3\). The local test of \(H_{23}\) is not part of the requirement for rejecting \(H_1\), because \(H_1\) is not one of its components.
A local test is a test applied to one intersection hypothesis. For closure at level \(\alpha\), each local test must have Type I error at most \(\alpha\) whenever its own intersection null is true. In an adaptive trial, that validity must apply to the complete procedure, including any data-dependent changes that affect the test. A set of ordinary fixed-design p-values is not automatically valid for an adaptive analysis.
A Direct Decision Rule
Suppose each local test reports a p-value \(p_S\) for \(H_S\), with each p-value valid for its intersection null. At level \(\alpha\), the local test rejects \(H_S\) when \(p_S\leq\alpha\). Define the closed-testing adjusted p-value for \(H_i\) as the largest local p-value among the intersections that contain \(i\). Then \(H_i\) is rejected exactly when this largest value is at most \(\alpha\).
The maximum captures the “every intersection” requirement: if even one required local test fails to reject, the adjusted p-value exceeds \(\alpha\), and closure does not permit the elementary rejection. The closed-testing adjusted p-value is a summary of the required local p-values; it is not a probability that \(H_i\) is true.
Worked Example: One Elementary Claim Passes Closure
Hypothetical setting. An adaptive trial has three treatment claims, represented by \(H_1,H_2,H_3\). Valid local p-values have been calculated for every intersection. The familywise significance level is \(\alpha=0.025\).
| Intersection | Local p-value | Local decision at 0.025 |
|---|---|---|
| \(H_1\) | 0.008 | Reject |
| \(H_2\) | 0.031 | Do not reject |
| \(H_3\) | 0.014 | Reject |
| \(H_{12}\) | 0.018 | Reject |
| \(H_{13}\) | 0.022 | Reject |
| \(H_{23}\) | 0.027 | Do not reject |
| \(H_{123}\) | 0.020 | Reject |
For \(H_1\), the required p-values are 0.008, 0.018, 0.022, and 0.020. Therefore, \(p_1^{\mathrm{closed}}=\max(0.008,0.018,0.022,0.020)=0.022\), which is at most 0.025. Reject \(H_1\).
For \(H_2\), the required p-values are 0.031, 0.018, 0.027, and 0.020. Thus, \(p_2^{\mathrm{closed}}=\max(0.031,0.018,0.027,0.020)=0.031\), which exceeds 0.025. Do not reject \(H_2\).
For \(H_3\), the required p-values are 0.014, 0.022, 0.027, and 0.020. Thus, \(p_3^{\mathrm{closed}}=\max(0.014,0.022,0.027,0.020)=0.027\), which exceeds 0.025. Do not reject \(H_3\). Although its singleton test rejects, the local test of \(H_{23}\) does not, so closure blocks the elementary rejection. The allowable elementary rejection set is \(\{H_1\}\).
Why Closure Controls Familywise Error
The decision rule has a direct error-control justification. Consider any configuration of true and false elementary null hypotheses. If no elementary null is true, there can be no false elementary rejection. Otherwise, let \(T\) be the nonempty set of indices for which \(H_i\) is true. The intersection \(H_T=\bigcap_{i\in T}H_i\) is then true.
If the procedure falsely rejects any true elementary hypothesis \(H_i\), where \(i\in T\), closure requires rejection of every intersection containing \(i\), including \(H_T\). Consequently, the event of making at least one false elementary rejection is contained in the event of rejecting the true intersection \(H_T\). Since the local test of \(H_T\) has Type I error at most \(\alpha\), the probability of the former event is also at most \(\alpha\). This is strong familywise error control: the bound holds whether none, some, or all of the elementary null hypotheses are false.
The justification depends on both parts of the construction: closure requires every relevant intersection test to reject, and the local test for the intersection of the true nulls must actually be level \(\alpha\). In an adaptive design, a local test that is invalid after selection or interim review cannot supply this guarantee merely because it has been placed in a closure table.
Worked Example: A Failed Global Intersection Blocks All Claims
Hypothetical setting. In a second adaptive trial, the seven valid local p-values are 0.012 for \(H_1\), 0.021 for \(H_2\), 0.018 for \(H_3\), 0.030 for \(H_{12}\), 0.044 for \(H_{13}\), 0.049 for \(H_{23}\), and 0.052 for \(H_{123}\). Let \(\alpha=0.05\).
Each singleton and pairwise local test rejects at level 0.05, but the test of the global intersection does not, because \(0.052>0.05\). For \(H_1\), the closed p-value is \(\max(0.012,0.030,0.044,0.052)=0.052\). For \(H_2\), it is \(\max(0.021,0.030,0.049,0.052)=0.052\). For \(H_3\), it is \(\max(0.018,0.044,0.049,0.052)=0.052\).
All three closed p-values exceed 0.05, so none of the elementary hypotheses can be rejected. The failure of the global intersection test blocks every elementary claim because \(H_{123}\) contains all three hypotheses. This is not a finding that all three nulls are true; it is the decision required by this procedure at the stated level.
Comparing Results and Checking the Analysis
The local tests and the closed decisions answer different questions. A local rejection says that one intersection test has crossed its own threshold. An elementary rejection under closure says that every required intersection test has crossed that threshold. Reporting local rejections as if they were automatically elementary conclusions omits the multiplicity protection.
Closed testing is a general framework: it can use different valid local tests for different intersections. Bonferroni and Holm procedures are more specific alternatives based on elementary p-values; Holm is equivalent to closure using Bonferroni local tests in the standard setting. With suitable valid intersection tests, closure can use information about joint hypotheses rather than relying only on Bonferroni bounds. That flexibility does not remove the need to validate each local test or to evaluate every required intersection.
For three hypotheses, an audit can be performed systematically:
Write the three singleton hypotheses, all three pairs, and the triple intersection. Confirm that none is missing.
Check that each p-value or rejection decision comes from a test valid at level \(\alpha\) for that intersection, including any adaptive steps.
For each \(H_i\), inspect only intersections containing \(i\). Reject \(H_i\) only if every one rejects.
State which elementary claims are rejected and, where useful, identify the intersection test that blocked a claim.
Common failures include omitting \(H_{123}\), testing the singleton hypotheses but not the pairwise intersections, comparing a local p-value to a different significance level without justification, or using an unadjusted final p-value after data-dependent selection. Another mistake is to infer that a rejected intersection means every component elementary null is false. Rejection of \(H_{12}\), for instance, is evidence against the joint statement that both \(H_1\) and \(H_2\) hold; it does not alone identify which one fails. Closure’s elementary rule instead asks whether all intersections containing a particular \(H_i\) have been rejected.
Closure can be conservative, and its conclusions are only as trustworthy as its local tests. It also does not repair poor endpoint definitions, an incomplete family of claims, or an adaptive analysis whose local error rates were not controlled. Record the family, the local test for each intersection, the level used, and the resulting closed decisions so that another analyst can reproduce the logic.
Worked Example: Three Claims Pass at a Boundary-Near Level
Hypothetical setting. A third adaptive trial reports valid local p-values of 0.006, 0.011, and 0.020 for \(H_1,H_2,H_3\), respectively; 0.018, 0.028, and 0.049 for \(H_{12},H_{13},H_{23}\); and 0.041 for \(H_{123}\). The prespecified level is \(\alpha=0.05\).
For \(H_1\), the maximum among its four required values is \(\max(0.006,0.018,0.028,0.041)=0.041\). For \(H_2\), it is \(\max(0.011,0.018,0.049,0.041)=0.049\). For \(H_3\), it is \(\max(0.020,0.028,0.049,0.041)=0.049\).
Each maximum is at most 0.05, so closure permits rejection of all three elementary hypotheses. The result for \(H_2\) and \(H_3\) is close to the threshold: their pairwise intersection \(H_{23}\), with p-value 0.049, is the limiting test. This example illustrates why decisions should use the specified comparison exactly and why rounding p-values before comparing them can change a borderline decision.
Check Your Understanding
For each question, use the closure rule and show which local tests determine the answer.
- At \(\alpha=0.05\), suppose \(p_1=0.01\), \(p_2=0.02\), \(p_3=0.03\), \(p_{12}=0.04\), \(p_{13}=0.02\), \(p_{23}=0.06\), and \(p_{123}=0.03\). Which elementary hypotheses can be rejected?
- If the singleton test of \(H_2\) rejects but the local test of \(H_{12}\) does not, may closure reject \(H_2\)? Explain.
- Why does the procedure control the probability of at least one false elementary rejection when some null hypotheses are true?
- In an adaptive trial, all seven numerical p-values are below \(\alpha\), but the p-value for \(H_{123}\) was computed without accounting for interim selection. What can be concluded from the closure table?
Solutions. (1) For \(H_1\), the maximum is \(\max(0.01,0.04,0.02,0.03)=0.04\), so reject \(H_1\). For \(H_2\), the maximum is \(\max(0.02,0.04,0.06,0.03)=0.06\), so do not reject \(H_2\). For \(H_3\), the maximum is \(\max(0.03,0.02,0.06,0.03)=0.06\), so do not reject \(H_3\). (2) No. \(H_{12}\) contains \(H_2\), and closure requires every intersection containing \(H_2\) to reject. (3) If \(T\) indexes the true elementary nulls, any false elementary rejection requires rejection of the true intersection \(H_T\). A level-\(\alpha\) local test of \(H_T\) therefore bounds the probability of any false rejection by \(\alpha\). (4) The table does not establish valid closed decisions. Every local test, including the test of \(H_{123}\), must be valid for its intersection under the adaptive procedure; an unadjusted p-value cannot be treated as a valid local result.