Why Test Intersections, Not Only the Global Null?
A MAMS trial may compare several treatments with a shared control, producing a family of elementary null hypotheses such as “treatment 1 has no effect” and “treatment 2 has no effect.” A test of the global null—that all treatments have no effect—addresses whether there is evidence against the whole family. It does not, by itself, determine which individual treatment claims are supported. Closed testing provides a general rule for connecting tests of joint hypotheses to decisions about individual hypotheses.
The introductory material on elementary and intersection null hypotheses defines an intersection as the hypothesis that all of its component elementary nulls are true. Here the key requirement is that the closure includes every nonempty intersection, and that each has a valid local test at the chosen significance level. The decision rule then protects the family of elementary claims, not merely the global null.
For \(m\) distinct elementary hypotheses, the closure has up to \(2^m-1\) intersections. The rule is intentionally demanding: rejecting \(H_i\) requires passing not just the test of \(H_i\), but every intersection test that includes it. An intersection may have more than one valid local test; the procedure uses the local test specified for that intersection.
The Strong Familywise Error Guarantee
A local test is valid at level \(\alpha\) when its probability of rejecting its intersection null is at most \(\alpha\) for every parameter value under which that intersection null is true. This is a property of the test under the intersection hypothesis, not a guarantee that the hypothesis itself is true. The familywise error rate (FWER) is the probability of rejecting at least one true elementary null.
The qualification “strongly” matters. The guarantee holds whether all elementary nulls are true, none are true, or only some are true. The proof uses the intersection of the true elementary nulls, rather than assuming that the global null is true.
Why the Closure Rule Works
Suppose the set of indices of the true elementary nulls is \(T\), and that \(T\) is nonempty. The intersection \(H_T=\bigcap_{i\in T}H_i\) is true: every component null in it is true. If closed testing rejects any true elementary null \(H_i\), then \(i\in T\). Its rejection requires rejection of every intersection containing it, including \(H_T\). Thus, making any false rejection among the true elementary hypotheses implies that the local test of the true intersection \(H_T\) rejected.
Taking probabilities and using validity of the local test for \(H_T\) gives the desired bound:
If \(T\) is empty, there are no true elementary nulls to reject, so the FWER is zero. These cases cover every configuration and complete the justification. The proof does not require independence among test statistics or p-values. It does require that the local test used for the true intersection really is valid under that intersection.
Worked Example: Applying the Rule to Three Treatment Claims
Suppose a hypothetical trial has elementary nulls \(H_1,H_2,H_3\) for three treatment comparisons. At a prespecified analysis, the valid local tests produce the following p-values. Assume all tests use \(\alpha=0.05\).
| Intersection tested | Local-test p-value |
|---|---|
| \(H_1\) | 0.008 |
| \(H_2\) | 0.021 |
| \(H_3\) | 0.180 |
| \(H_{12}\) | 0.014 |
| \(H_{13}\) | 0.032 |
| \(H_{23}\) | 0.110 |
| \(H_{123}\) | 0.025 |
The test of an intersection rejects at level \(0.05\) when its p-value is at most \(0.05\). For \(H_1\), all four required tests reject: \(H_1\) has p-value \(0.008\), \(H_{12}\) has \(0.014\), \(H_{13}\) has \(0.032\), and \(H_{123}\) has \(0.025\). Therefore closed testing rejects \(H_1\).
For \(H_2\), the required intersections include \(H_{23}\), whose p-value is \(0.110>0.05\). Thus \(H_2\) is not rejected, even though its singleton test has p-value \(0.021\). For \(H_3\), its own singleton p-value is \(0.180>0.05\), so it is not rejected. The closed-testing conclusions are that there is evidence to reject \(H_1\), but not enough to reject \(H_2\) or \(H_3\) under this procedure.
Closed-Testing Adjusted P-Values
For a fixed set of local tests, the closed-testing adjusted p-value for \(H_i\) is the largest local p-value among all intersections containing \(i\). This maximum is the smallest common significance level at which every required intersection test would reject. Therefore \(H_i\) is rejected at level \(\alpha\) exactly when its adjusted p-value is at most \(\alpha\).
This calculation assumes the local p-values correspond to valid local tests and are defined on the same significance scale. The maximum is not an average or a combination of the p-values: it records the most demanding intersection that must be passed for that elementary rejection.
Worked Example: Calculating Closed-Testing Adjusted P-Values
Consider three hypothetical elementary p-values \(0.006\), \(0.018\), and \(0.041\). Suppose each intersection is tested by a Bonferroni local test using its component elementary p-values. For a set \(I\), the local p-value is \(p_I=\min\{1,\ |I|\min_{i\in I}p_i\}\), where \(|I|\) is the number of nulls in that intersection. This is a valid level-\(\alpha\) intersection test if each component p-value is valid under its own null, because under \(H_I\) all \(|I|\) component nulls are true and the union bound applies.
The pairwise local p-values are \(p_{12}=2(0.006)=0.012\), \(p_{13}=2(0.006)=0.012\), and \(p_{23}=2(0.018)=0.036\). For the three-way intersection, \(p_{123}=3(0.006)=0.018\). The singleton local p-values are the given values.
For \(H_1\), the maximum over its containing intersections is \(\max(0.006,0.012,0.012,0.018)=0.018\). For \(H_2\), it is \(\max(0.018,0.012,0.036,0.018)=0.036\). For \(H_3\), it is \(\max(0.041,0.012,0.036,0.018)=0.041\). At \(\alpha=0.05\), all three adjusted p-values are below \(0.05\), so closed testing rejects all three elementary nulls. These decisions are familywise-error controlled because every intersection used the specified valid Bonferroni local test.
What Must Be Checked in Practice?
The theorem is only as sound as its local tests. For each intersection, the analyst must identify what null is being tested, specify the test statistic and rejection rule, and establish that its type I error is at most \(\alpha\) whenever that intersection is true. A test calibrated only under the global null is not automatically valid for every smaller intersection. Likewise, a p-value that is valid for a single elementary null cannot simply be treated as a valid p-value for an intersection without a justified construction.
In a MAMS setting, the analysis plan should also make clear which comparisons belong to the family and what data and analysis time each local test uses. Closed testing handles multiplicity across the specified hypotheses; it does not by itself correct invalid sampling assumptions, unplanned outcome selection, or repeated looks over time. If a local test is designed for a particular stage or analysis schedule, its validity must match that schedule.
A practical limitation is computational: the number of intersections grows exponentially with the number of elementary hypotheses. Some structures permit shortcuts or equivalent procedures, but any shortcut must preserve the closure decisions. Closed testing is also a framework rather than a single test: its power and conclusions depend on the local tests selected. A more powerful local test may help, but it must remain valid for its intersection null.
How It Differs from Other Multiplicity Procedures
Closed testing specifies a general decision principle. Holm’s step-down procedure is a particular method that can be understood through closed-testing logic with suitable Bonferroni local tests; it offers a convenient ordered calculation rather than requiring every intersection to be tested separately. The Bonferroni procedure tests elementary hypotheses using allocated thresholds, while closure can use different valid tests for different intersections. The Simes test, covered in the earlier material on the Simes test, can serve as a local test for an intersection when its assumptions and validity conditions hold; using it locally does not remove the requirement to apply the closure rule.
A significant test of the global intersection is therefore not an alternative to the full closed procedure when the goal is to identify individual treatments. It establishes evidence against the joint null, but elementary rejections still require all containing intersections to pass their local tests.
Worked Example: Why a Global Rejection Is Not Enough
For a hypothetical two-comparison family, suppose the global intersection \(H_{12}\) has a valid local p-value of \(0.020\), while the singleton p-values are \(0.090\) for \(H_1\) and \(0.300\) for \(H_2\). At \(\alpha=0.05\), the global test rejects because \(0.020\leq0.05\). But rejection of \(H_1\) requires both its singleton test and the global test to reject; since \(0.090>0.05\), \(H_1\) is not rejected. Rejection of \(H_2\) also fails because \(0.300>0.05\). Thus the global result supports a conclusion against the joint null, but it does not identify either individual comparison as significant under closed testing.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For four elementary nulls, how many nonempty intersections are in the closure if all are distinct?
- At \(\alpha=0.05\), the local p-values for all intersections containing \(H_2\) are \(0.012\), \(0.047\), \(0.051\), and \(0.030\). Is \(H_2\) rejected?
- Why does closed testing control FWER when some elementary nulls are false?
- In a two-hypothesis family, the global intersection test rejects, but the singleton test for \(H_1\) does not. Can closed testing reject \(H_1\)? Explain.
- What validity condition must hold for a local test of \(H_I\)?
Solutions. (1) There are \(2^4-1=15\) nonempty index sets. (2) No. At least one required local p-value is \(0.051>0.05\), so not every intersection containing \(H_2\) is rejected. (3) The intersection of all true elementary nulls is itself true. Rejecting any true elementary null requires rejecting that true intersection, whose valid local test rejects with probability at most \(\alpha\). (4) No. The singleton \(H_1\) is one of the intersections that must be rejected, and it failed its local test. (5) Under every parameter value for which \(H_I\) is true, its probability of rejection must be at most \(\alpha\).