From Local Tests to Elementary Decisions
A local test answers one question: whether to reject a particular null hypothesis, often an intersection null such as “none of these three treatments differs from control.” A family of elementary hypotheses, however, calls for decisions about individual treatment comparisons. The closed testing procedure connects these two levels: an elementary hypothesis is rejected only when the local tests support rejection of every intersection hypothesis that contains it.
The key practical task is to make the full set of intersections explicit before looking at the decisions. With three elementary hypotheses, that means seven local tests. Leaving out even one intersection breaks the stated closed-testing rule and can invalidate its familywise error guarantee.
For three comparisons, the intersections are \(H_1,H_2,H_3,H_{12},H_{13},H_{23},H_{123}\), where, for example, \(H_{12}=H_1\cap H_2\). An elementary hypothesis occurs in its own singleton intersection, in each pair that includes it, and in the full intersection. Thus, to reject \(H_1\), the procedure must reject \(H_1,H_{12},H_{13}\), and \(H_{123}\). Rejecting just the global intersection \(H_{123}\) is not enough.
Assumptions and the Decision Rule
Each local test must be valid for the null hypothesis it tests. In p-value terms, a local p-value \(p_I\) for \(H_I\) is valid if, whenever \(H_I\) is true, \(P(p_I\leq t)\leq t\) for every \(t\) between 0 and 1. In particular, the test that rejects \(H_I\) when \(p_I\leq\alpha\) has type I error probability no greater than \(\alpha\) under \(H_I\). The local tests may use different statistics and need not be independent. Their validity under their respective intersection nulls is the essential condition.
When every local test is expressed through a p-value and uses the same level \(\alpha\), the closure rule can be summarized with one number for each elementary hypothesis: the largest local p-value among intersections containing that hypothesis. This is the closed-testing adjusted p-value. The maximum appears because all of the relevant local tests must pass the threshold.
This adjusted p-value is a convenient summary of the closure decision, not a new local test. It is the smallest common level at which all local tests in the closure for \(H_i\) would reject. If one required local p-value exceeds the level, the maximum exceeds it too, and \(H_i\) is not rejected.
Worked Example: Listing and Applying All Seven Local Tests
Suppose a hypothetical three-arm trial compares each of three treatments with control. Let \(H_1,H_2,H_3\) denote the three elementary no-difference null hypotheses. At \(\alpha=0.05\), the prespecified, valid local tests produce the following hypothetical p-values:
| Intersection | \(H_1\) | \(H_2\) | \(H_3\) | \(H_{12}\) | \(H_{13}\) | \(H_{23}\) | \(H_{123}\) |
|---|---|---|---|---|---|---|---|
| Local p-value | 0.012 | 0.031 | 0.200 | 0.028 | 0.018 | 0.044 | 0.025 |
For \(H_1\), the relevant intersections are \(H_1,H_{12},H_{13},H_{123}\). Therefore, \( \widetilde p_1=\max(0.012,0.028,0.018,0.025)=0.028 \), which is less than 0.05, so reject \(H_1\).
For \(H_2\), the relevant intersections are \(H_2,H_{12},H_{23},H_{123}\). Thus, \( \widetilde p_2=\max(0.031,0.028,0.044,0.025)=0.044 \), so reject \(H_2\). For \(H_3\), the relevant intersections are \(H_3,H_{13},H_{23},H_{123}\), giving \( \widetilde p_3=\max(0.200,0.018,0.044,0.025)=0.200 \). Do not reject \(H_3\).
The closed procedure therefore rejects \(H_1\) and \(H_2\), but not \(H_3\). These decisions concern evidence against the corresponding no-difference nulls under the specified testing procedure; they do not establish that a treatment causes a benefit.
Why the Closure Controls Strong Familywise Error
Strong familywise error control means that, for every configuration of true and false elementary null hypotheses, the probability of rejecting at least one true elementary null is no greater than \(\alpha\). The argument uses the local test of the intersection of all true elementary hypotheses. It does not require independence among the local tests or assume that all elementary nulls are true.
Let \(T\) be the nonempty set of indices for which \(H_i\) is true in the actual parameter configuration. The intersection \(H_T=\bigcap_{i\in T}H_i\) is then true.
If the procedure falsely rejects any true \(H_i\), where \(i\in T\), it must reject every intersection containing \(i\). In particular, it must reject \(H_T\), since \(i\in T\).
The probability of rejecting \(H_T\) when it is true is at most \(\alpha\), because its local test is level \(\alpha\). The event of any false elementary rejection is contained in this event, so its probability is also at most \(\alpha\).
In probability notation, if \(E\) is the event of at least one false elementary rejection, then \(E\subseteq\{\text{reject }H_T\}\). Consequently, \(P(E)\leq P(\text{reject }H_T)\leq\alpha\). If no elementary null is true, there can be no false elementary rejection, and the familywise error probability is zero. This completes the justification for strong control under the stated validity condition.
Worked Example: A Pairwise Test Blocks Two Comparisons
In a second hypothetical three-treatment analysis, valid local tests at \(\alpha=0.05\) yield \(p_1=0.010\), \(p_2=0.040\), \(p_3=0.030\), \(p_{12}=0.060\), \(p_{13}=0.020\), \(p_{23}=0.049\), and \(p_{123}=0.040\). Apply closure to decide which elementary nulls can be rejected.
For \(H_1\), the maximum is \( \widetilde p_1=\max(0.010,0.060,0.020,0.040)=0.060 \). For \(H_2\), \( \widetilde p_2=\max(0.040,0.060,0.049,0.040)=0.060 \). Both exceed 0.05, so neither \(H_1\) nor \(H_2\) is rejected. In each case, the local test of \(H_{12}\) is the blocker.
For \(H_3\), \( \widetilde p_3=\max(0.030,0.020,0.049,0.040)=0.049 \). Since \(0.049\leq0.05\), reject \(H_3\). Thus only the third elementary null is rejected, even though the marginal p-values for the first and second comparisons are each below 0.05.
Reading the Results and Checking the Construction
The local p-values must be interpreted against their own nulls. For example, \(p_{12}\) is evidence assessed by a test of the joint intersection \(H_1\cap H_2\), not automatically a p-value for either elementary comparison by itself. A small p-value for an intersection can be necessary for an elementary rejection, but the closure rule still requires all the other local tests containing that elementary hypothesis to reject.
The closure is finite and systematic. A reliable implementation starts with a list of every nonempty subset of elementary hypotheses, records the test and p-value for each subset, and then checks the subsets containing each elementary hypothesis. For three hypotheses, the full calculation is short enough to audit directly. For larger families, software can automate the subset bookkeeping, but the input tests and the validity of each local test still require statistical justification.
A useful diagnostic is to ask what happens if a local p-value is just above \(\alpha\). Every elementary hypothesis contained in that intersection is blocked from rejection by that local test. Conversely, a rejected intersection alone does not mean that all of its elementary hypotheses are rejected: each one has its own full set of containing intersections to pass. Local p-values also need not be ordered from smaller to larger as intersections get larger. The closure calculation uses the values supplied by the prespecified tests; it does not silently repair them or infer missing tests.
Worked Example: The Global Intersection Can Block Every Claim
A third hypothetical set of valid local tests at \(\alpha=0.05\) gives \(p_1=0.004\), \(p_2=0.006\), \(p_3=0.020\), \(p_{12}=0.008\), \(p_{13}=0.030\), \(p_{23}=0.040\), and \(p_{123}=0.051\).
For each elementary hypothesis, the full intersection \(H_{123}\) is among the required tests. The adjusted values are \( \widetilde p_1=\max(0.004,0.008,0.030,0.051)=0.051 \), \( \widetilde p_2=\max(0.006,0.008,0.040,0.051)=0.051 \), and \( \widetilde p_3=\max(0.020,0.030,0.040,0.051)=0.051 \). All three are greater than 0.05, so none of the elementary nulls is rejected. The marginal and pairwise tests may reject at 0.05, but the full closure condition is not met.
Comparison, Limitations, and Common Pitfalls
Directly testing each elementary null at level \(\alpha\) is simpler, but it does not in general control the probability of one or more false rejections across the family. Holm’s procedure is a step-down procedure that offers strong familywise error control using the elementary p-values. Closed testing is more general: it permits different, tailored local tests for different intersections, provided each is valid at level \(\alpha\). That flexibility can be useful when the intersection hypotheses have meaningful joint tests, but it requires defining and validating all those tests. Depending on the family and the local tests, closure can also be computationally demanding because the number of nonempty intersections is \(2^m-1\).
A common mistake is to test only the global intersection and then treat its rejection as permission to reject every elementary null. That reasoning omits the other intersections in each elementary hypothesis’s closure. The correction is to list all nonempty intersections and reject \(H_i\) only if every local test for an intersection containing \(i\) rejects at the prespecified level. Another error is to assume that naming a test “local” guarantees validity. The test must actually have type I error no greater than \(\alpha\) under that specific intersection null; otherwise, the strong-control argument does not apply.
Closure gives a general validity framework, not a guarantee that a particular set of local tests is powerful, scientifically appropriate, or easy to calculate. It also does not remove the need to prespecify the multiplicity family and the tests. A well-reported analysis identifies the elementary hypotheses, defines the intersections, describes the local tests, states the common level, and shows which local results determine each elementary decision.
Check Your Understanding
Answer each question, then compare with the solutions below.
- For three elementary hypotheses, how many nonempty intersection hypotheses must be tested, and which intersections are required to reject \(H_2\)?
- At \(\alpha=0.05\), suppose the four local p-values containing \(H_1\) are 0.02, 0.04, 0.03, and 0.06. What is the closed-testing adjusted p-value, and is \(H_1\) rejected?
- Why does a collection of valid level-\(\alpha\) local tests control strong familywise error without an independence assumption?
- Suppose \(p_1=0.01\), \(p_{12}=0.03\), \(p_{13}=0.02\), and \(p_{123}=0.05\). At \(\alpha=0.05\), is \(H_1\) rejected? Explain the role of equality.
Solutions. (1) There are \(2^3-1=7\) nonempty intersections. For \(H_2\), check \(H_2,H_{12},H_{23},H_{123}\). (2) The adjusted value is \(\max(0.02,0.04,0.03,0.06)=0.06\); do not reject because it exceeds 0.05. (3) Any false elementary rejection implies rejection of the intersection of all true elementary nulls. Its valid local test rejects with probability at most \(\alpha\), so the probability of any false elementary rejection is also at most \(\alpha\). This containment argument does not require independence. (4) Yes. The largest of the four values is \(0.05\), and the stated rule rejects when \(p_I\leq\alpha\), including equality.