The Closure Principle

Why Test Intersections?

In a trial with several endpoints or treatment comparisons, a conclusion about any one hypothesis is part of a family of decisions. Controlling the chance of at least one false rejection in that family requires more than a test of the single hypothesis that all nulls hold. The closure principle gives a general construction: test every nonempty intersection of the elementary null hypotheses, and reject an elementary null only when every intersection containing it is rejected.

This is useful in adaptive trials because the design may involve multiple comparisons and data-dependent decisions. Closure provides an error-control framework, but it does not by itself make a test valid after adaptation. The local tests used within the framework must have the required error guarantee under the trial’s actual design.

Definition: Let \(H_1,\ldots,H_m\) be the elementary null hypotheses in a family. For each nonempty index set \(I\subseteq\{1,\ldots,m\}\), define the intersection hypothesis \(H_I=\bigcap_{i\in I}H_i\). A collection of tests is a closed testing procedure at level \(\alpha\) if it tests every \(H_I\) with a level-\(\alpha\) local test, and rejects \(H_i\) only if it rejects every \(H_I\) for which \(i\in I\).

A test of \(H_I\) is called a local test because its decision applies to that particular intersection null. “Level \(\alpha\)” means that, whenever \(H_I\) is true, the probability that its local test rejects is at most \(\alpha\). For a composite null, this guarantee is required for every parameter value allowed by that null:

\sup_{\theta\in H_I} P_\theta(\text{local test rejects }H_I)\leq \alpha.

The familywise error rate, or FWER, is the probability of rejecting at least one true elementary null. Strong FWER control means that this probability is at most \(\alpha\) for every configuration of true and false elementary nulls—not only when all nulls are true.

The Closure Rule in Practice

For a particular \(H_i\), the relevant set of local tests consists of \(H_i\) itself and every larger intersection that includes it. With four elementary hypotheses, for example, rejecting \(H_1\) requires rejecting \(H_1\), \(H_{\{1,2\}}\), \(H_{\{1,3\}}\), \(H_{\{1,4\}}\), \(H_{\{1,2,3\}}\), \(H_{\{1,2,4\}}\), \(H_{\{1,3,4\}}\), and \(H_{\{1,2,3,4\}}\). The local test of \(H_1\) alone is not sufficient.

The number of nonempty intersections is \(2^m-1\), so the full collection can become large. The rule is nevertheless simple: a single non-rejection among the intersections containing \(H_i\) prevents rejection of \(H_i\). That conservative-looking requirement is what links an elementary conclusion to tests of the broader null configurations in which it could be false.

Worked Example: One Intersection Blocks an Elementary Rejection

Suppose a hypothetical trial has four elementary null hypotheses and uses \(\alpha=0.05\). The local p-values for the eight intersections containing \(H_1\) are:

\(p_{\{1\}}=0.012\), \(p_{\{1,2\}}=0.021\), \(p_{\{1,3\}}=0.018\), \(p_{\{1,4\}}=0.031\);
\(p_{\{1,2,3\}}=0.027\), \(p_{\{1,2,4\}}=0.043\), \(p_{\{1,3,4\}}=0.064\), \(p_{\{1,2,3,4\}}=0.038\).

Under the usual p-value rule, a local test rejects its intersection when \(p_I\leq\alpha\). Seven of these p-values are at most \(0.05\), but \(p_{\{1,3,4\}}=0.064>0.05\). Therefore the local test of \(H_{\{1,3,4\}}\) does not reject, and the closure rule does not reject \(H_1\). The small p-value for \(H_1\) by itself cannot override that failure.

This example concerns decisions, not the calibration of the local tests. To use the closure guarantee, each reported local test must in fact have level at most \(0.05\) under its own intersection null.

Why Closure Gives Strong FWER Control

The principle applies regardless of which elementary nulls happen to be true. Suppose the set of true elementary nulls is \(T\), with \(T\neq\varnothing\). Their intersection \(H_T\) is one of the hypotheses included in the closure. If the procedure rejects any true \(H_i\), where \(i\in T\), it must have rejected every intersection containing that \(H_i\), including \(H_T\). Thus the event of at least one false elementary rejection is contained within the event that the local test of \(H_T\) rejects.

\{\text{at least one true }H_i\text{ is rejected}\} \subseteq \{\text{the local test of }H_T\text{ rejects }H_T\}.

Because \(H_T\) is true, its local test rejects with probability at most \(\alpha\). The containment therefore bounds the probability of any false rejection by \(\alpha\). This is the central justification for the principle: in every possible true-null configuration, an erroneous elementary conclusion requires rejection of a particular true intersection null. The formal proof can be written as an event argument; no independence among the local tests is required.

This reasoning also explains why “strong” matters. The relevant intersection is not necessarily the intersection of the whole family. If only some elementary nulls are true, the procedure still includes and tests the intersection of precisely those true nulls. A test of the global intersection alone would not supply that protection.

Closed-Test Adjusted P-Values

If each local test is represented by a valid p-value \(p_I\), the closed-test adjusted p-value for \(H_i\) is the largest local p-value among the intersections that contain \(i\). It is the least significance level at which all the required local tests would reject. Consequently, \(H_i\) is rejected at level \(\alpha\) exactly when its adjusted p-value is at most \(\alpha\).

p_{i,\mathrm{closed}}=\max_{I:\,i\in I}p_I.

Worked Example: Computing a Closed-Test Adjusted P-Value

Consider another hypothetical four-hypothesis family. Suppose the eight local p-values for intersections containing \(H_2\) are \(0.018, 0.026, 0.033, 0.041, 0.029, 0.037, 0.022,\) and \(0.046\). Then

p_{2,\mathrm{closed}} =\max(0.018,0.026,0.033,0.041,0.029,0.037,0.022,0.046) =0.046.

At \(\alpha=0.05\), \(0.046\leq0.05\), so the closure rule rejects \(H_2\): every required local test rejects. At \(\alpha=0.04\), it does not reject \(H_2\), because the local p-value \(0.046\) exceeds that level. The adjusted p-value summarizes the most restrictive required local test; it is not the p-value from a single test of \(H_2\).

Choice of Local Tests and a Comparison

The closure principle specifies how local decisions combine, not which test must be used for each intersection. Different choices of local tests can yield different power and different elementary decisions while retaining strong FWER control, provided every local test has level at most \(\alpha\) under its own intersection null. This flexibility is important: a local test can use the structure of a particular endpoint family or comparison, but its error guarantee must match the null it tests.

For a simple illustration, suppose each elementary hypothesis has a valid p-value \(p_i\), and use a Bonferroni local test for each intersection \(I\). Its local p-value is

p_I=\min\left(1,\ |I|\min_{i\in I}p_i\right),

where \(|I|\) is the number of elementary hypotheses in \(I\). This local test rejects \(H_I\) when \(p_I\leq\alpha\). The Bonferroni inequality ensures its level is at most \(\alpha\) under \(H_I\); the resulting closed procedure therefore has strong FWER control. Other local tests may use the joint structure of the data and can be more powerful, but their validity needs to be established for the relevant intersection.

Worked Example: Bonferroni Local Tests Inside Closure

Suppose a hypothetical family has four valid elementary p-values \(0.008, 0.021, 0.044,\) and \(0.200\), with \(\alpha=0.05\). Use the Bonferroni local p-value above. For the intersection containing hypotheses 2, 3, and 4,

p_{\{2,3,4\}}=\min(1,3\times\min(0.021,0.044,0.200)) =3\times0.021=0.063.

Since \(0.063>0.05\), this local test does not reject. It contains \(H_2\), so the closure rule cannot reject \(H_2\), even though its own elementary p-value is \(0.021\). For \(H_1\), the smallest p-value among all four is \(0.008\); every intersection containing \(H_1\) has size at most four, and its Bonferroni local p-value is at most \(4\times0.008=0.032\). Thus every required intersection test rejects, and closure rejects \(H_1\). These conclusions follow from the specified local tests and the closure rule, not from an assumption that the endpoint tests are independent.

A single test of the intersection of all hypotheses answers a narrower question: whether the complete global null can be rejected. It does not, on its own, justify rejecting a particular elementary null while controlling the chance of any false familywise rejection. Closed testing differs by requiring the relevant tests of all intersections. Bonferroni and related procedures can also control familywise error; closure is a general framework that can use Bonferroni local tests or other valid local tests. It is not automatically more powerful than every alternative—the result depends on the local tests chosen.

Assumptions, Diagnostics, and Limitations

The guarantee rests on a few precise requirements. The family of elementary hypotheses must be defined, every nonempty intersection in its closure must have a specified local test, and each local test must control its rejection probability at level \(\alpha\) whenever its intersection null is true. The closure argument does not require independent test statistics, but it does require correct local calibration.

In an adaptive trial, changes to allocation, sample size, treatment selection, or analysis can affect the distribution of a local test statistic. Closure does not repair that effect. If a local p-value is not valid under the design actually used, the step that bounds the rejection probability for \(H_T\) is no longer justified. A practical review should therefore check which data and design decisions each local test uses, and whether its stated level guarantee covers those decisions.

The main costs are computation and specification. A family of \(m\) hypotheses has \(2^m-1\) nonempty intersections, although structure or efficient algorithms may reduce the work needed in practice. Also, a rejection under closure need not be accompanied by rejection of every elementary hypothesis in the same intersection. Closure guarantees familywise error control; it does not promise a particular pattern of discoveries or make a non-rejection evidence that the null is true.

Common Mistake

Wrong reasoning: “The p-value for \(H_i\) is below \(\alpha\), so \(H_i\) is rejected under closure.”
Correction: Check every local test of an intersection containing \(H_i\). One non-rejected intersection is enough to prevent rejection of \(H_i\).
Wrong reasoning: “Because closure controls familywise error, any local p-value produced by an adaptive analysis is safe to use.”
Correction: The control guarantee applies only when each local test is valid at level \(\alpha\) under its own intersection null for the design and analysis actually used.

Key takeaway: Closure combines local level-\(\alpha\) tests into a familywise decision rule: reject an elementary null only if every intersection null containing it is rejected. Valid local tests then provide strong FWER control across all true-null configurations.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In a family of four hypotheses, what local tests must reject before the procedure can reject \(H_3\)?
At \(\alpha=0.05\), the local p-values for intersections containing \(H_1\) are \(0.01, 0.03, 0.049,\) and \(0.052\), with all other required values at most \(0.05\). Is \(H_1\) rejected?
Why does the closure principle not require independent local tests?
What must be checked before using closed testing after a data-dependent change to a trial?

Solutions. (1) The local tests of every nonempty intersection containing \(H_3\), including \(H_3\) itself and the full-family intersection, must reject. (2) No: the local p-value \(0.052\) exceeds \(0.05\), so at least one required intersection is not rejected. (3) The FWER argument uses event containment and the level guarantee for the intersection of the true nulls; it does not factor joint probabilities or assume independence. (4) Verify that every local test remains level \(\alpha\) under its intersection null for the design and analysis actually used; closure cannot compensate for invalid local calibration.

What You'll Learn