Closure for Three Hypotheses

Why Three Hypotheses Require Seven Tests

Suppose a trial has three confirmatory questions, represented by elementary null hypotheses $H_1$, $H_2$, and $H_3$. For example, these might be null hypotheses of no benefit on three prespecified clinical endpoints. Testing each one separately at significance level $\alpha$ does not, by itself, control the probability of making one or more false rejections across the family. Closed testing addresses this by requiring evidence against every relevant intersection null, not merely against the elementary null being considered.

With three hypotheses, there are seven nonempty subsets of $\{1,2,3\}$. Each subset defines an intersection hypothesis. A valid closed-testing decision requires a local test for each of these seven intersections. The tests can use different statistics, provided each is valid for its own null hypothesis at the stated level.

Definition: For every nonempty subset $I$ of $\{1,2,3\}$, define the intersection null hypothesis $H_I=\bigcap_{i\in I}H_i$. A level-$\alpha$ local test for $H_I$ must satisfy $P(\text{reject }H_I)\leq\alpha$ whenever $H_I$ is true. The closed procedure rejects an elementary hypothesis $H_i$ only if it rejects every intersection $H_I$ for which $i\in I$.

The seven tests are as follows. A “pair” test concerns the null that both named elementary nulls hold; the three-way test concerns the null that all three hold.

Local test	Intersection null	Elementary hypotheses included
Singleton 1	$H_{\{1\}}=H_1$	$H_1$
Singleton 2	$H_{\{2\}}=H_2$	$H_2$
Singleton 3	$H_{\{3\}}=H_3$	$H_3$
Pair 12	$H_{\{1,2\}}=H_1\cap H_2$	$H_1,H_2$
Pair 13	$H_{\{1,3\}}=H_1\cap H_3$	$H_1,H_3$
Pair 23	$H_{\{2,3\}}=H_2\cap H_3$	$H_2,H_3$
Three-way	$H_{\{1,2,3\}}=H_1\cap H_2\cap H_3$	$H_1,H_2,H_3$

Local Results and the Closure Rule

Let $p_I$ be the p-value from the local test of $H_I$. At a common significance level $\alpha$, that local test rejects when $p_I\leq\alpha$. For example, $H_1$ is rejected by the closed procedure only when the tests of $H_1$, $H_1\cap H_2$, $H_1\cap H_3$, and $H_1\cap H_2\cap H_3$ all reject. Rejection of the singleton test alone is not enough.

Key result: For each elementary hypothesis $H_i$, the closed-testing p-value is the largest p-value among the intersection tests whose index sets contain $i$: $$ p_{\mathrm{closed},i}=\max_{I:\,i\in I}p_I. $$ Reject $H_i$ at level $\alpha$ exactly when $p_{\mathrm{closed},i}\leq\alpha$. Equivalently, every required intersection test must reject.

For three hypotheses, the calculation is explicit: $$ p_{\mathrm{closed},1}=\max(p_1,p_{12},p_{13},p_{123}),\qquad p_{\mathrm{closed},2}=\max(p_2,p_{12},p_{23},p_{123}),\qquad p_{\mathrm{closed},3}=\max(p_3,p_{13},p_{23},p_{123}). $$

These maxima are useful summaries, but they do not replace the need for seven valid local tests. A small p-value for one intersection cannot compensate for a large p-value for another required intersection.

Worked Example: Two Elementary Nulls Survive Closure

Hypothetical data. A trial tests three prespecified endpoint hypotheses at $\alpha=0.05$. Suppose the seven valid local tests produce $p_1=0.012$, $p_2=0.041$, $p_3=0.20$, $p_{12}=0.030$, $p_{13}=0.018$, $p_{23}=0.049$, and $p_{123}=0.025$. Here $p_{12}$, for example, is the p-value for $H_1\cap H_2$.

For $H_1$, all four required local p-values are at most 0.05. The closed p-value is $$ p_{\mathrm{closed},1}=\max(0.012,0.030,0.018,0.025)=0.030. $$ Thus $H_1$ is rejected.

For $H_2$, the calculation is $$ p_{\mathrm{closed},2}=\max(0.041,0.030,0.049,0.025)=0.049. $$ Thus $H_2$ is also rejected, although its closed p-value is close to the 0.05 threshold.

For $H_3$, $$ p_{\mathrm{closed},3}=\max(0.20,0.018,0.049,0.025)=0.20. $$ The singleton test of $H_3$ does not reject, so closure does not reject $H_3$. The procedure’s decisions are to reject $H_1$ and $H_2$, but not $H_3$.

Why the Closed Decisions Control Familywise Error

The local tests must be valid at level $\alpha$ under their respective intersection nulls. More precisely, for every nonempty $I$, $$ \sup_{\theta\in H_I}P_{\theta}(\text{local test rejects }H_I)\leq\alpha. $$ This requirement allows the hypotheses to be composite and does not require the endpoint test statistics to be independent. It does require the p-values to be calibrated for the nulls they actually test.

A short argument explains why this condition is enough. Let $T$ be the set of indices of the elementary null hypotheses that are true. If at least one true elementary null is falsely rejected, choose an index $i\in T$ that was rejected. Closure then requires rejection of every intersection containing $i$, including $H_T=\bigcap_{j\in T}H_j$. Therefore, the event of one or more false elementary rejections is contained in the event that the local test of the true intersection $H_T$ rejects:

\{\text{at least one true }H_i\text{ is rejected}\} \subseteq \{\text{reject }H_T\}.

Since $H_T$ is true and its local test has level $\alpha$, the probability of the event on the right is at most $\alpha$. The probability of making one or more false elementary rejections is therefore also at most $\alpha$. This is strong familywise error control: the guarantee applies for every configuration of true and false elementary nulls.

Worked Example: A Nonrejected Global Intersection Blocks Every Claim

Hypothetical data. Consider a three-endpoint trial at $\alpha=0.05$, with valid local p-values $p_1=0.001$, $p_2=0.010$, $p_3=0.020$, $p_{12}=0.015$, $p_{13}=0.030$, $p_{23}=0.040$, and $p_{123}=0.080$.

Each singleton and pairwise local test rejects at 0.05, but the three-way test does not, because $0.080>0.05$. The global intersection is required for all three elementary decisions. For example, $$ p_{\mathrm{closed},1}=\max(0.001,0.015,0.030,0.080)=0.080. $$ Similarly, $p_{\mathrm{closed},2}=\max(0.010,0.015,0.040,0.080)=0.080$, and $p_{\mathrm{closed},3}=\max(0.020,0.030,0.040,0.080)=0.080$. None of the three elementary hypotheses is rejected by closure.

This result is not a contradiction: the local tests answer different null-hypothesis questions. Closure deliberately requires the global intersection to be rejected before any individual claim can pass.

Using the Seven Outcomes Systematically

A reliable analysis keeps the local decisions separate from the final elementary decisions. First record which of the seven intersection tests reject. Then, for each elementary hypothesis, check its four required tests. The flow below is a useful bookkeeping method.

List all seven intersections.
Include the three singletons, three pairs, and the three-way intersection. Confirm that each has a defined local test.

Apply the local decision rule.
At the chosen $\alpha$, mark intersection $I$ rejected exactly when $p_I\leq\alpha$.

Check each elementary hypothesis.
For $H_i$, inspect its singleton, both pair intersections containing it, and the three-way intersection.

Report only closed rejections.
Reject $H_i$ only if all four required tests reject; otherwise, do not reject it by the closed procedure.

Worked Example: Only the First Hypothesis Passes Closure

Hypothetical data. For three clinical endpoint hypotheses at $\alpha=0.05$, suppose the seven local p-values are $p_1=0.010$, $p_2=0.070$, $p_3=0.20$, $p_{12}=0.030$, $p_{13}=0.040$, $p_{23}=0.11$, and $p_{123}=0.049$.

The closed p-value for $H_1$ is $$ p_{\mathrm{closed},1}=\max(0.010,0.030,0.040,0.049)=0.049, $$ so $H_1$ is rejected. For $H_2$, $$ p_{\mathrm{closed},2}=\max(0.070,0.030,0.11,0.049)=0.11, $$ so $H_2$ is not rejected. For $H_3$, $$ p_{\mathrm{closed},3}=\max(0.20,0.040,0.11,0.049)=0.20, $$ so $H_3$ is not rejected. The fact that the three-way test rejects does not by itself establish any individual claim; each elementary hypothesis must pass its own four-test check.

Diagnostics, Alternatives, and Limitations

Several checks help catch errors before interpreting the results:

Check completeness. There must be seven local tests. Omitting a pairwise intersection can make an elementary decision impossible to verify correctly.
Check the null being tested. A local test must address the stated intersection null, not a different endpoint set or population. In particular, a test of one endpoint is not automatically a valid test of a conjunction of endpoint nulls.
Check local calibration. Every local test must control its own Type I error at $\alpha$. Closure cannot repair a local test whose p-value is not valid under its intersection null.
Check the boundary consistently. If the procedure uses $p_I\leq\alpha$, a p-value equal to $\alpha$ counts as rejection. Apply the same convention to all seven tests.
Do not infer an endpoint result from the global test alone. A rejected three-way intersection is only one of the four required rejections for each elementary hypothesis.

Compared with applying a simple Bonferroni adjustment to the three elementary p-values, closed testing can use the structure of each intersection. Its local tests may use procedures tailored to the hypotheses in that intersection, which can provide more informative decisions than a single uniform adjustment. If Bonferroni local tests are used for every intersection, the resulting closed procedure corresponds to the Holm step-down procedure for the elementary hypotheses. The choice of local tests therefore matters: closure is a framework for combining valid local tests, not a method that automatically makes arbitrary tests valid.

There are practical costs. Three hypotheses require seven local tests, and larger families require many more intersections. Moreover, the validity of each test must be justified under its own null, including any relevant data-dependent design features. A closed-testing calculation is only as trustworthy as those local tests and their calibration.

Common mistake: “The singleton test of $H_1$ has $p_1=0.01$, so $H_1$ is significant.” This ignores the other intersections containing $H_1$. The correction is to compute $\max(p_1,p_{12},p_{13},p_{123})$ and reject $H_1$ only if that maximum is at most $\alpha$.

Check Your Understanding

Answer the questions, then compare your reasoning with the solutions.

How many local tests are needed for closed testing with three elementary hypotheses, and what are the three types of intersections?
At $\alpha=0.05$, suppose $p_1=0.01$, $p_{12}=0.04$, $p_{13}=0.03$, and $p_{123}=0.06$. Can the closed procedure reject $H_1$?
Why does closure control the probability of one or more false elementary rejections when every local test is valid at level $\alpha$?
Suppose all seven local p-values are below 0.05. What are the closed-testing decisions for $H_1,H_2,H_3$?

Solutions. (1) Seven tests are needed: three singleton tests, three pairwise intersection tests, and one three-way intersection test. (2) No. The closed p-value for $H_1$ is $\max(0.01,0.04,0.03,0.06)=0.06$, which exceeds 0.05. (3) If any true elementary null is falsely rejected, closure also rejects the intersection of all true elementary nulls. That intersection is true, and its valid local test rejects with probability at most $\alpha$; hence the probability of any false elementary rejection is at most $\alpha$. (4) Each elementary hypothesis has four required intersections, all of which reject, so the procedure rejects all three elementary hypotheses.

Local test	Intersection null	Elementary hypotheses included
Singleton 1	\(H_{\{1\}}=H_1\)	\(H_1\)
Singleton 2	\(H_{\{2\}}=H_2\)	\(H_2\)
Singleton 3	\(H_{\{3\}}=H_3\)	\(H_3\)
Pair 12	\(H_{\{1,2\}}=H_1\cap H_2\)	\(H_1,H_2\)
Pair 13	\(H_{\{1,3\}}=H_1\cap H_3\)	\(H_1,H_3\)
Pair 23	\(H_{\{2,3\}}=H_2\cap H_3\)	\(H_2,H_3\)
Three-way	\(H_{\{1,2,3\}}=H_1\cap H_2\cap H_3\)	\(H_1,H_2,H_3\)

What You'll Learn