Nonconsonant Closed Tests

Why a Rejected Intersection Need Not Reject an Elementary Hypothesis

Closed testing can reject a composite intersection hypothesis without rejecting any of the individual hypotheses inside it. This is not a contradiction: the intersection test answers whether all of its component null hypotheses can remain true, while an elementary rejection requires stronger evidence across every intersection that contains that elementary hypothesis.

The distinction matters when interpreting a family of trial endpoints or treatment comparisons. A significant global test is not automatically evidence that any particular endpoint has passed the closed-testing procedure. Whether an elementary hypothesis is rejected depends on the full set of local tests, not just the test of the global intersection.

Definition: A closed-testing procedure specifies a level-\(\alpha\) local test for every nonempty intersection \(H_I=\bigcap_{i\in I} H_i\) of the elementary hypotheses. It rejects an elementary hypothesis \(H_i\) only if it rejects every intersection \(H_I\) for which \(i\in I\). The procedure is consonant if rejection of any intersection entails rejection of at least one elementary hypothesis within that intersection. It is nonconsonant if some intersection is rejected but none of its constituent elementary hypotheses is rejected.

Consonance is a property of the decisions produced by a particular closed-testing procedure, not a requirement for strong familywise error control. The closure principle gives strong control when every local test is valid at level \(\alpha\); it does not require that every rejected intersection lead to an elementary rejection. The earlier material on closure establishes the rejection rule: all intersections containing \(H_i\) must be rejected before \(H_i\) itself is rejected.

A Fisher Closed Test for Three Hypotheses

We use three elementary null hypotheses, \(H_1,H_2,H_3\), with corresponding p-values \(p_1,p_2,p_3\). For each nonempty subset \(I\) of \(\{1,2,3\}\), test \(H_I\) by combining the p-values whose indices are in \(I\). Fisher’s statistic is

T_I=-2\sum_{i\in I}\ln(p_i).

For this example, assume that whenever \(H_I\) is true, its component p-values are mutually independent and each has an exact Uniform\((0,1)\) distribution under the null. Under those conditions, \(T_I\) has a chi-square distribution with \(2|I|\) degrees of freedom under \(H_I\). Reject \(H_I\) when its Fisher-test p-value is at most \(\alpha=0.05\). The singleton test for \(H_i\) is just the ordinary p-value test \(p_i\leq 0.05\).

Assumption check: Fisher’s chi-square calibration here relies on independence and exact uniformity of the p-values under the relevant intersection null. Valid marginal p-values alone do not guarantee that the combined statistic has this chi-square distribution. Dependence, or discrete p-values that are only super-uniform, requires a justified calibration rather than automatically using the reference distribution below.

The reference distribution follows from a short argument. If \(U\sim\mathrm{Uniform}(0,1)\), then for \(x\geq 0\),

P(-2\ln U\geq x) =P(U\leq e^{-x/2}) =e^{-x/2}.

This is the survival function of a chi-square random variable with 2 degrees of freedom. Under the stated independence assumption, the terms \(-2\ln(p_i)\) for the true component nulls are independent, so their sum has a chi-square distribution with \(2|I|\) degrees of freedom. This justifies the local test’s reference distribution; it does not remove the need to check independence and null calibration.

Worked Example: The Global Intersection Is Rejected, but No Elementary Hypothesis Is

Worked Example: Three Equal P-Values

Suppose a hypothetical trial produces \(p_1=p_2=p_3=0.0600\), and the assumptions for Fisher’s test stated above hold. Use closed testing at \(\alpha=0.05\). Is the global intersection rejected, and are any elementary hypotheses rejected?

Singleton intersections. Each local singleton test has p-value \(0.0600\), which is greater than \(0.05\). Thus none of \(H_1,H_2,H_3\) is rejected by its singleton test.

Pair intersections. For any pair, the product of the two p-values is \(0.0600^2=0.003600\). For example, for \(H_{\{1,2\}}\),

T_{\{1,2\}}=-2\ln(0.003600)=11.253642867\ldots

This statistic has a chi-square distribution with 4 degrees of freedom under \(H_{\{1,2\}}\). For a chi-square variable with 4 degrees of freedom, the upper-tail probability at \(T\) is \(e^{-T/2}(1+T/2)\). Therefore,

p_{\{1,2\}} =0.003600\left(1+\frac{11.253642867\ldots}{2}\right) =0.0238566\ldots .

The pair intersection is rejected because its p-value is below \(0.05\). The same calculation applies to \(H_{\{1,3\}}\) and \(H_{\{2,3\}}\), since all three observed p-values are equal.

Global intersection. For \(H_{\{1,2,3\}}\), the product is \(0.0600^3=0.0002160\). Thus

T_{\{1,2,3\}}=-2\ln(0.0002160)=16.880464300\ldots .

This statistic has a chi-square distribution with 6 degrees of freedom. Its upper-tail probability is \(e^{-T/2}\{1+(T/2)+(T/2)^2/2\}\), so

p_{\{1,2,3\}} =0.0002160\left(1+8.440232150\ldots+ \frac{(8.440232150\ldots)^2}{2}\right) \approx 0.009733.

The global intersection is rejected. Yet no elementary hypothesis is rejected: each \(H_i\) fails its singleton test, which is one of the required tests in its closure. This is a nonconsonant outcome. It says the local test rejects the joint null that all three hypotheses are true; it does not identify an elementary hypothesis that passes closed testing.

Worked Example: A Pair Intersection Is Rejected Without Either Member

Worked Example: A Rejected Pair with a Third, Less Significant Result

Consider hypothetical p-values \(p_1=0.0700\), \(p_2=0.0700\), and \(p_3=0.300\), again using Fisher local tests under the stated assumptions. Focus on \(H_{\{1,2\}}\).

Neither \(H_1\) nor \(H_2\) passes its singleton test, since \(0.0700>0.0500\). For their intersection, the product is \(0.0700^2=0.004900\), giving

T_{\{1,2\}}=-2\ln(0.004900)=10.637040\ldots .

With 4 degrees of freedom, its p-value is

p_{\{1,2\}} =0.004900\left(1+\frac{10.637040\ldots}{2}\right) \approx 0.03096.

Thus \(H_{\{1,2\}}\) is rejected at level \(0.05\), even though neither of its elementary members is. The closure rule does not convert that pair rejection into rejection of \(H_1\) or \(H_2\): each elementary hypothesis already fails the singleton condition. The third p-value does not change this specific pair calculation.

Worked Example: Checking the Full Pattern of Decisions

Worked Example: Which Intersections Pass?

For another hypothetical dataset, let \(p_1=0.0600\), \(p_2=0.0600\), and \(p_3=0.200\). At level \(0.05\), determine whether the global intersection and each pair intersection are rejected, then decide whether any elementary hypothesis is rejected.

The singleton tests do not reject: \(0.0600>0.0500\) for \(H_1,H_2\), and \(0.200>0.0500\) for \(H_3\). The pair \(H_{\{1,2\}}\) has the same calculation as in the first example, so its p-value is \(0.0238566\) and it is rejected.

For the global intersection, the product is \(0.0600(0.0600)(0.200)=0.0007200\). Therefore,

T_{\{1,2,3\}}=-2\ln(0.0007200)=14.472519\ldots .

With 6 degrees of freedom, its p-value is

0.0007200\left(1+7.236259\ldots+ \frac{(7.236259\ldots)^2}{2}\right) \approx 0.02478.

So the global intersection is rejected. For either remaining pair, the product is \(0.0600(0.200)=0.01200\), and the 4-degree-of-freedom p-value is

0.01200\left(1+\frac{-2\ln(0.01200)}{2}\right) =0.01200\{1-\ln(0.01200)\} \approx 0.06507.

Neither \(H_{\{1,3\}}\) nor \(H_{\{2,3\}}\) is rejected. The full decision pattern is therefore: reject \(H_{\{1,2\}}\) and \(H_{\{1,2,3\}}\), but do not reject the other intersections or any singleton. In particular, the rejected global intersection does not make \(H_1\) or \(H_2\) rejectable, because their singleton tests fail; \(H_3\) also fails its singleton test.

Why Strong Error Control Does Not Require Consonance

The key distinction is between the validity of local tests and the logical pattern of their rejections. If every local test has type I error at most \(\alpha\) whenever its intersection null is true, closed testing controls the familywise error rate at level \(\alpha\). To see the logic briefly, consider any configuration of true and false elementary nulls. The intersection of all the true elementary nulls is itself true. If the procedure falsely rejects any true elementary hypothesis, the closure rule requires rejection of that all-true intersection. Its valid local test rejects with probability at most \(\alpha\). Hence the probability of at least one false elementary rejection is at most \(\alpha\).

This argument constrains false elementary rejections. It does not require that a rejected intersection contain a rejected elementary hypothesis. Nonconsonance can therefore occur while strong familywise error control remains intact. An intersection result may be informative about the family as a whole, but it does not by itself provide a closed-testing claim for a specific endpoint.

Comparison, Diagnostics, and Limitations

One common alternative is Bonferroni-based closed testing. For an intersection containing \(m\) hypotheses, its local Bonferroni test rejects when at least one component p-value is at most \(\alpha/m\). The resulting closed procedure is equivalent to the Holm step-down procedure, but it need not be consonant. With three p-values all equal to \(0.0600\), the pairwise Bonferroni threshold is \(0.0250\), and the global threshold is approximately \(0.01667\); neither the pairs nor the global intersection would be rejected. Fisher combination can detect a joint pattern of moderately small p-values that this Bonferroni test does not. The choice depends on the scientific question and the validity of the assumptions, not on a general preference for one combination method.

Consonance is not guaranteed merely by choosing a different local test. A max-type or other intersection test must be evaluated as part of the full closed procedure to determine its rejection pattern. Similarly, Fisher’s greater sensitivity to several modest p-values comes with a calibration requirement: its usual chi-square reference distribution is not justified by marginal p-value validity alone.

Useful checks when applying a closed test include:

List the local test and reference distribution for every intersection used in the closure decision.
Verify that the calibration assumptions apply under each intersection null, particularly dependence and discreteness.
Keep local intersection rejection separate from elementary rejection; for the latter, check every intersection containing the elementary hypothesis.
Do not infer a specific endpoint claim from rejection of the global intersection alone.

Common mistake: “The global intersection was rejected, so at least one elementary hypothesis must be rejected.” That conclusion assumes consonance and is not true for every closed-testing procedure. The correction is to apply the closure rule to each elementary hypothesis separately; if even its singleton test fails, that elementary hypothesis cannot be rejected, regardless of other intersection decisions.

Check Your Understanding

Answer each question using the distinction between intersection rejection and elementary rejection. Brief solutions follow the questions.

In the first worked example, why does rejection of \(H_{\{1,2,3\}}\) not imply rejection of \(H_1\)?
Under what assumptions does Fisher’s statistic for an intersection of \(m\) hypotheses have a chi-square distribution with \(2m\) degrees of freedom?
For p-values \(0.0700,0.0700,0.300\), what is the approximate Fisher p-value for the pair intersection of the first two hypotheses, and what is the decision at level \(0.05\)?
Does a nonconsonant rejection pattern by itself show that strong familywise error control has failed? Explain.
With three p-values all equal to \(0.0600\), what is the Bonferroni local threshold for the global intersection, and would its local test reject?

Solutions. (1) Rejection of an elementary hypothesis requires rejection of every intersection containing it, including its singleton test; \(p_1=0.0600>0.0500\), so \(H_1\) is not rejected. (2) The component p-values must be mutually independent and exactly Uniform\((0,1)\) under the intersection null. (3) The pair p-value is approximately \(0.03096\), so the pair intersection is rejected, although neither singleton is. (4) No. Strong control follows from validity of each local test and the closure rule; consonance is not required. (5) The threshold is \(0.05/3\approx0.01667\). Since the smallest p-value is \(0.0600\), the Bonferroni global local test does not reject.

What You'll Learn