Consonance in Closed Testing

Why Consonance Matters

Closed testing organizes a family of elementary hypotheses by testing their intersections. A rejection of an intersection says that its entire collection of component null hypotheses cannot all be retained under the specified local test. But that is not automatically the same as a rejection of any one component by the closed testing procedure: for an elementary hypothesis to be rejected, every intersection containing it must also be rejected. Consonance is the property that makes the link between these two kinds of rejection precise.

The distinction matters when interpreting results. A rejected intersection can be evidence against a collection of null hypotheses without identifying a component null hypothesis that passes the full closure requirement. If a closed test is consonant, that identification is guaranteed: every rejected intersection contains at least one elementary hypothesis that is itself rejected by closure.

Definition and Closed Rejection

Let the family contain \(m\) elementary hypotheses, indexed by \(M=\{1,\ldots,m\}\). For every nonempty subset \(I\subseteq M\), write \(H_I=\bigcap_{i\in I}H_i\) for the corresponding intersection hypothesis. A closed testing procedure specifies a local test for every \(H_I\), all conducted at significance level \(\alpha\). The family is closed because it includes every nonempty intersection in the family.

Definition: The closed testing procedure rejects an elementary hypothesis \(H_i\) if and only if it rejects every intersection hypothesis \(H_I\) for which \(i\in I\). The procedure is consonant at level \(\alpha\) if, whenever a local test rejects \(H_I\), at least one elementary hypothesis \(H_i\), with \(i\in I\), is rejected by the closed testing procedure.

The quantifier “every intersection containing \(i\)” is essential. Rejecting the singleton test of \(H_i\), or even the local test of one larger intersection containing \(i\), is not enough on its own to reject \(H_i\) by closure. Consonance concerns the complete pattern of local decisions, not just whether an individual local test rejects.

When local tests are reported through valid local p-values \(p_I\), the closed adjusted p-value for \(H_i\) is \(p_i^{\mathrm{cl}}=\max_{I:\,i\in I}p_I\). Thus \(H_i\) is rejected at level \(\alpha\) exactly when \(p_i^{\mathrm{cl}}\leq\alpha\). In this notation, consonance requires that every rejected intersection have at least one member \(i\) for which this maximum is at most \(\alpha\).

The Logical Connection

The definition can be written directly in terms of the sets of rejections. Let \(R(I)\) mean that the local test of \(H_I\) rejects, and let \(C_i\) mean that \(H_i\) is rejected by closure. By definition, \(C_i\) holds exactly when \(R(J)\) holds for every \(J\) containing \(i\). Consonance is therefore the implication \(R(I)\Rightarrow \text{there exists }i\in I\text{ such that }C_i\).

R(I)\ \Longrightarrow\ \bigvee_{i\in I} C_i, \qquad C_i=\bigwedge_{J:\,i\in J}R(J).

This gives a short justification of the key result. Suppose the procedure is consonant and a local test rejects \(H_I\). By consonance, some \(i\in I\) has \(C_i\). By the definition of \(C_i\), every intersection containing \(i\) is rejected. That is exactly the closure rule for rejecting \(H_i\). The implication is not that all members of \(I\) are rejected, nor that every rejected intersection produces a rejection of all its components; it guarantees at least one.

Validity and consonance answer different questions. For strong familywise error control, the standard closure result requires that each local test have type I error at most \(\alpha\) whenever its intersection null is true. Under this condition, the closed procedure controls the probability of rejecting one or more true elementary null hypotheses at no more than \(\alpha\), for every configuration of true and false nulls. Consonance is not needed for that error-control result. It describes how local intersection rejections translate into elementary rejections.

Worked Example: Checking a Three-Hypothesis Bonferroni Closure

Hypothetical data: Suppose three valid marginal p-values are \(p_1=0.004\), \(p_2=0.021\), and \(p_3=0.40\). Use the Bonferroni local test for each intersection: reject \(H_I\) when \(|I|\min_{i\in I}p_i\leq\alpha\), with \(\alpha=0.05\). Each local test is valid under its intersection null by the union bound. Compute all local p-values and determine whether \(H_1\) and \(H_2\) are rejected by closure.

For a singleton, the local p-value is the marginal p-value. For an intersection with more members, it is the smaller of 1 and the number of members multiplied by the smallest marginal p-value in that intersection: \(p_I=\min(1,|I|\min_{i\in I}p_i)\).

\begin{aligned} p_{\{1\}}&=0.004, & p_{\{2\}}&=0.021, & p_{\{3\}}&=0.40,\\ p_{\{1,2\}}&=2(0.004)=0.008, & p_{\{1,3\}}&=2(0.004)=0.008, & p_{\{2,3\}}&=2(0.021)=0.042,\\ p_{\{1,2,3\}}&=3(0.004)=0.012. \end{aligned}

The local tests reject all these intersections except \(H_{\{3\}}\), whose p-value is \(0.40>0.05\). For \(H_1\), the largest p-value among intersections containing 1 is \(0.012\), so \(p_1^{\mathrm{cl}}=0.012\leq0.05\) and closure rejects \(H_1\). For \(H_2\), that largest p-value is \(0.042\), so \(p_2^{\mathrm{cl}}=0.042\leq0.05\) and closure rejects \(H_2\). The rejected intersection \(H_{\{2,3\}}\), for example, contains the closed-rejected hypothesis \(H_2\). The calculations show the required implication for every rejected intersection in this example.

How to Check Consonance

For a finite family, consonance at a chosen \(\alpha\) can be checked from the local rejection decisions. For every rejected intersection \(H_I\), list the elementary hypotheses in \(I\), then check whether at least one of them passes all of its closure requirements. Equivalently, find whether at least one \(i\in I\) has \(p_J\leq\alpha\) for every \(J\) containing \(i\). A rejected local test alone does not complete this check.

Record local decisions.
For each nonempty \(I\), mark whether its local test rejects at the specified \(\alpha\). If using p-values, compare each \(p_I\) with \(\alpha\).

Apply closure to components.
For each elementary hypothesis \(H_i\), check all intersections \(H_J\) with \(i\in J\). Mark \(H_i\) closed-rejected only if all those local tests reject.

Check each rejected intersection.
For every rejected \(H_I\), verify that at least one \(i\in I\) was marked closed-rejected. If even one rejected intersection has no such member, the procedure is not consonant at that level for those decisions.

Worked Example: From Local p-Values to Closed Rejections

Hypothetical data: For another three-hypothesis family, use Bonferroni local tests at \(\alpha=0.05\), with marginal p-values \(p_1=0.003\), \(p_2=0.018\), and \(p_3=0.20\). Determine the closed decisions and verify the consonance implication.

The local p-values are \(p_{\{1\}}=0.003\), \(p_{\{2\}}=0.018\), and \(p_{\{3\}}=0.20\). For the pairs, \(p_{\{1,2\}}=2(0.003)=0.006\), \(p_{\{1,3\}}=2(0.003)=0.006\), and \(p_{\{2,3\}}=2(0.018)=0.036\). For the full intersection, \(p_{\{1,2,3\}}=3(0.003)=0.009\). These are Bonferroni local p-values, so the local tests are valid at level \(0.05\) under their respective intersection nulls.

The closed adjusted p-value for \(H_1\) is the maximum over its four supersets: \(\max(0.003,0.006,0.006,0.009)=0.009\), so \(H_1\) is rejected. For \(H_2\), the maximum is \(\max(0.018,0.006,0.036,0.009)=0.036\), so \(H_2\) is rejected. For \(H_3\), the maximum is \(\max(0.20,0.006,0.036,0.009)=0.20\), so \(H_3\) is not rejected. Every intersection test rejects except the singleton \(H_{\{3\}}\). Each rejected intersection contains at least one of \(H_1\) and \(H_2\), both of which are closed-rejected. The rejection pattern is consonant at \(\alpha=0.05\).

Why Strong Error Control Is Not Enough

A frequent mistake is to reason that, because a closed procedure controls familywise error, any rejected intersection must yield a rejected elementary hypothesis. This confuses an error-rate guarantee with a structural property of the rejection pattern. Error control limits the probability of false elementary rejections; it does not force every local intersection rejection to satisfy the closure requirements for one of its components.

For example, consider three valid Bonferroni local tests at \(\alpha=0.05\) with hypothetical marginal p-values \(0.02\), \(0.90\), and \(0.90\). The pair intersection \(H_{\{1,2\}}\) has local p-value \(2(0.02)=0.04\), so it is rejected. The full intersection has local p-value \(3(0.02)=0.06\), so it is not rejected. Neither \(H_1\) nor \(H_2\) can be rejected by closure, because each requires rejection of the full intersection. Thus this local rejection does not establish a closed rejection of either contained elementary hypothesis.

This calculation is a diagnostic, not a reason to disregard the local test or its error control. It shows why consonance must be checked as a separate property. In reporting a consonant closed test, the conclusion can identify at least one closed-rejected elementary hypothesis within any rejected intersection. Without consonance, that conclusion does not follow merely from the intersection rejection.

Assumptions, Scope, and Interpretation

The consonance definition applies to a finite family with a specified local test for each nonempty intersection, and the closure rule must be applied consistently to all such intersections. The logical implication itself requires no particular test statistic or data distribution. To interpret the procedure as controlling strong familywise error at level \(\alpha\), however, each local test must also be valid at level \(\alpha\) under its intersection null. If p-values are used, they must be valid p-values for those local nulls; the maximum formula for closed adjusted p-values presupposes that the local tests are represented on the same significance scale.

Consonance does not mean that all elementary hypotheses in a rejected intersection are false, nor does it identify the scientific effect size or establish a causal interpretation. It means only that at least one contained elementary null is rejected according to the full closed testing rule. As with any hypothesis test, the conclusion is governed by the model, endpoint definitions, and error-control plan used to construct the tests.

Key takeaway: A rejected intersection is a local decision. Consonance is the additional property that guarantees this decision implies a closed rejection of at least one elementary hypothesis inside that intersection. Check the closure requirements, not just the local test.

Check Your Understanding

Use the definition of consonance and the closure rule to answer each question. Solutions follow each prompt.

For a family of three hypotheses, a local test rejects \(H_{\{1,2\}}\). What additional fact is required to conclude that consonance gives a closed rejection of a contained elementary hypothesis?
Solution: At least one of \(H_1\) or \(H_2\) must be rejected by closure. For that hypothesis, every intersection containing it—not just \(H_{\{1,2\}}\)—must have a rejected local test.
At level \(\alpha=0.05\), suppose \(p_{\{1\}}=0.02\), \(p_{\{2\}}=0.03\), \(p_{\{1,2\}}=0.04\), and \(p_{\{1,2,3\}}=0.07\). Can either \(H_1\) or \(H_2\) be rejected by closure on the basis of these values?
Solution: Neither can be rejected by closure, because both require rejection of \(H_{\{1,2,3\}}\), and \(0.07>0.05\). The local rejection of \(H_{\{1,2\}}\) alone is insufficient.
In a three-hypothesis procedure, the local p-values for all intersections containing \(H_1\) are \(0.01\), \(0.03\), \(0.02\), and \(0.04\). What is the closed adjusted p-value for \(H_1\), and is \(H_1\) rejected at \(\alpha=0.05\)?
Solution: The closed adjusted p-value is the maximum, \(p_1^{\mathrm{cl}}=\max(0.01,0.03,0.02,0.04)=0.04\). Since \(0.04\leq0.05\), \(H_1\) is rejected by closure.
Does strong familywise error control alone imply consonance?
Solution: No. Strong control is a probability guarantee about false elementary rejections across true-null configurations. Consonance is a logical condition on the local and closed rejection pattern. The two properties are distinct.

What You'll Learn