Why Three Comparisons Require More Than Three Tests
Suppose a trial compares three treatments with a shared control and the confirmatory question is whether each treatment improves the outcome. Testing each treatment-versus-control hypothesis at level \(\alpha\) does not, by itself, ensure that the probability of making one or more false rejections across the family is at most \(\alpha\). Closed testing addresses this by requiring evidence against every intersection null hypothesis that contains the elementary hypothesis under consideration.
For three comparisons, there are seven nonempty intersections. The practical task is therefore twofold: construct those intersections correctly, then apply a valid local level-\(\alpha\) test to each one. The local tests are tests of particular intersection nulls; a small p-value from an elementary comparison is not enough on its own to establish that its elementary null may be rejected by closure.
This one-sided formulation is an example of a prespecified family, not a universal choice. A two-sided trial question would require two-sided elementary hypotheses and p-values throughout. The direction, estimand, analysis population, and endpoint must be defined consistently before interpreting the tests. The shared control affects the dependence among treatment-comparison statistics, but it does not change which logical intersections must be included.
The Seven Intersection Hypotheses
Each nonempty subset of \(\{1,2,3\}\) identifies one intersection. The singleton intersections are the elementary hypotheses themselves; the larger intersections assert that several elementary nulls hold together.
| Index set | Intersection null | Meaning |
|---|---|---|
| \(\{1\}\) | \(H_1\) | Treatment 1 has no positive effect versus control. |
| \(\{2\}\) | \(H_2\) | Treatment 2 has no positive effect versus control. |
| \(\{3\}\) | \(H_3\) | Treatment 3 has no positive effect versus control. |
| \(\{1,2\}\) | \(H_1\cap H_2\) | Neither treatment 1 nor treatment 2 has a positive effect. |
| \(\{1,3\}\) | \(H_1\cap H_3\) | Neither treatment 1 nor treatment 3 has a positive effect. |
| \(\{2,3\}\) | \(H_2\cap H_3\) | Neither treatment 2 nor treatment 3 has a positive effect. |
| \(\{1,2,3\}\) | \(H_1\cap H_2\cap H_3\) | None of the three treatments has a positive effect. |
Worked Example: Writing an Intersection in Context
A hypothetical trial compares three doses of an investigational treatment with the same control. Let \(\delta_i\) be the difference in mean symptom improvement, in points, for dose \(i\) versus control; larger values mean greater improvement. The trial uses the one-sided alternatives \(\delta_i>0\).
The pairwise intersection \(H_{\{1,3\}}\) is \(H_1\cap H_3\). Substituting the elementary nulls gives \(H_{\{1,3\}}:\delta_1\leq0\ \text{and}\ \delta_3\leq0\). It does not say that the two effects are equal, and it does not include treatment 2. It asserts that both of the specified treatment-versus-control effects are nonpositive.
For the full intersection, \(H_{\{1,2,3\}}\), the assertion is \(\delta_1\leq0,\ \delta_2\leq0,\ \text{and}\ \delta_3\leq0\). The word “and” matters: this is the null that all three elementary nulls hold, not the claim that at least one holds.
Local Bonferroni Tests for Every Intersection
A simple local test can be formed from the elementary p-values. Assume each \(p_i\) is a valid p-value for \(H_i\): under \(H_i\), \(P(p_i\leq u)\leq u\) for every \(u\) between 0 and 1. If \(H_I\) is true, then every \(H_i\) for \(i\in I\) is true, so each corresponding \(p_i\) remains valid under \(H_I\). No independence among these p-values is required for the Bonferroni local test.
The reason this is a valid local test is the union bound. For \(0\leq t<1\), if \(q_I\leq t\), clipping at 1 cannot be the reason for the inequality. It follows that \(m\min_{i\in I}p_i\leq t\), so at least one of the \(m\) p-values is at most \(t/m\). Therefore, when \(H_I\) is true,
At \(t=1\), \(P(q_I\leq1)=1\leq1\), so the validity condition holds there as well. In particular, taking \(t=\alpha<1\) shows that the local test has type I error at most \(\alpha\). The separate endpoint argument matters: the implication from \(q_I\leq t\) to a small component p-value need not hold at \(t=1\), because \(q_I\) is clipped at 1.
From Local Results to Elementary Rejections
The closed-testing rule is precise: an elementary hypothesis \(H_i\) may be rejected only if every intersection \(H_I\) with \(i\in I\) is rejected by its local test. Thus \(H_1\) requires rejection of \(H_1\), \(H_1\cap H_2\), \(H_1\cap H_3\), and \(H_1\cap H_2\cap H_3\). \(H_2\) and \(H_3\) each have their own corresponding four required tests.
This rule provides strong familywise error control when every local test is valid at level \(\alpha\). To see why, suppose at least one true elementary null is rejected. Let \(T\) be the nonempty set of all true elementary nulls. Any rejected true \(H_i\), with \(i\in T\), could be rejected under closure only if every intersection containing it was rejected, including \(H_T=\bigcap_{j\in T}H_j\). But \(H_T\) is true. Thus the event of any false elementary rejection is contained in the event that the local test rejects the true intersection \(H_T\), whose probability is at most \(\alpha\). This argument applies whichever subset of elementary nulls is true.
Worked Example: Closing Three Bonferroni Tests
Consider hypothetical valid one-sided p-values \(p_1=0.006\), \(p_2=0.024\), and \(p_3=0.061\), with \(\alpha=0.05\). Assume the p-values come from prespecified treatment-versus-control tests with the direction and endpoint defined above. Use the Bonferroni local p-value for each intersection.
For the three singleton intersections, \(q_{\{1\}}=0.006\), \(q_{\{2\}}=0.024\), and \(q_{\{3\}}=0.061\). The first two are at most 0.05, but the third is not.
For the pair intersections, the calculations are \(q_{\{1,2\}}=\min(1,2\min(0.006,0.024))=0.012\), \(q_{\{1,3\}}=\min(1,2\min(0.006,0.061))=0.012\), and \(q_{\{2,3\}}=\min(1,2\min(0.024,0.061))=0.048\). Each pair intersection is rejected at 0.05. For the triple intersection, \(q_{\{1,2,3\}}=\min(1,3\min(0.006,0.024,0.061))=0.018\), which is also rejected.
To decide about \(H_1\), check its four required intersections: \(H_1\) (0.006), \(H_1\cap H_2\) (0.012), \(H_1\cap H_3\) (0.012), and \(H_1\cap H_2\cap H_3\) (0.018). All pass, so closure permits rejection of \(H_1\). The corresponding checks for \(H_2\) are 0.024, 0.012, 0.048, and 0.018; all pass, so closure permits rejection of \(H_2\) as well. Closure does not permit rejection of \(H_3\), because its singleton test has p-value 0.061, above 0.05. The conclusion is that treatments 1 and 2 show evidence of benefit under this testing procedure, but treatment 3 does not meet the rejection rule.
Worked Example: A Small Elementary P-Value Is Not Sufficient
Now use hypothetical valid p-values \(p_1=0.012\), \(p_2=0.026\), and \(p_3=0.30\), with \(\alpha=0.05\), and again use Bonferroni local tests. The pair intersection \(H_2\cap H_3\) has \(q_{\{2,3\}}=\min(1,2\min(0.026,0.30))=0.052\), so it is not rejected.
Although the singleton test for \(H_2\) has \(p_2=0.026<0.05\), closed testing also requires rejection of \(H_2\cap H_3\), because this intersection contains \(H_2\). That required local test fails. Consequently, closure does not permit rejection of \(H_2\). The failure is not evidence that \(H_2\) is true; it means this procedure has not established a closed-testing rejection for it.
Choosing and Checking the Local Test
Bonferroni is attractive because its local validity follows from marginal p-value validity and the union bound, without an assumption of independence. Its cost is that it may be conservative, especially for an intersection with several hypotheses. A Simes local test is a possible alternative when its dependence conditions are justified. For \(m\) p-values in an intersection, order them as \(p_{(1)}\leq\cdots\leq p_{(m)}\) and calculate
The Simes test is valid under independence and under certain positive-dependence conditions, such as suitable positive regression dependence. Shared-control comparisons are generally dependent, so independence must not be assumed just because the treatment groups are distinct. A Simes local test should be used only when the relevant dependence condition is established for the p-values in the intersection. If that justification is unavailable, Bonferroni is a straightforward choice.
Worked Example: How the Local-Test Choice Can Matter
Suppose three hypothetical p-values are \(0.020\), \(0.020\), and \(0.90\), and suppose, for this illustration, that the conditions for a Simes local test are satisfied. For the full intersection, the ordered values are \(p_{(1)}=0.020\), \(p_{(2)}=0.020\), and \(p_{(3)}=0.90\).
The Bonferroni local p-value for the triple intersection is \(q_{\{1,2,3\}}=\min(1,3(0.020))=0.060\), so that local test does not reject at \(\alpha=0.05\). The Simes value is \(\min(1,3(0.020),3(0.020)/2,3(0.90)/3)=\min(1,0.060,0.030,0.90)=0.030\), so the Simes local test rejects the triple intersection at 0.05.
This difference concerns the local test of the triple intersection only; it does not by itself establish that any elementary hypothesis is rejectable under closure. Each elementary hypothesis still requires rejection of every intersection containing it, including its singleton. The example also depends on the stated Simes validity condition. It is not evidence that Simes is automatically valid for shared-control comparisons.
The main alternatives serve different purposes. A simple Bonferroni adjustment tests elementary hypotheses by allocating error across the family; Holm’s step-down procedure is another strong familywise-error-controlling approach. Closed testing makes the intersection structure explicit and can use different valid local tests at different intersections, but requires tracking all the intersections relevant to each rejection. A global test of only \(H_1\cap H_2\cap H_3\) cannot establish which individual treatment hypotheses may be rejected: rejection of the global null alone does not supply the other required local results.
Diagnostics and Common Pitfalls
Before interpreting the rejection set, verify that the p-values and local tests match the planned family. In particular, check that each elementary p-value tests the stated treatment-versus-control null, uses the prespecified direction, and is valid under its null. Confirm that every nonempty intersection appears exactly once and that each local test is applied at the intended \(\alpha\). For Bonferroni, confirm that the multiplier is the size of the intersection, not automatically the total number of treatment comparisons.
- Testing only the global null. Rejecting the triple intersection does not by itself license rejection of any particular \(H_i\). Check all intersections containing that \(H_i\).
- Stopping after an elementary p-value. A singleton p-value below \(\alpha\) is necessary but may not be sufficient. A required pair or triple intersection can still fail.
- Assuming independence because of separate treatment groups. Treatment-versus-control statistics share observations from the control group, so their dependence must be accounted for when choosing a local test that needs a dependence condition.
- Changing the comparison family after seeing results. Omitting an inconvenient comparison changes which intersections are required and can invalidate the intended error control.
- Interpreting nonrejection as proof of no effect. Failure to reject means the procedure did not establish evidence against that null at the chosen level; it does not establish that the treatment effect is zero or harmful.
The closure calculation is a decision rule, not an estimate of treatment effect. Report effect estimates and their units alongside the multiplicity-adjusted inferential conclusion when available. Also distinguish the statistical conclusion from clinical importance: a treatment effect can be statistically significant yet too small to matter clinically, or clinically relevant but estimated imprecisely.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- How many nonempty intersection hypotheses arise from three elementary treatment-versus-control hypotheses? Name the intersection hypotheses containing \(H_2\).
- For \(p_1=0.010\), \(p_2=0.040\), and \(p_3=0.20\), calculate the Bonferroni local p-value for \(H_1\cap H_2\).
- Suppose a local test rejects \(H_1\), \(H_1\cap H_2\), and \(H_1\cap H_3\), but does not reject \(H_1\cap H_2\cap H_3\). May closure reject \(H_1\)? Why?
- What assumption permits use of the Simes local test, and why should it not be assumed automatically for shared-control comparisons?
- For Bonferroni local testing with three valid p-values, explain why the probability bound at \(t=1\) is handled separately from the argument for \(t<1\).
Solutions. (1) There are \(2^3-1=7\) nonempty subsets. The intersections containing \(H_2\) are \(H_2\), \(H_1\cap H_2\), \(H_2\cap H_3\), and \(H_1\cap H_2\cap H_3\). (2) \(q_{\{1,2\}}=\min(1,2\min(0.010,0.040))=0.020\). (3) No. The triple intersection contains \(H_1\), and closure requires every intersection containing \(H_1\) to be rejected. (4) Simes validity requires independence or a suitable positive-dependence condition; shared-control comparisons are dependent, so the condition needs justification. (5) For \(t<1\), clipping at 1 cannot make \(q_I\leq t\) hold, which permits the implication to a component p-value at most \(t/m\). At \(t=1\), that implication can fail, but the required bound is immediate because \(P(q_I\leq1)=1\leq1\).