Why Closure Gives Strong Error Control
A multiple-testing procedure should limit the chance of making one or more false claims, not merely behave well when every null hypothesis is true. Strong familywise error control requires that limit to hold no matter which hypotheses are true. Closure achieves this by requiring evidence against a claim to pass a sequence of tests, including a test of the intersection of all the null hypotheses that are actually true.
The introductory material on the Closure Principle describes the rejection rule. Here the focus is its error-control proof: why the rule works for every configuration of true and false hypotheses, and exactly what must be valid for the proof to apply.
For any nonempty index set \(I\subseteq\{1,\ldots,m\}\), define the intersection hypothesis \(H_I=\bigcap_{i\in I}H_i\). It is true precisely when every elementary null hypothesis indexed by \(I\) is true. A local test for \(H_I\) is a test that decides whether to reject that particular intersection.
The local tests need not be independent, and their statistics need not have the same form. The requirement is that each intersection test is valid under its own null hypothesis. In particular, if an intersection is true, its local test must reject with probability no greater than \(\alpha\). The procedure may use the same data for many tests; dependence among the resulting decisions does not invalidate the argument below.
The Event-Containment Proof
Fix any configuration of elementary null hypotheses, and let \(T\) be the set of indices of the true ones. If \(T\) is empty, no elementary null is true, so the probability of falsely rejecting a true elementary null is zero. Suppose instead that \(T\) is nonempty. The intersection \(H_T\) is then true by definition.
If the procedure makes at least one false elementary rejection, it rejects some \(H_i\) with \(i\in T\).
Because \(i\in T\), the intersection \(H_T\) is among the intersections containing \(H_i\). Rejecting \(H_i\) therefore requires rejecting \(H_T\).
The event of at least one false elementary rejection is a subset of the event that the local test rejects the true \(H_T\). The latter event has probability at most \(\alpha\).
In symbols, writing \(E\) for the event of at least one false elementary rejection and \(R_T\) for rejection of \(H_T\), the closure rule gives \(E\subseteq R_T\). Therefore
The first inequality is event containment; it does not require independence or an assumption about the joint distribution of the local test statistics. The second inequality is exactly the local level-\(\alpha\) validity assumption, applied to the true intersection \(H_T\). Since the configuration was arbitrary, the bound holds under every configuration of true and false nulls. That is strong, rather than merely global-null, control.
Worked Decisions and Probability Bounds
Worked Example: Applying Closure to Three Endpoints
Consider a hypothetical trial with three elementary null hypotheses: \(H_1\), \(H_2\), and \(H_3\), corresponding to no treatment effect on three prespecified endpoints. Suppose valid local tests produce the following hypothetical p-values. Each p-value is for the intersection named in its row.
| Intersection | Local p-value |
|---|---|
| \(H_1\) | 0.012 |
| \(H_2\) | 0.200 |
| \(H_3\) | 0.031 |
| \(H_1\cap H_2\) | 0.018 |
| \(H_1\cap H_3\) | 0.044 |
| \(H_2\cap H_3\) | 0.080 |
| \(H_1\cap H_2\cap H_3\) | 0.040 |
At \(\alpha=0.05\), the closed-test adjusted p-value for \(H_i\) is the largest local p-value among intersections that contain \(H_i\). This follows because \(H_i\) is rejected only if every one of those local p-values is at most the chosen significance level.
For \(H_1\), the relevant intersections are \(H_1\), \(H_1\cap H_2\), \(H_1\cap H_3\), and \(H_1\cap H_2\cap H_3\). Thus its adjusted p-value is \(\max(0.012,0.018,0.044,0.040)=0.044\), so \(H_1\) is rejected.
For \(H_2\), the adjusted p-value is \(\max(0.200,0.018,0.080,0.040)=0.200\), so \(H_2\) is not rejected. For \(H_3\), it is \(\max(0.031,0.044,0.080,0.040)=0.080\), so \(H_3\) is not rejected. The calculation illustrates that a small singleton p-value alone is insufficient: all containing intersections must also pass.
Worked Example: Bounding Error When Only Some Nulls Are True
Suppose a hypothetical study tests three claims, and its true configuration is \(H_1\) true, \(H_2\) false, and \(H_3\) true. Assume the local test of \(H_1\cap H_3\) rejects with probability 0.041 under this configuration. This is a design property of the test, not an estimate from one observed dataset.
The true elementary nulls have index set \(T=\{1,3\}\), so the relevant true intersection is \(H_T=H_1\cap H_3\). A false elementary rejection can only be a rejection of \(H_1\), \(H_3\), or both. If the closed procedure rejects \(H_1\), it must reject \(H_1\cap H_3\), since that intersection contains \(H_1\). If it rejects \(H_3\), it must also reject \(H_1\cap H_3\). Consequently,
Taking probabilities gives \(P(\text{at least one false rejection})\leq 0.041\). Since \(0.041\leq 0.05\), this configuration has FWER at most 0.041, hence at most the target level 0.05. The conclusion does not depend on whether the false \(H_2\) is rejected.
Worked Example: Checking a Two-Hypothesis Local Test
For two hypothetical endpoints, suppose the p-values \(p_1\) and \(p_2\) are independent and uniformly distributed on \([0,1]\) when both null hypotheses are true. Use valid singleton tests and a Bonferroni local test for \(H_1\cap H_2\), with local p-value \(p_{12}=\min(1,2\min(p_1,p_2))\).
Under the global intersection null, this local test rejects at level 0.05 when \(p_{12}\leq0.05\), or equivalently when \(\min(p_1,p_2)\leq0.025\). Independence gives
Thus the intersection test rejects with probability 0.049375, which is below 0.05. Under the global null, any closed elementary rejection entails rejection of this intersection, so the FWER is at most 0.049375. In this example, whenever the intersection test rejects, at least one singleton p-value is at most 0.025 and therefore passes its own 0.05 test. The FWER is consequently exactly 0.049375. Independence was used to calculate this particular probability; the general closure proof does not require independence.
What the Guarantee Does—and Does Not—Require
The proof is deliberately short because its force comes from the structure of the rejection rule. It does not combine error probabilities across all intersections or add them together. Instead, for a fixed true configuration, it identifies one particular true intersection, \(H_T\), whose valid local test bounds the entire event of making any false elementary rejection.
This differs from testing only the intersection of all hypotheses. A level-\(\alpha\) test of the global intersection alone can control the probability of rejecting at least one hypothesis when all are true, but it does not by itself justify which individual hypotheses to reject. Closure uses local tests for every nonempty intersection and permits an elementary claim only when all tests required for that claim reject. Direct single-step procedures are another possible approach, but their validity must be established under their own assumptions; closure’s guarantee follows from validity of each local test.
Closure also does not repair a faulty local test. For instance, with two independent true nulls and valid uniform singleton p-values, using \(\min(p_1,p_2)\) itself as the global-intersection p-value would reject that intersection when either p-value is at most 0.05. Its Type I error would be
which exceeds 0.05. Closing around that invalid local test cannot establish level-0.05 strong control. The local test for the intersection must be valid under the intersection null; a correction such as the Bonferroni local test in the preceding example supplies one valid option in that setting.
Diagnostics and Limitations
When checking a closed-testing analysis, verify the assumptions that carry the proof:
- Every required intersection is represented. For each elementary claim, list all intersections containing it. An omitted intersection can make the actual rejection rule less restrictive than the closed rule.
- Each local test is valid under its intersection null. A test calibrated only for a different null configuration may not satisfy the needed Type I error bound.
- The rejection rule is applied consistently. Rejecting \(H_i\) requires rejection of every \(H_I\) with \(i\in I\), not just the singleton test and the global intersection test.
- The calibration matches the design. If the trial’s sampling procedure or analysis changes, local validity must still hold for the procedure actually used. Closure is an error-control argument conditional on that validity, not a substitute for establishing it.
A useful diagnostic is to identify the set \(T\) of true elementary nulls for a configuration under consideration and ask: “Does every false elementary rejection force rejection of \(H_T\), and is the local test of \(H_T\) level \(\alpha\)?” If either answer is no, this proof does not establish the claimed bound. The theorem guarantees an upper bound; it does not say the FWER equals \(\alpha\), that every true null will be retained, or that the procedure is powerful for every alternative.
Check Your Understanding
Answer each question using the event-containment argument; brief solutions follow the questions.
- In a family of four hypotheses, suppose \(H_2\) and \(H_4\) are true. Which intersection test bounds the probability of at least one false elementary rejection?
- Why does the closure proof not require independence among local test statistics?
- At \(\alpha=0.05\), a closed adjusted p-value for one elementary hypothesis is 0.047. Is that hypothesis rejected, and what must be true of its containing-intersection local p-values?
- If the local test of the intersection of all true nulls has Type I error 0.058, does the stated theorem establish FWER control at 0.05 for that configuration?
- What error-control conclusion follows if all elementary null hypotheses are false?
Solutions. (1) The test of \(H_2\cap H_4\), because this is \(H_T\), the intersection of all true nulls. (2) The proof uses set inclusion, \(E\subseteq R_T\), and the marginal bound \(P(R_T)\leq\alpha\), not a factorization of joint probabilities. (3) Yes; 0.047 is below 0.05, so every required containing-intersection local p-value is at most 0.047 and hence at most 0.05. (4) No; 0.058 exceeds 0.05, so the required local level bound is not established. (5) There are no true elementary nulls to reject falsely, so FWER is zero in that configuration.