Why the Local Test Matters
A closed procedure can use a different local test for each intersection hypothesis. The central design choice is not simply which test appears most powerful. Each chosen test must have the required level for its own intersection null under the full analysis procedure, including any data-dependent adaptations. If that guarantee holds, closure supplies strong control of the family-wise error rate; if it does not, closure cannot repair the problem.
When several candidate tests address the same intersection hypothesis, they may differ in assumptions, sensitivity to different patterns of evidence, and performance with nuisance parameters or adaptive data. Comparing them means checking their guarantees under the null first, then considering their power and practical limitations.
A closed procedure rejects an elementary hypothesis \(H_i\) only when it rejects every intersection hypothesis \(H_J\) for which \(i\in J\). Thus, a local test for \(H_{\{i\}}\) is not enough: the tests for the larger intersections containing \(i\) also matter. The common level \(\alpha\) is the local testing level; it is not divided by the number of intersections.
Why Level-Valid Local Tests Suffice
The closure guarantee follows by considering the set of hypotheses that are true at a particular parameter value. The argument does not require the tests for different intersections to be independent, nor does it require them to have the same form.
At a fixed parameter value, let \(I_0\) be the nonempty set of indices \(i\) for which \(H_i\) is true. Then the intersection \(H_{I_0}\) is true.
If the closed procedure falsely rejects any true \(H_i\), with \(i\in I_0\), it must have rejected every intersection containing \(i\), including \(H_{I_0}\).
The event of at least one false rejection is therefore contained in the event that the local test of the true \(H_{I_0}\) rejects. If that local test has level \(\alpha\), the family-wise error probability is at most \(\alpha\).
In symbols, if \(\mathcal{E}\) is the event of at least one false rejection, then $$ \mathcal{E}\subseteq\{\phi_{I_0}=1\}, \qquad P_\theta(\mathcal{E})\leq P_\theta(\phi_{I_0}=1)\leq\alpha. $$ This is why the choice of local test matters: closure provides the containment, while validity of the local test provides the probability bound.
Comparing Tests for the Same Intersection
Suppose an intersection null is \(H_{\{A,B\}}=H_A\cap H_B\), and \(p_A,p_B\) are valid p-values for their component hypotheses. Three common local approaches illustrate different trade-offs.
| Local approach | Rejection rule at level \(\alpha\) | Key requirement or feature |
|---|---|---|
| Intersection-union test | Reject if both \(p_A\leq\alpha\) and \(p_B\leq\alpha\) | Valid under the intersection null without requiring independence of the two p-values |
| Bonferroni test | Reject if \(\min(p_A,p_B)\leq\alpha/2\) | Valid under arbitrary dependence when each component p-value is valid under the intersection null |
| Fisher combination test | Reject for large \(T=-2\{\log(p_A)+\log(p_B)\}\), using its null reference distribution | With independent uniform p-values under the joint null, \(T\) has a chi-square distribution with 4 degrees of freedom |
The intersection-union test asks for evidence against both component nulls. Bonferroni can reject when one p-value is small enough by itself, with a threshold adjusted for two tests. Fisher combines evidence across both p-values; its usual chi-square calibration relies on their joint null distribution, commonly independence. Neither a smaller p-value nor a more permissive rejection pattern alone establishes that a candidate is suitable: its level condition must hold for every parameter value in the intersection null.
Worked Example: Comparing Three Local Tests
Suppose a hypothetical trial yields component p-values \(p_A=0.020\) and \(p_B=0.400\), and the goal is to test \(H_A\cap H_B\) at \(\alpha=0.05\). Assume both p-values are valid under this intersection null. For the Fisher calculation, additionally assume they are independent and uniform under the null.
Intersection-union test. It requires both p-values to be at most 0.05. Since \(0.400>0.05\), it does not reject.
Bonferroni test. Its threshold for either p-value is \(0.05/2=0.025\). Since \(0.020\leq0.025\), it rejects. Equivalently, the Bonferroni-adjusted p-value is $$ \min\{1,2\min(0.020,0.400)\}=2(0.020)=0.040. $$
Fisher test. The statistic is $$ T=-2\{\log(0.020)+\log(0.400)\}=-2\log(0.008)=9.657. $$ Under the stated independence assumption, the null distribution is chi-square with 4 degrees of freedom. Its upper-tail probability is $$ P(\chi^2_4\geq9.657)=e^{-9.657/2}(1+9.657/2)\approx0.0466. $$ Thus Fisher also rejects at 0.05. These decisions differ because the tests combine evidence differently, not because they use different intersection hypotheses. If independence is not justified, the stated Fisher reference calculation is not established.
Worked Example: Carrying a Local Test Through Closure
Consider two elementary hypotheses \(H_A\) and \(H_B\) in a hypothetical adaptive trial. Suppose their local tests use valid p-values \(p_A=0.020\) and \(p_B=0.400\). For the intersection \(H_A\cap H_B\), use the Fisher test from the previous calculation, with its required independent-uniform null assumption. At level 0.05, the singleton test of \(H_A\) rejects because \(0.020\leq0.05\), and the singleton test of \(H_B\) does not reject because \(0.400>0.05\). The joint Fisher test rejects because its p-value is approximately 0.0466.
To reject \(H_A\) under closure, both \(H_A\) and \(H_A\cap H_B\) must be rejected. Both are, so the closed procedure rejects \(H_A\). To reject \(H_B\), both \(H_B\) and \(H_A\cap H_B\) must be rejected. The singleton test of \(H_B\) does not reject, so the procedure does not reject \(H_B\). The joint result alone cannot reject either elementary hypothesis: each must pass every required local test.
The validity check is separate from these observed decisions. If each singleton test has level 0.05 under its own null and the Fisher local test has level 0.05 uniformly under \(H_A\cap H_B\), the local guarantees needed for this closed procedure are in place. The calculation of one observed p-value cannot establish those guarantees; they come from the tests’ assumptions and calibration.
Data-Dependent Choice Among Candidates
A crucial distinction is whether the candidate local test is fixed in advance or selected using the data that will also determine rejection. Individually valid candidates do not automatically remain valid after choosing whichever one rejects most readily. The relevant guarantee is for the selected procedure, not just for each candidate considered in isolation.
Worked Example: Why Choosing the Smaller P-Value Can Fail
Suppose two candidate tests address the same true intersection null. Under that null, their p-values \(U_1\) and \(U_2\) are independent Uniform(0,1) random variables. Each test, used on its own, rejects at level 0.05 when its p-value is at most 0.05, and therefore has rejection probability 0.05.
Now select the candidate with the smaller p-value and reject if that selected p-value is at most 0.05. This rule rejects whenever \(U_1\leq0.05\) or \(U_2\leq0.05\). Hence $$ P(\min(U_1,U_2)\leq0.05) =1-P(U_1>0.05,U_2>0.05) =1-(0.95)^2 =0.0975. $$ The selected rule has a 9.75% type I error probability, not 5%, despite both candidates being individually level 0.05. To use such a selection rule, its calibration must account for selection, or the choice must be made in a way that preserves a valid level guarantee.
More generally, if data determine which candidate \(\phi_J^{(k)}\) is used, verify the bound for the complete rule: $$ \sup_{\theta\in H_J}P_\theta\{\text{the selected local test rejects }H_J\}\leq\alpha. $$ A conditional error bound can sometimes establish this: if, for every relevant conditioning event \(S\), the conditional rejection probability under the null is at most \(\alpha\), then averaging over \(S\) also gives an unconditional bound of at most \(\alpha\). A bound averaged only over some nuisance-parameter values or selection outcomes is not a uniform level guarantee.
Assumptions, Diagnostics, and Limits
Before choosing among candidate local tests, make the conditions behind each guarantee explicit. In an adaptive trial, the probability calculation must correspond to the actual procedure, including interim decisions and any changes to allocation or analysis. A test calibrated for a fixed design is not automatically valid for a design whose data-dependent changes alter its null distribution.
- Check the exact null. Confirm that the test is calibrated for the intersection \(H_J\), not merely for one component hypothesis. State which parameter values and nuisance parameters are covered by the level bound.
- Check dependence assumptions. Fisher’s usual chi-square reference relies on the specified joint null distribution of the p-values. Valid marginal p-values alone do not establish independence.
- Check selection and adaptation. If interim information chooses the test, its critical value, or the analysis path, calibrate the resulting rule as a whole. Do not transfer the candidates’ individual guarantees to a data-selected rule without justification.
- Check discreteness and approximation. In small or discrete samples, a continuous reference distribution or asymptotic approximation may not give the claimed level. Verify exact or conservative calibration under the relevant null.
- Check operating characteristics as well as validity. Once level is established, compare power for scientifically relevant alternatives, interpretability, and robustness to plausible dependence or nuisance-parameter values.
The principal alternative to selecting a single candidate is to specify one local test for each intersection in advance. Another option is to build a selection-adjusted local test whose rejection probability includes the selection step. These approaches can both support closure if the resulting local tests meet their level requirements, but they may have different power and implementation costs. No single local test is uniformly best across all alternatives and dependence structures.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For \(H_A\cap H_B\), a candidate local test rejects only when both component p-values are at most \(\alpha\). What is this approach called, and why does it have level at most \(\alpha\) under the intersection null?
- At \(\alpha=0.05\), Bonferroni is applied to two valid component p-values \(0.012\) and \(0.070\). Calculate its adjusted p-value and state the local decision.
- In a closed procedure for \(H_A,H_B\), the singleton test of \(H_A\) and the joint test of \(H_A\cap H_B\) reject, but the singleton test of \(H_B\) does not. Which elementary hypotheses are rejected?
- Two independent Uniform(0,1) candidate p-values are observed under a true intersection null. What is the rejection probability if the smaller is selected and compared with 0.05?
Solutions. (1) This is the intersection-union test. Under \(H_A\cap H_B\), both component nulls are true, and the event that both p-values are at most \(\alpha\) is contained in the event that either specified valid component p-value is at most \(\alpha\). Its probability is therefore at most \(\alpha\). (2) The adjusted value is \(\min\{1,2(0.012)\}=0.024\), so the local test rejects at 0.05. (3) The procedure rejects \(H_A\), because both required tests for it reject; it does not reject \(H_B\), because its singleton test does not reject. (4) The probability is \(1-(0.95)^2=0.0975\), or 9.75%, so the selected rule is not level 0.05.