Why Treatment Selection Requires Intersection Tests
A multiarm trial may compare several experimental treatments with a shared control, then select one or more promising treatments for further study. A treatment that looks best among the candidates is not automatically confirmatory: selection can favor arms with unusually large observed effects, even when their true effects are not beneficial. Closed testing provides a systematic way to decide which treatment hypotheses can be rejected while controlling the probability of at least one false efficacy claim across the original family.
The key idea is that a claim about one treatment must survive more than its own test. It must also survive tests of every intersection hypothesis that includes it. These intersection tests make the procedure account for the possibility that several treatment null hypotheses are simultaneously true.
A closed testing procedure tests every nonempty intersection hypothesis at level \(\alpha\). It rejects an elementary hypothesis \(H_i\) only if it rejects every \(H_I\) for which \(i\in I\). In particular, it must reject \(H_i\) itself, the singleton intersection, as well as all larger intersections containing \(H_i\). If an arm was selected for a confirmatory claim, the same rule applies: selection does not remove any of the intersections required for that arm.
A Bonferroni Construction for the Intersections
One transparent way to construct the local tests is to begin with a valid one-sided p-value \(p_i\) for each elementary treatment hypothesis. For an intersection indexed by \(I\), let \(|I|\) be the number of hypotheses in that intersection and define its Bonferroni p-value as follows.
Reject \(H_I\) at level \(\alpha\) when \(p_I\leq\alpha\). Equivalently, reject if at least one of its component p-values is no greater than \(\alpha/|I|\). For a singleton, this reduces to the ordinary test, since \(p_{\{i\}}=p_i\).
This local test is valid under arbitrary dependence among the treatment statistics. Under \(H_I\), every \(H_i\) with \(i\in I\) is true, and each corresponding p-value is marginally valid: \(P(p_i\leq t)\leq t\) for \(0\leq t\leq1\). The union bound therefore gives
The intersection test may be conservative, because the union bound does not use the actual dependence structure. Its advantage is that the validity argument does not require independent treatment comparisons. Other valid local tests can also be used in a closed procedure; whichever tests are chosen, each must control its own intersection hypothesis at level \(\alpha\).
Worked Example: Two Treatments, One Selected for Confirmation
Suppose a hypothetical trial compares two treatments with control on a benefit scale measured in points. The prespecified one-sided tests yield valid p-values \(p_1=0.020\) and \(p_2=0.31\). Treatment 1 was selected for a confirmatory claim. Use Bonferroni local tests with \(\alpha=0.05\).
There are three nonempty intersections: \(H_1\), \(H_2\), and \(H_{\{1,2\}}=H_1\cap H_2\). The singleton p-values are 0.020 and 0.31. For the pair,
Treatment 1 is confirmed: its singleton test passes because \(0.020\leq0.05\), and the pair intersection also passes because \(0.040\leq0.05\). Treatment 2 is not confirmed because its singleton test fails, \(0.31>0.05\). The fact that treatment 2's singleton test fails does not prevent confirmation of treatment 1; the required pair intersection test passes. In particular, closure does not require every singleton hypothesis in the family to be rejected before any treatment can be confirmed.
How Closure Controls Familywise Error
Strong familywise error control means that, for every configuration of true and false elementary null hypotheses, the probability of rejecting at least one true null is no greater than \(\alpha\). The closure rule achieves this if every local intersection test is valid at level \(\alpha\). The justification identifies one particular intersection that must be rejected whenever any true elementary hypothesis is falsely rejected.
Let \(T\) be the nonempty set of indices for which \(H_i\) is true. If there are no true elementary nulls, a false rejection cannot occur.
If closure falsely rejects any true \(H_i\), it must reject every intersection containing \(i\), including \(H_T=\bigcap_{j\in T}H_j\).
Because \(H_T\) is true and its local test has size at most \(\alpha\), the probability of rejecting at least one true elementary hypothesis is at most \(P(\text{reject }H_T)\leq\alpha\).
This argument establishes strong control, not merely control when every treatment null is true. It also explains why the full family matters. If treatment 1 is tested after selection from several candidates, its required intersections include those involving treatment 1 and other original candidate treatments, even if some of those other treatments were not selected.
Worked Example: Three Treatments and the Full Closure
Consider a hypothetical three-treatment trial with valid one-sided p-values \(p_1=0.012\), \(p_2=0.020\), and \(p_3=0.11\). Suppose treatments 1 and 2 were selected. Calculate the Bonferroni local p-values for all seven nonempty intersections and determine which selected treatments are confirmed at \(\alpha=0.05\).
For the singleton intersections, the local p-values are \(p_{\{1\}}=0.012\), \(p_{\{2\}}=0.020\), and \(p_{\{3\}}=0.11\). For the three pairs,
For the three-treatment intersection,
Treatment 1 requires rejection of \(H_1\), \(H_{\{1,2\}}\), \(H_{\{1,3\}}\), and \(H_{\{1,2,3\}}\). Their p-values are all at most 0.05, so treatment 1 is confirmed. Treatment 2 requires rejection of \(H_2\), \(H_{\{1,2\}}\), \(H_{\{2,3\}}\), and \(H_{\{1,2,3\}}\); these also all pass, so treatment 2 is confirmed. Treatment 3 is not confirmed because its singleton p-value is 0.11. The calculations show that an unselected treatment can still occur in an intersection required for a selected treatment's closure decision.
Selection, Assumptions, and Practical Checks
The closure guarantee depends on the validity of the local tests, not on the fact that a treatment was selected. In this construction, each \(p_i\) must be a valid p-value for its elementary null under the trial's actual design and analysis. The family of candidate hypotheses must be specified, and all intersections in that family must be included. The Bonferroni argument also assumes that the component p-values are valid marginally; it does not require them to be independent.
Selection based on interim or final data can make a reported effect estimate look more favorable than it would in an unselected sample. It does not, by itself, invalidate a closed procedure whose tests remain valid for the design and whose closure is performed over the original family. However, if the adaptive design changes the data collection or analysis in a way that makes the ordinary p-values invalid, closure cannot repair that problem: valid design-specific p-values or valid local intersection tests are needed first. Likewise, testing only the selected subset as if it had been fixed in advance can omit intersections and lose the stated guarantee.
As a diagnostic, make an intersection checklist for each treatment proposed for confirmation. For a family of \(K\) treatments, each elementary hypothesis belongs to \(2^{K-1}\) nonempty intersections, including its singleton. Verify that every one of those intersections has a valid local test and that every required test passes. A missing intersection, a p-value computed under an incompatible design, or a test applied at a level larger than \(\alpha\) breaks the closure argument.
| Approach | What it does | Relation to closed testing |
|---|---|---|
| Bonferroni adjustment | Controls familywise error by dividing the error budget across hypotheses. | The Bonferroni local tests above yield a closed procedure; testing a single set of adjusted hypotheses is a simpler implementation in some settings. |
| Holm step-down procedure | Orders elementary p-values and applies sequential thresholds. | For Bonferroni local tests, Holm is a computationally convenient procedure with the same familywise error guarantee; closure makes the intersection logic explicit. |
| Dunnett adjustment | Uses the shared-control comparison structure to account for dependence among treatment statistics. | It can provide a less conservative alternative when its assumptions and local tests are appropriate; closed testing is the general framework for requiring valid intersection tests. |
Explicit closure is especially useful when the scientific question is which particular treatment effects can be confirmed, rather than only whether there is evidence somewhere in the family. Its cost is that the number of intersections grows exponentially: a family of \(K\) treatments has \(2^K-1\) nonempty intersections. For larger families, shortcut procedures may reduce computation, but a shortcut is justified only when it preserves the relevant closed-testing decisions or provides its own valid error-control argument.
Common Pitfalls
- Stopping after a selected arm's singleton test. A small \(p_i\) alone is not enough: every intersection containing \(i\) must also be rejected.
- Requiring all singleton tests to pass. That is not the closure rule. A selected treatment can be confirmed even when another treatment's singleton test fails, provided all intersections required for the selected treatment pass.
- Dropping unselected arms from the family after seeing the data. This may omit required intersections. Keep the original candidate family for closure unless a valid procedure explicitly justifies a different family.
- Treating closure as a fix for invalid p-values. The local tests must already be valid under the actual design. Multiplicity adjustment cannot correct a p-value whose null distribution is wrong.
Check Your Understanding
Use \(\alpha=0.05\) and Bonferroni local tests where requested. Each p-value below is assumed valid for its stated elementary null.
- For two treatments with p-values \(0.018\) and \(0.27\), calculate the pair-intersection p-value. Is treatment 1 confirmed?
- For two treatments with p-values \(0.030\) and \(0.030\), both singleton tests pass. Does closure confirm either treatment using the Bonferroni pair test?
- Why does a closed procedure control the probability of falsely rejecting any true elementary hypothesis when the set of true nulls is \(T\)?
- A trial selects one arm from four candidates, then tests only that arm at 0.05 using an ordinary p-value. Name one reason this is not, by itself, the closed procedure described here.
Solutions. (1) The pair p-value is \(\min(1,2(0.018))=0.036\). Treatment 1 is confirmed because its singleton p-value is 0.018 and the required pair intersection p-value is 0.036; both are at most 0.05. The second singleton need not pass. (2) The pair p-value is \(\min(1,2(0.030))=0.060\), which exceeds 0.05. Since each treatment requires rejection of the pair intersection, neither is confirmed. (3) Any false rejection of a true elementary hypothesis forces rejection of \(H_T\), the intersection of all true elementary nulls. The local test of \(H_T\) has rejection probability at most 0.05, so the probability of any false elementary rejection is also at most 0.05. (4) Testing only the selected arm does not test every required intersection from the original four-arm family; moreover, selection or adaptation may make the ordinary p-value invalid unless it is valid under the actual design.