Why Selecting a Subset Does Not End the Multiplicity Problem
An adaptive trial may use interim data to select a subset of treatment comparisons for further study or for final claims. That selection can be sensible, but it changes which hypotheses are of interest based on observed data. If the trial then tests only the selected hypotheses as though the subset had been fixed in advance, the resulting error control may be invalid. The key question is not only which hypotheses were selected; it is whether the evidence justifies each selected rejection against every relevant intersection null.
Closure provides a systematic way to audit those claims. The full family of hypotheses defines a collection of intersection hypotheses. To reject an individual hypothesis, the procedure must reject every intersection hypothesis that includes it. Selection determines which individual claims are being considered, but it does not, by itself, remove the required intersection tests from the original family.
Suppose an interim rule selects the random subset \(S\subseteq\{1,\ldots,m\}\), and final claims are allowed only for hypotheses in \(S\). For a selected \(H_i\), the required intersections are still indexed by every set \(J\) that contains \(i\), including sets that also contain hypotheses excluded from \(S\). Testing only intersections formed from subsets of \(S\) is not generally enough. It omits possible true intersection nulls in the original family.
This distinction is especially important when selection and testing use the same data. A local p-value that would be valid for a prespecified test may not remain valid after selecting hypotheses because their interim results looked favorable. The local tests used in the closure must be valid for the actual adaptive procedure. Depending on the design, that may require unconditional calibration that accounts for adaptation or tests valid conditional on the selection event.
Tracing the Required Intersections
For a selected set \(S\), write down the required tests separately for each selected hypothesis. For \(H_i\), the list consists of all \(H_J\) with \(i\in J\). The list is not limited to \(J\subseteq S\). Some required tests are shared: if two selected hypotheses are both in \(J\), the local test of \(H_J\) contributes to the decision for both.
If each local test supplies a p-value \(p_J\), and these p-values are valid for their corresponding intersection hypotheses under the adaptive procedure, the closure-adjusted p-value for \(H_i\) is
The maximum appears because every required local test must pass: \(H_i\) is rejected at level \(\alpha\) only when \(p_i^{\mathrm{closed}}\leq\alpha\). This calculation is meaningful only if the original family is specified and all relevant local p-values are valid. A small p-value for \(H_i\) alone cannot compensate for a large p-value for an intersection that also contains \(i\).
Worked Example: A Selected Pair and an Omitted Intersection
Consider three hypotheses, \(H_1,H_2,H_3\). An interim rule selects \(S=\{1,3\}\). For this hypothetical example, suppose the following are valid p-values for the indicated local tests at the final analysis.
| Intersection | Local p-value |
|---|---|
| \(H_1\) | 0.024 |
| \(H_2\) | 0.400 |
| \(H_3\) | 0.030 |
| \(H_{\{1,2\}}\) | 0.018 |
| \(H_{\{1,3\}}\) | 0.021 |
| \(H_{\{2,3\}}\) | 0.070 |
| \(H_{\{1,2,3\}}\) | 0.012 |
At \(\alpha=0.05\), the selected hypothesis \(H_1\) requires tests of \(H_1\), \(H_{\{1,2\}}\), \(H_{\{1,3\}}\), and \(H_{\{1,2,3\}}\). Thus \(p_1^{\mathrm{closed}}=\max(0.024,0.018,0.021,0.012)=0.024\), so \(H_1\) is rejected.
For \(H_3\), the required tests are \(H_3\), \(H_{\{1,3\}}\), \(H_{\{2,3\}}\), and \(H_{\{1,2,3\}}\). Therefore \(p_3^{\mathrm{closed}}=\max(0.030,0.021,0.070,0.012)=0.070\), so \(H_3\) is not rejected. Testing only intersections formed from the selected pair would omit \(H_{\{2,3\}}\) and would incorrectly make the closure audit appear to pass for \(H_3\).
Why the Full Set of Required Tests Matters
The logic of closure can be seen by considering what a false rejection would mean. Let \(T\) be the set of indices for which the null hypotheses are in fact true. If the procedure falsely rejects any \(H_i\) with \(i\in T\), closure requires that it also reject \(H_T\), the intersection of all true null hypotheses. Since \(H_T\) is true, that rejection must come from a local test of a true intersection. This is why the local tests for the required intersections, rather than just the selected individual tests, underpin the error-control argument.
The same logic exposes the problem with omitting intersections involving unselected hypotheses. If a selected hypothesis is false-rejected, the true intersection \(H_T\) may include indices outside \(S\). A test confined to intersections within \(S\) may never test that true \(H_T\). Consequently, the closure argument cannot use that omitted local test to control the error.
Worked Example: Computing a Closure-Adjusted P-Value
Suppose there are three original hypotheses and the selection rule leaves only \(H_2\) eligible for a final claim. The valid local p-values for the four intersections containing \(H_2\) are, hypothetically, \(p_2=0.031\), \(p_{\{1,2\}}=0.024\), \(p_{\{2,3\}}=0.041\), and \(p_{\{1,2,3\}}=0.018\).
The closure-adjusted p-value is \(p_2^{\mathrm{closed}}=\max(0.031,0.024,0.041,0.018)=0.041\). At \(\alpha=0.05\), the procedure rejects \(H_2\), because every required local p-value is at most 0.05. The claim is not based solely on the marginal p-value 0.031 for \(H_2\); the three intersection tests are also required.
If instead the local p-value for \(H_{\{2,3\}}\) were 0.058, the maximum would be 0.058 and \(H_2\) would not be rejected at level 0.05. A small p-value for the other intersections would not change that result.
Selection-Aware Local Tests
Closure specifies which local tests are needed; it does not make an invalid local test valid. When data-dependent selection affects the distribution of a test statistic, the p-value for an intersection must account for that selection. One possible requirement is conditional validity: for every selection outcome \(S=s\) with positive probability, a local test of a true \(H_J\) has conditional rejection probability at most \(\alpha\), given \(S=s\). Another possibility is an appropriately calibrated unconditional test that accounts for the selection rule. Which approach is appropriate depends on the design and the inferential target.
When conditional local tests are used, their coverage must match the intersections needed for the selected claims. In particular, it is not enough to have conditional tests only for intersections contained in \(S\) if an excluded hypothesis can belong to the true intersection that closure needs to test. The selection rule, the tested intersections, and the calibration of their local tests must fit together.
Worked Example: Auditing Two Claims in a Four-Hypothesis Family
In a hypothetical trial, an interim rule selects \(S=\{1,4\}\) from four hypotheses. Assume all listed local p-values are valid for the adaptive analysis. The required sets for \(H_1\) are \(\{1\},\{1,2\},\{1,3\},\{1,4\},\{1,2,3\},\{1,2,4\},\{1,3,4\},\{1,2,3,4\}\). Their p-values, in that order, are 0.012, 0.021, 0.028, 0.019, 0.035, 0.044, 0.032, and 0.027. Therefore \(p_1^{\mathrm{closed}}=0.044\), and \(H_1\) is rejected at \(\alpha=0.05\).
For \(H_4\), the required sets are \(\{4\},\{1,4\},\{2,4\},\{3,4\},\{1,2,4\},\{1,3,4\},\{2,3,4\},\{1,2,3,4\}\). Suppose their p-values are 0.016, 0.019, 0.061, 0.023, 0.044, 0.032, 0.052, and 0.027. Then \(p_4^{\mathrm{closed}}=0.061\), so \(H_4\) is not rejected. The p-value 0.061 for \(H_{\{2,4\}}\) is required even though \(H_2\) was not selected.
The result is one justified claim, \(H_1\), and one unsubstantiated selected claim, \(H_4\). Selection makes both hypotheses eligible for consideration; it does not guarantee that either one will pass closure.
Comparison, Diagnostics, and Limitations
Bonferroni and Holm procedures offer alternative ways to control family-wise error for a specified family. Holm’s procedure is a step-down method related to closed testing with Bonferroni local tests; it can be more convenient than explicitly listing every intersection. Neither method licenses silently dropping unselected hypotheses from the original family when the selection was data-dependent. Selection-adjusted approaches can sometimes use the selection information more directly, but they must provide a valid error-control argument for the actual selection rule. Closure is particularly useful as an audit because it makes the intersections required by each claim explicit.
A practical audit should check the following before reporting a selected rejection:
- Is the original family of hypotheses defined, rather than only the subset observed after selection?
- For each selected \(H_i\), are all intersections \(H_J\) with \(i\in J\) identified?
- Are intersections involving unselected hypotheses included when they contain a selected hypothesis?
- Are the local tests valid under the adaptive procedure, with selection accounted for where necessary?
- Does each claimed rejection require, and pass, every relevant local test?
A frequent failure is to treat the observed subset as if it had been chosen before any data were examined. For example, a report might say, “Only two treatments were selected, and both had p-values below 0.05, so both are significant.” The correction is to identify the original family, check every required intersection for each claimed hypothesis, and verify that the local tests remain valid after selection. If the necessary tests were not performed or cannot be justified, the closure argument does not support those rejections.
Closure can be computationally demanding because \(m\) hypotheses generate \(2^m-1\) nonempty intersections. Some procedures exploit structure to avoid evaluating every test separately, but any shortcut must preserve the same logical requirements. A selected subset may reduce the number of individual claims to audit, yet each claim can still require many local tests. The method also depends on prespecified, defensible local tests: closure cannot repair poor calibration or an unaccounted data-dependent analysis.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- There are four original hypotheses and \(S=\{2,4\}\). Which intersection tests are required to reject \(H_2\)?
- For a selected \(H_1\), the required local p-values are 0.01, 0.03, 0.04, and 0.06. What is its closure-adjusted p-value, and is it rejected at \(\alpha=0.05\)?
- Why can testing only intersections contained in the selected subset fail to justify a selected rejection?
- What additional issue arises if the same interim data both select hypotheses and determine ordinary final p-values?
Solutions. (1) Test every \(H_J\) with \(2\in J\), for every nonempty \(J\subseteq\{1,2,3,4\}\). This includes intersections involving \(H_1\), \(H_3\), or both, whether or not those hypotheses were selected. (2) The maximum is \(0.06\); the procedure does not reject at 0.05 because at least one required local test fails. (3) The true intersection associated with a false rejection may include unselected hypotheses. Omitting that intersection leaves a gap in the closure argument. (4) Selection can change the distribution of the test statistic or p-value. The local test must therefore be calibrated for the adaptive procedure, for example through a justified selection-aware or conditional test; ordinary fixed-analysis validity alone is not sufficient.