Why Adaptive Claims Need a Family-Wide Rule
An adaptive trial may end with several possible claims: a treatment benefits one population, another treatment benefits a different population, or both. The interim decision can change which claims are pursued or reported. Even if each individual test has a small false-rejection probability, making several claims creates a chance of at least one false claim. The relevant error rate is the family-wise error rate: the probability of rejecting one or more true null hypotheses in the family.
Closed testing provides a way to control that probability while allowing local tests to account for the trial’s adaptive design. Its central requirement is that the family of possible claims be represented by a set of hypotheses, and that every intersection of those hypotheses have a valid test. The local tests may be different for different intersections; each must control its own type I error at level \(\alpha\), including under the adaptation used in the trial.
The intersection is a logical conjunction, not necessarily a new scientific claim with a simple pooled parameter. For example, \(H_1\cap H_3\) says that both null statements hold. Its local test must test that joint null. In particular, a test of \(H_1\cap H_3\) is not automatically valid merely because separate tests of \(H_1\) and \(H_3\) are valid.
Worked Example: Translating Claims Into Intersections
Suppose the protocol permits three efficacy claims. Let \(\delta_i\) be the treatment effect in the population and endpoint specified for claim \(i\), with positive values representing benefit. Define \(H_1:\delta_1\leq 0\) as no benefit for treatment A in the adult population; \(H_2:\delta_2\leq 0\) as no benefit for treatment B in the adult population; and \(H_3:\delta_3\leq 0\) as no benefit for treatment A in the older population.
The pairwise intersections are:
- \(H_{12}\): neither treatment A nor treatment B benefits adults.
- \(H_{13}\): treatment A benefits neither the adult nor the older population.
- \(H_{23}\): treatment B does not benefit adults, and treatment A does not benefit the older population.
The triple intersection \(H_{123}\) says that all three no-benefit statements hold. If the interim review leads the team to focus on claims 1 and 3, that does not change the meaning of \(H_{13}\), nor does it remove \(H_{123}\) from the closure requirements for either claim.
The Closed-Testing Rule
For every nonempty subset \(K\), let the local test of \(H_K\) reject when its p-value \(p_K\) is at most \(\alpha\). The p-value must be valid for that intersection: for every parameter configuration in \(H_K\), \(P(p_K\leq u)\leq u\) for \(0\leq u\leq 1\). In an adaptive trial, this validity must hold for the procedure actually used, including its permitted interim decisions. A sufficient conditional guarantee is \(P(p_K\leq\alpha\mid\mathcal{F})\leq\alpha\) under \(H_K\) for every relevant interim history \(\mathcal{F}\); averaging over those histories then gives an unconditional rejection probability no greater than \(\alpha\).
Once the local tests are defined, the decision for an elementary claim is mechanical: claim \(i\) is rejected only if every local test for an intersection containing \(i\) rejects. For three hypotheses, claim 1 therefore requires rejection of \(H_1,H_{12},H_{13},H_{123}\). The other claims have analogous requirements. There is no division of \(\alpha\) by the number of intersections: each local test is level \(\alpha\), and the closure rule, rather than a Bonferroni allocation over all intersections, provides the family-wide guarantee.
Worked Example: Applying Closure to Local P-Values
Consider the three claims above at \(\alpha=0.05\). Suppose the adaptive design supplies the following valid local p-values. These are hypothetical outputs of the prespecified local tests, not unadjusted p-values assumed to remain valid automatically after adaptation.
| Hypothesis | Local p-value | Reject at 0.05? |
|---|---|---|
| \(H_1\) | 0.018 | Yes |
| \(H_2\) | 0.041 | Yes |
| \(H_3\) | 0.009 | Yes |
| \(H_{12}\) | 0.032 | Yes |
| \(H_{13}\) | 0.012 | Yes |
| \(H_{23}\) | 0.064 | No |
| \(H_{123}\) | 0.028 | Yes |
For claim 1, the required p-values are \(0.018,0.032,0.012,\) and \(0.028\). Each is at most \(0.05\), so \(H_1\) is rejected. For claim 2, \(H_{23}\) is required and \(0.064>0.05\); thus \(H_2\) is not rejected. Claim 3 also requires \(H_{23}\), so \(H_3\) is not rejected. The closed procedure therefore supports claim 1 only, despite the small marginal p-values for \(H_2\) and \(H_3\).
If the adaptive trial ends by presenting only claims 1 and 3, closure is still evaluated against the full candidate family. Claim 1 passes its required intersections; claim 3 does not. Dropping claim 2 from the final report does not make the failed \(H_{23}\) test disappear from claim 3’s closure requirements.
Why Closure Controls the Chance of Any False Claim
The error-control argument uses the true hypotheses at the parameter configuration under consideration. Let \(T\) be the set of indices whose elementary null hypotheses are true. If \(T\) is nonempty, then \(H_T\), the intersection of all true elementary nulls, is itself true. Any false rejection among the elementary hypotheses must include rejection of at least one true \(H_i\). By the closed-testing rule, rejecting that \(H_i\) requires rejecting every intersection that contains it, and in particular requires rejecting \(H_T\).
Because the local test of the true \(H_T\) has type I error at most \(\alpha\), the probability of the event on the left is no greater than \(\alpha\). If \(T\) is empty, there are no true elementary nulls to reject, so the family-wise error rate is zero. This establishes strong family-wise error control: the bound holds whether all, some, or none of the elementary null hypotheses are true.
The same reasoning applies when local tests account for interim adaptations. In the conditional case, if the local rejection probability for \(H_T\) is at most \(\alpha\) at every interim history, then
The closure proof does not require independence among tests, nor does it require the final set of reported claims to be chosen independently of the data. It does require that the local test for each relevant intersection really be valid under that intersection null, after accounting for the design’s data-dependent decisions.
Worked Example: Conditional Error Bound for Two True Claims
Suppose claims 1 and 3 are both true at the parameter configuration under consideration, so \(H_{13}\) is true. An interim rule has two possible histories, \(F=a\) and \(F=b\), with probabilities 0.60 and 0.40 under this configuration. A valid adaptive local test of \(H_{13}\) has conditional rejection probabilities 0.03 after history \(a\) and 0.05 after history \(b\).
The unconditional probability of rejecting this true intersection is
If either true claim 1 or true claim 3 is rejected by closure, \(H_{13}\) must also have been rejected. Consequently, the probability of falsely rejecting at least one of these two true claims is at most \(0.038\), which is below \(\alpha=0.05\). The bound remains valid even if the interim history influences which claims are emphasized in the final report, provided all required local tests retain their stated validity.
Building and Auditing the Family
In practice, the claim family should include every elementary claim the adaptive procedure could make. For a finite family of \(m\) claims, there are \(2^m-1\) nonempty intersections. The required local tests can be organized before the trial and implemented using the conditional-error or other adaptive-testing methods appropriate to the design. The key is not that the same test must be used at every intersection, but that each intersection has a valid test at the stated level.
Write down every claim the adaptive procedure is permitted to report, not only the claims expected to be selected.
Specify the treatment, population, endpoint, effect direction, and null boundary that make the claim false or unsupported.
For each subset of claims, state the joint null as the conjunction of its elementary nulls.
Confirm that the test for each intersection controls type I error under that intersection and under every allowed design adaptation.
Reject an elementary null only when every intersection containing it is rejected by its valid local test.
There are useful alternatives, but they answer the problem in different ways. Testing each claim at level \(\alpha\) without adjustment is simpler, but does not generally control the probability of any false claim when there are multiple claims. Bonferroni-style adjustment can control that probability when its component p-values are valid, but can be conservative and may not use the structure of the joint nulls. Closed testing instead uses intersection-specific tests and can exploit their design and logical structure; its computational cost can grow rapidly as the family becomes large.
Neither closure nor a multiplicity adjustment repairs an invalid local p-value. For example, if an ordinary fixed-design p-value is used after an adaptation that changes the test’s error distribution, the fact that it appears in a closed-testing table does not make it valid. Similarly, computing tests only for the intersections among claims selected at interim is not enough unless the procedure has separately justified why every closure requirement for a reported claim is met.
Common Mistakes and Limitations
- Testing only the selected claims. The wrong reasoning is that unselected claims cannot affect the final error rate. The correction is to retain the full possible family and all intersections required by closure for any claim that may be rejected.
- Treating a small elementary p-value as sufficient. A claim is not rejected just because its own local test rejects. Every intersection containing it must also reject.
- Assuming closure makes any test valid. Closure controls family-wise error only when the local test for each true intersection is valid under the adaptive design.
- Confusing the intersection with a union. \(H_{12}\) means both \(H_1\) and \(H_2\) are true. It is not the statement that at least one of them is true.
A complete audit should therefore check the mapping from scientific claims to null hypotheses, the set of possible claims, the validity of every local intersection test, and the final closure decisions. A finite candidate family may be costly to test when it is large; reducing the family is defensible only if the reduced set still contains every claim the procedure could make. The statistical interpretation also remains limited to the hypotheses and effect definitions specified: closure does not turn a test into evidence for a different endpoint, population, or effect direction.
Check Your Understanding
Answer each question, then compare with the solutions below.
- For two elementary nulls \(H_1\) and \(H_2\), which local tests must reject before closed testing can reject \(H_1\)?
- At \(\alpha=0.05\), the local p-values for \(H_1,H_2,H_{12}\) are 0.02, 0.03, and 0.07. Which elementary nulls can closure reject?
- Why does testing each of four possible claims at level 0.05 not, by itself, establish family-wise error control at 0.05?
- If the local test of a true intersection has conditional rejection probability at most 0.04 after every interim history, what upper bound follows for its unconditional type I error?
Solutions. (1) It must reject both \(H_1\) and \(H_{12}\), since these are exactly the nonempty intersections containing claim 1. (2) Neither elementary null is rejected: \(H_1\) requires rejection of \(H_{12}\), but \(0.07>0.05\); \(H_2\) has the same failed intersection requirement. (3) The chance of at least one false rejection can exceed the per-test level when several true nulls are tested; validity of each test separately is not a family-wide guarantee. (4) The unconditional type I error is at most 0.04, since averaging conditional probabilities bounded by 0.04 cannot exceed 0.04.