Why Treatment Selection Calls for Closed Testing
A seamless trial may begin with several candidate treatments and use interim information to decide which will continue. The final report may then make more than one treatment-specific claim. The difficulty is not only that several hypotheses are tested: the treatment set and the evidence may depend on the same trial data. A procedure that controls error for a fixed set of tests does not automatically retain that control after adaptive selection.
Closed testing gives a general way to organize such claims. Its error-control guarantee depends on valid tests of every relevant intersection of hypotheses. Adaptation can be part of the trial design, but it must be accounted for when those local tests are calibrated. Applying a multiplicity adjustment after selection cannot repair a p-value that was already invalid because of the selection process.
The elementary hypotheses correspond to the individual claims, such as “treatment \(i\) is superior to control on the prespecified outcome.” An intersection null \(H_I\) says that all the elementary nulls indexed by \(I\) are true. With \(m\) treatments, the full closed family has \(2^m-1\) nonempty intersections. The tests need not all use the same statistic, but each must be valid at level \(\alpha\) whenever its own intersection null is true.
Local Tests and Adaptive Selection
A local test addresses one intersection at a time. For example, a test of \(H_{\{1,2\}}=H_1\cap H_2\) must control its probability of rejection when both treatment-specific nulls are true. It is not enough for that test to work only when one of the two nulls is false.
In a seamless design, the local test may use evidence from multiple stages. Stagewise p-values or a combination test can be used, but the test must be valid under the intersection being tested for the complete adaptive procedure. This includes the selection rule, any prespecified changes at the stage boundary, and the use of interim data. If a p-value is valid only for a treatment fixed in advance, selecting that treatment based on interim outcomes may invalidate it for the selected treatment.
A safe general formulation is to define, for each nonempty \(I\), a local p-value \(p_I\) satisfying \(P(p_I\leq u\mid H_I)\leq u\) for every \(u\in[0,1]\), under the trial’s specified design and analysis. Here the probability is understood under any configuration of parameters for which \(H_I\) is true. Such p-values are called valid or super-uniform under their corresponding intersection nulls. The closure rule then operates on these local tests.
If each treatment has a p-value \(p_i\) that is valid under \(H_i\) for the complete adaptive design, a simple local test for an intersection is the Bonferroni test. Under \(H_I\), each \(H_i\) with \(i\in I\) is true, so each corresponding \(p_i\) is valid there as well. No independence assumption among those p-values is required.
This construction is reliable only if the component p-values are valid for the adaptive design. If treatment selection makes a selected treatment’s p-value anti-conservative under its null, multiplying it by \(|I|\) does not restore validity. One option is to use selection-valid treatment p-values supplied by a prespecified stagewise method. Another is to construct and calibrate local tests directly for the adaptive procedure. In either case, the hypothesis family and the local-test validity conditions must be clear before interpreting the closure results.
Worked Example: Applying Closure to Three Treatment Claims
Hypothetical trial. Three candidate treatments, A, B, and C, are compared with control on a prespecified outcome. Suppose the complete trial analysis yields p-values \(p_A=0.008\), \(p_B=0.022\), and \(p_C=0.20\). Assume these p-values are valid under their respective treatment nulls for the design used. Use Bonferroni local tests at \(\alpha=0.05\).
For a singleton intersection, the local p-value is the component p-value: \(p_{\{A\}}=0.008\), \(p_{\{B\}}=0.022\), and \(p_{\{C\}}=0.20\). For the pairwise intersections, \(p_{\{A,B\}}=\min(1,2\min(0.008,0.022))=0.016\), \(p_{\{A,C\}}=\min(1,2\min(0.008,0.20))=0.016\), and \(p_{\{B,C\}}=\min(1,2\min(0.022,0.20))=0.044\).
For the three-way intersection, \(p_{\{A,B,C\}}=\min(1,3\min(0.008,0.022,0.20))=0.024\). Treatment A can be claimed because every intersection containing A is rejected: its singleton, the AB intersection, the AC intersection, and the ABC intersection all have p-values at most 0.05. Treatment B also passes all its required intersections: its singleton p-value is 0.022, and the AB, BC, and ABC values are 0.016, 0.044, and 0.024. Treatment C cannot be claimed because its singleton null is not rejected: \(0.20>0.05\).
The conclusions are individual claims, not a statement that all three treatments work. The calculation also shows why testing only the smallest p-value or adjusting only the final reported claims would not describe the closed procedure.
Why Closure Controls Familywise Error
The familywise error rate (FWER) is the probability of rejecting at least one true elementary null hypothesis. Closure controls this probability if every local test controls its own type I error at level \(\alpha\). The key is to consider the intersection of all elementary nulls that are actually true, rather than to assume the tests are independent.
For a particular parameter configuration, let \(T\) be the nonempty set of indices \(i\) for which \(H_i\) is true. If there are no true nulls, a false rejection is impossible.
The intersection \(H_T=\bigcap_{i\in T}H_i\) is true in this configuration. Its local test therefore has rejection probability at most \(\alpha\).
If closure rejects any true elementary hypothesis \(H_i\), then it must reject every intersection containing \(i\), including \(H_T\), since \(i\in T\).
The event of at least one false rejection is contained in the event that the true intersection \(H_T\) is rejected. Its probability is therefore at most \(\alpha\).
In symbols, for nonempty \(T\), the closure rule gives \[ \{\text{at least one true }H_i\text{ is rejected}\} \subseteq \{\text{reject }H_T\}, \qquad P(\text{at least one false rejection})\leq P(\text{reject }H_T)\leq\alpha. \] This argument makes no independence assumption. It does require that the local test of \(H_T\) has level-\(\alpha\) validity under the full design, including any adaptive treatment selection. Adaptation does not break the closure argument; invalid local calibration does.
Worked Example: Selection at the Stage Transition
Hypothetical trial. A trial starts with three candidate treatments. A prespecified interim rule selects A and C for continued study. Suppose the final, selection-valid p-values for the full adaptive design are \(p_A=0.004\), \(p_B=0.040\), and \(p_C=0.11\). The reporting rule permits claims only for selected treatments. Apply the closed Bonferroni procedure at \(\alpha=0.05\) to the original three-treatment family.
The pairwise local p-values are \(p_{\{A,B\}}=2(0.004)=0.008\), \(p_{\{A,C\}}=2(0.004)=0.008\), and \(p_{\{B,C\}}=2(0.040)=0.080\). The three-way value is \(p_{\{A,B,C\}}=3(0.004)=0.012\). For A, all intersections containing A have p-values 0.004, 0.008, 0.008, and 0.012, so the procedure rejects \(H_A\). For C, the singleton intersection has p-value 0.11, so it does not reject \(H_C\). Thus the permitted selected-treatment claim is supported for A but not C.
The closure family still includes B even though B was not selected. The protocol’s reporting rule prevents making a claim for B; its presence in the intersection family protects the error guarantee for the original candidate-treatment family. Crucially, the numbers above must be valid after applying the selection rule. If they were ordinary fixed-treatment p-values that ignored interim selection, this calculation would not establish FWER control.
Bonferroni Closure, Holm’s Procedure, and Other Local Tests
Closed testing is a framework, not one fixed choice of local test. Bonferroni tests are attractive when dependence among treatment comparisons is unknown or complicated: they need valid component p-values but make no independence assumption. They can be conservative, especially when several treatment statistics are strongly correlated or when the number of candidates is large.
For a fixed family of valid p-values, closure using Bonferroni local tests gives the same elementary rejection decisions as Holm’s step-down procedure. Holm’s procedure sorts the p-values from smallest to largest and compares the \(j\)th smallest with \(\alpha/(m-j+1)\), stopping at the first failure. It is often quicker to calculate than all \(2^m-1\) intersections. Closed testing is more flexible when different intersections have different valid local tests, for example tests that use a prespecified joint distribution or a tailored combination procedure.
A single-step Bonferroni adjustment instead compares each p-value with \(\alpha/m\). It is valid under the same basic p-value validity conditions but may reject fewer hypotheses than Holm’s procedure. None of these procedures makes selection ignorable: the p-values or local tests still need to be calibrated for the adaptive design. Nor does closure establish that the outcome, estimand, or treatment comparison is clinically appropriate; those are separate design and interpretation questions.
Worked Example: Checking a Bonferroni Intersection p-Value
Suppose an intersection \(H_I\) contains \(k=2\) true treatment nulls, and the component p-values are 0.030 and 0.060. The Bonferroni intersection p-value is \(p_I^{\mathrm{Bonf}}=\min(1,2\min(0.030,0.060))=\min(1,0.060)=0.060\). At level 0.05, this local test does not reject the intersection.
To verify validity, let \(0\leq u<1\). Then \[ \{p_I^{\mathrm{Bonf}}\leq u\} = \{\min_{i\in I}p_i\leq u/|I|\}. \] Under \(H_I\), every \(H_i\) for \(i\in I\) is true. By the union bound and validity of each component p-value, \[ P(p_I^{\mathrm{Bonf}}\leq u\mid H_I) \leq \sum_{i\in I}P(p_i\leq u/|I|\mid H_I) \leq |I|\frac{u}{|I|}=u. \] At \(u=1\), handle the endpoint separately: \(P(p_I^{\mathrm{Bonf}}\leq1\mid H_I)=1\leq1\). Thus the local p-value is valid over the full range \(u\in[0,1]\). No independence assumption was used.
Diagnostics and Common Pitfalls
Before interpreting closed-testing results, check that the analysis has a well-defined family of hypotheses and a local test for every required intersection. In particular, verify that the p-values or test statistics were calibrated for the actual stage-transition and selection rules. If the method relies on conditional validity given interim data, check that this condition is stated and justified, rather than assuming that validity conditional on a fixed treatment automatically carries over to a selected treatment.
A frequent mistake is to test only the treatments chosen at the interim look, apply a standard multiplicity correction to their final p-values, and declare FWER control. That reasoning can fail twice: selection may have affected the p-values, and the intersection tests for the original family may have been omitted. The correction is to specify the intended claim family, ensure the local tests are valid under each relevant intersection for the adaptive design, and apply the closure rule to every intersection required for each claim.
Another pitfall is to treat non-rejection of an intersection as proof that all its component nulls are true. A local test’s failure to reject is not evidence that the intersection has been established; it simply blocks the elementary claims that require that intersection to be rejected. Finally, closure controls the probability of one or more false rejections at the stated level. It does not guarantee high power, eliminate bias in effect estimates after selection, or replace prespecification of the clinical estimand and reporting rules.
Check Your Understanding
Try each question, then compare your reasoning with the solutions below.
- With three treatment hypotheses, how many nonempty intersection hypotheses are included in the closed family?
- At \(\alpha=0.05\), suppose a treatment’s singleton null is rejected, but one larger intersection containing it has local p-value 0.07. Can the treatment claim be rejected by closure? Why?
- State the event-containment argument that establishes FWER control when all local intersection tests have level-\(\alpha\) validity.
- Why is multiplying a selected treatment’s ordinary p-value by the number of selected treatments not, by itself, a guarantee of adaptive FWER control?
Solutions. (1) There are \(2^3-1=7\) nonempty subsets, hence seven intersection hypotheses. (2) No. Closure requires rejection of every intersection containing that treatment, and the intersection with p-value 0.07 is not rejected at 0.05. (3) Let \(T\) index all true elementary nulls. Any false rejection under closure entails rejection of \(H_T\), so the probability of at least one false rejection is at most \(P(\text{reject }H_T)\leq\alpha\). (4) Selection may make the ordinary p-value invalid under its null; a multiplicity adjustment cannot repair that invalidity. The p-value or local test must first be calibrated for the full adaptive procedure.