From Local Sequential Tests to Arm-Level Decisions
In a multi-arm, multi-stage trial, evidence for a treatment may be assessed at more than one interim analysis. A treatment can look promising at one stage and less promising at another; meanwhile, testing several treatment hypotheses creates a familywise error problem. Closed testing combines the two tasks: it uses a valid sequential test for each intersection of treatment hypotheses, then uses all relevant intersection-test results to decide whether an individual treatment hypothesis can be rejected.
The analysis below concerns four prespecified treatment-versus-control hypotheses. Its key inputs are the local sequential p-values for the intersections. These must already account for the planned stages, the stopping rules, and the local test calibration. A raw p-value from the final look alone is not a substitute.
With four treatment hypotheses there are \(2^4-1=15\) nonempty intersections: four singletons, six pairs, four triples, and the global intersection \(H_{\{1,2,3,4\}}\). At familywise significance level \(\alpha\), reject \(H_i\) precisely when \(p_{i,\mathrm{adj}}\leq\alpha\). Equivalently, every local sequential test whose intersection contains \(H_i\) must reject at that level.
What the Sequential Local p-Values Must Represent
For a particular intersection \(H_I\), the local test is a rule that can evaluate evidence at the planned interim and final stages. Its boundaries, or an equivalent sequential p-value calculation, must control the probability of rejecting \(H_I\) when it is true at the specified local level. The calibration must use the design’s actual stage structure and account for dependence between looks, such as the correlation induced by using accumulating data. If the analysis permits stopping early, that stopping rule must also be included.
A sequential local p-value can be defined as the smallest local significance level at which the calibrated sequential test would reject on the observed path of stagewise results. This interpretation requires a calibration that is nested in the local significance level: increasing the level cannot make a rejection less attainable. The p-value summarizes the complete observed path, not just the final statistic. When a valid sequential p-value is available, comparison with \(\alpha\) gives the same decision as applying the corresponding calibrated local test at level \(\alpha\).
Worked Example: Reading a Two-Stage Local Result
Suppose a hypothetical two-stage trial has a local sequential test of \(H_{\{1,2\}}\), the intersection null for treatments 1 and 2. The prespecified local test uses boundaries calibrated for both looks. On the observed path, the interim result does not cross its boundary, and the calibrated sequential p-value after the final look is \(0.018\). For this local test, the p-value calculation uses both stages; the final-look-only p-value would be \(0.014\).
At local level \(\alpha=0.025\), compare the sequential p-value with the level: \(0.018\leq0.025\). Thus the local sequential test rejects \(H_{\{1,2\}}\). The final-look-only value \(0.014\) is not the value used for this decision, because it does not, by itself, account for the planned opportunity to reject at the interim look.
This is only one of the intersections needed for a closed-testing decision. Rejecting \(H_{\{1,2\}}\) is not sufficient to reject either treatment hypothesis: all other intersections containing that treatment must also be rejected.
Calculating Adjusted p-Values
For each treatment \(i\), list every nonempty subset \(I\) that contains \(i\), then take the largest local sequential p-value on that list. The maximum is necessary because the closed procedure requires every such local test to reject. A single non-rejected intersection blocks the treatment-level rejection, even if the singleton test has a very small p-value.
Worked Example: Finding One Treatment’s Adjusted p-Value
Suppose a hypothetical analysis of four treatments produces these local sequential p-values for the eight intersections containing treatment 1: \(p_{\{1\}}^{\mathrm{seq}}=0.003\), \(p_{\{1,2\}}^{\mathrm{seq}}=0.008\), \(p_{\{1,3\}}^{\mathrm{seq}}=0.019\), \(p_{\{1,4\}}^{\mathrm{seq}}=0.012\), \(p_{\{1,2,3\}}^{\mathrm{seq}}=0.020\), \(p_{\{1,2,4\}}^{\mathrm{seq}}=0.016\), \(p_{\{1,3,4\}}^{\mathrm{seq}}=0.021\), and \(p_{\{1,2,3,4\}}^{\mathrm{seq}}=0.023\).
The adjusted p-value is the maximum of these eight values: \[ p_{1,\mathrm{adj}}=\max(0.003,0.008,0.019,0.012,0.020,0.016,0.021,0.023)=0.023. \] At familywise level \(\alpha=0.025\), treatment 1’s hypothesis is rejected because \(0.023\leq0.025\). The singleton result alone would not determine this conclusion; the other seven tests are required by closure.
Complete Four-Treatment Analysis
Consider the following fully hypothetical results. The design has two planned looks and familywise level \(\alpha=0.025\). The entries are local sequential p-values from tests calibrated for the respective intersections under the planned stage structure. Thus, each entry already reflects sequential monitoring; the table is not a list of unadjusted final-look p-values.
| Intersection | Local sequential p-value |
|---|---|
| \(H_{\{1\}}\) | 0.003 |
| \(H_{\{2\}}\) | 0.010 |
| \(H_{\{3\}}\) | 0.041 |
| \(H_{\{4\}}\) | 0.014 |
| \(H_{\{1,2\}}\) | 0.008 |
| \(H_{\{1,3\}}\) | 0.019 |
| \(H_{\{1,4\}}\) | 0.012 |
| \(H_{\{2,3\}}\) | 0.030 |
| \(H_{\{2,4\}}\) | 0.017 |
| \(H_{\{3,4\}}\) | 0.026 |
| \(H_{\{1,2,3\}}\) | 0.020 |
| \(H_{\{1,2,4\}}\) | 0.016 |
| \(H_{\{1,3,4\}}\) | 0.021 |
| \(H_{\{2,3,4\}}\) | 0.024 |
| \(H_{\{1,2,3,4\}}\) | 0.023 |
For treatment 1, the largest value among its eight containing intersections is \(0.023\), so \(p_{1,\mathrm{adj}}=0.023\). For treatment 2, the values include \(0.030\) for \(H_{\{2,3\}}\); the maximum is \(0.030\), so \(p_{2,\mathrm{adj}}=0.030\). For treatment 3, its singleton value is already \(0.041\), the largest among its containing intersections, giving \(p_{3,\mathrm{adj}}=0.041\). For treatment 4, the largest value is \(0.026\), from \(H_{\{3,4\}}\), giving \(p_{4,\mathrm{adj}}=0.026\).
Comparing each result with \(0.025\), reject \(H_1\), because \(0.023\leq0.025\). Do not reject \(H_2\), \(H_3\), or \(H_4\), because their adjusted p-values exceed \(0.025\). The conclusion is that the closed-testing procedure rejects the hypothesis for treatment 1 at the specified familywise level; it does not establish that treatment 1 is effective in a causal or clinical sense beyond what the trial’s hypothesis and estimand specify.
Why the Maximum Gives Strong Familywise Error Control
The closure principle provides a short justification for the procedure. Strong familywise error control means that, for every configuration of true and false treatment hypotheses, the probability of rejecting at least one true treatment hypothesis is at most \(\alpha\). The argument depends on valid local tests for all intersection hypotheses, not on independence among the treatment comparisons.
Let \(T\) be the set of indices for which \(H_i\) is true. If \(T\) is empty, there is no true treatment hypothesis to reject. Otherwise, \(H_T=\bigcap_{i\in T}H_i\) is a true intersection hypothesis.
If any true \(H_i\), with \(i\in T\), is rejected by closed testing, every intersection containing \(i\) must be rejected. In particular, \(H_T\), which contains \(i\), must be rejected.
The probability of rejecting the true \(H_T\) is at most \(\alpha\), by validity of its local sequential test. Therefore the probability of rejecting at least one true treatment hypothesis is no greater than \(\alpha\).
The adjusted p-value formula is the matching numerical version of the same logic. Treatment \(i\) is rejected at level \(\alpha\) exactly when every containing intersection has \(p_I^{\mathrm{seq}}\leq\alpha\). That condition is equivalent to their maximum being at most \(\alpha\). Taking the maximum therefore reports the smallest common level at which the full closed-testing requirement for that treatment is met.
Diagnostics, Alternatives, and Limitations
A reliable analysis should make it possible to trace each treatment decision back to the intersections that determine it. Check that all 15 nonempty subsets appear, that each local p-value belongs to the correct intersection, and that the local sequential calibration matches the planned looks and stopping rules. Recomputing a selected subset with a different boundary or a different stopping convention can make the collection incoherent.
- Check the local input. Confirm that each p-value is sequentially calibrated for its intersection, rather than copied from an ordinary final-look test.
- Check the subset bookkeeping. A treatment’s maximum must include its singleton, all pairs containing it, all triples containing it, and the four-arm intersection.
- Check the decision threshold. Use the prespecified familywise level consistently; do not compare some intersections with a different level without a prespecified procedure.
- Check the boundary convention. If the local procedure rejects at a boundary, use a consistent inclusive comparison, \(p\leq\alpha\), in the adjusted-p-value calculation.
A simpler alternative is to allocate error across treatments, for example with a Bonferroni procedure, and apply valid sequential tests at the allocated levels. This can be easier to implement but may be more conservative because it does not use the full set of intersection-specific tests. Closed testing is more flexible, but it requires valid local tests and careful bookkeeping for every intersection. Neither approach repairs a misspecified sequential calibration, and closed testing can be computationally demanding as the number of treatment hypotheses grows. The four-arm calculation is manageable by hand; for larger families, software should be checked against a documented subset and decision audit.
Check Your Understanding
Use the definitions and procedure above. The solutions follow each question.
How many nonempty intersection hypotheses are tested for four treatment hypotheses?
Solution: There are \(2^4-1=15\): four singletons, six pairs, four triples, and one four-way intersection.
For one treatment, suppose the local sequential p-values across its containing intersections are \(0.006, 0.014, 0.021, 0.018, 0.027, 0.016, 0.023,\) and \(0.020\). Find its adjusted p-value and decision at \(\alpha=0.025\).
Solution: The maximum is \(0.027\), so the adjusted p-value is \(0.027\). Since \(0.027>0.025\), do not reject that treatment hypothesis.
Why is a final-look-only p-value generally not an appropriate replacement for a local sequential p-value in this analysis?
Solution: It may ignore the planned interim opportunity to reject and the dependence between looks. The local sequential calibration must account for the complete monitoring and stopping plan.
If any true treatment hypothesis is rejected under closed testing, which local intersection test must also have rejected?
Solution: The test of the intersection of all true treatment hypotheses must have rejected, because that intersection contains every true treatment hypothesis. Its validity bounds the probability of any false treatment-level rejection by \(\alpha\).