Why a Global-Null Test Is Not Enough
In a multi-arm trial, a type I error occurs if the analysis declares at least one ineffective treatment arm effective. The familywise error rate (FWER) is the probability of making one or more such false rejections. A procedure that controls this probability when every arm is ineffective controls error under the global null. Strong control is more demanding: the FWER must be at most the prespecified \(\alpha\) under every configuration of true and false arm hypotheses.
Sequential monitoring adds a further requirement. A local test for a hypothesis must control its probability of rejection over the entire trial path, including every planned look and any permitted adaptation. A test that has level \(\alpha\) at each look separately does not, on that fact alone, have level \(\alpha\) for the event of crossing at least one boundary.
The family of intersection hypotheses contains every nonempty subset. The closed-testing rule rejects an elementary hypothesis \(H_j\) only when every intersection hypothesis containing it has been rejected by its own local test. Thus rejecting \(H_1\), for example, requires rejecting \(H_{\{1\}}\), \(H_{\{1,2\}}\), and every other intersection whose index set includes arm 1.
The Strong-Control Argument
The proof is short once the event of a familywise error is linked to one particular local test. Let \(T\) be the set of indices for which the elementary null hypothesis is true in the actual parameter configuration. If \(T\) is empty, there is no true null to reject, so the probability of a familywise error is zero. If \(T\) is nonempty, the intersection \(H_T\) is true.
Suppose a familywise error occurs. Then some true elementary hypothesis \(H_j\), with \(j\in T\), is rejected. By the closed-testing rule, every intersection containing \(j\) must have been rejected. In particular, \(T\) contains \(j\), so \(H_T\) is one of those intersections. Therefore a familywise error implies rejection of the true intersection \(H_T\). The event inclusion gives the bound:
The final inequality is exactly the local validity assumption, applied to the true intersection \(H_T\). It must apply to the actual configuration, including any effects on arms outside \(T\). This establishes strong FWER control; the argument does not depend on how many elementary hypotheses are true or which ones they are.
What Sequential Validity Requires
For a sequential local test, “rejects” refers to the event that its rejection rule is met at any permitted look—not merely at the final analysis. If \(R_I\) denotes that event for \(H_I\), the required condition is \(P(R_I)\leq\alpha\) whenever \(H_I\) is true. The probability is calculated under the sequential design, including its stopping and arm-dropping rules. The earlier material on stagewise p-values and conditional-error recursion supplies ways to construct or assess such local tests; closed testing uses their validity rather than replacing their boundary derivations.
A local test for \(H_I\) must also remain valid when arms outside \(I\) have effects. For example, if \(H_{\{1,2\}}\) is true but arm 3 is effective, the local test of \(H_{\{1,2\}}\) must still have rejection probability at most \(\alpha\). It is not enough to calibrate that test only when all arms are null. The proof above applies a local test under the actual configuration, so global-null calibration alone cannot support its last inequality.
A practical way to record a local sequential analysis is with an intersection-specific sequential p-value \(p_I\), defined so that under \(H_I\) it is super-uniform: \(P(p_I\leq u)\leq u\) for every \(0\leq u\leq1\). Rejecting \(H_I\) when \(p_I\leq\alpha\) then gives the required level-\(\alpha\) local test. Alternatively, a calibrated boundary-crossing rule can be used directly; its probability of crossing at any look must be at most \(\alpha\).
Worked Example: Identifying the True Intersection
Consider a hypothetical three-arm trial with \(\alpha=0.05\). Suppose the true elementary nulls are \(H_1\) and \(H_3\), while \(H_2\) is false. Assume every intersection has a valid sequential local test. The observed local results are that \(H_{\{1,3\}}\) was rejected, while \(H_{\{1\}}\) was not rejected.
Here the true-index set is \(T=\{1,3\}\), so the true intersection relevant to the proof is \(H_T=H_{\{1,3\}}\). Rejection of \(H_1\) requires rejection of every intersection containing arm 1, including \(H_{\{1\}}\). Since that local test did not reject, closed testing does not reject \(H_1\), despite rejection of \(H_{\{1,3\}}\). A familywise error would require rejection of \(H_1\) or \(H_3\), and either rejection would require rejection of \(H_{\{1,3\}}\).
The general bound in this configuration is \(P(\text{familywise error})\leq P(\text{reject }H_{\{1,3\}})\leq0.05\). The first inequality follows from event inclusion; the second follows from validity of the sequential local test under the true intersection. No assumption about the effect on arm 2 is needed beyond the local test being valid when \(H_{\{1,3\}}\) is true and \(H_2\) is false.
Applying the Rule to Local Sequential Results
For a fixed data set, the closure rule can be checked mechanically. First determine which local tests rejected, using the prespecified sequential boundary or local p-value threshold. Then, for each arm, check all intersections containing it. The arm is rejected only if every one of those tests rejected. The observed decisions determine which hypotheses the procedure rejects; the error-control proof concerns the probabilities of those decisions over repeated trials.
Worked Example: Three Arms and Local P-Values
Suppose a hypothetical three-arm sequential trial uses \(\alpha=0.05\) for every intersection test. At the end of the trial, valid sequential p-values are \(p_{\{1\}}=0.012\), \(p_{\{2\}}=0.041\), \(p_{\{3\}}=0.080\), \(p_{\{1,2\}}=0.030\), \(p_{\{1,3\}}=0.044\), \(p_{\{2,3\}}=0.020\), and \(p_{\{1,2,3\}}=0.036\). Each local test rejects when its p-value is at most \(0.05\).
For arm 1, the relevant values are \(0.012\), \(0.030\), \(0.044\), and \(0.036\). Each is at most \(0.05\), so \(H_1\) is rejected. For arm 2, the values are \(0.041\), \(0.030\), \(0.020\), and \(0.036\); each is also at most \(0.05\), so \(H_2\) is rejected. For arm 3, the values include \(p_{\{3\}}=0.080\), and \(0.080>0.05\). Thus \(H_3\) is not rejected, even though the other intersections containing arm 3 were rejected.
The local p-values are not combined by taking their average or by applying another multiplicity correction at this step. The decision follows the closure rule: every required local test must reject. For example, arm 1 passes because \(0.012\leq0.05\), \(0.030\leq0.05\), \(0.044\leq0.05\), and \(0.036\leq0.05\).
Worked Example: Why Testing Only the Global Null Is Insufficient
Imagine a hypothetical three-arm procedure that tests only the global intersection \(H_{\{1,2,3\}}\) at level \(0.05\), then declares all three arms effective whenever that test rejects. Even if the global test is valid when all three nulls are true, the procedure has no corresponding protection when only \(H_1\) is true and arms 2 and 3 are effective. A rejection could then lead directly to the false declaration for arm 1.
As a numerical illustration, suppose that under this partial-null configuration the global test rejects with probability \(0.12\). If its rejection causes rejection of \(H_1\), then the FWER in this configuration is \(0.12\), which exceeds \(\alpha=0.05\). Global-null control does not rule out this result because the configuration with only \(H_1\) true is not the global null.
Closed testing prevents this inference unless the local test for \(H_{\{1\}}\) and every other intersection containing arm 1 also rejects. In particular, the true intersection \(H_{\{1\}}\) must have a valid sequential local test. If that local test has rejection probability at most \(0.05\) under the configuration, the event of falsely rejecting \(H_1\) is contained in an event with probability at most \(0.05\). The protection comes from testing the relevant true intersection, not from the global test alone.
Assumptions, Diagnostics, and Limits
The closure theorem is a logical result: its proof needs event inclusion and valid local tests. It does not require independent arm statistics. Nor does it require local tests for different intersections to be independent, because the proof bounds the familywise-error event by one true-intersection rejection event rather than adding probabilities across intersections.
The difficult work is verifying the local validity condition for the actual sequential design. In particular, check that:
- the local rejection event includes crossings at every monitored look, not just a single-look error calculation;
- the test for each \(H_I\) is calibrated when that intersection is true, including relevant configurations of effects on arms outside \(I\);
- the boundary or local p-value matches the planned information times, stopping rules, and permitted arm-dropping or adaptation; and
- the closed-testing implementation actually requires every intersection containing an arm, rather than only the global intersection or a selected subset.
A diagnostic warning sign is a local test justified only under the global null. Another is a nominal level assigned separately at each look without accounting for the probability of crossing over the whole trial. Either issue breaks the premise needed for the strong-control proof. Simulating several null configurations can help reveal a problem, but simulation alone does not establish the mathematical validity condition for all configurations.
Closed testing can require many local tests: for \(K\) arms there are \(2^K-1\) nonempty intersections. Some designs use a shortcut to obtain the same decisions with less computation, but a shortcut must still implement the closed-testing decisions or provide a separate strong-control argument. Closed testing also does not make an invalid local test valid. Its guarantee is only as sound as the intersection tests it relies on.
The contrast with a single global test is therefore precise. A level-\(\alpha\) global test protects against false rejection when all elementary nulls are true. Strong control requires protection in every partial-null configuration. Closed testing achieves that by requiring the relevant true intersection test to be rejected whenever any true elementary null is rejected.
Check Your Understanding
Answer each question using the closed-testing rule and the sequential local-test condition.
- In a four-arm trial, the true elementary nulls are \(H_2\) and \(H_4\). Which intersection test is used directly in the strong-control event-inclusion argument?
- In a two-arm trial with \(\alpha=0.05\), the local tests reject \(H_{\{1\}}\) and \(H_{\{1,2\}}\), but do not reject \(H_{\{2\}}\). Which elementary hypotheses are rejected by closure?
- Why does local validity only under the global null fail to establish strong FWER control?
- Suppose there are three looks, and a boundary has crossing probability \(0.02\) at each look considered separately. Does that alone establish that its sequential local test has level \(0.05\)? Explain.
Solutions. (1) The true-index set is \(T=\{2,4\}\), so use \(H_{\{2,4\}}\). A false rejection of either true elementary hypothesis requires rejection of this intersection. Its valid local test bounds the familywise-error probability by \(0.05\) (or by the specified \(\alpha\)). (2) \(H_1\) is rejected because both intersections containing arm 1, namely \(H_{\{1\}}\) and \(H_{\{1,2\}}\), reject. \(H_2\) is not rejected because \(H_{\{2\}}\) does not reject. (3) The true intersection in a partial-null configuration may be smaller than the global intersection. The strong-control proof needs the local test for that actual true intersection to be valid in that configuration; global-null validity does not supply this. (4) No. Separate-look probabilities do not determine the probability of crossing at least one of the three looks. The sequential crossing probability must itself be calibrated to be at most \(0.05\), accounting for the joint path across looks.