Why Repeated Looks Must Be Covered by Closure
An interim review can lead to stopping, continuing, or changing which treatment claims are pursued. With several reviews, a false rejection may occur at any look along any permitted path. Controlling error at one look is therefore not, by itself, a guarantee about the complete trial. When there are several claims, closed testing adds another requirement: a claim can be rejected only when the relevant intersection hypotheses have also been rejected.
The key task is to define the error event over the full procedure. For each intersection hypothesis, ask whether its local test could reject at any look or permitted path. Closure then links those complete local rejection events to the familywise error event for the elementary claims.
Elementary Hypotheses, Intersections, and Closure
A local test is a test of one particular \(H_I\). In a trial with repeated looks, “reject \(H_I\)” means that the local procedure has rejected it at some point during the trial, under its specified stopping and adaptation rules. Its type I error must be controlled for that whole procedure, not just for an isolated look.
Closure involves \(2^m-1\) nonempty intersections. This can be many tests, but the principle is simple: every elementary rejection must be supported by tests of all intersections that contain it. Rejection of one local intersection alone does not authorize an elementary claim.
How Repeated-Look Error Enters the Local Tests
Suppose a local test of \(H_I\) can reject at \(K\) planned looks. Write \(R_{I,k}\) for its rejection event at look \(k\). The complete local rejection event is \(R_I=\bigcup_{k=1}^{K}R_{I,k}\). If, under \(H_I\), each look-specific test has rejection probability at most \(c_k\), then the union bound gives
Thus, choosing nonnegative thresholds whose sum is at most \(\alpha\) makes the local test level at most \(\alpha\) over all \(K\) looks. No independence between looks is needed. The bound may be conservative, but it remains valid when the look-specific statistics are dependent.
This argument assumes that each look-specific test is valid under its intersection null and that the stated thresholds cover all looks at which it could reject. If an adaptation changes the data or test in a way that invalidates a look-specific test, the union bound cannot repair that invalidity. The local procedure must still have a valid error guarantee under \(H_I\).
Worked Example: Spending Error Across Three Looks
A hypothetical trial has three planned interim looks. For every intersection hypothesis, suppose the local test uses a p-value \(p_{I,k}\) at look \(k\), with \(P(p_{I,k}\leq u\mid H_I)\leq u\) for each fixed look and each \(u\) between 0 and 1. The trial targets \(\alpha=0.05\). Find a simple threshold that controls the complete local rejection probability.
Use the same threshold at each of the three looks. The union-bound requirement is \(3c\leq 0.05\), so choose the exact threshold
The local rule rejects \(H_I\) if \(p_{I,k}\leq 1/60\) at any look. Under \(H_I\), each look-specific rejection probability is at most \(1/60\). Therefore,
The implemented cutoff must be \(1/60\) or another value no greater than it. Writing \(0.01667\) as the actual cutoff would round upward: \(3(0.01667)=0.05001\), which exceeds the target. The same locally valid repeated-look rule is used for every intersection; closure then determines which elementary hypotheses can be rejected.
Why Closure Controls Strong Familywise Error
The central justification is that any false elementary rejection forces rejection of an intersection hypothesis that is true. Let \(T\) be the set of indices of the true elementary nulls. If \(T\) is nonempty, \(H_T=\bigcap_{i\in T}H_i\) is true. Suppose at least one true elementary null \(H_i\), with \(i\in T\), is rejected. By the closed-testing rule, every intersection containing \(i\) must have been rejected. In particular, \(H_T\) contains \(i\), so the local test of \(H_T\) must have rejected.
Taking probabilities under the parameter configuration in question, the familywise error probability is no greater than the probability that the local test rejects \(H_T\). If every local procedure controls its complete repeated-look type I error at level \(\alpha\) under its intersection null, that probability is at most \(\alpha\). If \(T=\varnothing\), there are no true elementary nulls to reject, so the familywise error event is impossible and its probability is zero. This establishes strong familywise error control for every configuration, including the case in which no elementary null is true.
The argument does not require the local tests for different intersections to be independent, nor does it require independence across looks. It does require that the local test of each true intersection is valid over its complete procedure. Closure cannot compensate for a local test whose repeated-look error exceeds its stated bound.
Worked Example: Tracing Two Claims Through Closure
Consider two elementary null hypotheses, \(H_A\) and \(H_B\), and their joint intersection \(H_{AB}=H_A\cap H_B\). There are two looks. For each of the three local tests, use a threshold of \(0.025\) at each look. Assume each p-value is valid at its fixed look under the corresponding null. Since \(0.025+0.025=0.05\), the union bound gives complete local type I error at most \(0.05\) for each intersection.
Suppose the hypothetical p-values are as follows:
| Local hypothesis | Look 1 | Look 2 |
|---|---|---|
| \(H_A\) | 0.012 | 0.006 |
| \(H_B\) | 0.040 | 0.021 |
| \(H_{AB}\) | 0.018 | 0.011 |
At look 1, the local tests reject \(H_A\) and \(H_{AB}\), because \(0.012\leq0.025\) and \(0.018\leq0.025\). They do not reject \(H_B\), because \(0.040>0.025\). Closure therefore permits rejection of \(H_A\): both intersections containing \(A\), namely \(H_A\) and \(H_{AB}\), have been rejected. It does not permit rejection of \(H_B\), because the local test of \(H_B\) has not rejected.
At look 2, all three p-values are at most \(0.025\). Consequently, all three local tests have rejected by the end of the trial, and closure permits rejection of both elementary nulls. The conclusion concerns the rejection rule, not the probability that either null is true. Error control follows from the assumed validity of each local test across both looks and the closure argument.
Every Permitted Rejection Path Must Be Covered
In an adaptive trial, the route through the looks may depend on interim data: the trial may stop early, continue to a later look, or select among allowed actions. For each intersection, define its complete rejection event to include every way the local procedure could reject under the design. A path that is impossible under the design need not be counted; a path that is permitted must not be silently omitted.
This gives a practical audit. List the looks and decision paths on which a local intersection test can reject. Verify that the local error guarantee covers the union of those events. Then verify that each elementary claim is rejected only after every intersection containing it has rejected. The local tests can use different procedures, but each must cover its own complete rejection event at level \(\alpha\) or less.
A simple spending rule is one option: choose look-specific bounds \(c_1,\ldots,c_K\) with \(\sum_k c_k\leq\alpha\). Equal allocation is not required. For example, with three looks, \(0.025\), \(0.015\), and \(0.010\) sum to \(0.050\). The union-bound guarantee is the same even if the look statistics are dependent. The spending pattern affects when rejection is easier, not the total bound.
Worked Example: Why Reusing Alpha at Every Look Fails
Suppose a local test of the full intersection \(H_{AB}\) uses a nominal \(0.05\) cutoff at each of three looks, with independent Uniform(0,1) p-values under \(H_{AB}\). Each individual look has type I error \(0.05\), but the local procedure rejects if any look crosses the cutoff.
The probability of no crossing is the probability that all three p-values exceed \(0.05\). Independence gives
Therefore, the probability of at least one crossing is
The complete local test has error about \(0.1426\), not \(0.05\). If \(H_A\) and \(H_B\) are both true, rejecting either elementary null through closure requires rejecting \(H_{AB}\). Thus, a procedure using this invalid local test cannot claim the usual level-\(0.05\) guarantee merely because it applies closure. This calculation uses independence to get the exact probability; with dependent looks the exact value may differ, but testing at \(0.05\) repeatedly still does not establish a \(0.05\) bound.
Comparison, Diagnostics, and Limitations
A single-look closed test and a repeated-look closed test share the same closure rule. The difference is what must be controlled for each local intersection: the single-look procedure has one rejection event, while the repeated-look procedure has the union of rejection events across looks and permitted paths. A Bonferroni approach across elementary hypotheses instead divides error directly among the claims. Repeated-look spending addresses error over time; closure addresses which multiple claims may be rejected. These are distinct layers, and a design may need both.
Useful diagnostics include checking that the set of local tests includes every nonempty intersection, that every permitted rejection route is included in each local error calculation, and that the thresholds actually sum to no more than \(\alpha\). Also check whether the local p-values or tests are valid under their intersection nulls at the looks where they are used. If a stopping rule or adaptation creates an additional opportunity to reject, that opportunity belongs in the local rejection event.
- Using alpha at every look: a per-look level does not equal a complete repeated-look level.
- Checking only elementary tests: an elementary rejection is not allowed unless every containing intersection test has rejected.
- Checking only one realized path: error control concerns the procedure’s possible paths, not just the path observed in one trial.
- Rounding a cutoff upward: even a small increase can make a stated union-bound guarantee false.
The union bound can be conservative, particularly when looks are strongly dependent. It is nevertheless useful because it needs no independence assumption. More efficient repeated-look procedures may be possible, but they still need a valid complete local error guarantee for every intersection before closure can provide strong familywise control.
Check Your Understanding
Answer each question, then compare with the solutions below.
- There are four planned looks. A local intersection test uses thresholds \(0.01\), \(0.015\), \(0.01\), and \(0.015\). What complete local error bound follows from the union bound?
- For two elementary nulls, the local tests of \(H_A\) and \(H_{AB}\) reject, but the local test of \(H_B\) does not. Which elementary hypotheses may closure reject?
- Why does the closure proof need a separate case when no elementary null is true?
- Three independent uniform p-values are tested at \(0.02\) at each of three looks. What is the probability of at least one crossing under the null?
Solutions. (1) The thresholds sum to \(0.01+0.015+0.01+0.015=0.05\), so the complete local error is at most \(0.05\). (2) \(H_A\) may be rejected because both intersections containing \(A\), \(H_A\) and \(H_{AB}\), rejected. \(H_B\) may not be rejected because its own local test did not reject. (3) If the set \(T\) of true elementary nulls is empty, \(H_T\) is not a defined nonempty intersection. Instead, note directly that no true elementary null can be rejected, so familywise error is zero. (4) The probability is \(1-(1-0.02)^3=1-0.98^3=0.058808\), approximately \(0.05881\), which exceeds \(0.02\).